Skip to content

Commit

Permalink
Revamp of the FSBrowser and SDWebServer examples (#7182)
Browse files Browse the repository at this point in the history
* Minimal file with a few ESP8266-specific keywords - github issue #3701

* Renamed "SDWebServer" to the more universal "WebFileManager"

* SD was replaced by SDFS, and sketch now works on either SDFS, SPIFFS or
LittleFS based on a #define logic (required adding a second param to open() and
replacing 'FILE_WRITE' by "w") + Added size information to file list and a /status request handler to return filesystem status

* Tree panel width is now proportional to window. Changed icons (lighter and
more neutral), including one for files. Show size of files. Fill
"filename" box upon clicking on a file. Sort files alphabetically.

* Replaced by a lighter version

* Return the filesystem time in the status object
+ Massive cleanup/merge/align with some code from the FSBrowser example
and misc refactorings

* Fixed folder handling

* Replaced the FILESYSTEM #define by a filesystem variable, and introduced FSConfig to prevent FS formating.
Fixed recursive deletion.
Got rid of specific isDir() for SPIFFS.

* Made 8.3 lowercase filenames formating optional (disabled by default).
Refresh only part of the tree when possible.
Selecting a file for upload defaults to the same folder as the last
clicked file.
Removed the Mkdir button on SPIFFS.

* Added 'wait' cursor during asynchronous operations.
Slight refactoring of XMLHttpRequest completion handling

* Removed limitation "files must have an extension, folders may not".
Case insensivity of the extension for the editor and preview.

* Support Filenames without extension, Dirnames with extension.
Added Save/Discard/Help buttons to Editor, discard confirmation on leave, and refresh tree/status upon save.
Removed redundant Ctrl-Z + Ctrl-Shift-Z shortcut declarations.
Small bug fixes.
+ some refactoring

* Fixed tree refresh on delete in all cases by returning the remaining path as response to the delete request.
Refactoring

* Changed FS status in text by a percentage graph, with numbers as tooltip.
Unsupported files on SPIFFS (files at root not sarting with "/", files with double "/", files ending with "/") are now detected and reported in the page.

* Small fix + refactoring

* Restrict filename support check to SPIFFS.

* Implemented Move/Rename.
Added "loading" screen during async operations (dim with spinner and status).
Fixed "discard" feature that kept prompting even after an image was loaded.
Improved refresh of parts of the tree, with recursive listing.
Moved the "path" id attribute to the "li" elements for folders (was already the case for files).
Refactoring and cleanup.

* Fixed broken spinner

* Cosmetic improvements.
Removed non-functional Upload context menu.
Fixed error in response to move requests.
Added minified version.

* Added specific icons for text and image files.
Fixed incompatibilities with SPIFFS.
Fixed a race condition between deletion and reinsertion of nodes when multiple folders are refreshed.
Fixed missing URL decoding for files with special chars (e.g. space char).
Moved info from source code comment to a readme.md file.
Added source PNG to git.
Cleanup.

* Added favicon.ico.

* Renamed project

* Small changes

* Add a note about the ace.js dependency

* Minor changes

* Define LittleFS by default.
If both uncompressed and gz versions exist, use uncompressed version.
Small fixes.

* Define LittleFS by default.
If both uncompressed and gz versions exist, use uncompressed version.
Small fixes.

* Restyled version

* (dummy edit to retrigger broken CI)

* Using unsigned int for comparison with String.length()

* Return an error when upload fails (e.g. filesystem full)

* Trying to reorder functions to please CI

* Reordered functions to please CI.

* Moved file

* Renamed "SDWebServer" to the more universal "WebFileManager"

* SD was replaced by SDFS, and sketch now works on either SDFS, SPIFFS or
LittleFS based on a #define logic (required adding a second param to open() and
replacing 'FILE_WRITE' by "w") + Added size information to file list and a /status request handler to return filesystem status

* Tree panel width is now proportional to window. Changed icons (lighter and
more neutral), including one for files. Show size of files. Fill
"filename" box upon clicking on a file. Sort files alphabetically.

* Replaced by a lighter version

* Return the filesystem time in the status object
+ Massive cleanup/merge/align with some code from the FSBrowser example
and misc refactorings

* Fixed folder handling

* Replaced the FILESYSTEM #define by a filesystem variable, and introduced FSConfig to prevent FS formating.
Fixed recursive deletion.
Got rid of specific isDir() for SPIFFS.

* Made 8.3 lowercase filenames formating optional (disabled by default).
Refresh only part of the tree when possible.
Selecting a file for upload defaults to the same folder as the last
clicked file.
Removed the Mkdir button on SPIFFS.

* Added 'wait' cursor during asynchronous operations.
Slight refactoring of XMLHttpRequest completion handling

* Removed limitation "files must have an extension, folders may not".
Case insensivity of the extension for the editor and preview.

* Support Filenames without extension, Dirnames with extension.
Added Save/Discard/Help buttons to Editor, discard confirmation on leave, and refresh tree/status upon save.
Removed redundant Ctrl-Z + Ctrl-Shift-Z shortcut declarations.
Small bug fixes.
+ some refactoring

* Fixed tree refresh on delete in all cases by returning the remaining path as response to the delete request.
Refactoring

* Changed FS status in text by a percentage graph, with numbers as tooltip.
Unsupported files on SPIFFS (files at root not sarting with "/", files with double "/", files ending with "/") are now detected and reported in the page.

* Small fix + refactoring

* Restrict filename support check to SPIFFS.

* Implemented Move/Rename.
Added "loading" screen during async operations (dim with spinner and status).
Fixed "discard" feature that kept prompting even after an image was loaded.
Improved refresh of parts of the tree, with recursive listing.
Moved the "path" id attribute to the "li" elements for folders (was already the case for files).
Refactoring and cleanup.

* Fixed broken spinner

* Cosmetic improvements.
Removed non-functional Upload context menu.
Fixed error in response to move requests.
Added minified version.

* Added specific icons for text and image files.
Fixed incompatibilities with SPIFFS.
Fixed a race condition between deletion and reinsertion of nodes when multiple folders are refreshed.
Fixed missing URL decoding for files with special chars (e.g. space char).
Moved info from source code comment to a readme.md file.
Added source PNG to git.
Cleanup.

* Added favicon.ico.

* Renamed project

* Small changes

* Add a note about the ace.js dependency

* Minor changes

* Define LittleFS by default.
If both uncompressed and gz versions exist, use uncompressed version.
Small fixes.

* Define LittleFS by default.
If both uncompressed and gz versions exist, use uncompressed version.
Small fixes.

* Restyled version

* (dummy edit to retrigger broken CI)

* Using unsigned int for comparison with String.length()

* Return an error when upload fails (e.g. filesystem full)

* Trying to reorder functions to please CI

* Reordered functions to please CI.

* Update to use chunked response API

* Removed temp files commited by mistake

* Avoid using args() as requested

* Use html entity for non-breaking space to avoid losing char when minifying

* Script to preprocess index.htm

* (reformated code)

* (comments)

* Preprocessed files

* Fixed dump to create an actual include file

* Optionally embed index.htm in code.
(+ documentation and preprocessing script)

* (reformated)

* If editor cannot be loaded from the web, try a local version, or default
to a text viewer if not present

* (removed a TODO item :-))

* (forgot to reprocess files after last commit)

* (reprocess should be ok this time)

* Return error 500 when upload fails immediately (e.g. filesystem full)

* Use standard <meter> tag for filesystem use

* (updated following changes to index.htm)

* Do not include gzipped version in the data folder by default. Leave it in the extras folder and change readme accordingly (plus some reformatingi of the readme file)

* Gzipped index file not included in data/edit by default. It is now left in the extras folder.
Readme file was updated accordingly (+ some reformating)

* Reduce String clutter by reserving and concatenating elements one by one.

* Use clear() to reset String.

* Avoid comparisons against empty String.

* Use char instead of single-char String where possible.

* Prefer direct logic over inverted.

* Rename returnBlah to replyBlah.

* Renamed h2int to hexDigitToInt

* Renamed getFileError() to checkForUnsupportedPath(), to avoid confusion
with a getter.

* Misc improvements.

* Added comments about mandatory rebuilding gz and h files in case of update
to index.htm.

* Addressed a few comments.

* Improve replies: bad requests vs server error

* (reformated)

* Reduce clutter by reserving String size beforehand.

* Moved most Strings of more than 10 chars to flash.

* Use lib version of urlDecode() instead of a local one, and only call it when required.

* Added a comment about the dangers of recursion on embedded devices.

* Added a more explicit warning in the .h header comment.

* Added a typical set of required files to load ace editor from the ESP.

* (reformated)

* More explicit warning at the beginning of the .h version.

Co-authored-by: david gauchard <[email protected]>
Co-authored-by: Earle F. Philhower, III <[email protected]>
Co-authored-by: Develo <[email protected]>
  • Loading branch information
4 people committed Apr 29, 2020
1 parent ce200ed commit 668b33d
Show file tree
Hide file tree
Showing 15 changed files with 2,418 additions and 1,287 deletions.
735 changes: 551 additions & 184 deletions libraries/ESP8266WebServer/examples/FSBrowser/FSBrowser.ino

Large diffs are not rendered by default.

Binary file not shown.
1,128 changes: 1,128 additions & 0 deletions libraries/ESP8266WebServer/examples/FSBrowser/data/edit/index.htm

Large diffs are not rendered by default.

Binary file not shown.
99 changes: 12 additions & 87 deletions libraries/ESP8266WebServer/examples/FSBrowser/data/index.htm
Original file line number Diff line number Diff line change
@@ -1,97 +1,22 @@
<!--
FSWebServer - Example Index Page
Copyright (c) 2015 Hristo Gochkov. All rights reserved.
This file is part of the ESP8266WebServer library for Arduino environment.
This library is free software; you can redistribute it and/or
modify it under the terms of the GNU Lesser General Public
License as published by the Free Software Foundation; either
version 2.1 of the License, or (at your option) any later version.
This library is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
Lesser General Public License for more details.
You should have received a copy of the GNU Lesser General Public
License along with this library; if not, write to the Free Software
Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
-->
<!DOCTYPE html>
<html>
<head>
<meta http-equiv="Content-type" content="text/html; charset=utf-8">
<title>ESP Monitor</title>
<script type="text/javascript" src="graphs.js"></script>
<script type="text/javascript">
var heap,temp,digi;
var reloadPeriod = 1000;
var running = false;

function loadValues(){
if(!running) return;
var xh = new XMLHttpRequest();
xh.onreadystatechange = function(){
if (xh.readyState == 4){
if(xh.status == 200) {
var res = JSON.parse(xh.responseText);
heap.add(res.heap);
temp.add(res.analog);
digi.add(res.gpio);
if(running) setTimeout(loadValues, reloadPeriod);
} else running = false;
}
};
xh.open("GET", "/all", true);
xh.send(null);
};

function run(){
if(!running){
running = true;
loadValues();
}
<title>ESP Index</title>
<style>
body {
background-color:black;
color:white;
}

</style>
<script type="text/javascript">
function onBodyLoad(){
var refreshInput = document.getElementById("refresh-rate");
refreshInput.value = reloadPeriod;
refreshInput.onchange = function(e){
var value = parseInt(e.target.value);
reloadPeriod = (value > 0)?value:0;
e.target.value = reloadPeriod;
}
var stopButton = document.getElementById("stop-button");
stopButton.onclick = function(e){
running = false;
}
var startButton = document.getElementById("start-button");
startButton.onclick = function(e){
run();
}

// Example with 10K thermistor
//function calcThermistor(v) {
// var t = Math.log(((10230000 / v) - 10000));
// t = (1/(0.001129148+(0.000234125*t)+(0.0000000876741*t*t*t)))-273.15;
// return (t>120)?0:Math.round(t*10)/10;
//}
//temp = createGraph(document.getElementById("analog"), "Temperature", 100, 128, 10, 40, false, "cyan", calcThermistor);

temp = createGraph(document.getElementById("analog"), "Analog Input", 100, 128, 0, 1023, false, "cyan");
heap = createGraph(document.getElementById("heap"), "Current Heap", 100, 125, 0, 30000, true, "orange");
digi = createDigiGraph(document.getElementById("digital"), "GPIO", 100, 146, [0, 4, 5, 16], "gold");
run();
console.log("we are loaded!!");
}
</script>
</head>
<body id="index" style="margin:0; padding:0;" onload="onBodyLoad()">
<div id="controls" style="display: block; border: 1px solid rgb(68, 68, 68); padding: 5px; margin: 5px; width: 362px; background-color: rgb(238, 238, 238);">
<label>Period (ms):</label>
<input type="number" id="refresh-rate"/>
<input type="button" id="start-button" value="Start"/>
<input type="button" id="stop-button" value="Stop"/>
</div>
<div id="heap"></div>
<div id="analog"></div>
<div id="digital"></div>
<body id="index" onload="onBodyLoad()">
<h1>ESP8266 Pin Functions</h1>
<img src="pins.png" />
</body>
</html>
</html>
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file not shown.
529 changes: 529 additions & 0 deletions libraries/ESP8266WebServer/examples/FSBrowser/extras/index_htm.h

Large diffs are not rendered by default.

Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
#/bin/sh

# Processing script to optionally reduce filesystem use by miniying, gzipping and preparing index.htm for embedding in code.
# Please see readme.md for more information.

# Requires xdd which is part of the VIM package
# Requires npm
# sudo apt install npm
# ln -s /usr/bin/nodejs /usr/bin/node
# Requires html-minifier
# sudo npm install html-minifier -g

html-minifier \
--case-sensitive \
--collapse-boolean-attributes \
--collapse-whitespace \
--minify-css true \
--minify-js true \
--process-conditional-comments \
--remove-attribute-quotes \
--remove-comments \
--remove-empty-attributes \
--remove-optional-tags \
--remove-redundant-attributes \
--remove-script-type-attributes \
--remove-style-link-type-attributes \
-o index.htm \
../data/edit/index.htm

if [ $? -ne 0 ]
then
echo "Error minifying index.htm"
exit -1
fi

if [ -e index.htm.gz ]
then
rm index.htm.gz
fi

gzip index.htm
if [ $? -ne 0 ]
then
echo "Error gzipping minified index.htm"
exit -1
fi

echo '// WARNING: Auto-generated file. Please do not modify by hand.' > index_htm.h
echo '// This file is an embeddable version of the gzipped index.htm file.' >> index_htm.h
echo '// To update it, please rerun the `reduce_index.sh` script located in the `extras` subfolder' >> index_htm.h
echo '// then recompile the sketch after each change to the `index.html` file.' >> index_htm.h
xxd -i index.htm.gz >> index_htm.h
if [ $? -ne 0 ]
then
echo "Error creating include file from index.htm.gz"
exit -1
fi

echo Reduce complete.

138 changes: 138 additions & 0 deletions libraries/ESP8266WebServer/examples/FSBrowser/readme.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,138 @@
# FSBrowser readme

## What is this sketch about ?

This example is a FileSystem Browser for the ESP8266 using http requests and a html/javascript frontend,
working for all of SPIFFS, LittleFS and SDFS.
This unified version is based on the previous examples named FSWebServer, FSBrowser and SDWebServer, Copyright (c) 2015 Hristo Gochkov. All rights reserved.

## How to use it ?
1. Uncomment one of the `#define USE_xxx` directives in the sketch
2. Add the credentials of your WiFi network (search for `STASSID`)
3. Compile and upload the sketch to your ESP8266 device
4. For normal use, copy the contents of the `data` folder to the filesystem. To do so:
- for SDFS, copy that contents (not the data folder itself, just its contents) to the root of a FAT/FAT32-formated SD card connected to the SPI port of the ESP8266
- for SPIFFS or LittleFS, please follow the instructions at https://arduino-esp8266.readthedocs.io/en/latest/filesystem.html#uploading-files-to-file-system
5. Once the data and sketch have been uploaded, access the editor by pointing your browser to http://fsbrowser.local/edit

## Options
If you need to free some space on the ESP filesystem, you can delete all the sample files at the root but also replace the `index.htm` file in the `data/edit` subfolder by the `index.htm.gz` file from the `extras` folder. That compressed version is not suited for learning or debugging, but will bring the total FS usage under 7KB.
If you want to use the browser on a an existing filesystem or don't want to perform step 4 above, you have two possibilities :
- either upload the `index.htm` file to the filesystem by opening a command shell in the `data` folder and running the following cURL command:
`curl -F file=@edit/index.htm;filename=/edit/index.htm fsbrowser.local/edit`
- or embed a version of the html page in the source code itself by uncommenting the following line in the sketch and rebuilding:
`#define INCLUDE_FALLBACK_INDEX_HTM`
That embedded version is functionally equivalent and will be returned if no `/edit/index.htm` or `/edit/index.htm.gz` file can be found on the filesystem, at the expense of a higher binary size.

If you use the gzipped or `INCLUDE_FALLBACK_INDEX_HTM` options, please remember to rerun the `reduce_index.sh` script located in the `extras` subfolder and recompile the sketch after each change to the `index.html` file.

## Dependency
The html page uses the [Ace.js](https://ace.c9.io/) (v1.4.9 at the time of writing) text editor which is loaded from a CDN. Consequently, internet access from your web browser is required for the FSBrowser editing feature to work as-is.

If your browser has no web access (e.g. if you are connected to the ESP8266 as an access-point), you can copy the `ace.js` file to the `edit` subfolder of the ESP filesystem, along with optional plugins etc. according to your needs. A typical set might be:
```
ace.js
ext-keybinding_menu.js
ext-searchbox.js
mode-html.js
worker-html.js
worker-css.js
worker-javascript.js
mode-xml.js
worker-xml.js
mode-json.js
worker-json.js
```
(see https://github.com/ajaxorg/ace-builds for a full list).

If `ace.js` cannot be found on the ESP filesystem either, the page will default to a plain text viewer, with a warning message.

## Notes
- See https://arduino-esp8266.readthedocs.io/en/latest/filesystem.html for more information on FileSystems supported by the ESP8266.
- For SDFS, if your card's CS pin is not connected to the default pin (4), uncomment the `fileSystemConfig.setCSPin(chipSelectPin);` line, specifying the GPIO the CS pin is connected to
- `index.htm` is the default index returned if your URL does not end with a filename (works on subfolders as well)
- Filesystem limitations apply. For example, FAT16 is limited to 8.3 filenames - see https://en.wikipedia.org/wiki/8.3_filename - SPIFFS and LittleFS also have limitations, please see https://arduino-esp8266.readthedocs.io/en/latest/filesystem.html#spiffs-file-system-limitations
- Directories are supported on SDFS and LittleFS. On SPIFFS, all files are at the root, although their names may contain the "/" character.
- The convention here is that the root of the filesystem is "/". On SPIFFS, paths not started with a slash are not supported
- For creation, the convention is that a path ending with a "/" means create a folder, while without a "/" we create a file. Having an extension or not does not matter.

## Changelog since original FSBrowser

### Fixes to work on LittleFS based on SDFS
- #define logic to select FS
- switched from SD to SDFS
- begin() does not support parameters > removed SS and added optional config
- LittleFS.open() second parametsr is mandatory > specified "r" where needed
- 'FILE_WRITE' was not declared in this scope > replaced by "w"

### UI/usability improvements
- Never format filesystem, just return "FS INIT ERROR" when FS cannot be mounted
- Tree panel width is now proportional (20%) to see long names on big screens
- Added icons for files, and indented them to the same level as folders
- Changed file/folder icon set to use a lighter and more neutral one, and added specific "text" and "image" icons for formats recognized as such
- Items are now sorted (folders first, then plain files, each in alphabetic order)
- Added file size after each file name
- Added FS status information at the top right
- Made clear that an async operation is in progress by dimming screen and showing operation status
- Filled filename box in header with the name of the last clicked file
- Selecting a file for upload defaults to putting it in the same folder as the last clicked file
- Removed limitation to 8.3 lowercase filenames
- Support Filenames without extension, Dirnames with extension
- Improved recursive refresh of parts of the tree (e.g. refresh folder upon file delete, show last folder upon creating nested file)
- Added Save/Discard/Help buttons to ACE editor, discard confirmation on leave, and refresh tree and status upon save
- Removed "Upload" from context menu (which didn't work anyway)
- Added "Rename/Move" feature to context menu
- Sketch can be used on a preexisting filesystem by embedding the index.htm file in the program

## TODO (maybe)
- ? How can we query the fatType of the SDFS (FAT16 or FAT32) to limit filenames to 8.3 on FAT16 ?
- ? Add a visible root node "/" (with no delete option) + add the FS type next to it, like <i>LittleFS</i>
- ? move "Mkdir" and "MkFile" to context menu, with prompt like for Rename/Move
- ? implement drag/drop for move + make "rename" only a local rename operation (no move)
- ? Optionally present SPIFFS as a hierarchical FS too
- ? Optionally mount several filesystems at the same time (SPIFFS + SDFS or LittleFS + SDFS)

## Test suite
These tests are a checklist of operations to verify the FSBrowser behaviour.
### On SPIFFS
#### 8.3 filenames
- At root : MkFile '/1.txt' / List / Edit / Download / Delete / Upload '/1.png' / View image / Delete image
- In subdir : MkFile '/dir/2.txt' / List / Edit / Download / Delete / Upload '/dir/2.png' / View image
- Create nested file '/a/b.txt' and delete it
- Attempt creation of unsupported filenames
#### Long filenames
- At root : MkFile '/My text file 1.txt' / List / Edit / Download / Delete / Upload '/My image file 1.png' / View image / Delete image
- In subdir : MkFile '/My Directory/My text 2.txt' / List / Edit / Download / Delete / Upload '/My Directory/My image 2.png' / View image
- Create nested file '/My folder/My test file.txt' and delete it

### On LittleFS
#### 8.3 filenames
- At root : MkFile '/1.txt' / List / Edit / Download / Delete / Upload '/1.png' / View image / Delete image / Mkdir '/dir'
- In subdir : MkFile '/dir/2.txt' / List / Edit / Download / Delete / Upload '/dir/2.png' / View image / Mkdir '/dir/sub'
- Delete root folder '/dir'
- Create nested file '/a/b.txt' and delete file 'b.txt'
#### Long filenames
- At root : MkFile '/My text file 1.txt' / List / Edit / Download / Delete / Upload '/My image file 1.png' / View image / Delete image / Mkdir '/My Directory'
- In subdir : MkFile '/My Directory/My text file 2.txt' / List / Edit / Download / Delete / Upload '/My Directory/My image file 2.png' / View image / Mkdir '/My Directory/My Subdirectory'
- Delete root folder '/My Directory'
- Create nested file '/My folder/My test file.txt' and delete file 'My test file.txt'

### On SDFS
#### 8.3 filenames
- At root : MkFile '/1.txt' / List / Edit / Download / Delete / Upload '/1.png' / View image / Delete image / Mkdir '/dir'
- In subdir : MkFile '/dir/2.txt' / List / Edit / Download / Delete / Upload '/dir/2.png' / View image / Mkdir '/dir/sub'
- Delete root folder '/dir'
- Create nested file '/a/b.txt' and delete file 'b.txt', then delete '/a'
#### Long filenames
- At root : MkFile '/My text file 1.txt' / List / Edit / Download / Delete / Upload '/My image file 1.png' / View image / Delete image / Mkdir '/My Directory'
- In subdir : MkFile '/My Directory/My text file 2.txt' / List / Edit / Download / Delete / Upload '/My Directory/My image file 2.png' / View image / Mkdir '/My Directory/My Subdirectory'
- Delete root folder '/My Directory'
- Create nested file '/My folder/My test file.txt' and delete file 'My test file.txt'

## Credits
- Original version of FSBrowser written by me-no-dev, contributions over time by various contributors
- Icons are from https://feathericons.com/ . The resulting PNG is passed first through https://compresspng.com/ before being converted to base64 using https://www.base64-image.de/
- The spinner is based on https://github.com/jlong/css-spinners
- Minifiying of index.htm is done using the command line version of https://kangax.github.io/html-minifier/
- Idea of embedding webpage in code borrowed from https://github.com/me-no-dev/ESPAsyncWebServer

Loading

0 comments on commit 668b33d

Please sign in to comment.