Ota commands in flash #6538
Ota commands in flash #6538
Conversation
…should mean nothing else moves)
|
Good useful PR, so a CI that does not fail and the PR goes through would be great... |
|
Travis is next on the list, along with some tests on a couple of different
esps that I've got.
The other thing I need to check, since I'm listing todo's is ensure I
finished the eboot doc changes.
|
|
What would happen if I deployed the current bootloader and had it update itself to the new version later? Would it brick itself because the running code would be overwritten? |
While I don't have my notes handy at the moment I'm fairly sure that self-updating was supported, if not already (I forget whether the writing in eboot is already totally run-from-RAM) then certainly as an achievable improvement - switching to the flash-using bootloader entirely OTA is an intended use-case. |
…ault off. Readded the start of the eboot docs in README.md (still WIP, but it's a start) Adjusted the default number of commands that are cycled through for wear levelling to 32 Fixed a 🤦 bug in the loop to find the first available flash-based command block
|
Sorry for jumping in...
I just want to report that I am using flash stored OTA commands (a fork of
the PR) on my devices, and I updated (without compression) tens of times
more than 200 devices, without reporting a single issue.
Il giorno ven 16 ott 2020 alle ore 11:27 Julian Davison <
[email protected]> ha scritto:
… Oh, potentially getting rid of the existing RTC stuff will help, the
original approach left that intact to provide a path of least change, but
if we were to opt to go with just flash storage, it might all fit. Will
explore that next.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#6538 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/ACBHPUCSPU2EPEFJWTUGZO3SLAGWVANCNFSM4IYSVVSA>
.
--
*Luigi R.*
|
Instead of using either a series of etc_putc or setting a series of bytes one by one, use a simple macro to define 32b constants to build up strings. Saves ~30 bytes of program code in eboot for #6538 to work with.
|
@lrodorigo it is always a good thing to have other testers confirming that something works! |
|
@davisonja that's only 152 bytes over, not a lot. I don't think it makes sense to keep both rtc and flash commands, so I'd say you're on the right path. |
|
@devyte @davisonja This code exploits the same write without erase "feature" that SPI does, it may stop working at some point with new flashes. from what I can see it "should" work on PUYA because of read-then-write, but we need to be careful here. |
…lash * upstream/master: (72 commits) Typo error in ESP8266WiFiGeneric.h (esp8266#7797) lwip2: use pvPortXalloc/vPortFree and "-free -fipa-pta" (esp8266#7793) Use smarter cache key, cache Arduino IDE (esp8266#7791) Update to SdFat 2.0.2, speed SD access (esp8266#7779) BREAKING - Upgrade to upstream newlib 4.0.0 release (esp8266#7708) mock: +hexdump() from debug.cpp (esp8266#7789) more lwIP physical interfaces (esp8266#6680) Rationalize File timestamp callback (esp8266#7785) Update to LittleFS v2.3 (esp8266#7787) WiFiServerSecure: Cache SSL sessions (esp8266#7774) platform.txt: instruct GCC to perform more aggressive optimization (esp8266#7770) LEAmDNS fixes (esp8266#7786) Move uzlib to master branch (esp8266#7782) Update to latest uzlib upstream (esp8266#7776) EspSoftwareSerial bug fix release 6.10.1: preciseDelay() could delay() for extremely long time, if period duration was exceeded on entry. (esp8266#7771) Fixed OOM double count in umm_realloc. (esp8266#7768) Added missing check for failure on umm_push_heap calls in Esp.cpp (esp8266#7767) Fix: cannot build after esp8266#7060 on Win64 (esp8266#7754) Add the missing 'rename' method wrapper in SD library. (esp8266#7766) i2s: adds i2s_rxtxdrive_begin(enableRx, enableTx, driveRxClocks, driveTxClocks) (esp8266#7748) ...
Are there flash systems where this doesn't apply, @drzony ? If I follow your comment it's the behaviour where we're modifying already written blocks to mark commands as needing action? |
Finally actually had a first run at removing the RTC stuff, and it looks like that does the trick, so compressed images and flash storage can both run, without the RTC support (and also without serial debugging on, but with #7545 applied more liberally through the other debug code it might yet be possible to get that in too. I've yet to verify the code actually works after all these random code removals, but the fact that it fits is a good start :) |
|
@davisonja Currently I don't think there are any flashes that we don't handle. But the workaround is based on flash ID. If another flash comes out that works in the same manner as PUYA ones, then it will break silently (without read-then-write PUYA flashes corrupt the bytes written). |
|
Your CI failures are related to you having a ESP8266SDFat submodule in your commit. The easiest thing at this point would be to just update to the proper ESP8266SDFat commit |
|
@drzony I might need to find the PUYA info on a computer - I had a quick look last night from my phone. My impression of flash systems were they (all) require an erase cycle (usually a sequence of consecutive addresses - a page or block) which resets the storage to a known value (typically 1's IIRC); from there you can write actual values which is actually achieved by updating bits from the erase-value. The write process can be repeated as many times as you like, but you can only ever toggle a bit from the erase-value to the other one. So in the case of an erase-value of 1, you can only ever clear bits, but you can do so at anytime - there's no concept of write-count, so 'twice' doesn't really apply. From a (fairly quick) look around this seems to be fairly standard, though the exact amount you have to 'write' varies (as in, a byte, or several bytes) the ability to update any 1 to a 0 holds. It's definitely a common 'trick'. Worth a note in the info on the eboot docs, tho. |
|
@davisonja
That's the problem I'm highlighting, that this is a 'common trick', but it's not really documented by any manufacturer and may not work on all flashes. Some of it is already described in #7644 |
|
@drzony that's an interesting collection of reading, ta. Do you think the current scheme (as in this incomplete PR) is going to be ok to proceed with given that it looks as though it's compatible with the flash implementations we've dealt with so far? I'm not clear whether 'be careful here' is aimed at that being kept in mind, or in changing the process the code is using to mark commands... |
|
@davisonja I would change the way of writing the commands. The simplest idea would be to have each flag as a separate |
|
@drzony so the considered wisdom is that a full 32-bits is safest - is that for flash in general, or the current scheme of writing used in this project? |
|
@davisonja It's the limitation of
I wrote workarounds for this, you can see them here |
|
Poor bug, still not fixed |
|
It is still on my list, there's been too little time to redo it. I have a
working solid solution, that needs slight tweaking to account for some
specific flash behaviour mentioned in the discussion.
Unfortunately the code base has moved significantly since it was first
built and we're at the point where it's simpler to re-apply the concept to
the current code, and sort out the specifics of fitting everything in (or a
migration path).
…On Sat, 9 Apr. 2022, 12:29 zhfei1979, ***@***.***> wrote:
Poor bug, still not fixed
—
Reply to this email directly, view it on GitHub
<#6538 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AABX6A3SOPJATQD3PV2HVQTVEDFPBANCNFSM4IYSVVSA>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
We can help with this if needed |
Currently OTA updates involve the use of (volatile) RTC memory to keep track of update commands across a chip reset. If there is a powerloss part-way through the flashing process the result is likely to be corrupt flash. Ideally these commands would be persisted in something less volatile - like flash.
This PR adds that functionality, and resolves #905.
Update commands are stored in a reserved area of flash that has been inserted just before the start of the FS in flash. This ensures that other things don't end up moving (like sdkwifi pages), that we don't lose FS space, but does cost 2 pages of potential sketch space. This may be an issue on smaller sized devices and it is possible to make the reservation optional - though the last consensus was it was a small enough loss so as not to be an issue.
A pointer to the location of the flash store has been added to the start of the flash, along with a magic number that the bootloader can use to recognise whether or not flash-storage should be checked. In this way it's possible to use the updated bootloader with older sketches without any issues (and means that control over using flash storage should be kept within the sketch).
As it stands the location of the update data hasn't been changed, and so is still in the first 1MB of flash, meaning you have a maximum sketch size half of that (less reserved areas). Once this has received some meaningful testing beyond my boards I anticipate updating the system to allow update data to be stored further on in flash, thus allowing larger OTA updates - assuming it's not already been done; I've not been through the current state beyond what was required to merge in the flash update solution.
boards.txt.pynow generates .ld files that should include all the necessary information, and in the right places.There is now a
README.mdfile in the eboot folder as the start of some descriptions of how it works. It still needs some details added, but realistically all that info is already in the source itself.I've now tested it on real hardware, which works, after a few fixes. 😄