Add a CRC32 over progmem and ESP.checkFlashCRC #6566

earlephilhower · 2019-09-29T08:35:18Z

Automatically embed a CRC32 of the program memory (including bootloader
but excluding any filesystems) in all images in unused space in the
bootloader block.

Add a call, ESP.checkFlashCRC() which returns false if the calculated
CRC doesn't match the one stored in the image (i.e. flash corruption).

Fixes #4165

Automatically embed a CRC32 of the program memory (including bootloader but excluding any filesystems) in all images in unused space in the bootloader block. Add a call, ESP.checkFlashCRC() which returns false if the calculated CRC doesn't match the one stored in the image (i.e. flash corruption). Fixes esp8266#4165

cores/esp8266/Esp.cpp

Show CRC checking catch a 1-bit error in program code by corrupting a large array, and then return it to clean and verify the CRC matches once again. Add comments to the CRC check routine Clean up pylint complaints on crc32bin.py

Add an assertion in the eboot linker file to guarantee that we have at least 8 bytes of unused space at the end of the boot sector to patch in the CRC. If not, the eboot link will fail.

@d-a-v

Per discussion with @d-a-v. When the CRC check fails, you could *try* to do certain things (but may not succeed since there is known flash corruption at that point). List a few ideas for application authors.

platform.txt

@mcspr

Per discussion w/@mcspr, combine the CRC calculation with the binary generation, removing the additional build step.

dok-net · 2019-12-21T14:33:13Z

@earlephilhower Is this supposed to work out of the box just yet? Like pulling master, restarting the IDE, rebuilding my sketch.

I get on one board:

esptool.py v2.8
Serial port COM3
Connecting....
Chip is ESP8266EX
Features: WiFi
Crystal is 26MHz
MAC: […]
Uploading stub...
Running stub...
Stub running...
Configuring flash size...
Auto-detected Flash size: 16MB
Flash params set to 0x0290
Compressed 278784 bytes to 205050...
Writing at 0x00000000... (7 %)
[…]
Writing at 0x00030000... (100 %)
Wrote 278784 bytes (205050 compressed) at 0x00000000 in 18.9 seconds (effective 118.2 kbit/s)...
	The upload process has finished.
Hash of data verified.
Leaving...
Hard resetting via RTS pin...

Repeater example for EspSoftwareSerial
CRC check: ERROR
Vcc = 3020V

But, on another board:

esptool.py v2.8
Serial port COM5
Connecting....
Chip is ESP8266EX
Features: WiFi
Crystal is 26MHz
MAC: […]
Uploading stub...
Running stub...
Stub running...
Configuring flash size...
Auto-detected Flash size: 4MB
Compressed 278784 bytes to 205049...
Writing at 0x00000000... (7 %)
[…]
Writing at 0x00030000... (100 %)
Wrote 278784 bytes (205049 compressed) at 0x00000000 in 18.1 seconds (effective 123.3 kbit/s)...
	The upload process has finished.
Hash of data verified.
Leaving...
Hard resetting via RTS pin...

Repeater example for EspSoftwareSerial
CRC check: OK
Vcc = 3041V

Immediately noticeable is the 1 byte difference in compressed size.

mcspr · 2019-12-21T15:08:43Z

tools/elf2bin.py

@@ -30,6 +30,9 @@
 ffreqb = { '40': 0, '26': 1, '20': 2, '80': 15 }
 fsizeb = { '512K': 0, '256K': 1, '1M': 2, '2M': 3, '4M': 4, '8M': 8, '16M': 9 }

+crcsize_offset = 4088
+crcval_offset = 4092


Hm. I think it is off by one error, starting index should be 4087?

@mcspr Build+upload with crcsize_offset = 4087 crashes on boot.

Right, nvm :)

dok-net · 2019-12-21T15:20:43Z

@earlephilhower Ah, OK, I never knew that I HAVE TO select the precise right board w.r.t. flash size. Building for LOLIN D1 mini Pro (16MB flash settings in config) it's fine:

Repeater example for EspSoftwareSerial
CRC check: OK
Vcc = 3018V

Please allow one remark: could you be more precise in the readme on the use of
ADC_MODE(ADC_VCC); - my first quite naïve take on "at the top of your sketch" was to place it inside setup() ;-)
How does this CRC complement, or is superseded, by signing the sketch? Would using an auto-generated keypair not have been an equal solution, albeit more tamper proof, but perhaps more resource consuming?

earlephilhower · 2019-12-21T17:25:01Z

@dok-net, your comment explains it! If you send the wrong flash size, the Updater will change one byte silently to match what's already set. So the CRC check did succeed and found an error...what was in flash was not what you sent in! :)

This was related to another bug (already fixed) where the Updater was always changing 1 byte of an uploaded image, even when it was a FS.

Signing guarantees that you created the bin being uploaded, and is a 1-time check on upload only.

CRC checks dynamically (when you call it) for changes to the flash vs. when you generated the bitstream. Random bit flips, flash page failures, etc.

d-a-v reviewed Sep 30, 2019

View reviewed changes

cores/esp8266/Esp.cpp Outdated Show resolved Hide resolved

earlephilhower added 7 commits September 30, 2019 16:18

Add example that currupts itself, comments

79d0197

Show CRC checking catch a 1-bit error in program code by corrupting a large array, and then return it to clean and verify the CRC matches once again. Add comments to the CRC check routine Clean up pylint complaints on crc32bin.py

Merge branch 'master' into crc32

7011265

Merge branch 'master' into crc32

b7e9f01

Merge branch 'master' into crc32

4a50bab

Update Esp.cpp

3fca89e

Update Esp.cpp

fcfa48c

Fix example astyle problems

00929d1

s0170071 mentioned this pull request Nov 7, 2019

Run CRC at build time letscontrolit/ESPEasy#1598

Closed

earlephilhower added this to the 2.7.0 milestone Nov 11, 2019

earlephilhower added 3 commits December 9, 2019 15:43

Check linker script for CRC space in bootsector

49180aa

Add an assertion in the eboot linker file to guarantee that we have at least 8 bytes of unused space at the end of the boot sector to patch in the CRC. If not, the eboot link will fail.

Merge branch 'master' into crc32

95e64e4

Fix order of crc/len in linker script

800ebe5

devyte assigned earlephilhower Dec 18, 2019

Merge branch 'master' into crc32

23ecb8e

earlephilhower requested review from devyte and d-a-v December 18, 2019 17:23

d-a-v approved these changes Dec 19, 2019

View reviewed changes

Add note about what to do if CRC check fails

77d617a

Per discussion with @d-a-v. When the CRC check fails, you could *try* to do certain things (but may not succeed since there is known flash corruption at that point). List a few ideas for application authors.

mcspr reviewed Dec 19, 2019

View reviewed changes

platform.txt Outdated Show resolved Hide resolved

earlephilhower added 4 commits December 20, 2019 08:11

Only single, flash/ram friendly crc32() function

985b6c5

Combine the CRC calc and bin generation in 1 step

379b167

Per discussion w/@mcspr, combine the CRC calculation with the binary generation, removing the additional build step.

Merge branch 'master' into crc32

51fec2c

Merge branch 'master' into crc32

365a9d0

mcspr approved these changes Dec 20, 2019

View reviewed changes

earlephilhower merged commit 52d84b1 into esp8266:master Dec 20, 2019

earlephilhower deleted the crc32 branch December 20, 2019 22:38

mcspr reviewed Dec 21, 2019

View reviewed changes

pidloop mentioned this pull request Oct 9, 2020

Is OTA detecting bad FLASH now? #7647

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add a CRC32 over progmem and ESP.checkFlashCRC #6566

Add a CRC32 over progmem and ESP.checkFlashCRC #6566

earlephilhower commented Sep 29, 2019

dok-net commented Dec 21, 2019

mcspr Dec 21, 2019

dok-net Dec 21, 2019

mcspr Dec 21, 2019

dok-net commented Dec 21, 2019 •

edited

Loading

earlephilhower commented Dec 21, 2019

Add a CRC32 over progmem and ESP.checkFlashCRC #6566

Add a CRC32 over progmem and ESP.checkFlashCRC #6566

Conversation

earlephilhower commented Sep 29, 2019

dok-net commented Dec 21, 2019

mcspr Dec 21, 2019

Choose a reason for hiding this comment

dok-net Dec 21, 2019

Choose a reason for hiding this comment

mcspr Dec 21, 2019

Choose a reason for hiding this comment

dok-net commented Dec 21, 2019 • edited Loading

earlephilhower commented Dec 21, 2019

dok-net commented Dec 21, 2019 •

edited

Loading