Document how to use flash programming

[sbridger]

Can you make a wiki page / doc about using flash write feature?

How does a program page erase and byte write the flash?
How does program find the addresses to call? (in a version independent fashion)
How does it tell if the installed optiboot version has support for flash writing?

I have just been discussing/proposing making Flashforth optiboot loadable. The issue being that Forth must write its flash to work (being interactive).

[MCUdude]

I don't have time to do this at the moment. You can find an example on how to use the flash write functionality here: https://github.com/MCUdude/MiniCore/tree/master/avr/libraries/Optiboot_flasher

[sbridger]

OK thanks, not urgent at all. Optiboot.h seems reasonably clear.

So the do_spm entry vector is 2bytes above the bottom of the bootloader, and that entry point works for all the functions.
Where are the constants __BOOT_PAGE_XXXX defined [boot.h + ?.h ]
(Do you know the values for 328P off the top of your head) ?

What does this optiboot do with addresses inside itself? i.e if the hexfile also contains optiboot, what will happen? [i.e can we have a single hex file including optiboot, flash it with programmer, then send it to optiboot later]

Can this optiboot modify its own address space i.e. be used to update/replace itself (by bootloading)?
Can these flash routines be used by a program to update/replace optiboot?

---

Of course the real sticking point is that without this optiboot version installed in the arduino factory, you have to resort to a programmer to replace the bootloader anyway.

I suppose that if there was a standard version bootloader from the factory(s) , and if you can call routines into it at known addresses, then in theory you could make it overwrite itself in two steps until a new optiboot was fitted.

[DRSDavidSoft]

@MCUdude Sorry to post here, but I have a burning question, if the program has the ability to write flash pages, how do we guarantee that writing to flash doesn't corrupt the running program code?

In bootloader mode, we can guarantee this by always writing to the application sectors (from 0x0000), but if the code is running from the application section, what happens if we write to that flash area?

[MCUdude]

The bootloader is located in the upper part of flash memory (upper 512 or 1024 bytes depending on the AVR in use). This section is assigned using the fuse bits. A bootloader cannot erase itself, because the bootloader section is protected, which means that code running in the bootloader memory section can only modify flash pages outside the bootloader area.

if the code is running from the application section, what happens if we write to that flash area?

An AVR that supports a bootloader section (Line the ATmega328P) can't write to the flash memory without utilizing a bootloader that supports this, for instance, Optiboot flash or Urboot, which is better an Optiboot in any way. However, if you utilize this flash memory write function to write to arbitrary flash addresses, you may risk breaking the running program due to "sawing off the branch you're sitting on". What you should do instead is to allocate som PROGMEM space in your code that's "reserved" for this kind of use.

See the example here: https://github.com/MCUdude/MiniCore/blob/master/avr/libraries/Optiboot_flasher/examples/Flash_put_get/Flash_put_get.ino

(BTW I'm soon going to release a new version of all my Arduino cores where I've replaced Optiboot flash with Urboot. Urboot occupies less flash space (384 bytes instead of 512 bytes), has auto baud support (no need to reflash the bootloader if you want to use a different baud rate), supports EEPROM read/write, supports metadata for the uploaded program and more). Avrdude is also backwards compatible, so Avrdude doesn't care which bootloader you're using; it will work regardless)

[DRSDavidSoft]

@MCUdude Thanks for the information and speedy reply! By upper flash memory, I assume the last 512 or 1024 bytes that comes at the end of the flash space, correct? So, it starts at either 32K-512 (32256, or 7E00h) or 32K-1024 (31744, or 7C00h) and will end right on 8000h.

The major concern was the application space, from 0x0000 to the start of the boot sector, namely what would happen if I tried to update the entire firmware containing the application code from the beginning to the end, while the code was running. If I were to allocate some PROGMEM for this purpose, I'd need to write some complex logic to handle firmware updates this way. As I understand it, in order to perform SPM instructions, I explicitly need to be in the bootloader space.

Thanks for introducing Urboot, I hadn't heard of it earlier. It seems to be a promising bootloader! 😄 I've been meaning to fork Optiboot to add some missing functionality for my project, such as receiving the firmware from a RS-485 BUS and/or driving a LCD to report update progress. I guess it'd be better to understand and work on Urboot instead of Optiboot. Optiboot served the community well these past years.

On another note, a big thanks for all the work you've done on these cores, I've used most of them in my projects and I always wanted to thank you for your amazing work, so thank you! 🤝 Looking forward to the future releases of your cores!

P.S. I would love it if you could please add some examples to the Urboot-supported edition/branch of the cores that would allow the functionality of updating the firmware while the application code is running, if it would work. I'm desperately in need of such feature... 😞

[MCUdude]

By upper flash memory, I assume the last 512 or 1024 bytes that comes at the end of the flash space, correct? So, it starts at either 32K-512 (32256, or 7E00h) or 32K-1024 (31744, or 7C00h) and will end right on 8000h.

Yes, that's correct. And the size of the bootloader section is decided by the fuse bits.

The major concern was the application space, from 0x0000 to the start of the boot sector, namely what would happen if I tried to update the entire firmware containing the application code from the beginning to the end, while the code was running.

The ArduinoOTA project supports over-the-air updating AVRs using Optiboot flash. I've never tried it, but I think it works well. I'm not sure what the best solution for self-updating AVRs would be. I know others have designed dedicated bootloaders where the AVR runs code from within the bootloader section when reprogramming itself. If it falls for some reason, it will just stay in the bootloader memory waiting for a new program to be received. I'm not sure how it would be accomplished using "write to flash" functionality from the user application. @stefanrueger, the author of Urboot may have a clue.

On another note, a big thanks for all the work you've done on these cores, I've used most of them in my projects and I always wanted to thank you for your amazing work, so thank you! 🤝 Looking forward to the future releases of your cores!

You're welcome! They have always just been side projects, but it looks like they are still very popular among seasoned hobbyists, even though Arduino themselves are doing their best to move away from AVRs. I learned to program microcontrollers on an Arduino Duemilanove, and I've never been particularly interested in dealing with the Arduino board itself, with shields and all that. I'd much rather have a convenient breakout board of some sort, and connect external things to the GPIO pins I wanted.

The next release of my cores would, as mentioned, bundle Urboot in favor of Optiboot flash. I still have a lot of work to do, and I also have to write the entire PlatformIO integration as well.

P.S. I would love it if you could please add some examples to the Urboot-supported edition/branch of the cores that would allow the functionality of updating the firmware while the application code is running, if it would work. I'm desperately in need of such feature... 😞

I'm not really a bootloader expert, and I've never had the need for a device that has to be able to program itself. Urboot states that it supports...

Dual programming from external SPI flash memory for over-the-air programming

And:

Watchdog timeout resets the MCU just like an external reset. When the bootloader is entered under this condition, though, it then normally jumps directly to the application. However, when the bootloader was compiled with dual-boot support, it first checks external SPI flash memory to see it contains a new sketch, in which case a copy of it is burned from external memory onto internal flash. The idea is that the application sketch could have been written to receive a new version via a radio and have placed it onto the external SPI flash memory before issuing a WDT reset. Hence, this dual-boot property is also called over the air programming, although one could conceivably also plug a new external SPI flash into a board for programming.

But I've never tried it and don't know how it works. An external SPI flash chip is dirt cheap, so that shouldn't be a problem. I think this sounds like a good solution! An external flash chip that can store the incoming program, and a bootloader that can deal with the rest?