esp8266 Arduino package can't be used as referenced package

[JAndrassy]

Arduino IDE 3rd party Hardware specification section "Referencing another core, variant or tool" specifies that a boards package can be referred from custom hardware definitions or other boards packages.

esp8266 package can't be used now as referred, because platform.txt file entries use variable {runtime.platform.path}, which points to the folder of selected board's boards.txt file. In case of a referring definition it is not the right folder to access sdk and tools.

solved in PR #5770

[JAndrassy]

the PR did not solve it. the solution was to use {build.core.path}/../.. instead of {runtime.platform.path} but in git version not all platform.txt keys can work with {build.core.path}.

other option is to do the replace as part of release preparation

[JAndrassy]

@d-a-v David, after the new release I again try to solve the "referred package" issue. The main problem are paths to the tools folder in platform folder (hardware/esp8266/esp8266/tools) used in platform.txt as {runtime.platform.path}/tools/
I discovered that Arduino IDE creators thought about need of a folder of this kind, but they prescribed a name system to it and then it is accessible as {build.system.path}. This works good if platform is used as referred package.

So if the tools folder would be renamed to system and all (note*) {runtime.platform.path}/tools would be replaced to {build.system.path} we would get closer to make Arduino esp8266 core work as referred.
The {build.system.path} works good in git install too.

note*: there is a new problem in 2.5.1. the

tools.esptool.upload.network_pattern="{network_cmd}" "{runtime.platform.path}/tools/espota.py"

doesn't evaluate anything else then the problematic runtime.platform.path

arduino-sam package (Due) uses the system folder https://github.com/arduino/ArduinoCore-sam/tree/master/system

[devyte]

@JAndrassy I suspect this isn't an issue with wide impact, so you're the main interested party. You started a proposal with #5770, but then dropped it, and it's not clear to me why. Will you be continuing your work, or should this issue just be closed?

[JAndrassy]

the maintainers should be interested, because now every manufacturer of a esp8266 board must add the board to this package and can't maintain own board definitions.

it is hard to work on this if the platform.txt file changes so much like in last months.

would the maintainers accept renaming of the tools folder to system? it would solve problems with the path to tools because system is what Arduino builder expects as name for this kind of folder.

[devyte]

@JAndrassy please make your proposal in a PR, or update #5770. We can discuss further there.

[JAndrassy]

I would need a release version of the current platform.txt. How is it transformed for release?

[d-a-v]

Only version will change in that file, in the next release.

[JAndrassy]

no. there is some edit not tracked in git. tools removed, paths of tools changed

and this line was removed in packaged platform.txt in 2.5.2

recipe.hooks.core.prebuild.2.pattern="{runtime.tools.python.path}/python" "{runtime.tools.makecorever}" --build_path "{build.path}" --platform_path "{runtime.platform.path}" --version "unix-{version}"

[d-a-v]

You are right, sorry.
Changes are made by package/build_boards_manager_package.sh.
You can run it locally, it will create a subdir with the installable archive including the updated platform.txt in it.

[JAndrassy]

So when I opened this issue, esptool was installed as IDE tool, next to mkspiffs and xtensa-lx106-elf-gcc. But then the upload tool changed and is with SDK in tools folder next to core folder. And this path can't be used from a referring package to run commands, because there is no build variable which would evaluate to this path for a command. That is why my PR to solve the referring of the package failed.

So first thing to solve would be to make the upload.py & co installable as IDE tool like mkspiffs or the old esptool.

[JAndrassy]

Of course the problem with the build variables evaluation at Upload is a problem of the Arduino IDE. I investigated it and reported it in May but then new version of IDE doesn't have it fixed.
https://github.com/arduino/Arduino/issues/8886

@earlephilhower please could the py scripts from tools folder be 'external' tools in boards manager package (like the old esptool was or mkspiffs is)?

[d-a-v]

@JAndrassy is that what you need achieved by #5970 #5770 ?
Why did you close it ?
Can you explain what is a referred package and when it is needed ?

[earlephilhower]

@d-a-v , maybe you have the wrong bug? #5970 is GPS and SW serial related.

[earlephilhower]

One gotcha I see here is if @JAndrassy does move tools to system, does that mean the SDK .a and .h includes and all the other bits we have in tools move with them, and can only be updated via a new tools package release?

@JAndrassy, the .pys in the tools folder under git control really need to track with the core because they do things like make the ld files, etc. It's possible, but I think it would be a pain to manage.

[JAndrassy]

sorry for the confusion. the renaming of tools to system doesn't help. it would create nicer paths in platform txt but that is all.

Before the old esptool was replaced with the esptool.py wrapped in upload.py, I made the PR which changed the paths in platform.txt to use other variable then {runtime.platform.path}. But in the same time as I made the PR the esptool was replaced and the change in my PR was not solving the path for upload.py. (the espota.py had the problem too, but for that I could document a workaround for referred packages.)

Arduino builder enables to create boards packages or custom board definitions which refer core and tools of some other boards package. This was made to avoid copies of complete board packages for definition of board variants. Here I have examples of custom boards definitions referring AVR and SAMD boards packages. And here is example of Sparkfun AVR boards package which refers the Arduino AVR package. Only board variants are defined. The core and tools are from the Arduino package.

The problem with esp8266 boards package as referred is the use of {runtime.platform.path} to evaluate the tools folder. The {runtime.platform.path} evaluates to referring package, not to referred where the tools folder is. I attempted to replace it with {build.core.path}/../../tools or {build.system.path}, but because of the implementation in the IDE they are not evaluated when running commands. {runtime.platform.path} is evaluated.

I want to solve it only for Boards Manager installed package, not for 'git install'.

[earlephilhower]

I think @JAndrassy has a point here, but I'm not sure what the underlying issue is or how to go about fixing is now. Marking for 3.0.0

[JAndrassy]

first step is to make upload.py and espota as tools installed in packages/esp8266/tools. the old esptool was installed this way.

it is not a requirement of the Arduino IDE, but a workaround for a 'feature' (old bug?). the Arduino IDE can't access them in the current location packages/esp8266/hardware/esp8266/x.x.x/tools if the package is reffered.

[earlephilhower]

Do you have, or can you make a sample of what a referred package would be using the 8266 core? I'm thinking of something to test against as the tweaks are done. (i.e. it fails now, but when the core is in some "good" state it magically starts working again)

GOOG doesn't give anything on the Arduino site about it (maybe I'm using the wrong term), so I really don't know what kind of spec we're working towards.

[JAndrassy]

a simple example is a custom boards definition like this one
https://github.com/jandrassy/my_boards/tree/7733d94503afc157bd2651636ee1b7c3db9d840f
(it is a link to an older revision of the repo because I deleted the esp8266 part in August)

the link to specification is in platform.txt at the top
https://github.com/arduino/Arduino/wiki/Arduino-IDE-1.5-3rd-party-Hardware-specification

[d-a-v]

platform specification has moved to https://arduino.github.io/arduino-cli/0.23/platform-specification/

[d-a-v]

@JAndrassy
In your repo, you tell this:

esp8266
For this to work you must replace in installed official platform.txt {runtime.platform.path} to {build.core.path}/../../. (Only until next version of the esp8266 package.)

On a linux installation, both point to the same location.
Are you telling that by simply replacing all occurences of {runtime.platform.path} by {build.core.path}/../../ the issue is solved ?

[JAndrassy]

On a linux installation, both point to the same location.

it is not the same location if 'board platform' is different from 'core platform'

Are you telling that by simply replacing all occurences of {runtime.platform.path} by {build.core.path}/../../ the issue is solved ?

no. because there is a second problem: esptool.py and espota.py don't work from different 'board platform', because Arduino IDE upload doesn't have build.core.path (and build.system.path) evaluated (arduino/arduino-cli#1012).

(Upload tools shouldn't be at that location. They should be installed as 'global' tools like the old esptool was, back then when I made the first PR about this. In esp32 core esptool,py is installed as 'global' tool)

a first good step would be to rename /tools folder to /system and use build.system.path in platform.txt

[d-a-v]

a first good step would be to rename /tools folder to /system and use build.system.path in platform.txt

@JAndrassy Do you plan to propose a PR for this ?

(that you you have been able to test with your global use-case)