Cmake source prep and building #46

mcspr · 2020-02-10T12:35:51Z

As mentioned in the other issue, this is general cmake setup to handle lwip2 source directory, it's patching and building liblwip.a for the esp8266/Arduino project.
How this works:

$ mkdir build
$ cd build/
$ cmake ../ -DARDUINO_DIR=<PATH_TO_THE_ARDUINO_CORE_ROOT> -DCMAKE_PROGRAM_PATH=<PATH_TO_TOOLCHAIN_BIN_DIRECTORY>
$ cmake --build .
# or just `make`
# also, see `make help`

By default, build script will place the resulting library into the output/lib. This can be changed via -DLIB_INSTALL_DIR=<PATH-TO-TOOLS-SDK> (without the lib/, which is appended implicitly by the cmake tools. Not yet sure what is up with that and how to have a single one prefix configuration)
edit: or, DESTDIR=..., which I even forgot about. but that need undo'ing the manual directory management in builder externalproject declaration

cmake/toolchain.cmake, cmake/esp8266-platform.cmake handle the toolchain part. cflags are taken from an existing makefile definitons
cmake/builder.cmake builds glue+lwip in the same step, combining .o as liblwip. Following lwip2 build guidelines:
http://git.savannah.nongnu.org/cgit/lwip.git/tree/BUILDING
http://git.savannah.nongnu.org/cgit/lwip/lwip-contrib.git/tree/ports/unix/example_app/CMakeLists.txt
lwip project already provides lwip2-src/src/Filelists.cmake, and simply by including it registers lwipcore library with a proper source list. Additional modules config is taken from the same file definitions, simply appending module variable to the lwipcore target sources list
cmake/lwip<name>/CMakeLists.txt are supposed to be for-loop replacement from the current makefile, listing things instead of dynamically generating them each time. see scripts/cmake-generate
Root CMakeLists.txt manages generated headers (err_t and hash string, see glue-lwip/*.h.in) and lwip2-src (upkeep, checkout, patch) and building each cmake/lwip<name>
scripts/clean-lwip2-src and scripts/patch-lwip2-src are moved from makefile steps to handle patch invocation and git checkout & git clean

I don't think it is intended to replace existing setup (and with opensdk even wise to do so), but simply to have a general view of how the stuff works. Things are commented as much as possible, pinpointing some questions / problems / quirks with an existing setup.

…ly fix include order

mcspr · 2020-02-11T11:58:57Z

edit: see README-CMake.md

d-a-v · 2020-02-17T11:20:12Z

Should the windows instructions be stored in this repo, in PIO repo, or in Tasmota repo ?

Should the basic instructions be put on a README.cmake file ?

I just tried on linux and compilation works (I have not tested the generated files though).

However I would rename the variable ARDUINO_DIR to ESP8266_ARDUINO_CORE_DIR (because I made it point to the arduino IDE directory the first time I tried - until I realised that one of the points of PIO is to avoid the Arduino IDE :).

I don't know in PIO how things work, but if tools/get.py script is run, compiler is then installed in $(ESP8266_ARDUINO_CORE_DIR)/tools/xtensa-lx106-elf/bin which makes (in that case) CMAKE_PROGRAM_PATH a duplicate configuration parameter.

mcspr · 2020-02-17T14:25:55Z

Should the windows instructions be stored in this repo, in PIO repo, or in Tasmota repo ?
Should the basic instructions be put on a README.cmake file ?

Makes sense to update Building section? PIO is just a shortcut for getting toolchain + esp8266 Core files in the same step. edit: In any case, I will move build instructions (keeping in mind both IDE and PIO)

Windows part too, I think? But I am failing hard at patching though:

when git clones lwip2 source with crlf line endings.
here is also something about git-apply that makes it fail on 0003-dhcp-dhcp6-Ensure-that-DHCP-DHCPv6-DNS-server-update.patch (assuming it is the correct order: 0001, 0002, 0003, everything else alphabetically). Generating full patch from git-format worked though, after using patch and committing the result

I just tried on linux and compilation works (I have not tested the generated files though).

Resulting firmware worked for me btw.

However I would rename the variable ARDUINO_DIR to ESP8266_ARDUINO_CORE_DIR (because I made it point to the arduino IDE directory the first time I tried - until I realised that one of the points of PIO is to avoid the Arduino IDE :).

I don't know in PIO how things work, but if tools/get.py script is run, compiler is then installed in $(ESP8266_ARDUINO_CORE_DIR)/tools/xtensa-lx106-elf/bin which makes (in that case) CMAKE_PROGRAM_PATH a duplicate configuration parameter.

Makes sense, will rename and try to search for files there beforehand. And I also haven't done "release" install target yet to copy headers to the Core directory

PIO toolchain and framework are separate packages:

~/.platformio/packages/toolchain-xtensa
~/.platformio/packages/framework-arduinoespressif8266
As there could be multiple versions of the same package installed (with @version suffixes) and user can easily swap those in platformio.ini project config file

patches/0003-dhcp-dhcp6-Ensure-that-DHCP-DHCPv6-DNS-server-update.patch

mcspr · 2020-02-21T13:57:29Z

put generated, glue dhcpserver and lwip2 headers into the tools/sdk/lwip2

"this was generated" README

cmake/package now handles header generation and installation and is loaded as externalproject. Reason for doing that is that external project target will automatically install everything and we don't need to have to do make install, everything is installed just by running make

…tion

mcspr · 2021-07-29T15:17:43Z

Have not tested that the binaries actually work, but patches apply and everything seems to be merged together correctly on Windows. And b/c it was 2y ago, need to go over the CFLAGS, lwipopts.h, etc. and check whether those are actually the same as the Makefile build

There are also these after the install phase, in the esp8266 repo:

Untracked files:
  (use "git add <file>..." to include in what will be committed)
        tools/sdk/lwip2/include/compat/
        tools/sdk/lwip2/include/lwip/altcp.h
        tools/sdk/lwip2/include/lwip/altcp_tcp.h
        tools/sdk/lwip2/include/lwip/altcp_tls.h
        tools/sdk/lwip2/include/lwip/apps/altcp_proxyconnect.h
        tools/sdk/lwip2/include/lwip/apps/altcp_tls_mbedtls_opts.h
        tools/sdk/lwip2/include/lwip/apps/fs.h
        tools/sdk/lwip2/include/lwip/apps/http_client.h
        tools/sdk/lwip2/include/lwip/apps/httpd.h
        tools/sdk/lwip2/include/lwip/apps/httpd_opts.h
        tools/sdk/lwip2/include/lwip/apps/lwiperf.h
        tools/sdk/lwip2/include/lwip/apps/mqtt.h
        tools/sdk/lwip2/include/lwip/apps/mqtt_opts.h
        tools/sdk/lwip2/include/lwip/apps/mqtt_priv.h
        tools/sdk/lwip2/include/lwip/apps/netbiosns.h
        tools/sdk/lwip2/include/lwip/apps/netbiosns_opts.h
        tools/sdk/lwip2/include/lwip/apps/smtp.h
        tools/sdk/lwip2/include/lwip/apps/smtp_opts.h
        tools/sdk/lwip2/include/lwip/apps/snmp.h
        tools/sdk/lwip2/include/lwip/apps/snmp_core.h
        tools/sdk/lwip2/include/lwip/apps/snmp_mib2.h
        tools/sdk/lwip2/include/lwip/apps/snmp_opts.h
        tools/sdk/lwip2/include/lwip/apps/snmp_scalar.h
        tools/sdk/lwip2/include/lwip/apps/snmp_snmpv2_framework.h
        tools/sdk/lwip2/include/lwip/apps/snmp_snmpv2_usm.h
        tools/sdk/lwip2/include/lwip/apps/snmp_table.h
        tools/sdk/lwip2/include/lwip/apps/snmp_threadsync.h
        tools/sdk/lwip2/include/lwip/apps/snmpv3.h
        tools/sdk/lwip2/include/lwip/apps/tftp_opts.h
        tools/sdk/lwip2/include/lwip/apps/tftp_server.h
        tools/sdk/lwip2/include/netif/bridgeif.h
        tools/sdk/lwip2/include/netif/bridgeif_opts.h
        tools/sdk/lwip2/include/netif/ieee802154.h
        tools/sdk/lwip2/include/netif/lowpan6.h
        tools/sdk/lwip2/include/netif/lowpan6_ble.h
        tools/sdk/lwip2/include/netif/lowpan6_common.h
        tools/sdk/lwip2/include/netif/lowpan6_opts.h
        tools/sdk/lwip2/include/netif/ppp/ccp.h
        tools/sdk/lwip2/include/netif/ppp/chap-md5.h
        tools/sdk/lwip2/include/netif/ppp/chap-new.h
        tools/sdk/lwip2/include/netif/ppp/chap_ms.h
        tools/sdk/lwip2/include/netif/ppp/eap.h
        tools/sdk/lwip2/include/netif/ppp/ecp.h
        tools/sdk/lwip2/include/netif/ppp/eui64.h
        tools/sdk/lwip2/include/netif/ppp/ipv6cp.h
        tools/sdk/lwip2/include/netif/ppp/magic.h
        tools/sdk/lwip2/include/netif/ppp/mppe.h
        tools/sdk/lwip2/include/netif/ppp/polarssl/
        tools/sdk/lwip2/include/netif/ppp/ppp_impl.h
        tools/sdk/lwip2/include/netif/ppp/pppapi.h
        tools/sdk/lwip2/include/netif/ppp/pppcrypt.h
        tools/sdk/lwip2/include/netif/ppp/pppdebug.h
        tools/sdk/lwip2/include/netif/ppp/pppoe.h
        tools/sdk/lwip2/include/netif/ppp/pppol2tp.h
        tools/sdk/lwip2/include/netif/ppp/upap.h
        tools/sdk/lwip2/include/netif/slipif.h
        tools/sdk/lwip2/include/netif/zepif.h

not sure what's the difference there just yet, I don't remember these extra includes copied

d-a-v · 2021-09-13T21:05:01Z

There are also these after the install phase, in the esp8266 repo:

These harmless files are unused copied from lwip's iclude files.
Those which are not currently added / committed in esp8266/Arduino are useless.

I understand that this PR needs some more checks or am I wrong ?

mcspr · 2021-09-14T07:31:05Z

These harmless files are unused copied from lwip's iclude files.
Those which are not currently added / committed in esp8266/Arduino are useless.

Right now it simply does PATTERN "*.h" for everything under include/. It could either be a PATTERN "paths/we/dont/like" EXCLUDE" or a list of required files.

I understand that this PR needs some more checks or am I wrong ?

I was able to bootstrap it again from the README, so I think it's done. Let me do a once-over and wip could be removed. Have you tried it btw?

and idk about the open-sdk part, but that may be added later.

patches/dhcpv6-dns-fix-v2.patch

mcspr added 12 commits February 8, 2020 05:59

Build with cmake

785de0a

generate cmake variant projects

2764c55

add .in headers

b4be8fc

use ExternalProject for git tracking

b2b2981

add sntp and mdns to app list

e33e602

point to the same lwip2-src as makefile, add install lib argument

0028e8e

(?) -debug and -esp targets, fix not having esp side of funcs. possib…

792a2db

…ly fix include order

always run ep cmake

8c911f6

note about the approach with ep

0b67903

Nicer names, notes

838ea2d

ignore output dir

51cb38f

tweak comments

7cd9cca

mcspr added 14 commits February 18, 2020 14:55

ARDUINO_DIR -> ESP8266_ARDUINO_CORE_DIR

513936d

less args, try to support win32

a1f561e

native paths

ffe9bb4

there is no WIN32

540cd9e

adjust patch series

d538abc

fix win32 detection (again)

47f7915

win32 can finally patch

3ccc318

fix win32, use what cmake generated earlier

0ffcf44

add notice about the PIO toolchain

143b198

readme-cmake

38c820f

detect CORE_DIR too, group settings

b18bfe9

adjust relative path of core root

fd1e56f

mention wsl, note about the nmake option

567170c

mark both vars as filepaths

f918e64

mcspr commented Feb 18, 2020

View reviewed changes

patches/0003-dhcp-dhcp6-Ensure-that-DHCP-DHCPv6-DNS-server-update.patch Outdated Show resolved Hide resolved

mcspr added 5 commits February 18, 2020 23:16

vs c++ dev tools also install ninja

64df783

readme, variants, autogenerated warnings, nicer include path

3ad0af5

paths

87a19b1

separate cmakelist for packaging

6153509

should depend on lwip2-src

fc76fc1

mcspr added 8 commits July 28, 2021 18:14

Merge remote-tracking branch 'origin/master' into cmake-source-prep

d52e903

wip

a0c7a4b

oops

d13c677

cmake toolchain fixes

5b51f91

no need for extra dirs

7b2992c

notice about line-endings on windows

a40c3d3

up-to-date cmake toolchain instructions, add manual windows build sec…

d70337a

…tion

correctly install everything through install()

fafa183

mcspr mentioned this pull request Sep 9, 2021

DHCPS included in the Core and the old SDK implementation esp8266/Arduino#8307

Closed

6 tasks

arduino-core already has a server

e542c8f

mcspr added 4 commits September 14, 2021 11:09

mention parallel jobs

cbc2482

duplicate url

6066444

quickfix for unused headers that are distributed with the Core

a0b0067

fixup! quickfix for unused headers that are distributed with the Core

801c729

d-a-v reviewed Sep 21, 2021

View reviewed changes

patches/dhcpv6-dns-fix-v2.patch Outdated Show resolved Hide resolved

mcspr added 3 commits September 28, 2021 08:20

Merge remote-tracking branch 'origin/master' into cmake-source-prep

8e8b9f5

revert to original patch

d84f6e0

version bump

c7d3372

d-a-v merged commit 99a47a9 into d-a-v:master Nov 29, 2021

d-a-v mentioned this pull request Nov 29, 2021

lwIP: v2.1.3 + dhcp fixes esp8266/Arduino#8319

Merged

d-a-v changed the title ~~[wip?] Cmake source prep and building~~ Cmake source prep and building Nov 30, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Cmake source prep and building #46

Cmake source prep and building #46

mcspr commented Feb 10, 2020 •

edited

Loading

mcspr commented Feb 11, 2020 •

edited

Loading

d-a-v commented Feb 17, 2020

mcspr commented Feb 17, 2020 •

edited

Loading

mcspr commented Feb 21, 2020

mcspr commented Jul 29, 2021 •

edited

Loading

d-a-v commented Sep 13, 2021

mcspr commented Sep 14, 2021 •

edited

Loading

Cmake source prep and building #46

Cmake source prep and building #46

Conversation

mcspr commented Feb 10, 2020 • edited Loading

mcspr commented Feb 11, 2020 • edited Loading

d-a-v commented Feb 17, 2020

mcspr commented Feb 17, 2020 • edited Loading

mcspr commented Feb 21, 2020

mcspr commented Jul 29, 2021 • edited Loading

d-a-v commented Sep 13, 2021

mcspr commented Sep 14, 2021 • edited Loading

mcspr commented Feb 10, 2020 •

edited

Loading

mcspr commented Feb 11, 2020 •

edited

Loading

mcspr commented Feb 17, 2020 •

edited

Loading

mcspr commented Jul 29, 2021 •

edited

Loading

mcspr commented Sep 14, 2021 •

edited

Loading