Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

The C++ extension seems not able to recognise the toolchain installed via xpm #28

Open
ilg-ul opened this issue Dec 12, 2021 · 39 comments
Assignees
Labels
bug Something isn't working

Comments

@ilg-ul
Copy link
Contributor

ilg-ul commented Dec 12, 2021

The problem was reported as #27.

The output console displays:

[12/12/2021, 18:20:24] Unable to resolve configuration with compilerPath: "C:\Users\ilg\Desktop\gcc-test\xpacks\.bin\g++"

In case a different toolchain is installed somewhere, it will be picked, and IntelliSense will be able to process the header, but it is not correct.

@ilg-ul ilg-ul self-assigned this Dec 12, 2021
@ilg-ul ilg-ul added the bug Something isn't working label Dec 12, 2021
@ilg-ul
Copy link
Contributor Author

ilg-ul commented Dec 12, 2021

@nicreuss, I suspect the problem to be caused by VS Code not being able to start the applications via the .cmd shims required on Windows, a solution used by npm/xpm to compensate for the lack of symbolic links to files.

In other words, the program referred in compile_commands.json as ...\xpacks\.bin\g++, does not exist, nor a g++.exe exists in that folder; instead, there is a g++.cmd script, which must be started via cmd.exe, and which will start the actual g++.exe.

I will investigate during the next days, but if a patch in the C++ extension is needed, it'll take some time.

@nicreuss
Copy link

I share your suspicions.

The error seems to occur when the C/C++ extension queries the compiler.

With the following configuration in c_cpp_properties.json

        {
            "name": "Debug",
            "compileCommands": "${workspaceFolder}\\build\\debug\\compile_commands.json",
            "compilerPath": "${workspaceFolder}\\xpacks\\.bin\\gcc"
        },

The C/C++ extension debug log gives:

Attempting to get defaults from C compiler in "compilerPath" property: 'C:\Users\nic\Desktop\gcc-test\xpacks\.bin\gcc'
Compiler query command line: "C:\\Users\\nic\\Desktop\\gcc-test\\xpacks\\.bin\\gcc" -std=c17 -Wp,-v -E -dD -x c -m64  nul
Failed to spawn process. Error: 2 (The system cannot find the file specified.)
Failed to query default include paths and defines for C:\Users\nic\Desktop\gcc-test\xpacks\.bin\gcc.

@ilg-ul
Copy link
Contributor Author

ilg-ul commented Dec 12, 2021

Thank you for investigating.

The solution in these cases is to start the executable via a sub-shell, like cmd.exe /C "command".

If you know some magic setting in VS Code to do this, it would be useful.

The more elaborate solution requires a patch in the C/C++ extension and a test, if the command exists and is an .exe, the process can be spawned directly, otherwise if there is a related .cmd, it must be started via cmd.exe.

I don't know the workflow for submitting patches to VS Code; first I have to find a solution and then to figure out how to send a pull request.

@ilg-ul
Copy link
Contributor Author

ilg-ul commented Dec 12, 2021

Another alternate solution would be to create symlinks for files, as on Linux/macOS. This requires enabling developer mode, and works only on recent Windows 10 and newer.

I avoided doing so for now, because the idea to have to enable a developer mode is kind of stupid, but in some cases it might make sense.

VS Code must be fixed anyway, but it might take some time.

@nicreuss
Copy link

By explicitly including .cmd in the configuration, all IntelliSense errors disappear

{
            "name": "Debug",
            "compileCommands": "${workspaceFolder}\\build\\debug\\compile_commands.json",
            "compilerPath": "${workspaceFolder}\\xpacks\\.bin\\gcc.cmd"
}

image

This does not seem ideal for cross platform development. I will check if the configuration can be tailored to each platform.

@ilg-ul
Copy link
Contributor Author

ilg-ul commented Dec 12, 2021

Ok, that's one step forward. Try to update the CMake configuration to make it write the extension.

This is not ideal for cross platform development in general, but in this case it does not matter, since the file is generated in the build folder, which is not persistent, it is not committed in repos, and not moved to a different platform, it is simply regenerated on a new platforms.

@nicreuss
Copy link

That is a very good point!

Your suggestion solves the issue. I'm not well practiced with CMake but the code below seems to do the trick.
If there is a better way of doing it?

if("${CMAKE_HOST_SYSTEM_NAME}" STREQUAL "Windows")
  set(CMAKE_C_COMPILER "${CMAKE_C_COMPILER}.cmd")
  set(CMAKE_CXX_COMPILER "${CMAKE_CXX_COMPILER}.cmd")
endif()

@ilg-ul
Copy link
Contributor Author

ilg-ul commented Dec 12, 2021

CMake is not my thing either, but your code seems fine.

Until a solution will be available, I'll update the template to use this workaround.

@nicreuss
Copy link

Awesome! Thank you @ilg-ul for all your hard work on the xPack ecosystem.

@ilg-ul
Copy link
Contributor Author

ilg-ul commented Dec 12, 2021

Thank you for your help to improve it!

@ilg-ul
Copy link
Contributor Author

ilg-ul commented Dec 13, 2021

I'll update the template to use this workaround

@AaronFontaine-DojoFive
Copy link

The workaround described above did not work for me (I will give details on that below).

However, I have a different workaround. I added compilerPath to c_cpp_properties.json as follows. When set in c_cpp_properties.json, it will override the compiler paths provided by compiler_commands.json.

    {
      "name": "Debug",
      "intelliSenseMode": "linux-gcc-arm",
      "compilerPath": "${workspaceFolder}/xpacks/xpack-dev-tools-arm-none-eabi-gcc/.content/bin/arm-none-eabi-gcc"
    },

I've noticed that the xPack C/C++ Managed Build Tools extension will not interfere with the compilerPath setting even though it updates c_cpp_properties.json.

This form is cross-platform compatible. The symlink is now on the xpack-dev-tools-arm-none-eabi-gcc folder instead of the final executable. Invocation through cmd.exe is no longer required. VS Code appears able to follow folder paths that consist of symlinks. It is also not necessary to append .exe to the name of the compiler. The C/C++ extension (I'm assuming) is able to figure that out.

This is a non-ideal solution however as it requires adding c_cpp_properties.json to the repo. This file is modified by the xPacks extension and should be considered a build output.
Build from Windows:

      "compileCommands": "${workspaceFolder}\\build\\stm\\debug\\compile_commands.json"

Build from WSL2/Ubuntu 22.04:

      "compileCommands": "${workspaceFolder}/build/stm/debug/compile_commands.json"

Would it be possible to modify the xPacks extension so that it sets compilerPath in c_cpp_properties.json as follows?

      "compilerPath": "${workspaceFolder}/xpacks/xpack-dev-tools-arm-none-eabi-gcc/.content/bin/arm-none-eabi-gcc"

@AaronFontaine-DojoFive
Copy link

As far as modifying the toolchain file to append .cmd to the compiler and assembler names, this appears to interfere with CMake's autodetection of the binutils. This autodetection appears to be an undocumented feature of CMake.

If I set only CMAKE_C_COMPILER, CMAKE_CXX_COMPILER, and CMAKE_ASM_COMPILER in the toolchain file, then CMakeCache.txt magically gets populated with entries for CMAKE_AR, CMAKE_OBJCOPY, CMAKE_OBJDUMP and so forth:

//Path to a program.
CMAKE_OBJCOPY:FILEPATH=C:/Users/AFont/Documents/Projects/DojoFive/templates/cmake/stm32-template/xpacks/.bin/arm-none-eabi-objcopy

//Path to a program.
CMAKE_OBJDUMP:FILEPATH=C:/Users/AFont/Documents/Projects/DojoFive/templates/cmake/stm32-template/xpacks/.bin/arm-none-eabi-objdump

If I try to set them explicitly, then they do not show up in CMakeCache.txt.

This auto-detection scheme seems to choke when .cmd is appended, at least when using the arm-none-eabi-gcc set of compilers.

-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - failed
-- Check for working C compiler: C:/Users/AFont/Documents/Projects/DojoFive/templates/cmake/stm32-template/xpacks/.bin/arm-none-eabi-gcc.cmd
-- Check for working C compiler: C:/Users/AFont/Documents/Projects/DojoFive/templates/cmake/stm32-template/xpacks/.bin/arm-none-eabi-gcc.cmd - broken
CMake Error at C:/Users/AFont/AppData/Roaming/xPacks/@xpack-dev-tools/cmake/3.20.6-2.1/.content/share/cmake-3.20/Modules/CMakeTestCCompiler.cmake:66 (message):
  The C compiler

    "C:/Users/AFont/Documents/Projects/DojoFive/templates/cmake/stm32-template/xpacks/.bin/arm-none-eabi-gcc.cmd"

  is not able to compile a simple test program.

  It fails with the following output:

    Change Dir: C:/Users/AFont/Documents/Projects/DojoFive/templates/cmake/stm32-template/build/stm/debug/CMakeFiles/CMakeTmp

    Run Build Command(s):C:/Users/AFont/Documents/Projects/DojoFive/templates/cmake/stm32-template/xpacks/.bin/ninja cmTC_605b6 && [1/2] Building C object CMakeFiles/cmTC_605b6.dir/testCCompiler.c.obj
    [2/2] Linking C static library libcmTC_605b6.a
    FAILED: libcmTC_605b6.a
    cmd.exe /C "cd . && C:\Users\AFont\AppData\Roaming\xPacks\@xpack-dev-tools\cmake\3.20.6-2.1\.content\bin\cmake.exe -E rm -f libcmTC_605b6.a && CMAKE_AR-NOTFOUND qc libcmTC_605b6.a  CMakeFiles/cmTC_605b6.dir/testCCompiler.c.obj && cd ."
    'CMAKE_AR-NOTFOUND' is not recognized as an internal or external command,
    operable program or batch file.
    ninja: build stopped: subcommand failed.

You can see that it attempted to find the ar binutil, but failed as indicated by the name being set to CMAKE_AR-NOTFOUND. CMakeTestCCompiler.cmake attempted to use it anyway, which failed.

@ilg-ul
Copy link
Contributor Author

ilg-ul commented May 3, 2022

Would it be possible to modify the xPacks extension so that it sets compilerPath in c_cpp_properties.json as follows?

I need to think about it, but at first sight I am not convinced that this is a generic solution.

What is the criteria to decide on using that specific path? My projects use multiple toolchains, with different build configurations having their own toolchains, so there is no unique compiler path.

The symlink is now on the xpack-dev-tools-arm-none-eabi-gcc folder instead of the final executable. Invocation through cmd.exe is no longer required. VS Code appears able to follow folder paths that consist of symlinks. It is also not necessary to append .exe to the name of the compiler. The C/C++ extension (I'm assuming) is able to figure that out.

I'm not sure I understand this.

For Windows, links are very tricky, links to folders are functional, but links to files are very problematic, thus the need for .cmd forwarders (shims).

If you have a better idea on how to deal with them, please open separate topics in the Discussions section.

@ilg-ul
Copy link
Contributor Author

ilg-ul commented May 3, 2022

This auto-detection scheme seems to choke when .cmd is appended

Yes, unfortunately CMake is not happy with .cmd in general. But, for Windows, .cmd forwarders seem the only way to overcome the lack of links to files; at least this is the solution used by npm and I don't know a better one.

@AaronFontaine-DojoFive
Copy link

I can open a separate topic in the Discussions section, but I may not be able to get to it for a while.

I think you're right that the compiler path would need to come from compiler_commands.json if there are multiple toolchains. What I'm seeing in the xpacks subfolder is a subfolder called xpack-dev-tools-arm-none-eabi-gcc which itself seems to be a shortcut to the xpacks ARM GCC installation in C:\User\<name>\AppData\Roaming. So, that's what I showed above. Using a path that contains that folder shortcut in the middle to get to the .exe rather than using the .cmd link.

@AaronFontaine-DojoFive
Copy link

cmd.exe knows they're symlinks:

C:\Users\AFont\Documents\Projects\DojoFive\templates\cmake\stm32-template\xpacks>dir
 Volume in drive C is OS
 Volume Serial Number is 7E46-4D9D

 Directory of C:\Users\AFont\Documents\Projects\DojoFive\templates\cmake\stm32-template\xpacks

05/03/2022  03:57 PM    <DIR>          .
05/03/2022  03:57 PM    <DIR>          ..
05/03/2022  03:57 PM    <DIR>          .bin
05/03/2022  03:57 PM    <JUNCTION>     xpack-dev-tools-arm-none-eabi-gcc [C:\Users\AFont\AppData\Roaming\xPacks\@xpack-dev-tools\arm-none-eabi-gcc\10.3.1-2.3.1]
05/03/2022  03:57 PM    <JUNCTION>     xpack-dev-tools-cmake [C:\Users\AFont\AppData\Roaming\xPacks\@xpack-dev-tools\cmake\3.20.6-2.1]
05/03/2022  03:57 PM    <JUNCTION>     xpack-dev-tools-ninja-build [C:\Users\AFont\AppData\Roaming\xPacks\@xpack-dev-tools\ninja-build\1.10.2-5.1]
               0 File(s)              0 bytes
               6 Dir(s)  393,627,623,424 bytes free

@ilg-ul
Copy link
Contributor Author

ilg-ul commented May 3, 2022

I think you're right that the compiler path would need to come from compiler_commands.json if there are multiple toolchains.

Yes, compiler_commands.json should be the authoritative source for IntelliSense, and do not require any manually set properties.

cmd.exe knows they're symlinks

On Windows, links to folders are called junctions, and are functional.

Can you rephrase your proposal, and try to make it clearer? I probably have a bad day and I don't understand your point.

@ilg-ul
Copy link
Contributor Author

ilg-ul commented May 3, 2022

@aaronf-at-d5, did you see a functional project using xpm for managing building and running unit tests? (https://github.com/micro-os-plus/utils-lists-xpack/blob/xpack/README-MAINTAINER.md#testing)

@AaronFontaine-DojoFive
Copy link

Hi @ilg-ul, thank you for sharing that link. I will need some time to go over it. I'm fairly new to xPack. I've only been using it so far for a couple of weeks and only to manage dev environments.

I may be posting on the wrong place here. I am still learning how all these pieces work together. If the xPack C/C++ Managed Build Tools extension shouldn't be populating compilerPath, then the problem may not be resolvable in this extension.

As far as using junctions to link to another location in the system, this seems to work just fine. I decided to do an experiment to test this. Rather than run xpm run build --config Debug, I did the following (I am using Git Bash BTW):

$ cd xpacks/xpack-dev-tools-arm-none-eabi-gcc/.content/bin
$ export PATH=$(pwd):$PATH
$ cd ../../../..
$ cd xpacks/xpack-dev-tools-ninja-build/.content/bin
$ export PATH=$(pwd):$PATH
$ cd ../../../..
$ cd xpacks/xpack-dev-tools-cmake/.content/bin
$ export PATH=$(pwd):$PATH
$ cd ../../../..
$ make debug

Note that I'm using a Makefile shim here as described in the Embedded Artistry CMake course, so make debug runs the CMake configure and Ninja build steps. I did this build from scratch (rm -rf build first) and this built correctly.

In compile_commands.json I now have lines starting with

"command": "C:\\Users\\AFont\\Documents\\Projects\\DojoFive\\templates\\cmake\\stm32-template\\xpacks\\xpack-dev-tools-arm-none-eabi-gcc\\.content\\bin\\arm-none-eabi-gcc.exe

IntelliSense is working correctly with only the following in c_cpp_properties.json:

{
  "configurations": [
    {
      "name": "Debug",
      "compileCommands": "${workspaceFolder}\\build\\stm\\debug\\compile_commands.json"
    }
  ],
  "version": 4
}

@ilg-ul
Copy link
Contributor Author

ilg-ul commented May 4, 2022

If the xPack C/C++ Managed Build Tools extension shouldn't be populating compilerPath, then the problem may not be resolvable in this extension.

The xPack extension is invoking CMake to create the compile_commands.json, and IntelliSense should parse it to get the actual compiler path.

If this mechanism works properly, there is no need to set compilerPath.

If this mechanism does not work properly, either CMake does not generate the correct compile_commands.json, or VS Code does not understand it, and one of them (or both) must be fixed.

On macOS I did not encounter such problems, so they might be Windows specific.

BTW, setting the compiler path is not enough, for accurate IntelliSense you also need to configure the include paths, the defined macros and the compiler options. That's exactly the point why the xPack extension uses compile_commands.json, since this greatly automates the process and guarantees accurate IntelliSense.

@AaronFontaine-DojoFive
Copy link

Hi @ilg-ul, please let me explain my motivation. I understand that the VS Code xPack extension is not working properly on Windows, specifically with regards to IntelliSense. I have been looking into this for a few days with the aim of finding a way to resolve or mitigate the issue. I know that it works correctly on Mac and Linux. That is my context for posting here.

The xPack extension is invoking CMake to create the compile_commands.json, and IntelliSense should parse it to get the actual compiler path.

Yes, I understand this. As I said in my previous post:

If the xPack C/C++ Managed Build Tools extension shouldn't be populating compilerPath, then the problem may not be resolvable in this extension.

This is why I went on to conduct an experiment to see how compile_commands.json could be generated differently while still using the dev environment installed by xpm install. Please review my above post where I repath the developer tools manually rather than using xpm run build. That experiment implies that through different setup of $PATH it is possible to have the build stage run exactly the same but have different compiler paths appear in compile_commmands.json.

Specifically, here is the difference:

When using xpm run build, compile_commands.json ends up with the following path to the compiler:

C:\\Users\\AFont\\Documents\\Projects\\DojoFive\\templates\\cmake\\stm32-template\\xpacks\\.bin\\arm-none-eabi-gcc

When performing the experiment I outlined above, it ends up with this path:

C:\\Users\\AFont\\Documents\\Projects\\DojoFive\\templates\\cmake\\stm32-template\\xpacks\\xpack-dev-tools-arm-none-eabi-gcc\\.content\\bin\\arm-none-eabi-gcc.exe

Both work properly for building the project. The second is interpreted correctly by VS Code IntelliSense.

What this implies is that rather than prepending xpacks/.bin to $PATH, it would work to separately prepend each tool specified in devDependencies using folder paths that involve the junction folders. Again, please see the experiment I demonstrated in the previous post. As I understand, this would imply a change to how xpm run build operates (probably here), but not to this extension.

BTW, setting the compiler path is not enough, for accurate IntelliSense you also need to configure the include paths, the defined macros and the compiler options. That's exactly the point why the xPack extension uses compile_commands.json, since this greatly automates the process and guarantees accurate IntelliSense.

Yes, I understand this. That is the whole reason I've pivoted from my original idea of modifying c_cpp_properties.json and am instead now trying to affect the compiler paths as they appear in compile_commands.json. That's the whole point of the experiment that I described in the previous post.

@ilg-ul
Copy link
Contributor Author

ilg-ul commented May 4, 2022

When using xpm run build, compile_commands.json ends up with the following path to the compiler:

C:\\Users\\AFont\\Documents\\Projects\\DojoFive\\templates\\cmake\\stm32-template\\xpacks\\.bin\\arm-none-eabi-gcc

This is not correct, it should end with arm-none-eabi-gcc.cmd, since on Windows there is no arm-none-eabi-gcc.

@ilg-ul
Copy link
Contributor Author

ilg-ul commented May 4, 2022

If you generate the hello world project on Windows, using the template, as shown in the xCode extension README, does IntelliSense properly parse the compile_commands.json file?

@ilg-ul
Copy link
Contributor Author

ilg-ul commented May 4, 2022

After re-reading your message, I think I understood your proposal. You suggest that, on Windows, to no longer use .cmd shims in xpacks/.bin to start executables, and instead add all folders to the path.

This would introduce an inconsistency, probably even an incompatibility, between platforms, and an inconsistency with npm, which uses exactly the same .cmd mechanism.

I know that the .cmd mechanism has some issues, but I don't think it is unusable; I'll try to run some more tests on Windows, to validate this.

Before further tests, please fix your tools to generate valid compile_commands.json files (with the .cmd extension), and also check if IntelliSense works properly in the hello world project generated by the template.

@AaronFontaine-DojoFive
Copy link

I think I understand what you're saying. You're saying to add the .cmd extension in the toolchain file like so:

set(CMAKE_C_COMPILER    arm-none-eabi-gcc.cmd)

To make this still work cross-platform, I'd have to conditionally add the .cmd suffix for Windows only. I see the hello world project does this. Martin Bond describes a slightly different method for appending .exe here.

Thing is... I've never had to append anything. I just set CMAKE_C_COMPILER as follows.

set(CMAKE_C_COMPILER    arm-none-eabi-gcc)

And CMake just figures it out. Here is what happens running without a suffix on arm-none-eabi-gcc and without a dev environment.

Re-run cmake no build system arguments
-- Building for: Ninja
-- The C compiler identification is GNU 10.2.1
-- The CXX compiler identification is GNU 10.2.1
-- The ASM compiler identification is GNU
-- Found assembler: C:/Program Files (x86)/GNU Arm Embedded Toolchain/10 2020-q4-major/bin/arm-none-eabi-gcc.exe

The question is, why does that continue to work when using xpm run build? I took a look in the xpacks/.bin folder and there is arm-none-eabi-gcc and arm-none-eabi-gcc.cmd. The first is a shell script that passes through to the .exe file using exec, and that is what CMake was finding in my case:

Re-run cmake no build system arguments
-- Building for: Ninja
-- The C compiler identification is GNU 10.3.1
-- The CXX compiler identification is GNU 10.3.1
-- The ASM compiler identification is GNU
-- Found assembler: C:/Users/AFont/Documents/Projects/DojoFive/templates/cmake/stm32-template/xpacks/.bin/arm-none-eabi-gcc

So... does that work as a compile command? Yes, it does. Testing a single command out of the compile-commands.json is successful:

C:\Users\AFont\Documents\Projects\DojoFive\templates\cmake\stm32-template\build\stm\debug>C:\\Users\\AFont\\Documents\\Projects\\DojoFive\\templates\\cmake\\stm32-template\\xpacks\\.bin\\arm-none-eabi-gcc -I../../../device/stm32l4/include/cmsis -I../../../device/stm32l4/include/stm32l4xx -mcpu=cortex-m4 -mthumb -DSTM32L476xx -mfloat-abi=hard -mfpu=fpv4-sp-d16  -fdata-sections -ffunction-sections --specs=nano.specs -O0 -g3 -DDEBUG -Wall -std=gnu11 -o device\\stm32l4\\CMakeFiles\\STM32L4.dir\\system\\system_stm32l4xx.c.obj -c C:\\Users\\AFont\\Documents\\Projects\\DojoFive\\templates\\cmake\\stm32-template\\device\\stm32l4\\system\\system_stm32l4xx.c

C:\Users\AFont\Documents\Projects\DojoFive\templates\cmake\stm32-template\build\stm\debug>

I tested the hello world project and the IntelliSense does appear to be working correctly. I'm not sure why I'm not seeing any complaints or VS Code reverting to cl.exe. It may have something to do with using gcc instead of arm-none-eabi-gcc. The commands in compile_commands.json do have the .cmd suffix.

@ilg-ul
Copy link
Contributor Author

ilg-ul commented May 5, 2022

... It may have something to do with using gcc instead of arm-none-eabi-gcc. The commands in compile_commands.json do have the .cmd suffix.

No, it has nothing to do with using gcc, it has with having the correct file names.

The issue has nothing to do with CMake, xPacks, GCC, etc, it is a limitation of the VS Code IntelliSense code, which, on Windows, is not able to start extension-less commands from compile_commands.json via cmd.exe, thus the need to explicitly use the .cmd extension.

This is just one of the many joys of using Windows.

I guess VS Code can be fixed, but I had no time to further investigate and submit a pull request.

@AaronFontaine-DojoFive
Copy link

Okay, I think I finally understand what you're saying. IntelliSense works as long as there's an extension on the compile commands in compile_commands.json. It does not matter if it's .exe or .cmd, IntelliSense will work either way. If there are no extensions, IntelliSense will fail.

This makes sense to me. Windows has always used file extensions as opposed to file contents to identify file type. I don't see this as good or bad, just different.

I think where my disconnect came in here is that I've been relying on CMake to add the extension. If I set CMAKE_C_COMPILER to arm-none-eabi-gcc, then on Windows CMake will automatically figure out and add the implied .exe and it will show up that way in both compile_commands.json and CMakeCache.txt.

Using this method with xPack doesn't work. It will compile, but the IntelliSense breaks. Why does it compile? I did some further digging and what I found is not intuitive.

CMake tries to figure out the path and extension of the compiler. The existence of both xpacks/.bin/arm-none-eabi-gcc and xpacks/.bin/arm-none-eabi-gcc.cmd is confusing to CMake. Since the former exactly matches, that is what CMake attempts to use.

However, the Windows CLI is adding .cmd suffix implicitly. So, although CMake sees the shell script and thinks it's invoking that file, Windows is actually switching it under the hood.

There is a conflict here between how CMake and xpack are trying to manage cross-platform support on Windows. Adding the .cmd suffix in the toolchain file resolves this conflict and works for both compilation and IntelliSense. I didn't understand this earlier and I apologize for not picking up on it the first time.

At this point I'm not sure any change should be made to xpack. I don't understand enough about npm or the motivation for the use of .cmd files. I do know that prepending xpacks/xpack-dev-tools-arm-none-eabi-gcc/.content/bin to the PATH is enough to allow CMake to operate normally and allows both IntelliSense and compilation to work. I'm not sure that that's an acceptable solution for xpm-js.

@ilg-ul
Copy link
Contributor Author

ilg-ul commented May 5, 2022

IntelliSense works as long as there's an extension on the compile commands in compile_commands.json. It does not matter if it's .exe or .cmd, IntelliSense will work either way. If there are no extensions, IntelliSense will fail.

I would phrase it differently: IntelliSense parses compile_commands.json ad literam, without assuming any extension.

CMake tries to figure out the path and extension of the compiler. The existence of both xpacks/.bin/arm-none-eabi-gcc and xpacks/.bin/arm-none-eabi-gcc.cmd is confusing to CMake.

I personally highly dislike many of CMake 'automation' features, since they try to outsmart the user, and cause so much confusion.

The former is a shell script required when npm/xpm are invoked from a shell in a mingw-like environment. There is not much we can do about it.

... the use of .cmd files. ... I do know that prepending xpacks/xpack-dev-tools-arm-none-eabi-gcc/.content/bin to the PATH is enough to allow CMake to operate normally and allows both IntelliSense and compilation to work. I'm not sure that that's an acceptable solution for xpm-js.

I think that both are workarounds conceived to overcome some of the many Windows limitations. I would not recommend messing with the PATH, I think that adding .cmd is the lesser of two evils.

If you think that manually managing the path is ok for your projects, feel free to use it, but maintaining the projects will not be easy.

Take a look at a complex project package.json, with multiple dependencies, and try to replicate the functionality without xpm run.

@AaronFontaine-DojoFive
Copy link

Thank you for the feedback and perspective. Whether it's apparent or not, I appreciate the time you've taken to offer explanation.

I agree that adding the .cmd suffix is the lesser of two evils and that's what I'll move forward with.

As far as CMake over-automating things, yes it is problematic. Especially when such behavior is undocumented. I have spent the last few months educating myself about CMake and how to use it correctly. I had been avoiding suffixes in the toolchain file because that was my impression of "correct" CMake. I have no problem changing that view.

Also, thank you for the complex example project. I will make time to review it.

@ilg-ul
Copy link
Contributor Author

ilg-ul commented May 5, 2022

You're welcome!

I'm currently split between maintaining the Eclipse Embedded CDT (which I think is on the downslope now) and improving the VS Code xPack extension (which I'm optimistic about), so any suggestions are welcome.

@ilg-ul
Copy link
Contributor Author

ilg-ul commented May 5, 2022

The existence of both xpacks/.bin/arm-none-eabi-gcc and xpacks/.bin/arm-none-eabi-gcc.cmd is confusing to CMake.

BTW, I think that this can be fixed. Try to discuss this in the CMake forum and explain as clear as possible that, on Windows, when CMake encounters a name without extension, if that file is not present or it is not an executable, CMake should check if a similar .exe is available, and, if not, check for a similar .cmd, and only if none is available, pick the name without extension.

If they agree on such an enhancement, we can experiment in my CMake fork and eventually provide a pull request.

@AaronFontaine-DojoFive
Copy link

when CMake encounters a name without extension, if that file is not present or it is not an executable, CMake should check if a similar .exe is available, and, if not, check for a similar .cmd, and only if none is available, pick the name without extension.

and only if none is available, pick the name without extension. I think that's it right there. It can auto-append both the .exe and .cmd extensions, but if it sees that extensionless shell script first... that's what it goes with. Thank you for the link. I will look through their contribution and forum guidelines.

@AaronFontaine-DojoFive
Copy link

As far as the Eclipse Embedded CDT, I'm a big fan and plan to continue using it. I don't think the VS Code embedded experience has reached the maturity level of Eclipse, nor do I think VS Code's UI is as well thought out.

My opinion is a minority opinion though ... by a wide margin. Even the MCU on Eclipse blog is on that bandwagon. My suggestion is to focus on VS Code. I may love Eclipse, but as far as the templates I'm working on, I'm focusing on VS Code. That's what my coworkers use. That's what clients ask about.

One aspect of modern embedded is management of dev environments. That's something that this extension is helping to make very easy. It's literally "Check out repo. Click button." If you make xpm install a post-checkout hook, it gets even better.

Being able to have a simple tasks.json or package.json to fully configure VS Code is a powerful value proposition. The parsing of compile_commands.json to autoconfigure the IntelliSense is also significant. Reentering build info from CMake into the IDE is definitely a pain point for some of my coworkers. I'm not aware whether Eclipse has similar capability.

@ilg-ul
Copy link
Contributor Author

ilg-ul commented May 9, 2022

as far as the templates I'm working on ...

Just curious, did you try the xPack project templates? There are two of them now, one with the @xpack/hello-world-template also used by the xPack extension, and an older one, the @sifive/templates.

I plan to further enhance this style of templates, so if you have any suggestions, they are welcomed.

@AaronFontaine-DojoFive
Copy link

The templates I'm working on are based heavily on the CMake courses from Embedded Artistry. I found out about the xPack build framework only after starting work on the templates. I was originally looking at Conda to manage dev environments as described by Tyler Hoffman. xPack is a better solution.

I have looked through the hello-world template, but like I said, the design of my templates is already heavily influenced by Embedded Artistry.

There is one comment I would make.

I do not have a prepare stage in my actions. Only build and clean. build is a one-touch mechanism that runs CMake only if necessary. This is achieved via a "Makefile shim", a concept which comes from Phillip Johnston of Embedded Artistry. In my Makefile, this looks something like:

OPTIONS ?=
INTERNAL_OPTIONS += -DCMAKE_TOOLCHAIN_FILE=device/stm32l4/toolchains/cross/stm32l476rg.cmake

debug: INTERNAL_OPTIONS += -DCMAKE_BUILD_TYPE=Debug
debug: build/stm/debug/build.ninja
	$(Q)ninja -C build/stm/debug
build/stm/debug/build.ninja:
	$(Q)cmake -B build/stm/debug $(OPTIONS) $(INTERNAL_OPTIONS)

.PHONY: release
release: INTERNAL_OPTIONS += -DCMAKE_BUILD_TYPE=Release
release: build/stm/release/build.ninja
	$(Q)ninja -C build/stm/release
build/stm/release/build.ninja:
	$(Q)cmake -B build/stm/release $(OPTIONS) $(INTERNAL_OPTIONS)

Take that for what it's worth. Everything else I've seen treats CMake builds as two separate stages that need to be run independently.

As far as # TODO: understand why CMAKE_<lang>_OUTPUT_EXTENSION is not effective, I have a thought here. I do the following in my main CMakeLists.txt and it works:

set(CMAKE_EXECUTABLE_SUFFIX ".elf")

Maybe it can't be done in the toolchain file?

@AaronFontaine-DojoFive
Copy link

I do like the scripting for template setup. There's a lot of work here.

Like I said. I'm fairly new to Node.js and am still wrapping my head around package.json. I still need to take time to review the complex project you linked above.

@ilg-ul
Copy link
Contributor Author

ilg-ul commented May 13, 2022

I do not have a prepare stage in my actions.

Yeah, I know that you can do lots of tricks with CMake, but the xPack framework is neutral to the build system, you can use any tools, and the general case is with separate steps.

Actually I plan to implement an integrated build system in xPack and VS Code, similar to the Eclipse managed build, with the configurations stored in JSON files and editable via nice GUI pages, as in Eclipse.

I do like the scripting for template setup. There's a lot of work here.

Indeed. I think that scripted templates, used with xpm init are a very convenient way of generating projects.

@ilg-ul
Copy link
Contributor Author

ilg-ul commented May 14, 2022

I still need to take time to review the complex project you linked above.

It shows how to organise multi-platform unit-tests for a source library project intended to be included in embedded projects.

The library being highly portable (it manages intrusive lists), the tests can be compiled and executed both as native processes on the development platform, or as embedded applications. For now only Cortex-M4 tests are included, but once the new toolchains I'm working on right now will be available, I'll extend the tests with Cortex-A and RISC-V applications.

I doubt you saw many embedded projects running such extensive multi-platform tests and in such a convenient way.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Development

No branches or pull requests

3 participants