Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Failed to cross-compile CPU binaries - kr260 #59

Open
mohammedrafi-sk opened this issue Jun 29, 2022 · 11 comments
Open

Failed to cross-compile CPU binaries - kr260 #59

mohammedrafi-sk opened this issue Jun 29, 2022 · 11 comments

Comments

@mohammedrafi-sk
Copy link
Contributor

mohammedrafi-sk commented Jun 29, 2022

The following errors are seen while building the cross-compile CPU binaries for kr260
KRS1.0_kr260_crosscompile_cpu.txt

Steps to reproduce the issue:

I am trying out the kr260 cross-compilation-development. The setup of KRS went fine without any issues (steps 1-6).

While running step 7A, I am seeing the above-mentioned dependency errors.

colcon build --build-base=build-kr260-ubuntu --install-base=install-kr260-ubuntu --merge-install --mixin kr260 --cmake-args -DNOKERNELS=true
@mohammedrafi-sk
Copy link
Contributor Author

mohammedrafi-sk commented Jun 29, 2022

The errors related to aarch64-linux-gnu-g++ and aarch64-linux-gnu-gcc are resolved after installing
gcc-aarch64-linux-gnu and g++-aarch64-linux-gnu

Pull request to modify docs: #60

@mohammedrafi-sk
Copy link
Contributor Author

mohammedrafi-sk commented Jun 29, 2022

For resolving the error with package "ament_cmake_core", I cross-compiled the ament_cmake,

Could not find a package configuration file provided by "ament_cmake_core"
  with any of the following names:

Now, I am seeing a similar issue but with ament_cmake_ros

Failed   <<< tracetools_image_pipeline [0.73s, exited with code 1]
--- stderr: tracetools_acceleration
CMake Error at CMakeLists.txt:15 (find_package):
  By not providing "Findament_cmake_ros.cmake" in CMAKE_MODULE_PATH this
  project has asked CMake to find a package configuration file provided by
  "ament_cmake_ros", but CMake did not find one.

  Could not find a package configuration file provided by "ament_cmake_ros"
  with any of the following names:

    ament_cmake_rosConfig.cmake
    ament_cmake_ros-config.cmake
---
Failed   <<< tracetools_acceleration [0.74s, exited with code 1]

Added ament_cmake_ros to cross-compile and observed a new error

--- stderr: ros2_kria_msgs                                                                                                                                                                                                                    
CMake Error at CMakeLists.txt:15 (find_package):
  By not providing "Findbuiltin_interfaces.cmake" in CMAKE_MODULE_PATH this
  project has asked CMake to find a package configuration file provided by
  "builtin_interfaces", but CMake did not find one.

  Could not find a package configuration file provided by
  "builtin_interfaces" with any of the following names:

    builtin_interfacesConfig.cmake
    builtin_interfaces-config.cmake
---
Failed   <<< ros2_kria_msgs [0.34s, exited with code 1]

@vmayoral
Copy link
Contributor

ament_cmake_core should be installed in the Ubuntu sysroot if deployed appropriately. Check it it’s there please and otherwise report what went wrong in the deployment process (and posterior patching).

Also, to help assess the issues, it’d be useful to get the whole commands with their corresponding errors, as well as instructions on how to reproduce the errors. Otherwise providing help is hard.

@mohammedrafi-sk
Copy link
Contributor Author

Hi @vmayoral

I can't seem to find any ROS related CMake files in the ubuntu sysroot ~/krs_ws/acceleration/firmware/kr260/sysroots .

spa@ros2server:~/krs_ws/acceleration/firmware/kr260/sysroots$ find . -name ament_*
spa@ros2server:~/krs_ws/acceleration/firmware/kr260/sysroots$ 

I found this patch which seems to be installing ROS onto sysroot, can you confirm?
~/krs_ws/acceleration/firmware/kr260/patches/customize.patch

@vmayoral
Copy link
Contributor

I found this patch which seems to be installing ROS onto sysroot, can you confirm?
~/krs_ws/acceleration/firmware/kr260/patches/customize.patch

That's correct, but the patching and install should be automated by the CMake logic of acceleration_firmware_kr260. See https://github.com/ros-acceleration/acceleration_firmware_kr260/blob/main/cmake/ros2_cross.cmake. This is known to work when fetching appropriately v1.0.0 of such package.

Try the following:

# clean up your overlay workspace
rm -rf ~/krs_src

# re-start instructions according to https://xilinx.github.io/KRS/sphinx/build/html/docs/install.html#cross-compilation-development

###################################################
# 1. install some dependencies you might be missing
#
# NOTE: gcc-multilib conflicts with Yocto/PetaLinux 2022.1 dependencies
# so you can't have both paths simultaneously enabled in a single
# development machine
###################################################
sudo apt-get -y install curl build-essential libssl-dev git wget \
                          ocl-icd-* opencl-headers python3-vcstool \
                          python3-colcon-common-extensions python3-colcon-mixin \
                          kpartx u-boot-tools pv gcc-multilib

###################################################
# 2. create a new ROS 2 workspace with examples and
#    firmware for KR260
###################################################
mkdir -p ~/krs_ws/src; cd ~/krs_ws

###################################################
# 3. Create file with KRS 1.0 additional repos
###################################################
cat << 'EOF' > krs_humble.repos
repositories:  
  perception/image_pipeline:
    type: git
    url: https://github.com/ros-acceleration/image_pipeline
    version: ros2

  tracing/tracetools_acceleration:
    type: git
    url: https://github.com/ros-acceleration/tracetools_acceleration
    version: humble

  firmware/acceleration_firmware_kr260:
    type: zip
    url: https://github.com/ros-acceleration/acceleration_firmware_kr260/releases/download/v1.0.0/acceleration_firmware_kr260.zip

  acceleration/adaptive_component:
    type: git
    url: https://github.com/ros-acceleration/adaptive_component
    version: humble
  acceleration/ament_acceleration:
    type: git
    url: https://github.com/ros-acceleration/ament_acceleration
    version: humble
  acceleration/ament_vitis:
    type: git
    url: https://github.com/ros-acceleration/ament_vitis
    version: humble
  acceleration/colcon-hardware-acceleration:
    type: git
    url: https://github.com/colcon/colcon-hardware-acceleration
    version: main
  acceleration/ros2_kria:
    type: git
    url: https://github.com/ros-acceleration/ros2_kria
    version: main
  acceleration/ros2acceleration:
    type: git
    url: https://github.com/ros-acceleration/ros2acceleration
    version: humble
  acceleration/vitis_common:
    type: git
    url: https://github.com/ros-acceleration/vitis_common
    version: humble
  acceleration/acceleration_examples:
    type: git
    url: https://github.com/ros-acceleration/acceleration_examples
    version: main
EOF

###################################################
# 4. import repos of KRS 1.0 release
###################################################
vcs import src --recursive < krs_humble.repos  # about 3 mins in an AMD Ryzen 5 PRO 4650G

###################################################
# 5. build the workspace and deploy firmware for hardware acceleration
###################################################
source /tools/Xilinx/Vitis/2022.1/settings64.sh  # source Xilinx tools
source /opt/ros/humble/setup.bash  # Sources system ROS 2 installation.

# Note: The path above is valid if one installs ROS 2 from a pre-built debian
# packages. If one builds ROS 2 from the source the directory might
# vary (e.g. ~/ros2_humble/ros2-linux).
export PATH="/usr/bin":$PATH  # FIXME: adjust path for CMake 3.5+
colcon build --merge-install  # about 18 mins in an AMD Ryzen 5 PRO 4650G

(this has been dumped directly from https://xilinx.github.io/KRS/sphinx/build/html/docs/install.html#cross-compilation-development)

⚠️ NOTE ⚠️ : make sure you type your sudo password while executing the last command (colcon build). That's needed for the sysroot to get patched appropriately and for ROS 2 to be installed successfully (you need to use sudo to enter into the chroot)

@mohammedrafi-sk
Copy link
Contributor Author

Hi @vmayoral

I tried rebuilding the KRS as you mentioned, and found the chroot failed and so the ros packages didn't install to sysroot
Here is the log

mr@krs:~/krs_ws$ colcon build --merge-install
Starting >>> ament_acceleration
Starting >>> tracetools_image_pipeline
Starting >>> tracetools_acceleration
Starting >>> adaptive_component
Starting >>> camera_calibration
Starting >>> depth_image_proc
Starting >>> image_publisher
Starting >>> image_rotate
/usr/lib/python3/dist-packages/setuptools/dist.py:723: UserWarning: Usage of dash-separated 'script-dir' will not be supported in future versions. Please use the underscore name 'script_dir' insteadtion - 0.2s] [depth_image_proc - 0.2s] [image_publisher - 0.2s] [image_rotate - 0.2s]
  warnings.warn(
/usr/lib/python3/dist-packages/setuptools/dist.py:723: UserWarning: Usage of dash-separated 'install-scripts' will not be supported in future versions. Please use the underscore name 'install_scripts' instead
  warnings.warn(
--- stderr: camera_calibration
/usr/lib/python3/dist-packages/setuptools/dist.py:723: UserWarning: Usage of dash-separated 'script-dir' will not be supported in future versions. Please use the underscore name 'script_dir' instead
  warnings.warn(
/usr/lib/python3/dist-packages/setuptools/dist.py:723: UserWarning: Usage of dash-separated 'install-scripts' will not be supported in future versions. Please use the underscore name 'install_scripts' instead
  warnings.warn(
/usr/lib/python3/dist-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
  warnings.warn(
---
Finished <<< camera_calibration [6.90s]
Starting >>> image_view
Finished <<< ament_acceleration [7.49s]
Starting >>> ament_vitis
Finished <<< tracetools_image_pipeline [7.36s]
Starting >>> ros2_kria_msgs
Finished <<< tracetools_acceleration [7.38s]
Starting >>> FortranCInterface
Finished <<< ament_vitis [1.58s]
Starting >>> vitis_common
Finished <<< ros2_kria_msgs [1.55s]
Starting >>> IntelFortranImplicit
[23.011s] WARNING:colcon.colcon_cmake.task.cmake.build:Could not run installation step for package 'IntelFortranImplicit' because it has no 'install' target
Finished <<< IntelFortranImplicit [1.26s]
Starting >>> acceleration_firmware_kr260
Finished <<< vitis_common [1.36s]
Starting >>> image_proc
[23.331s] WARNING:colcon.colcon_cmake.task.cmake.build:Could not run installation step for package 'FortranCInterface' because it has no 'install' target
Finished <<< FortranCInterface [3.11s]
Starting >>> colcon-hardware-acceleration
--- stderr: colcon-hardware-acceleration
/usr/lib/python3/dist-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
  warnings.warn(
---
Finished <<< colcon-hardware-acceleration [1.00s]
Starting >>> publisher_xilinx
[Processing: acceleration_firmware_kr260, adaptive_component, depth_image_proc, image_proc, image_publisher, image_rotate, image_view, publisher_xilinx]
[Processing: acceleration_firmware_kr260, adaptive_component, depth_image_proc, image_proc, image_publisher, image_rotate, image_view, publisher_xilinx]
--- stderr: acceleration_firmware_kr260
+ set -o pipefail
+ trap 'finish "$chroot_d"' EXIT
+ '[' 5 -lt 4 ']'
+ '[' 0 -ne 0 ']'
+ chroot_d=/home/mr/krs_ws/install/../acceleration/firmware/kr260/sysroots/aarch64-xilinx-linux
+ suite=jammy
+ ppa_src=http://packages.ros.org/ros2/ubuntu
+ ppa_token=
++ dirname /home/mr/krs_ws/install/../acceleration/firmware/kr260/sysroots/bin/customize.sh
+ scripts_d=/home/mr/krs_ws/install/../acceleration/firmware/kr260/sysroots/bin
+ shift 4
+ pkgs_in_private_archive=(xrt linux-headers-xilinx-zynqmp)
+ cp /home/mr/krs_ws/install/../acceleration/firmware/kr260/sysroots/bin/cleanup-chroot.sh /home/mr/krs_ws/install/../acceleration/firmware/kr260/sysroots/aarch64-xilinx-linux
+ cd /home/mr/krs_ws/install/../acceleration/firmware/kr260/sysroots/aarch64-xilinx-linux
+ chroot_d=/home/mr/krs_ws/acceleration/firmware/kr260/sysroots/aarch64-xilinx-linux
+ prepare_chroot
+ mount --rbind --make-rslave /dev dev
+ mount -t proc none proc
+ mount -t sysfs none sys
+ mount -t tmpfs none tmp
+ mount -t tmpfs none var/lib/apt
+ mount -t tmpfs none var/cache/apt
+ mv etc/resolv.conf etc/resolv.conf.bak
+ cp /etc/resolv.conf etc/resolv.conf
+ install_ros2_humble jammy
+ local 'sources=deb [arch=arm64 signed-by=/usr/share/keyrings/ros-archive-keyring.gpg] http://packages.ros.org/ros2/ubuntu jammy main'
+ setup_pkgs=('software-properties-common' 'curl' 'gnupg' 'lsb-release')
+ local setup_pkgs
+ local key_hash=C1CF6E31E6BADE8868B172B4F42ED6FBAB17C654
+ pkgs=('ros-humble-desktop' 'ros-humble-image-common')
+ local pkgs
+ local 'opts=--no-install-recommends -y'
+ chroot . apt update
chroot: failed to run command ‘apt’: Exec format error
+ finish /home/mr/krs_ws/acceleration/firmware/kr260/sysroots/aarch64-xilinx-linux
+ local chroot_d=/home/mr/krs_ws/acceleration/firmware/kr260/sysroots/aarch64-xilinx-linux
+ shutdown_chroot /home/mr/krs_ws/acceleration/firmware/kr260/sysroots/aarch64-xilinx-linux
+ local chroot_d=/home/mr/krs_ws/acceleration/firmware/kr260/sysroots/aarch64-xilinx-linux
+ local mountpoint_match submount
+ rm -f etc/resolv.conf
+ mv etc/resolv.conf.bak etc/resolv.conf
++ printf %s /home/mr/krs_ws/acceleration/firmware/kr260/sysroots/aarch64-xilinx-linux
++ sed '-es,/$,,; s,/,\\/,g;'
+ mountpoint_match='\/home\/mr\/krs_ws\/acceleration\/firmware\/kr260\/sysroots\/aarch64-xilinx-linux\/'
+ read -r submount
++ awk '$2 ~ /\/home\/mr\/krs_ws\/acceleration\/firmware\/kr260\/sysroots\/aarch64-xilinx-linux\// { print $2 }'
++ LC_ALL=C
++ sort -r
+ umount /home/mr/krs_ws/acceleration/firmware/kr260/sysroots/aarch64-xilinx-linux/var/lib/apt
+ read -r submount
+ umount /home/mr/krs_ws/acceleration/firmware/kr260/sysroots/aarch64-xilinx-linux/var/cache/apt
+ read -r submount
+ umount /home/mr/krs_ws/acceleration/firmware/kr260/sysroots/aarch64-xilinx-linux/tmp
+ read -r submount
+ umount /home/mr/krs_ws/acceleration/firmware/kr260/sysroots/aarch64-xilinx-linux/sys
+ read -r submount
+ umount /home/mr/krs_ws/acceleration/firmware/kr260/sysroots/aarch64-xilinx-linux/proc
+ read -r submount
+ umount /home/mr/krs_ws/acceleration/firmware/kr260/sysroots/aarch64-xilinx-linux/dev/shm
+ read -r submount
+ umount /home/mr/krs_ws/acceleration/firmware/kr260/sysroots/aarch64-xilinx-linux/dev/pts
+ read -r submount
+ umount /home/mr/krs_ws/acceleration/firmware/kr260/sysroots/aarch64-xilinx-linux/dev/mqueue
+ read -r submount
+ umount /home/mr/krs_ws/acceleration/firmware/kr260/sysroots/aarch64-xilinx-linux/dev/hugepages
+ read -r submount
+ umount /home/mr/krs_ws/acceleration/firmware/kr260/sysroots/aarch64-xilinx-linux/dev
+ read -r submount
---
Finished <<< acceleration_firmware_kr260 [2min 6s]

But, after installing qemu on the Ubuntu host, the chroot worked, and I don't see any of the above ros package dependency issues.

sudo apt install qemu-user-static 

I have the vitis installed, but qemu is not part of it. I believe qemu is part of petalinux and not Vitis
@jasvinderkhurana Can you confirm if qemu is also part of Vitis installation?

mohammedrafi-sk added a commit to mohammedrafi-sk/KRS that referenced this issue Jul 4, 2022
gcc-multilib is conflicting with gcc-aarch64-linux-gnu and g++-aarch64-linux-gnu packages. 
Which are required: Xilinx#59 (comment)
@elimkwan
Copy link

elimkwan commented Aug 24, 2022

Hi @vmayoral @mohammedrafi-sk,

I have encountered the same chroot error when trying to cross-compile the CPU binaries for kr260.

+ chroot . apt update
chroot: failed to run command ‘apt’: Exec format error

Unfortunately, the fix sudo apt install qemu-user-static did not work for me.

So far I have not been able to find a fix, except to compute the CPU binaries on target. I have tried it on a Windows machine with wsl2 and a Linux machine (Ubuntu 22.04), and both of them result in the same chroot error, so it doesn't seem to be a wsl2 issue. Any insight would be appreciated, thanks.

@verderog
Copy link

verderog commented Oct 28, 2022

This thread has been very helpful! I'm building KRS using Docker and it turns out that when launching the container, I did not enable the --privileged flag. This in turn caused the...

/krs_ws/install/../acceleration/firmware/kr260/sysroots/bin/customize.sh

...script to silently fail during the prepare_chroot() function. This failure caused errors relating to ament_cmake_core and similar ament_cmake* packages when executing:

colcon build --build-base=build-kr260-ubuntu --install-base=install-kr260-ubuntu --merge-install --mixin kr260 --cmake-args -DNOKERNELS=true

Posting this in case anyone else runs into this in the future.

So, the fix for me was to simply launch via docker run --privileged <image-name-here>.

@lanyangyang
Copy link

@verderog Could you please explain more clearly about how you fix this issue ?? I met the same issue as you mentioned.

@verderog
Copy link

verderog commented Nov 22, 2022

For context, I've created a Docker container in order to build KRS. This Docker container performs the operations that the KRS build instructions call out (see more here).

It turns out that within a Docker container running the prepare_chroot() function that is within the customize.sh script will fail, due to some of the mount operations that are performed. By launching the container with the --privileged flag, that causes the prepare_chroot() to execute successfully. If the function fails (because the mount operations can't complete), the side effect is that you run into those various ament_cmake* errors.

More info on the Docker privileged flag here. One of the key take-aways:

...most potentially dangerous kernel capabilities are dropped; including cap_sys_admin (which is required to mount filesystems). However, the --privileged flag will allow it to run...

@lanyangyang I hope this helps!

@vmayoral
Copy link
Contributor

It turns out that within a Docker container running the prepare_chroot() function that is within the customize.sh script will fail, due to some of the mount operations that are performed. By launching the container with the --privileged flag, that causes the prepare_chroot() to execute successfully. If the function fails (because the mount operations can't complete), the side effect is that you run into those various ament_cmake* errors.

Right, this requires sudo privileges to successfully install ROS 2 in the chroot. The reason why some people is experiencing failures is because these privileges are not being given appropriately (either inside or outside of the container). By adding the --privileged flag (and by default executing with root privileges inside) you're addressing this.

Outside of the container folks should pay attention to this (and/or sudo -s before colcon build --merge-installing). There's a user experience gap in here that I agree needs to be addressed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants