-
Notifications
You must be signed in to change notification settings - Fork 17
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
ziti-edge-tunnel enroll "COULD_NOT_PROCESS_CSR" on Red Hat 9 / Rocky 9 #517
Comments
I ran the enroll command with strace and I see this message:
Same message showing the value of {ca certs}:
This could point toward the problem with processing the CSR or it may be a red herring message having to do with strace printing long filenames. |
I ran the enroll command with valgrind which reveals the context in which the failure occurs.
|
that is a red herring. OpenSSL makes several tries loading certs from a byte buffer: file, PEM, DER |
the error comes back from controller |
James and I both found that the redhat8 build is able to run the same enroll command successfully on redhat9. When I come back to this issue I'll try to reproduce the error with a redhat9 build and the main release build updated to 0.20.2. |
I can not reproduce this issue with the release binary or RedHat 8 package 0.20.4 running on RedHat 9. It seems to only present when running the new RedHat 9 build. I will try to gain access to the controller log in hopes there is a more helpful error message emitted there. The main difference between the two builds is that we're running gcc 11 on Red Hat 9 instead of gcc 10 on Red Hat 8. |
[rocky@ip-172-31-4-195 ~]$ /opt/openziti/bin/ziti-edge-tunnel enroll -j ./client8.jwt -i ./client8.json
(28800)[ 0.492] ERROR ziti-sdk:ziti_enroll.c:227 enroll_cb() failed to enroll with controller: https://7ce7e424-6a92-4ff2-9459-ebbba32346fa.production.netfoundry.io:443 COULD_NOT_PROCESS_CSR (The supplied csr could not be processed)
(28800)[ 0.492] ERROR ziti-edge-tunnel:ziti-edge-tunnel.c:1994 enroll_cb() enrollment failed: COULD_NOT_PROCESS_CSR(-3)
[rocky@ip-172-31-4-195 ~]$ ldd /opt/openziti/bin/ziti-edge-tunnel
linux-vdso.so.1 (0x00007fffa2dfe000)
libssl.so.3 => /lib64/libssl.so.3 (0x00007facd17d3000)
libcrypto.so.3 => /lib64/libcrypto.so.3 (0x00007facd13a7000)
libatomic.so.1 => /lib64/libatomic.so.1 (0x00007facd139e000)
libm.so.6 => /lib64/libm.so.6 (0x00007facd12c3000)
libresolv.so.2 => /lib64/libresolv.so.2 (0x00007facd12af000)
libc.so.6 => /lib64/libc.so.6 (0x00007facd10a5000)
libz.so.1 => /lib64/libz.so.1 (0x00007facd1089000)
/lib64/ld-linux-x86-64.so.2 (0x00007facd187e000) |
Built on RH9 OS with this command: cmake \
-DCMAKE_BUILD_TYPE=Release \
-DCMAKE_TOOLCHAIN_FILE=./toolchains/default.cmake \
-DBUILD_DIST_PACKAGES=ON \
-DUSE_OPENSSL=ON \
-S . \
-B ./build
cmake \
--build ./build \
--target package \
--verbose |
I found that I can set env var
|
The controller process is running at the default log level, and there were no messages emitted at the time of the failed enrollment. |
Steps to reproducebuild git checkout issue-514-package-for-redhat9
(cd ./.github/actions/openziti-tunnel-build-action/redhat-9/ && docker buildx build -t rh9-builder . --load ; )
docker run --rm -ti -v "${PWD}:/github/workspace" rh9-builder run cat > /tmp/client10.jwt
# paste contents of JWT, press ctrl-D to send EOF
docker run -i --rm \
-v "${PWD}/build/programs/Release/ziti-edge-tunnel:/mnt" \
-e ZITI_LOG=4 docker.io/library/rockylinux:9 \
bash -c '{
dnf install -yq libatomic \
&& /mnt/ziti-edge-tunnel enroll \
--jwt - --identity /mnt/client10.json;
}' < /tmp/client10.jwt |
I verified the steps to reproduce are still applicable in v0.20.20. |
I can still reproduce this in v0.21.0. |
This issue no longer occurs after adapting RH9 builder to the new VCPKG preset. ❯ docker run -i --rm \
-v "${PWD}/build:/mnt" \
-e ZITI_LOG=4 docker.io/library/rockylinux:9 \
bash -euxc '{
dnf install -yq /mnt/ziti-edge-tunnel-0.21.4-1.x86_64.rpm \
&& /opt/openziti/bin/ziti-edge-tunnel version \
&& /opt/openziti/bin/ziti-edge-tunnel enroll \
--jwt - --identity /mnt/client10.json;
}' < /tmp/rh9.jwt
+ dnf install -yq /mnt/ziti-edge-tunnel-0.21.4-1.x86_64.rpm
Importing GPG key 0x350D275D:
Userid : "Rocky Enterprise Software Foundation - Release key 2022 <[email protected]>"
Fingerprint: 21CB 256A E16F C54C 6E65 2949 702D 426D 350D 275D
From : /etc/pki/rpm-gpg/RPM-GPG-KEY-Rocky-9
Installed:
acl-2.3.1-3.el9.x86_64 dbus-1:1.12.20-7.el9_1.x86_64
dbus-broker-28-7.el9.x86_64 dbus-common-1:1.12.20-7.el9_1.noarch
iproute-6.1.0-1.el9.x86_64 kmod-libs-28-7.el9.x86_64
libatomic-11.3.1-4.3.el9.x86_64 libbpf-2:1.0.0-2.el9.x86_64
libmnl-1.0.4-15.el9.x86_64 libseccomp-2.5.2-2.el9.x86_64 psmisc-23.4-3.el9.x86_64 systemd-252-13.el9_2.x86_64
systemd-pam-252-13.el9_2.x86_64 systemd-rpm-macros-252-13.el9_2.noarch ziti-edge-tunnel-0.21.4-1.x86_64
+ /opt/openziti/bin/ziti-edge-tunnel version
v0.21.4-19-gd0c5eff-local
+ /opt/openziti/bin/ziti-edge-tunnel enroll --jwt - --identity /mnt/client10.json
(178)[ 0.000] INFO ziti-sdk:utils.c:188 ziti_log_set_level() set log level: root=4/DEBUG (178)[ 0.000] INFO ziti-sdk:utils.c:188 ziti_log_set_level() set log level: root=4/DEBUG
(178)[ 0.000] INFO ziti-sdk:ziti_enroll.c:90 ziti_enroll() Ziti C SDK version 0.32.6 @2fc3556(HEAD) starting enrollment at (2023-06-08T20:40:50.906)
(178)[ 0.000] DEBUG ziti-sdk:jwt.c:106 load_jwt() filename is: -
(178)[ 0.000] DEBUG ziti-sdk:jwt.c:69 load_jwt_file() reading JWT from standard input
(178)[ 0.000] DEBUG ziti-sdk:jwt.c:99 load_jwt_file() jwt file content is: eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJlbSI6Im90dCIsImV4cCI6MTY4NjQyODk1OSwiaXNzIjoiaHR0cHM6Ly83Y2U3ZTQyNC02YTkyLTRmZjItOTQ1OS1lYmJiYTMyMzQ2ZmEucHJvZHVjdGlvbi5uZXRmb3VuZHJ5LmlvOjQ0MyIsImp0aSI6IjdjZTU3OWQwLTFmNTYtNDZjZC05NjcyLTM1NmUyOWJiN
zAwZCIsInN1YiI6IjVQN0haQlN5cjkifQ.H2lLBrdUoIAoocPcAgRWKXoKyxR83tKsl65UeY6VJBt3a-ZNEYdgGDgLnTa8VyGBZ3HEUNRUq8gjx-0Wtg1JcmsGCOKCV_TT6LEy18uXk_pR5cHdHLeiVYoaNDSmXDaSJCz-SDMRPZDNmyQjY-Dydt6ewryOpnkAuvlAr3GpHcdZXKAO4BmeA7EdmCj_zjQey_3bgmBdSgD-
v6s4cToCJwuuPNvVqOImHD6Kb7TZBJPnFcnInzuBsnwUBOhGEzjbGnDGejRugrelT0qb-wBOi-tzGBethZIdJDUBQiYWu1A7EXj38WnEDrrc4V4j3uLMFDtSqBpkIfXdFxeckiftBNoMzS6LudVqWcn0w8lixMNrXoJE_5Wsxkg1p8exzVDwx15NymkqzNhzff7svsrSS-HiTZL8tZk7XLPwIk38T2a9SGzLnL5bPMpgb2
DtJBYSk_i4dyrzrTe9z45-RBswJRQ8iEiI3dN90CTK5opLTTQW-9ZlcNc-zvm721o4HAkGXEjjXq01ej-KE05HG15KEEsyREEpwa_r9ioXOOvW-djFA24m70R3bchIYqr0RPR4nq39Gwfj37vGoczJnCMKtn0x5--gk-w4FQLyL3_tflNq6gCDduJL8MxMrYXatlDBDA7yNGrtX5cnQBCwj5fh29yefjuHM5FMD230NPXx
hXY
(178)[ 0.000] DEBUG ziti-sdk:jwt.c:36 parse_jwt_content() ecfg->jwt_signing_input is:
eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJlbSI6Im90dCIsImV4cCI6MTY4NjQyODk1OSwiaXNzIjoiaHR0cHM6Ly83Y2U3ZTQyNC02YTkyLTRmZjItOTQ1OS1lYmJiYTMyMzQ2ZmEucHJvZHVjdGlvbi5uZXRmb3VuZHJ5LmlvOjQ0MyIsImp0aSI6IjdjZTU3OWQwLTFmNTYtNDZjZC05NjcyLTM1NmUyOWJiN
zAwZCIsInN1YiI6IjVQN0haQlN5cjkifQ
(178)[ 0.000] DEBUG ziti-sdk:ziti_ctrl.c:408 ziti_ctrl_init() ctrl[7ce7e424-6a92-4ff2-9459-ebbba32346fa.production.netfoundry.io] ziti controller client initialized
(178)[ 0.084] DEBUG ziti-sdk:ziti_enroll.c:39 verify_controller_jwt() verifying JWT signature
(178)[ 0.084] DEBUG ziti-sdk:ziti_enroll.c:67 verify_controller_jwt() JWT verification succeeded!
(178)[ 0.160] DEBUG ziti-sdk:ziti_enroll.c:157 well_known_certs_cb() CA PEM len = 21925
(178)[ 0.160] DEBUG ziti-sdk:ziti_ctrl.c:408 ziti_ctrl_init() ctrl[7ce7e424-6a92-4ff2-9459-ebbba32346fa.production.netfoundry.io] ziti controller client initialized
(178)[ 0.315] DEBUG ziti-sdk:ziti_ctrl.c:325 ctrl_body_cb() ctrl[7ce7e424-6a92-4ff2-9459-ebbba32346fa.production.netfoundry.io] completed POST[/enroll?method=ott&token=7ce579d0-1f56-46cd-9672-356e29bb700d] in 0.142 s
(178)[ 0.315] DEBUG ziti-sdk:ziti_enroll.c:242 enroll_cb() successfully enrolled with controller https://7ce7e424-6a92-4ff2-9459-ebbba32346fa.production.netfoundry.io:443 |
Will resolve in #515 |
This issue is recurring with the RedHat9 release RPM v0.22.5.
[root@rocky9 ~]# ldd /opt/openziti/bin/ziti-edge-tunnel
linux-vdso.so.1 (0x00007ffffd5de000)
libz.so.1 => /lib64/libz.so.1 (0x00007f4ce7fb5000)
libssl.so.3 => /lib64/libssl.so.3 (0x00007f4ce7f0f000)
libcrypto.so.3 => /lib64/libcrypto.so.3 (0x00007f4ce7a00000)
libatomic.so.1 => /lib64/libatomic.so.1 (0x00007f4ce7f06000)
libm.so.6 => /lib64/libm.so.6 (0x00007f4ce7925000)
libresolv.so.2 => /lib64/libresolv.so.2 (0x00007f4ce7ef2000)
libc.so.6 => /lib64/libc.so.6 (0x00007f4ce7600000)
/lib64/ld-linux-x86-64.so.2 (0x00007f4ce7fd4000) |
The statically-linked release binary doesn't have this problem on RedHat9. |
Possible cause addressed in openziti/ziti-sdk-c#551 |
I reproduced this with a build against shared libssl v3 on the rocky9 CPack image on the v0.22.21 tag. I built the RPM with this branch (link to pull request), which enables overriding the TLS library with an env var. (
cd ./.github/actions/openziti-tunnel-build-action/redhat-9/ \
&& docker buildx build --platform linux/amd64 --tag rh9-builder . --load ;
) Then, checkout the v0.22.21 tag and run the CPack builder image for RedHat 9 with the TLS library env var set docker run \
--rm \
--platform linux\amd64 \
--volume "${PWD}:/github/workspace" --workdir "/github/workspace" \
--env TLSUV_TLSLIB=openssl \
rh9-builder ci-linux-x64 Release Finally, attempt enrollment on a vanilla rocky9 image. I got the same result with the almalinux/9-base image. docker run \
--network=host --rm --platform linux/amd64 \
--volume ./build/ziti-edge-tunnel-0.22.21-1.x86_64.rpm:/tmp/ziti-edge-tunnel.rpm \
--volume /tmp/miniziti-client.jwt:/tmp/ziti-id.jwt \
--entrypoint=/bin/bash rockylinux/rockylinux:9 \
-c 'dnf install -y /tmp/ziti-edge-tunnel.rpm && ldd /usr/bin/ziti-edge-tunnel && TLSUV_DEBUG=6 ZITI_LOG=6 ziti-edge-tunnel enroll --jwt /tmp/ziti-id.jwt --identity /tmp/ziti-id.json'
|
@scareything I reproduced this with latest ZET 0.22.21 built w/ OpenSSL on Rocky 9 and Alma 9. |
shared object links from the OpenSSL test build I created by running the RedHat9 CPack builder image with override TLS lib build param:
I found these versions of OpenSSL to be available for the shown container images from their respective, default repositories.
|
I tried the 0.20.0 release binary andI builtziti-edge-tunnel
from source tag v0.20.0 on Rocky 9 and theenroll
command always gets this error:I was able to enroll with the same binary artifact running on another Linux system (not Red Hat 9) without encountering this error, and I was able to enroll the same JWT with the release binary 0.20.0 running on another Linux system, so it doesn't seem to be a problem with the controller or the JWT or the binary itself, except when it's running on Red Hat 9.
The release binary is built with Mbed-TLS, and I set USE_OPENSSL=ON when I built from source, so both TLS implementations have been tried.
The text was updated successfully, but these errors were encountered: