ziti-edge-tunnel enroll "COULD_NOT_PROCESS_CSR" on Red Hat 9 / Rocky 9 #517
Assignees
Comments
|
I ran the enroll command with strace and I see this message: Same message showing the value of {ca certs}: This could point toward the problem with processing the CSR or it may be a red herring message having to do with strace printing long filenames. |
|
I ran the enroll command with valgrind which reveals the context in which the failure occurs. |
|
that is a red herring. OpenSSL makes several tries loading certs from a byte buffer: file, PEM, DER |
|
the error comes back from controller |
|
James and I both found that the redhat8 build is able to run the same enroll command successfully on redhat9. When I come back to this issue I'll try to reproduce the error with a redhat9 build and the main release build updated to 0.20.2. |
|
I can not reproduce this issue with the release binary or RedHat 8 package 0.20.4 running on RedHat 9. It seems to only present when running the new RedHat 9 build. I will try to gain access to the controller log in hopes there is a more helpful error message emitted there. The main difference between the two builds is that we're running gcc 11 on Red Hat 9 instead of gcc 10 on Red Hat 8. |
[rocky@ip-172-31-4-195 ~]$ /opt/openziti/bin/ziti-edge-tunnel enroll -j ./client8.jwt -i ./client8.json
(28800)[ 0.492] ERROR ziti-sdk:ziti_enroll.c:227 enroll_cb() failed to enroll with controller: https://7ce7e424-6a92-4ff2-9459-ebbba32346fa.production.netfoundry.io:443 COULD_NOT_PROCESS_CSR (The supplied csr could not be processed)
(28800)[ 0.492] ERROR ziti-edge-tunnel:ziti-edge-tunnel.c:1994 enroll_cb() enrollment failed: COULD_NOT_PROCESS_CSR(-3)
[rocky@ip-172-31-4-195 ~]$ ldd /opt/openziti/bin/ziti-edge-tunnel
linux-vdso.so.1 (0x00007fffa2dfe000)
libssl.so.3 => /lib64/libssl.so.3 (0x00007facd17d3000)
libcrypto.so.3 => /lib64/libcrypto.so.3 (0x00007facd13a7000)
libatomic.so.1 => /lib64/libatomic.so.1 (0x00007facd139e000)
libm.so.6 => /lib64/libm.so.6 (0x00007facd12c3000)
libresolv.so.2 => /lib64/libresolv.so.2 (0x00007facd12af000)
libc.so.6 => /lib64/libc.so.6 (0x00007facd10a5000)
libz.so.1 => /lib64/libz.so.1 (0x00007facd1089000)
/lib64/ld-linux-x86-64.so.2 (0x00007facd187e000) |
|
Built on RH9 OS with this command: cmake \
-DCMAKE_BUILD_TYPE=Release \
-DCMAKE_TOOLCHAIN_FILE=./toolchains/default.cmake \
-DBUILD_DIST_PACKAGES=ON \
-DUSE_OPENSSL=ON \
-S . \
-B ./build
cmake \
--build ./build \
--target package \
--verbose |
|
I found that I can set env var |
|
The controller process is running at the default log level, and there were no messages emitted at the time of the failed enrollment. |
Steps to reproducebuild git checkout issue-514-package-for-redhat9
(cd ./.github/actions/openziti-tunnel-build-action/redhat-9/ && docker buildx build -t rh9-builder . --load ; )
docker run --rm -ti -v "${PWD}:/github/workspace" rh9-builderrun cat > /tmp/client10.jwt
# paste contents of JWT, press ctrl-D to send EOF
docker run -i --rm \
-v "${PWD}/build/programs/Release/ziti-edge-tunnel:/mnt" \
-e ZITI_LOG=4 docker.io/library/rockylinux:9 \
bash -c '{
dnf install -yq libatomic \
&& /mnt/ziti-edge-tunnel enroll \
--jwt - --identity /mnt/client10.json;
}' < /tmp/client10.jwt |
|
I verified the steps to reproduce are still applicable in v0.20.20. |
|
I can still reproduce this in v0.21.0. |
|
This issue no longer occurs after adapting RH9 builder to the new VCPKG preset. ❯ docker run -i --rm \
-v "${PWD}/build:/mnt" \
-e ZITI_LOG=4 docker.io/library/rockylinux:9 \
bash -euxc '{
dnf install -yq /mnt/ziti-edge-tunnel-0.21.4-1.x86_64.rpm \
&& /opt/openziti/bin/ziti-edge-tunnel version \
&& /opt/openziti/bin/ziti-edge-tunnel enroll \
--jwt - --identity /mnt/client10.json;
}' < /tmp/rh9.jwt
+ dnf install -yq /mnt/ziti-edge-tunnel-0.21.4-1.x86_64.rpm
Importing GPG key 0x350D275D:
Userid : "Rocky Enterprise Software Foundation - Release key 2022 <[email protected]>"
Fingerprint: 21CB 256A E16F C54C 6E65 2949 702D 426D 350D 275D
From : /etc/pki/rpm-gpg/RPM-GPG-KEY-Rocky-9
Installed:
acl-2.3.1-3.el9.x86_64 dbus-1:1.12.20-7.el9_1.x86_64
dbus-broker-28-7.el9.x86_64 dbus-common-1:1.12.20-7.el9_1.noarch
iproute-6.1.0-1.el9.x86_64 kmod-libs-28-7.el9.x86_64
libatomic-11.3.1-4.3.el9.x86_64 libbpf-2:1.0.0-2.el9.x86_64
libmnl-1.0.4-15.el9.x86_64 libseccomp-2.5.2-2.el9.x86_64 psmisc-23.4-3.el9.x86_64 systemd-252-13.el9_2.x86_64
systemd-pam-252-13.el9_2.x86_64 systemd-rpm-macros-252-13.el9_2.noarch ziti-edge-tunnel-0.21.4-1.x86_64
+ /opt/openziti/bin/ziti-edge-tunnel version
v0.21.4-19-gd0c5eff-local
+ /opt/openziti/bin/ziti-edge-tunnel enroll --jwt - --identity /mnt/client10.json
(178)[ 0.000] INFO ziti-sdk:utils.c:188 ziti_log_set_level() set log level: root=4/DEBUG (178)[ 0.000] INFO ziti-sdk:utils.c:188 ziti_log_set_level() set log level: root=4/DEBUG
(178)[ 0.000] INFO ziti-sdk:ziti_enroll.c:90 ziti_enroll() Ziti C SDK version 0.32.6 @2fc3556(HEAD) starting enrollment at (2023-06-08T20:40:50.906)
(178)[ 0.000] DEBUG ziti-sdk:jwt.c:106 load_jwt() filename is: -
(178)[ 0.000] DEBUG ziti-sdk:jwt.c:69 load_jwt_file() reading JWT from standard input
(178)[ 0.000] DEBUG ziti-sdk:jwt.c:99 load_jwt_file() jwt file content is: eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJlbSI6Im90dCIsImV4cCI6MTY4NjQyODk1OSwiaXNzIjoiaHR0cHM6Ly83Y2U3ZTQyNC02YTkyLTRmZjItOTQ1OS1lYmJiYTMyMzQ2ZmEucHJvZHVjdGlvbi5uZXRmb3VuZHJ5LmlvOjQ0MyIsImp0aSI6IjdjZTU3OWQwLTFmNTYtNDZjZC05NjcyLTM1NmUyOWJiN
zAwZCIsInN1YiI6IjVQN0haQlN5cjkifQ.H2lLBrdUoIAoocPcAgRWKXoKyxR83tKsl65UeY6VJBt3a-ZNEYdgGDgLnTa8VyGBZ3HEUNRUq8gjx-0Wtg1JcmsGCOKCV_TT6LEy18uXk_pR5cHdHLeiVYoaNDSmXDaSJCz-SDMRPZDNmyQjY-Dydt6ewryOpnkAuvlAr3GpHcdZXKAO4BmeA7EdmCj_zjQey_3bgmBdSgD-
v6s4cToCJwuuPNvVqOImHD6Kb7TZBJPnFcnInzuBsnwUBOhGEzjbGnDGejRugrelT0qb-wBOi-tzGBethZIdJDUBQiYWu1A7EXj38WnEDrrc4V4j3uLMFDtSqBpkIfXdFxeckiftBNoMzS6LudVqWcn0w8lixMNrXoJE_5Wsxkg1p8exzVDwx15NymkqzNhzff7svsrSS-HiTZL8tZk7XLPwIk38T2a9SGzLnL5bPMpgb2
DtJBYSk_i4dyrzrTe9z45-RBswJRQ8iEiI3dN90CTK5opLTTQW-9ZlcNc-zvm721o4HAkGXEjjXq01ej-KE05HG15KEEsyREEpwa_r9ioXOOvW-djFA24m70R3bchIYqr0RPR4nq39Gwfj37vGoczJnCMKtn0x5--gk-w4FQLyL3_tflNq6gCDduJL8MxMrYXatlDBDA7yNGrtX5cnQBCwj5fh29yefjuHM5FMD230NPXx
hXY
(178)[ 0.000] DEBUG ziti-sdk:jwt.c:36 parse_jwt_content() ecfg->jwt_signing_input is:
eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJlbSI6Im90dCIsImV4cCI6MTY4NjQyODk1OSwiaXNzIjoiaHR0cHM6Ly83Y2U3ZTQyNC02YTkyLTRmZjItOTQ1OS1lYmJiYTMyMzQ2ZmEucHJvZHVjdGlvbi5uZXRmb3VuZHJ5LmlvOjQ0MyIsImp0aSI6IjdjZTU3OWQwLTFmNTYtNDZjZC05NjcyLTM1NmUyOWJiN
zAwZCIsInN1YiI6IjVQN0haQlN5cjkifQ
(178)[ 0.000] DEBUG ziti-sdk:ziti_ctrl.c:408 ziti_ctrl_init() ctrl[7ce7e424-6a92-4ff2-9459-ebbba32346fa.production.netfoundry.io] ziti controller client initialized
(178)[ 0.084] DEBUG ziti-sdk:ziti_enroll.c:39 verify_controller_jwt() verifying JWT signature
(178)[ 0.084] DEBUG ziti-sdk:ziti_enroll.c:67 verify_controller_jwt() JWT verification succeeded!
(178)[ 0.160] DEBUG ziti-sdk:ziti_enroll.c:157 well_known_certs_cb() CA PEM len = 21925
(178)[ 0.160] DEBUG ziti-sdk:ziti_ctrl.c:408 ziti_ctrl_init() ctrl[7ce7e424-6a92-4ff2-9459-ebbba32346fa.production.netfoundry.io] ziti controller client initialized
(178)[ 0.315] DEBUG ziti-sdk:ziti_ctrl.c:325 ctrl_body_cb() ctrl[7ce7e424-6a92-4ff2-9459-ebbba32346fa.production.netfoundry.io] completed POST[/enroll?method=ott&token=7ce579d0-1f56-46cd-9672-356e29bb700d] in 0.142 s
(178)[ 0.315] DEBUG ziti-sdk:ziti_enroll.c:242 enroll_cb() successfully enrolled with controller https://7ce7e424-6a92-4ff2-9459-ebbba32346fa.production.netfoundry.io:443 |
|
Will resolve in #515 |
|
This issue is recurring with the RedHat9 release RPM v0.22.5. [root@rocky9 ~]# ldd /opt/openziti/bin/ziti-edge-tunnel
linux-vdso.so.1 (0x00007ffffd5de000)
libz.so.1 => /lib64/libz.so.1 (0x00007f4ce7fb5000)
libssl.so.3 => /lib64/libssl.so.3 (0x00007f4ce7f0f000)
libcrypto.so.3 => /lib64/libcrypto.so.3 (0x00007f4ce7a00000)
libatomic.so.1 => /lib64/libatomic.so.1 (0x00007f4ce7f06000)
libm.so.6 => /lib64/libm.so.6 (0x00007f4ce7925000)
libresolv.so.2 => /lib64/libresolv.so.2 (0x00007f4ce7ef2000)
libc.so.6 => /lib64/libc.so.6 (0x00007f4ce7600000)
/lib64/ld-linux-x86-64.so.2 (0x00007f4ce7fd4000) |
|
The statically-linked release binary doesn't have this problem on RedHat9. |
|
Possible cause addressed in openziti/ziti-sdk-c#551 |
|
I reproduced this with a build against shared libssl v3 on the rocky9 CPack image on the v0.22.21 tag. I built the RPM with this branch (link to pull request), which enables overriding the TLS library with an env var. (
cd ./.github/actions/openziti-tunnel-build-action/redhat-9/ \
&& docker buildx build --platform linux/amd64 --tag rh9-builder . --load ;
)Then, checkout the v0.22.21 tag and run the CPack builder image for RedHat 9 with the TLS library env var set docker run \
--rm \
--platform linux\amd64 \
--volume "${PWD}:/github/workspace" --workdir "/github/workspace" \
--env TLSUV_TLSLIB=openssl \
rh9-builder ci-linux-x64 ReleaseFinally, attempt enrollment on a vanilla rocky9 image. I got the same result with the almalinux/9-base image. docker run \
--network=host --rm --platform linux/amd64 \
--volume ./build/ziti-edge-tunnel-0.22.21-1.x86_64.rpm:/tmp/ziti-edge-tunnel.rpm \
--volume /tmp/miniziti-client.jwt:/tmp/ziti-id.jwt \
--entrypoint=/bin/bash rockylinux/rockylinux:9 \
-c 'dnf install -y /tmp/ziti-edge-tunnel.rpm && ldd /usr/bin/ziti-edge-tunnel && TLSUV_DEBUG=6 ZITI_LOG=6 ziti-edge-tunnel enroll --jwt /tmp/ziti-id.jwt --identity /tmp/ziti-id.json' |
|
@scareything I reproduced this with latest ZET 0.22.21 built w/ OpenSSL on Rocky 9 and Alma 9. |
|
shared object links from the OpenSSL test build I created by running the RedHat9 CPack builder image with override TLS lib build param: I found these versions of OpenSSL to be available for the shown container images from their respective, default repositories. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
I tried the 0.20.0 release binary andI builtziti-edge-tunnelfrom source tag v0.20.0 on Rocky 9 and theenrollcommand always gets this error:I was able to enroll with the same binary artifact running on another Linux system (not Red Hat 9) without encountering this error, and I was able to enroll the same JWT with the release binary 0.20.0 running on another Linux system, so it doesn't seem to be a problem with the controller or the JWT or the binary itself, except when it's running on Red Hat 9.
The release binary is built with Mbed-TLS, and I set USE_OPENSSL=ON when I built from source, so both TLS implementations have been tried.
The text was updated successfully, but these errors were encountered: