Cannot install pod5 through pip on ARM due to dependency issues

[hasindu2008]

We are using pod5 pip package in blue-crab to enable BLOW5<->POD5 conversion, however, pod5 installation on ARM64 is failing due to a dependency called polars failing. @Psy-Fer may add more info if required, as I am not too familiar with python package managers.

Python 3.8.10
Cloning into 'slow5lib'...
WARNING: The directory '/github/home/.cache/pip' or its parent directory is not owned or is not writable by the current user. The cache has been disabled. Check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
Processing /github/workspace/slow5lib
Requirement already satisfied: numpy in /usr/local/lib/python3.8/dist-packages (from pyslow5==1.2.0b0) (1.24.4)
Building wheels for collected packages: pyslow5
  Building wheel for pyslow5 (setup.py): started
  Building wheel for pyslow5 (setup.py): still running...
  Building wheel for pyslow5 (setup.py): finished with status 'done'
  Created wheel for pyslow5: filename=pyslow5-1.2.0b0-cp38-cp38-linux_aarch64.whl size=2364754 sha256=f9c8eb99b2948898fa830231cff42de6137d032048c0ea16f1b70328f75fa56f
  Stored in directory: /tmp/pip-ephem-wheel-cache-cm98_g4r/wheels/a8/65/f3/9a470dd09cd16d981c947b80a4b07ba270193ac95329813e57
Successfully built pyslow5
Installing collected packages: pyslow5
Successfully installed pyslow5-1.2.0b0
WARNING: The directory '/github/home/.cache/pip' or its parent directory is not owned or is not writable by the current user. The cache has been disabled. Check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
Processing /github/workspace
Requirement already satisfied: numpy in /usr/local/lib/python3.8/dist-packages (from blue-crab==0.1.0) (1.24.4)
Collecting pod5
  Downloading pod5-0.3.2-py3-none-any.whl (69 kB)
Requirement already satisfied: pyslow5>=1.0.1b0 in /usr/local/lib/python3.8/dist-packages (from blue-crab==0.1.0) (1.2.0b0)
Collecting vbz-h5py-plugin
  Downloading vbz_h5py_plugin-1.0.1-py3-none-any.whl (1.4 MB)
Collecting pyarrow~=14.0.0
  Downloading pyarrow-14.0.2-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (36.0 MB)
Collecting more-itertools
  Downloading more_itertools-10.1.0-py3-none-any.whl (55 kB)
Collecting polars~=0.19
  Downloading polars-0.20.2.tar.gz (3.0 MB)
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
    Preparing wheel metadata: started
    Preparing wheel metadata: finished with status 'error'
    ERROR: Command errored out with exit status 1:
     command: /usr/bin/python3 /tmp/tmpip4m_pj0 prepare_metadata_for_build_wheel /tmp/tmpt17wjlj3
         cwd: /tmp/pip-install-x8q6a8ai/polars
    Complete output (6 lines):
    
    Cargo, the Rust package manager, is not installed or is not on PATH.
    This package requires Rust and Cargo to compile extensions. Install it through
    the system's package manager or via https://rustup.rs/
    
    Checking for Rust toolchain....
    ----------------------------------------
ERROR: Command errored out with exit status 1: /usr/bin/python3 /tmp/tmpip4m_pj0 prepare_metadata_for_build_wheel /tmp/tmpt17wjlj3 Check the logs for full command output.

[HalfPhoton]

Hi @hasindu2008,
Have you tried following the instructions in the error message?

    Cargo, the Rust package manager, is not installed or is not on PATH.
    This package requires Rust and Cargo to compile extensions. Install it through
    the system's package manager or via https://rustup.rs/

[hasindu2008]

I am not keen to install another package manager, because I belive an embedded system with a lightweight ARM processor should not be bloated with many dependencies. But I just tried now, yet to end up in another error:

Python 3.8.10
Cloning into 'slow5lib'...
WARNING: The directory '/github/home/.cache/pip' or its parent directory is not owned or is not writable by the current user. The cache has been disabled. Check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
Processing /github/workspace/slow5lib
Requirement already satisfied: numpy in /usr/local/lib/python3.8/dist-packages (from pyslow5==1.2.0b0) (1.24.4)
Building wheels for collected packages: pyslow5
  Building wheel for pyslow5 (setup.py): started
  Building wheel for pyslow5 (setup.py): still running...
  Building wheel for pyslow5 (setup.py): finished with status 'done'
  Created wheel for pyslow5: filename=pyslow5-1.2.0b0-cp38-cp38-linux_aarch64.whl size=2364745 sha256=54c1a68dc9a182125ba8f3ce66ce27e72363da75c34fb5390ffe092c18ac7518
  Stored in directory: /tmp/pip-ephem-wheel-cache-i5atf7b0/wheels/a8/65/f3/9a470dd09cd16d981c947b80a4b07ba270193ac95329813e57
Successfully built pyslow5
Installing collected packages: pyslow5
Successfully installed pyslow5-1.2.0b0
WARNING: The directory '/github/home/.cache/pip' or its parent directory is not owned or is not writable by the current user. The cache has been disabled. Check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
Processing /github/workspace
Requirement already satisfied: numpy in /usr/local/lib/python3.8/dist-packages (from blue-crab==0.1.0) (1.24.4)
Collecting pod5
  Downloading pod5-0.3.2-py3-none-any.whl (69 kB)
Requirement already satisfied: pyslow5>=1.0.1b0 in /usr/local/lib/python3.8/dist-packages (from blue-crab==0.1.0) (1.2.0b0)
Collecting vbz-h5py-plugin
  Downloading vbz_h5py_plugin-1.0.1-py3-none-any.whl (1.4 MB)
Collecting tqdm
  Downloading tqdm-4.66.1-py3-none-any.whl (78 kB)
Collecting packaging
  Downloading packaging-23.2-py3-none-any.whl (53 kB)
Collecting lib-pod5==0.3.2
  Downloading lib_pod5-0.3.2-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (2.5 MB)
Collecting polars~=0.19
  Downloading polars-0.20.2.tar.gz (3.0 MB)
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
    Preparing wheel metadata: started
    Preparing wheel metadata: finished with status 'done'
Collecting iso8601
  Downloading iso8601-2.1.0-py3-none-any.whl (7.5 kB)
Collecting more-itertools
  Downloading more_itertools-10.1.0-py3-none-any.whl (55 kB)
Collecting pytz
  Downloading pytz-2023.3.post1-py2.py3-none-any.whl (502 kB)
Collecting pyarrow~=14.0.0
  Downloading pyarrow-14.0.2-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (36.0 MB)
Collecting h5py~=3.10.0
  Downloading h5py-3.10.0-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (9.0 MB)
Building wheels for collected packages: blue-crab, polars
  Building wheel for blue-crab (setup.py): started
  Building wheel for blue-crab (setup.py): finished with status 'done'
  Created wheel for blue-crab: filename=blue_crab-0.1.0-py3-none-any.whl size=18839 sha256=3f215af733bb2c77e8194da6abb45[1218](https://github.com/Psy-Fer/blue-crab/actions/runs/7286874303/job/19856422254#step:6:1219)870b1a15638378b5f3c7827b7ccbcf4
  Stored in directory: /tmp/pip-ephem-wheel-cache-kazqni0j/wheels/a0/95/bc/28c8fad24675c98206b85c82c40eae6717d4e0f16cb0a0e0bc
  Building wheel for polars (PEP 517): started
  Building wheel for polars (PEP 517): still running...
  Building wheel for polars (PEP 517): still running...
  Building wheel for polars (PEP 517): still running...
  Building wheel for polars (PEP 517): finished with status 'error'
  ERROR: Command errored out with exit status 1:
   command: /usr/bin/python3 /tmp/tmp_z27vahw build_wheel /tmp/tmpwmlblph8
       cwd: /tmp/pip-install-0zjy83x6/polars
  Complete output (110 lines):
  Running `maturin pep517 build-wheel -i /usr/bin/python3 --compatibility off`
  🍹 Building a mixed python/rust project
  🔗 Found pyo3 bindings with abi3 support for Python ≥ 3.8
  🐍 Not using a specific python interpreter
     Compiling libc v0.2.151
     Compiling proc-macro2 v1.0.70
     Compiling unicode-ident v1.0.12
     Compiling autocfg v1.1.0
     Compiling cfg-if v1.0.0
     Compiling version_check v0.9.4
     Compiling serde v1.0.193
     Compiling memoffset v0.9.0
     Compiling quote v1.0.33
     Compiling getrandom v0.2.11
     Compiling once_cell v1.19.0
     Compiling syn v2.0.41
     Compiling jobserver v0.1.27
     Compiling crossbeam-utils v0.8.17
     Compiling cc v1.0.83
     Compiling libm v0.2.8
     Compiling ahash v0.8.6
     Compiling num-traits v0.2.17
     Compiling zerocopy v0.7.31
     Compiling pin-project-lite v0.2.13
     Compiling crossbeam-epoch v0.9.16
     Compiling futures-core v0.3.29
     Compiling rayon-core v1.12.0
     Compiling pkg-config v0.3.27
     Compiling crossbeam-deque v0.8.4
     Compiling memchr v2.6.4
     Compiling itoa v1.0.10
     Compiling either v1.9.0
     Compiling slab v0.4.9
     Compiling futures-channel v0.3.29
     Compiling rayon v1.8.0
     Compiling futures-task v0.3.29
     Compiling allocator-api2 v0.2.16
     Compiling serde_derive v1.0.193
     Compiling futures-sink v0.3.29
     Compiling ring v0.17.7
     Compiling equivalent v1.0.1
     Compiling futures-util v0.3.29
     Compiling bytes v1.5.0
     Compiling hashbrown v0.14.3
  error[E0554]: `#![feature]` may not be used on the stable release channel
    --> /github/home/.cargo/registry/src/index.crates.io-6f17d22bba15001f/hashbrown-0.14.3/src/lib.rs:15:5
     |
  15 | /     feature(
  16 | |         test,
  17 | |         core_intrinsics,
  18 | |         dropck_eyepatch,
  ...  |
  24 | |         strict_provenance
  25 | |     )
     | |_____^
  
  error[E0554]: `#![feature]` may not be used on the stable release channel
    --> /github/home/.cargo/registry/src/index.crates.io-6f17d22bba15001f/hashbrown-0.14.3/src/lib.rs:16:9
     |
  16 |         test,
     |         ^^^^
  
  error[E0554]: `#![feature]` may not be used on the stable release channel
    --> /github/home/.cargo/registry/src/index.crates.io-6f17d22bba15001f/hashbrown-0.14.3/src/lib.rs:17:9
     |
  17 |         core_intrinsics,
     |         ^^^^^^^^^^^^^^^
  
  error[E0554]: `#![feature]` may not be used on the stable release channel
    --> /github/home/.cargo/registry/src/index.crates.io-6f17d22bba15001f/hashbrown-0.14.3/src/lib.rs:20:9
     |
  20 |         extend_one,
     |         ^^^^^^^^^^
  
  error[E0554]: `#![feature]` may not be used on the stable release channel
    --> /github/home/.cargo/registry/src/index.crates.io-6f17d22bba15001f/hashbrown-0.14.3/src/lib.rs:21:9
     |
  21 |         allocator_api,
     |         ^^^^^^^^^^^^^
  
  error[E0554]: `#![feature]` may not be used on the stable release channel
    --> /github/home/.cargo/registry/src/index.crates.io-6f17d22bba15001f/hashbrown-0.14.3/src/lib.rs:22:9
     |
  22 |         slice_ptr_get,
     |         ^^^^^^^^^^^^^
  
  error[E0554]: `#![feature]` may not be used on the stable release channel
    --> /github/home/.cargo/registry/src/index.crates.io-6f17d22bba15001f/hashbrown-0.14.3/src/lib.rs:23:9
     |
  23 |         maybe_uninit_array_assume_init,
     |         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  
  error[E0658]: use of unstable library feature 'build_hasher_simple_hash_one'
     --> /github/home/.cargo/registry/src/index.crates.io-6f17d22bba15001f/hashbrown-0.14.3/src/map.rs:262:18
      |
  262 |     hash_builder.hash_one(val)
      |                  ^^^^^^^^
      |
      = note: see issue #86161 <https://github.com/rust-lang/rust/issues/86161> for more information
  
     Compiling futures-macro v0.3.29
     Compiling heck v0.4.1
  Some errors have detailed explanations: E0554, E0658.
  For more information about an error, try `rustc --explain E0554`.
  error: could not compile `hashbrown` (lib) due to 8 previous errors
  warning: build failed, waiting for other jobs to finish...
  💥 maturin failed
    Caused by: Failed to build a native library through cargo
    Caused by: Cargo build finished with "exit status: 101": `PYO3_ENVIRONMENT_SIGNATURE="cpython-3.8-64bit" PYO3_PYTHON="/usr/bin/python3" PYTHON_SYS_EXECUTABLE="/usr/bin/python3" "cargo" "rustc" "--message-format" "json-render-diagnostics" "--manifest-path" "/tmp/pip-install-0zjy83x6/polars/py-polars/Cargo.toml" "--release" "--lib"`
  Error: command ['maturin', 'pep517', 'build-wheel', '-i', '/usr/bin/python3', '--compatibility', 'off'] returned non-zero exit status 1
  ----------------------------------------
  ERROR: Failed building wheel for polars
Successfully built blue-crab
Failed to build polars
ERROR: Could not build wheels for polars which use PEP 517 and cannot be installed directly

[Psy-Fer]

Might have something to do with this

#94

I hadn't gotten around to fixing it yet.

[hasindu2008]

@Psy-Fer I already fixed that setup.py install to be pip install . on the 21st of December and sent a PR here Psy-Fer/blue-crab#10. So that is not the issue.

[iiSeymour]

polars provides polars-0.19.19-cp38-abi3-manylinux_2_24_aarch64.whl so you shouldn't need a toolchain for install - see https://pypi.org/project/polars/0.19.19/#files

What OS + libc version is the target machine?

[HalfPhoton]

@Psy-Fer I already fixed that setup.py install to be pip install . on the 21st of December and sent a PR here Psy-Fer/blue-crab#10. So that is not the issue.

@hasindu2008, has this been merged in? The PR shows as open and master doesn't have the changes.

[iiSeymour]

Okay I see the failing jobs (OS / libc should be fine) - can you make sure you upgrade pip as @HalfPhoton suggests. This is often the issue with Python package issues.

[hasindu2008]

oooooooooooooh. The pip install --upgrade pip?
Let me try 🎉

[hasindu2008]

I might not be doing it right, but seems to still giving a cargo error:

Python 3.8.10
Cloning into 'slow5lib'...
WARNING: The directory '/github/home/.cache/pip' or its parent directory is not owned or is not writable by the current user. The cache has been disabled. Check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
Processing /github/workspace/slow5lib
Requirement already satisfied: numpy in /usr/local/lib/python3.8/dist-packages (from pyslow5==1.2.0b0) (1.24.4)
Building wheels for collected packages: pyslow5
  Building wheel for pyslow5 (setup.py): started
  Building wheel for pyslow5 (setup.py): still running...
  Building wheel for pyslow5 (setup.py): finished with status 'done'
  Created wheel for pyslow5: filename=pyslow5-1.2.0b0-cp38-cp38-linux_aarch64.whl size=2364724 sha256=22e124ac3828ebc8555c6f7b70c4ef05669978eb7437a31bbca1e62c8f59b863
  Stored in directory: /tmp/pip-ephem-wheel-cache-uipxecl2/wheels/a8/65/f3/9a470dd09cd16d981c947b80a4b07ba270193ac95329813e57
Successfully built pyslow5
Installing collected packages: pyslow5
Successfully installed pyslow5-1.2.0b0
WARNING: The directory '/github/home/.cache/pip' or its parent directory is not owned or is not writable by the current user. The cache has been disabled. Check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
Collecting pip
  Downloading pip-24.0-py3-none-any.whl (2.1 MB)
Installing collected packages: pip
  Attempting uninstall: pip
    Found existing installation: pip 20.0.2
    Not uninstalling pip at /usr/lib/python3/dist-packages, outside environment /usr
    Can't uninstall 'pip'. No files were found to uninstall.
Successfully installed pip-24.0
WARNING: The directory '/github/home/.cache/pip' or its parent directory is not owned or is not writable by the current user. The cache has been disabled. Check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
Processing /github/workspace
Requirement already satisfied: numpy in /usr/local/lib/python3.8/dist-packages (from blue-crab==0.1.0) (1.24.4)
Collecting pod5
  Downloading pod5-0.3.6-py3-none-any.whl (69 kB)
Requirement already satisfied: pyslow5>=1.0.1b0 in /usr/local/lib/python3.8/dist-packages (from blue-crab==0.1.0) (1.2.0b0)
Collecting polars~=0.19
  Downloading polars-0.20.9.tar.gz (3.3 MB)
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
    Preparing wheel metadata: started
    Preparing wheel metadata: finished with status 'error'
    ERROR: Command errored out with exit status 1:
     command: /usr/bin/python3 /tmp/tmpjw1nfay2 prepare_metadata_for_build_wheel /tmp/tmprnzjb7h8
         cwd: /tmp/pip-install-j8uislqq/polars
    Complete output (6 lines):
    
    Cargo, the Rust package manager, is not installed or is not on PATH.
    This package requires Rust and Cargo to compile extensions. Install it through
    the system's package manager or via https://rustup.rs/
    
    Checking for Rust toolchain....
    ----------------------------------------
ERROR: Command errored out with exit status 1: /usr/bin/python3 /tmp/tmpjw1nfay2 prepare_metadata_for_build_wheel /tmp/tmprnzjb7h8 Check the logs for full command output.

It says Successfully installed pip-24.0,

Workflow file says:

  arm64:
    name: pyslow5 ubuntu arm
    runs-on: ubuntu-latest
    steps:
    - run: docker run --rm --privileged multiarch/qemu-user-static:register --reset
    - uses: docker://multiarch/ubuntu-core:arm64-focal
      with:
        args: 'uname -a'
    - uses: actions/checkout@v1
    - uses: docker://multiarch/ubuntu-core:arm64-focal
      with:
        args: >
          bash -c "apt-get update &&
          apt-get install -y zlib1g-dev gcc make python3 python3-pip git wget &&
          pip3 install setuptools cython numpy &&
          ldd --version && gcc --version && python3 --version && 
          git clone -b dev https://github.com/hasindu2008/slow5lib && cd slow5lib && pip3 install . && cd .. &&
          pip3 install --upgrade pip && pip3 install . && blue-crab --help && test/test.sh"

[hasindu2008]

@Psy-Fer I already fixed that setup.py install to be pip install . on the 21st of December and sent a PR here Psy-Fer/blue-crab#10. So that is not the issue.

@hasindu2008, has this been merged in? The PR shows as open and master doesn't have the changes.

It is not merged, but the updated workflows are running from this branch:
https://github.com/Psy-Fer/blue-crab/tree/linkupdate

[HalfPhoton]

Can you try again and add pip3 --version after the upgrade?
Even though it says Successfully installed pip-24.0 - Not uninstalling pip at /usr/lib/python3/dist-packages, outside environment /usr is a bit suspicious. 🤞