Skip to content

Commit

Permalink
Bump Python 3.9 → 3.10
Browse files Browse the repository at this point in the history
  • Loading branch information
@maresb
maresb committed May 26, 2022
1 parent 1392542 commit 46c4128
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion base-notebook/Dockerfile
View file
Original file line number Diff line number Diff line change
Expand Up @@ -85,7 +85,7 @@ RUN echo "auth requisite pam_deny.so" >> /etc/pam.d/su && \
USER ${NB_UID}

# Pin python version here, or set it to "default"
ARG PYTHON_VERSION=3.9
ARG PYTHON_VERSION=3.10

# Setup work directory for backward-compatibility
RUN mkdir "/home/${NB_USER}/work" && \
Expand Down

10 comments on commit 46c4128

@slmg
Copy link

@slmg slmg commented on 46c4128 May 31, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just to let you know (not sure if it's intentional or not)

docker-stacks/mypy.ini

Line 11 in 5e1a27a

python_version = 3.9

@mathbunnyru
Copy link
Member

@mathbunnyru mathbunnyru commented on 46c4128 May 31, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@slmg this is intentional and this doesn't have any effect on the images.
mypy.ini is only used for checking our python files and we support >=3.9 (some devs might not have python 3.10 and it's not really worth it for our code to move to 3.10 at all).

@slmg
Copy link

@slmg slmg commented on 46c4128 Jun 1, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would it be possible to trigger a new build with python-3.9? An unlucky timing made TensorFlow 2.8 available with python-3.10 only.

Date Image
2022-05-29T17:23:10Z jupyter/tensorflow-notebook:872b0256f345
jupyter/tensorflow-notebook:python-3.10.4
jupyter/tensorflow-notebook:tensorflow-2.8.1
2022-05-28T16:07:09Z jupyter/tensorflow-notebook:872b0256f345
jupyter/tensorflow-notebook:python-3.9.13
jupyter/tensorflow-notebook:tensorflow-2.6.2

@slmg
Copy link

@slmg slmg commented on 46c4128 Jun 1, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Answering my own question. I can see there is no easy way to do this in the current workflow. inputs would need to be defined on the workflow_dispatch event.

docker-stacks/.github/workflows/docker.yml

Line 44 in 5e1a27a

workflow_dispatch:

Triggering a workflow with python version as an arg would however make images tags collide. How about supporting python 3.9 alongside python 3.10 instead? Kinda blue-green deployment, it would ensure a smoother transition between two major envs.

Happy to create a feature request if you deem it's reasonable.

@slmg
Copy link

@slmg slmg commented on 46c4128 Jun 1, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

About tags collision, I just realized jupyter/tensorflow-notebook:872b0256f345 exists for both pyhon-3.10 and the latest python-3.9, as shown in my first comment. 👀

@mathbunnyru
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@slmg I think right now it's impossible to have many python versions simultaneously.
Our pipeline is one threaded and it takes so much time already.
And it also uses images from dockerhub instead of freshly built ones (it's so bad, but we will fix it).
And this is also the reason for the tag collision.

@slmg
Copy link

@slmg slmg commented on 46c4128 Jun 2, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @mathbunnyru, yes I could see you're in the middle of a big refactor of the workflows.:superhero:
No worries I'll adapt to this new world via the addition of a python 3.9 kernel then.
Thanks again for the good work on this repo!

@jakob-keller
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No worries I'll adapt to this new world via the addition of a python 3.9 kernel then.

We are forced to go down that route as well. Could anyone share their experience or steps to make this happen? Thanks!

@slmg
Copy link

@slmg slmg commented on 46c4128 Jun 11, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey @jakob-keller , because I use certain Jupyter extensions not yet supporting python-3.10, I ended up using the latest available python-3.9 image instead, and installed a pytthon-3.10 kernel side-by-side to transition peacefully. The process would be almost identical if you'd rather have the base env running on 3.10 and an additional 3.9 kernel though.

# Latest pythton-3.9 available before the world changed.
FROM jupyter/scipy-notebook:1c40816d7f7d

USER jovyan

WORKDIR /home/jovyan

# Name of the additional mamba env.
ARG NEW_ENV=py3.10

RUN \
    # Create a new conda env first cloning all packages.
    # Do this before the next mamba install cmd to avoid cloning un-necessary packages.
    mamba create --quiet --yes --name "${NEW_ENV}" --clone base && \
    \
    # Install Jupyterlab enhancements in base env.
    mamba install --quiet --yes \
        # Version .ipynb files as python files.
        jupytext \
        # Static code analysis.
        black \
        isort \
        jupyterlab_code_formatter \
        # Cells vim-bindings.
        jupyterlab_vim && \
    \
    # Upgrade the new env to use python3.10 and upgrades all relevant packages.
    mamba update --quiet --yes --name "${NEW_ENV}" python=3.10 && \
    # Install ML research stack.
    mamba install --quiet --yes --name "${NEW_ENV}" \
        ipykernel \
        tensorflow \
        # Leverages graphviz to produce model graph.
        pydot \
        # Hyper-parameters tuning with ease.
        keras-tuner && \
    "${CONDA_DIR}/envs/${NEW_ENV}/bin/python" -m pip install --no-cache-dir \
        # TODO when tensorflow 2.9.1 becomes available in conda-forge:
        #   * Pin tensorflow-addons to 0.17 (0.16 was built against tf 2.8)
        #   * Un-comment tensorflow_decision_forests (only available for tf 2.9)
        # https://github.com/conda-forge/tensorflow-feedstock/pull/240
        tensorflow-addons~=0.16.1 \
        # tensorflow_decision_forests \
        xgboost && \
    \
    # Rename existing kernel and install new kernel for Python 3.10.
    sed -i 's/"Python 3 (ipykernel)"/"Python 3.9"/' /opt/conda/share/jupyter/kernels/python3/kernel.json && \
    "${CONDA_DIR}/envs/${NEW_ENV}/bin/python" -m ipykernel install \
        --user \
        --name "${NEW_ENV}" \
        --display-name "Python 3.10" \
        # Dynamically point to DATA_DIR in notebooks.
        --env DATA_DIR "/home/${NB_USER}/data" \
        # Ensure that the environment is "activated" in jupyter.
        # Some packages like pydot rely on discovering bins (graphviz) from PATH.
        --env PATH "${CONDA_DIR}/envs/${NEW_ENV}/bin:${PATH}" && \
    \
    # Allow to run `mamba env activate` interactively later on.
    mamba init bash && \
    # Create volume dirs to ensure correct permissions is applied at first mount.
    mkdir -p logs/tensorflow data && \
    # Tidy up.
    mamba clean --all --quiet -f -y && \
    fix-permissions "${CONDA_DIR}" && \
    fix-permissions "/home/${NB_USER}"

@jakob-keller
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey @jakob-keller , because I use certain Jupyter extensions not yet supporting python-3.10, I ended up using the latest available python-3.9 image instead, and installed a pytthon-3.10 kernel side-by-side to transition peacefully. The process would be almost identical if you'd rather have the base env running on 3.10 and an additional 3.9 kernel though.

Thank you so much for your thorough response. Python 3.10 is not yet a requirement for us and I am now also pinning the last available Python 3.9 based image.

Please sign in to comment.