Support Python 3.12

[chenrui333]

Description

Python 3.12 has been released on 2023-10-02: https://www.python.org/downloads/release/python-3120/

Azure CLI needs to support Python 3.12 like how we supported 3.10 (#19857), 3.11 (#24494).

TODOs

• Knack supports Python 3.12
  • Mark Python 3.12 support microsoft/knack#279
• azdev supports Python 3.12
  • Bump pylint to 3 azure-cli-dev-tools#458
• Dependencies are updated to the Python 3.12 compatible version
  • pymsalruntime (dependency of msal[broker]) {Core} Bump MSAL to 1.27.0 and pin pymsalruntime to 0.14.0 #28556
  • {Packaging} Bump cffi to 1.16.0 #29466
  • {Packaging} Bump PyWin32 to 306 #29486
• CI passes with Python 3.12
  • Remove distutils {Core} Remove deprecated usages of distutils to support Python 3.12 #28796
  • {Packaging} Add setuptools into dependency #27196
  • {Misc.} Remove deprecated assertRaisesRegexp #29504
  • {Misc.} Remove deprecated assertEquals #29515
  • {AKS} Fix enum check in test #29517
  • Remove assertDictContainsSubset and assertNotEquals Please port to Python3.12 #28194
    • {Misc.} Remove all occurrences of assertNotEquals #29120
    • {Misc.} Remove all occurrences of assertDictContainsSubset #29121
  • Hundreds of "SyntaxWarning: invalid escape sequence" will break in future python versions #28829
  • {Misc.} Improve code style and prepare for pylint 3 #29373
  • [Packaging] Support Python 3.12 #29465

[azure-client-tools-bot-prd]

Hi @chenrui333
Find similar issue #19857.

Issue title	Support Python 3.10
Create time	2021-10-12
Comment number	6

Possible solution:
As we did for Python 3.10 and 3.11, Azure CLI needs to support Python 3.12. We can follow the same steps we did for Python 3.10 support, which includes updating dependencies like pytest, pylint, and ensuring all test cases pass. Additionally, we need to make sure that Knack and azdev also support Python 3.12. We can refer to the pull requests for Knack and azdev that added support for Python 3.10 and update them accordingly for Python 3.12. Once we have updated the dependencies and ensured that all test cases pass, we can release a new version of Azure CLI that supports Python 3.12.

---

Please confirm if this resolves your issue.

[yonzhan]

Thank you for opening this issue, we will look into it.

[chenrui333]

I find there is no open issue to support py3.12, create this one for the team.

[Gyarbij]

Any progress on this and will this impact App Service Python 3.12 availability?

[borjamunozf]

Could be this the explanation why installing azure-cli with Python 3.12 throw this error complining about pymsalruntime?

building 'pymsalruntime.pymsalruntime' extension
      error: Microsoft Visual C++ 14.0 or greater is required. Get it with "Microsoft C++ Build Tools": https://visualstudio.microsoft.com/visual-cpp-build-tools/
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for pymsalruntime
Successfully built antlr4-python3-runtime
Failed to build pymsalruntime
ERROR: Could not build wheels for pymsalruntime, which is required to install pyproject.toml-based projects

[spauka]

Indeed it seems like the dependency of pymsalruntime means that azure-cli is not compatible with Python 3.12. The dependancy chain seems to be:
pymsalruntime<0.14,>=0.13.2 (from msal[broker]==1.24.0b2->azure-cli-core==2.53.1->azure-cli->az-cli)

The pymsalruntime library seems to use the deprecated and removed functions PyUnicode_AsUnicode and PyUnicode_FromUnicode, which means that the library cannot be built on the latest python.

[mikeeq]

Fedora 39 was just released, and it's containing Python 3.12. It is a huge blocker now for all Fedora users, as it's not possible to run Azure CLI "as is" without any additional hassle (running it in a container or with older version of python).

❯ az
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/opt/virtualenv/azure-cli/lib/python3.12/site-packages/azure/cli/__main__.py", line 39, in <module>
    az_cli = get_default_cli()
             ^^^^^^^^^^^^^^^^^
  File "/opt/virtualenv/azure-cli/lib/python3.12/site-packages/azure/cli/core/__init__.py", line 917, in get_default_cli
    from azure.cli.core.azlogging import AzCliLogging
  File "/opt/virtualenv/azure-cli/lib/python3.12/site-packages/azure/cli/core/azlogging.py", line 30, in <module>
    from azure.cli.core.commands.events import EVENT_INVOKER_PRE_CMD_TBL_TRUNCATE
  File "/opt/virtualenv/azure-cli/lib/python3.12/site-packages/azure/cli/core/commands/__init__.py", line 25, in <module>
    from azure.cli.core.extension import get_extension
  File "/opt/virtualenv/azure-cli/lib/python3.12/site-packages/azure/cli/core/extension/__init__.py", line 11, in <module>
    from distutils.sysconfig import get_python_lib  # pylint: disable=deprecated-module
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ModuleNotFoundError: No module named 'distutils'
❯ uname -a
Linux 6.5.10-300.fc39.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Nov  2 20:01:06 UTC 2023 x86_64 GNU/Linux
❯ python --version
Python 3.12.0

Links:

• https://fedoraproject.org/wiki/Changes/Python3.12
• https://peps.python.org/pep-0632/

[SumedhSingh12]

Do we have an ETA for this? This is blocking our customers who are using python 3.12 from using our azure cli extension.

[jiasli]

The pymsalruntime library seems to use the deprecated and removed functions PyUnicode_AsUnicode and PyUnicode_FromUnicode, which means that the library cannot be built on the latest python.

I have created AzureAD/microsoft-authentication-library-for-python#624 to track this issue.

[jiasli]

Do we have an ETA for this? This is blocking our customers who are using python 3.12 from using our azure cli extension.

While we will plan this feature as soon as possible, we can't provide an ETA at the moment.

Azure CLI is a product, not a library, so only MSI, DEB, RPM packages and docker images released by Microsoft are officially supported: https://learn.microsoft.com/en-us/cli/azure/install-azure-cli.

Also, in the source code, there is no declaration that Python 3.12 is supported:

azure-cli/src/azure-cli/setup.py

Lines 39 to 50 in f74a18b

	CLASSIFIERS = [
	'Development Status :: 5 - Production/Stable',
	'Intended Audience :: Developers',
	'Intended Audience :: System Administrators',
	'Programming Language :: Python',
	'Programming Language :: Python :: 3',
	'Programming Language :: Python :: 3.8',
	'Programming Language :: Python :: 3.9',
	'Programming Language :: Python :: 3.10',
	'Programming Language :: Python :: 3.11',
	'License :: OSI Approved :: MIT License',
	]

We don't guarantee Azure CLI works on all Python versions, such as the latest Python 3.12 or the deprecated Python 3.6. Installing Azure CLI with pip (#20476) is only a supplemental feature we provide for unsupported platforms, but without any warranty.

Thanks for understanding.

[MrMarkPower]

@jiasli its now over 2 months since this issue was first raised, are you any closer to knowing when this feature release (i.e. support for python 3.12) will be available please?

[mkleinbort-ic]

Any updates?

[AkechiShiro]

@mkleinbort-ic there is a workaround to install this dependency : AzureAD/microsoft-authentication-library-for-python#624 (comment)

But I haven't tested it yet, I don't have the time to meddle with this at the moment.

[DMiradakis]

The suggested workaround from #27673 (comment) above did not work for me. However, simply installing the setuptools library did the trick.

Python version: 3.12
pip install setuptools

[cbassett-silverstream]

The suggested workaround from #27673 (comment) above did not work for me. However, simply installing the setuptools library did the trick.

Python version: 3.12 pip install setuptools

Sadly, this didn't work for me either (I'm on Windows). Same error as before.
Thanks for sharing though.

[ddl-kfrench]

hey -- just wanted to bump this issue again to see if there is any estimate that can be provided for when (or if) the azure CLI will be compatible with python 3.12.

[PeterBosch2]

I saw that somebody states above that azure-cli does not support the installation via pip for Python 3.12. So, this message is just to spread the information, but no action is required.

A problem arose for the azure-cli feature in the DevContainer setup, as Debian Bookworm was not in the list of native installations, and a fallback to go via pip popped in. -> devcontainers/features#899

As I understand, the problem will be fixed over there, adding bookworm to the list of native installations.

[MrMarkPower]

Just to provide quick update re: latest release of azure-cli i.e. version 2.58.0 (5th March 2024)
This version of azure-cli still fails to install on Windows with the same pymsalruntime fatal error LNK1120 errors that were reported back in November 2023

C:\windows\system32>pip install azure-cli --upgrade
Collecting azure-cli
Downloading azure_cli-2.58.0-py3-none-any.whl.metadata (8.4 kB)
...
Collecting pymsalruntime<0.14,>=0.13.2 (from msal[broker]==1.26.0->azure-cli-core==2.58.0->azure-cli)
Using cached pymsalruntime-0.13.12.tar.gz (1.1 MB)
Installing build dependencies ... done
Getting requirements to build wheel ... done
Preparing metadata (pyproject.toml) ... done
...
Building wheels for collected packages: pymsalruntime
Building wheel for pymsalruntime (pyproject.toml) ... error
error: subprocess-exited-with-error

× Building wheel for pymsalruntime (pyproject.toml) did not run successfully.
│ exit code: 1
╰─> [50 lines of output]
...
"-IC:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\include\um" /Tcpymsalruntime/PyMsalRuntime.c /Fobuild\temp.win-amd64-cpython-312\Release\pymsalruntime/PyMsalRuntime.obj
PyMsalRuntime.c
pymsalruntime/PyMsalRuntime.c(2888): warning C4013: 'PyUnicode_FromUnicode' undefined; assuming extern returning int
pymsalruntime/PyMsalRuntime.c(2888): warning C4047: '=': 'PyObject *' differs in levels of indirection from 'int'
pymsalruntime/PyMsalRuntime.c(3230): warning C4013: 'PyUnicode_AsUnicode' undefined; assuming extern returning int
...
Creating library build\temp.win-amd64-cpython-312\Release\pymsalruntime\pymsalruntime.cp312-win_amd64.lib and object build\temp.win-amd64-cpython-312\Release\pymsalruntime\pymsalruntime.cp312-win_amd64.exp
PyMsalRuntime.obj : error LNK2001: unresolved external symbol PyUnicode_AsUnicode
PyMsalRuntime.obj : error LNK2001: unresolved external symbol PyUnicode_FromUnicode
build\lib.win-amd64-cpython-312\pymsalruntime\pymsalruntime.cp312-win_amd64.pyd : fatal error LNK1120: 2 unresolved externals
error: command 'C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Tools\MSVC\14.37.32822\bin\HostX86\x64\link.exe' failed with exit code 1120

Attempting to perform a manual upgrade of dependency pymsalruntime... pip install pymsalruntime --upgrade and pip install pymsalruntime==0.14.0 also fails with the same error. I see the latest release (0.14.1) was yanked back in January 2024 https://pypi.org/project/pymsalruntime/0.14.1/

[rakotomandimby]

pip install setuptools made Azure CLI work on my workstation, though I did not depply test. I just made simple az invocation tests, such as --version and so on.

[MrMarkPower]

Finally found some time to return to this and have thankfully made some progress. I've installed the latest pymsalruntime which was released last month...
https://pypi.org/project/pymsalruntime/0.14.2/
pip install pymsalruntime==0.14.2
and the current release of azure_cli which was release 9 days ago...
https://pypi.org/project/azure-cli/
pip install azure-cli==2.61.0
and azure_cli has finally installed successfully with python 3.12 on Windows Server 2022
Thank you to all involved in sorting this out

[onlined]

@bebound can you also add microsoft/knack#279 to TODO list at description?

[ringerc]

This breaks Azure CLI on Ubuntu 24.04 LTS (Noble Numbat) when using the system Python, e.g. installing with pipx. The install succeeds, but the component fails at runtime as it fails to declare a needed dependency:

...
    from distutils.sysconfig import get_python_lib  # pylint: disable=deprecated-module
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ModuleNotFoundError: No module named 'distutils'

If you installed the Azure CLI with pipx (pipx install azure-cli), you can

pipx inject azure-cli setuptools

to work around this.

It looks like the package fails to declare a dependency on setuptools.

[jiasli]

The latest pymsalruntime now supports Python 3.12: https://pypi.org/project/pymsalruntime/0.16.2/#files

[Goldziher]

Is there already work to ensure compatibility with python 3.13?

[jiasli]

Is there already work to ensure compatibility with python 3.13?

Feel free to create a new issue.