venv module does not copy the correct python exe

[inorton]

BPO	43749
Nosy	@pfmoore, @vsajip, @tjguk, @zware, @eryksun, @zooba, @miss-islington, @NiklasRosenstein, @inorton, @mitchhentges
PRs	bpo-43749: Ensure current exe is copied when using venv on windows #25216 [3.9] bpo-43749: Ensure current exe is copied when using venv on windows (GH-25216) (GH-30033) #30033 [3.10] bpo-43749: Ensure current exe is copied when using venv on windows (GH-25216) (GH-30034) #30034

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2021-04-06.13:27:05.454>
labels = ['3.10', 'library', '3.9', 'OS-windows', '3.11']
title = 'venv module does not copy the correct python exe'
updated_at = <Date 2021-12-11.03:21:25.207>
user = 'https://github.com/inorton'

bugs.python.org fields:

activity = <Date 2021-12-11.03:21:25.207>
actor = 'eryksun'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)', 'Windows']
creation = <Date 2021-04-06.13:27:05.454>
creator = 'Ian Norton'
dependencies = []
files = []
hgrepos = []
issue_num = 43749
keywords = ['patch']
message_count = 13.0
messages = ['390329', '390330', '396286', '396287', '408224', '408228', '408229', '408231', '408233', '408234', '408247', '408279', '408289']
nosy_count = 10.0
nosy_names = ['paul.moore', 'vinay.sajip', 'tim.golden', 'zach.ware', 'eryksun', 'steve.dower', 'miss-islington', 'n_rosenstein', 'Ian Norton', 'mitchhentges']
pr_nums = ['25216', '30033', '30034']
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = None
url = 'https://bugs.python.org/issue43749'
versions = ['Python 3.9', 'Python 3.10', 'Python 3.11']

[inorton]

On windows, the venv module does not copy the correct python exe if the current running exe (eg sys.executable) has been renamed (eg, named python3.exe)

venv will only make copies of python.exe, pythonw.exe, python_d.exe or pythonw_d.exe.

If for example the python executable has been renamed from python.exe to python3.exe (eg, to co-exist in a system where multiple pythons are on PATH) then this can fail with errors like:

Error: [WinError 2] The system cannot find the file specified

When venv tries to run pip in the new environment.

If the running python executable is a differently named copy then errors like the one described in https://bugs.python.org/issue40588 are seen.

[inorton]

This may also cause https://bugs.python.org/issue35644

[NiklasRosenstein]

I just encountered the same behavior. Steps to reproduce:

• Install Python 3 (I used 3.9.5) on Windows

• Use the Admin console to create a python3.exe link, e.g.

  cd "c:/Users/niklas/AppData/Local/Programs/Python/Python39"
  mklink python3.exe python.exe

• Use python3.exe to create a venv, e.g.

  python3 -m venv

The environment is created up until the point _setup_pip() is called in venv, which causes the following error to be printed:

Error: [WinError 2] The system cannot find the file specified

Running venv from the interpreter gives the full stack trace.

    $ python3 -c 'import venv; venv.main([".venv"])'
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "C:\Users\niklas\AppData\Local\Programs\Python\Python39\lib\venv\__init__.py", line 491, in main
        builder.create(d)
      File "C:\Users\niklas\AppData\Local\Programs\Python\Python39\lib\venv\__init__.py", line 75, in create
        self._setup_pip(context)
      File "C:\Users\niklas\AppData\Local\Programs\Python\Python39\lib\venv\__init__.py", line 299, in _setup_pip
        subprocess.check_output(cmd, stderr=subprocess.STDOUT)
      File "C:\Users\niklas\AppData\Local\Programs\Python\Python39\lib\subprocess.py", line 424, in check_output
        return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
      File "C:\Users\niklas\AppData\Local\Programs\Python\Python39\lib\subprocess.py", line 505, in run
        with Popen(*popenargs, **kwargs) as process:
      File "C:\Users\niklas\AppData\Local\Programs\Python\Python39\lib\subprocess.py", line 951, in __init__
        self._execute_child(args, executable, preexec_fn, close_fds,
      File "C:\Users\niklas\AppData\Local\Programs\Python\Python39\lib\subprocess.py", line 1420, in _execute_child
        hp, ht, pid, tid = _winapi.CreateProcess(executable, args,
    FileNotFoundError: [WinError 2] The system cannot find the file specified

[NiklasRosenstein]

To complete my previous comment, patching the function to see the command that is invoked, you can see it tries to invoke 'python3.exe' (seemingly derived from sys.executable) in the venv but the venv only contains 'python.exe'.

['C:\\Users\\niklas\\dotfiles\\.venv\\Scripts\\python3.exe', '-Im', 'ensurepip', '--upgrade', '--default-pip']

[mitchhentges]

Hey, can I request a backport of this change? For my use case, 3.9+ would be great, but going back to 3.7 should be viable? I trust your judgement :)

I'll happily create the PRs for the backport, but it looks like the bug has to be indicated as "needs backport to X.Y" by Python maintainers first (https://devguide.python.org/gitbootcamp/?highlight=backport#backporting-merged-changes)

[vsajip]

3.10 and 3.9 - OK, but 3. is security fixes only, I'm afraid.

[vsajip]

*3.8, I meant.

[mitchhentges]

That's great, thanks!

[vsajip]

New changeset bb8d645 by Miss Islington (bot) in branch '3.10':
[3.10] bpo-43749: Ensure current exe is copied when using venv on windows (GH-25216) (GH-30034)
bb8d645

[vsajip]

New changeset bad16f0 by Miss Islington (bot) in branch '3.9':
[3.9] bpo-43749: Ensure current exe is copied when using venv on windows (GH-25216) (GH-30033)
bad16f0

[eryksun]

The Windows implementation of symlink_or_copy() actually copies "python.exe" and "pythonw.exe" launchers from "Lib/venv/scripts/nt". One cannot simply copy the "python3.exe" executable because the required DLLs aren't copied. If this seemed to work when testing, it was only because the installation directory was in PATH. The solution that actually works is to copy the launcher as "python3.exe".

[zooba]

We'd also need to update the launcher to launch the executable with its name, which it currently doesn't do. I was looking at this recently for some other reason.

[eryksun]

The code to copy a file could be rewritten to use a regex match. For example:

        # Copy src to dst. If src is a base executable, copy a launcher.
        dirname, filename = os.path.split(src)
        m = re.match(r'(pythonw?)[0-9._]*((?:_d)?(?:\.exe|\.pdb))$',
                filename.lower())
        if m is not None:
            src = os.path.join(os.path.dirname(__file__),
                    'scripts', 'nt', m.group(1) + m.group(2))
            # If the base environment is a Python source build, use
            # the launcher in the build directory instead of
            # "[Lib/venv/scripts/nt](https://github.com/python/cpython/blob/main/Lib/venv/scripts/nt)".
            if sysconfig.is_python_build(True) or not os.path.isfile(src):
                basename = ('venvlauncher' if m.group(1) == 'python' else
                            'venvwlauncher')
                src = os.path.join(dirname, basename + m.group(2))

[lmy668]

To complete my previous comment, patching the function to see the command that is invoked, you can see it tries to invoke 'python3.exe' (seemingly derived from sys.executable) in the venv but the venv only contains 'python.exe'.

['C:\Users\niklas\dotfiles\.venv\Scripts\python3.exe', '-Im', 'ensurepip', '--upgrade', '--default-pip']

hi, this problem is still existed in recently, how to solve it?

[xingdongzhe]

@lmy668 what version do you use?

[lmy668]

@xingdongzhe i tried to python3.8.10 and 3.10.5, it's both not .

[xingdongzhe]

@lmy668 I do these on my windows , it works fine. Hope that help you too
I use gitbash

python3.8.10 -m venv venv # this will error with [WinError 2], but has venv dir
cp venv/Scripts/python.exe venv/Scripts/python3.8.10
python3.8.10 -m venv venv

[zooba]

cp venv/Scripts/python.exe venv/Scripts/python3.8.10

I would suggest at least adding a .exe to the end of this command. Windows may not technically need it, but we almost certainly assume it will be there, and because it's not the .10 may be treated as an extension at certain points.

But also, Python 3.8 is not getting bugfixes anymore unless there's a security aspect. You won't see any fixes there. 3.10.5 ought to have the first fix, but I don't recall what else needed to be done. In any case, demonstrating a problem in anything besides 3.11, 3.12 or 3.13 is not going to help move this issue forward.

[lmy668]

any reply is appreciating.