Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Breaks with requests 2.32.0: Not supported URL scheme http+docker #3256

Closed
rra opened this issue May 20, 2024 · 60 comments · Fixed by #3257, griptape-ai/griptape#820 or harsha-simhadri/big-ann-benchmarks#296

Comments

@rra
Copy link

rra commented May 20, 2024

With requests 2.32.0 (released about an hour ago as I write this), the docker library as called by tox-docker fails with the following exception:

Traceback (most recent call last):
  File "/home/eagle/dvl/venvs/gafaelfawr/lib/python3.12/site-packages/requests/adapters.py", line 532, in send
    conn = self._get_connection(request, verify, proxies=proxies, cert=cert)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/eagle/dvl/venvs/gafaelfawr/lib/python3.12/site-packages/requests/adapters.py", line 400, in _get_connection
    conn = self.poolmanager.connection_from_host(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/eagle/dvl/venvs/gafaelfawr/lib/python3.12/site-packages/urllib3/poolmanager.py", line 304, in connection_from_host
    return self.connection_from_context(request_context)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/eagle/dvl/venvs/gafaelfawr/lib/python3.12/site-packages/urllib3/poolmanager.py", line 326, in connection_from_context
    raise URLSchemeUnknown(scheme)
urllib3.exceptions.URLSchemeUnknown: Not supported URL scheme http+docker

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/eagle/dvl/venvs/gafaelfawr/lib/python3.12/site-packages/docker/api/client.py", line 214, in _retrieve_server_version
    return self.version(api_version=False)["ApiVersion"]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/eagle/dvl/venvs/gafaelfawr/lib/python3.12/site-packages/docker/api/daemon.py", line 181, in version
    return self._result(self._get(url), json=True)
                        ^^^^^^^^^^^^^^
  File "/home/eagle/dvl/venvs/gafaelfawr/lib/python3.12/site-packages/docker/utils/decorators.py", line 46, in inner
    return f(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/eagle/dvl/venvs/gafaelfawr/lib/python3.12/site-packages/docker/api/client.py", line 237, in _get
    return self.get(url, **self._set_request_timeout(kwargs))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/eagle/dvl/venvs/gafaelfawr/lib/python3.12/site-packages/requests/sessions.py", line 602, in get
    return self.request("GET", url, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/eagle/dvl/venvs/gafaelfawr/lib/python3.12/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/eagle/dvl/venvs/gafaelfawr/lib/python3.12/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/eagle/dvl/venvs/gafaelfawr/lib/python3.12/site-packages/requests/adapters.py", line 534, in send
    raise InvalidURL(e, request=request)
requests.exceptions.InvalidURL: Not supported URL scheme http+docker

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/eagle/dvl/venvs/gafaelfawr/lib/python3.12/site-packages/tox/session/cmd/run/single.py", line 48, in _evaluate
    code, outcomes = run_commands(tox_env, no_test)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/eagle/dvl/venvs/gafaelfawr/lib/python3.12/site-packages/tox/session/cmd/run/single.py", line 79, in run_commands
    MANAGER.tox_before_run_commands(tox_env)
  File "/home/eagle/dvl/venvs/gafaelfawr/lib/python3.12/site-packages/tox/plugin/manager.py", line 88, in tox_before_run_commands
    self.manager.hook.tox_before_run_commands(tox_env=tox_env)
  File "/home/eagle/dvl/venvs/gafaelfawr/lib/python3.12/site-packages/pluggy/_hooks.py", line 513, in __call__
    return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/eagle/dvl/venvs/gafaelfawr/lib/python3.12/site-packages/pluggy/_manager.py", line 120, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/eagle/dvl/venvs/gafaelfawr/lib/python3.12/site-packages/pluggy/_callers.py", line 139, in _multicall
    raise exception.with_traceback(exception.__traceback__)
  File "/home/eagle/dvl/venvs/gafaelfawr/lib/python3.12/site-packages/pluggy/_callers.py", line 103, in _multicall
    res = hook_impl.function(*args)
          ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/eagle/dvl/venvs/gafaelfawr/lib/python3.12/site-packages/tox_docker/tox4/plugin.py", line 73, in tox_before_run_commands
    docker_build_or_pull(container_config, log)
  File "/home/eagle/dvl/venvs/gafaelfawr/lib/python3.12/site-packages/tox_docker/plugin.py", line 57, in docker_build_or_pull
    docker_pull(container_config, log)
  File "/home/eagle/dvl/venvs/gafaelfawr/lib/python3.12/site-packages/tox_docker/plugin.py", line 65, in docker_pull
    docker = docker_module.from_env(version="auto")
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/eagle/dvl/venvs/gafaelfawr/lib/python3.12/site-packages/docker/client.py", line 96, in from_env
    return cls(
           ^^^^
  File "/home/eagle/dvl/venvs/gafaelfawr/lib/python3.12/site-packages/docker/client.py", line 45, in __init__
    self.api = APIClient(*args, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/eagle/dvl/venvs/gafaelfawr/lib/python3.12/site-packages/docker/api/client.py", line 197, in __init__
    self._version = self._retrieve_server_version()
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/eagle/dvl/venvs/gafaelfawr/lib/python3.12/site-packages/docker/api/client.py", line 221, in _retrieve_server_version
    raise DockerException(
docker.errors.DockerException: Error while fetching server API version: Not supported URL scheme http+docker

Reverting to requests 2.31.0 without any other changes fixes the problem.

@zhiyue
Copy link

zhiyue commented May 20, 2024

same problem

In [1]: import docker
   ...: client = docker.from_env()
---------------------------------------------------------------------------
URLSchemeUnknown                          Traceback (most recent call last)
File /usr/local/lib/python3.8/dist-packages/requests/adapters.py:532, in HTTPAdapter.send(self, request, stream, timeout, verify, cert, proxies)
    531 try:
--> 532     conn = self._get_connection(request, verify, proxies=proxies, cert=cert)
    533 except LocationValueError as e:

File /usr/local/lib/python3.8/dist-packages/requests/adapters.py:400, in HTTPAdapter._get_connection(self, request, verify, proxies, cert)
    398 else:
    399     # Only scheme should be lower case
--> 400     conn = self.poolmanager.connection_from_host(
    401         **host_params, pool_kwargs=pool_kwargs
    402     )
    404 return conn

File /usr/local/lib/python3.8/dist-packages/urllib3/poolmanager.py:304, in PoolManager.connection_from_host(self, host, port, scheme, pool_kwargs)
    302 request_context["host"] = host
--> 304 return self.connection_from_context(request_context)

File /usr/local/lib/python3.8/dist-packages/urllib3/poolmanager.py:326, in PoolManager.connection_from_context(self, request_context)
    325 if not pool_key_constructor:
--> 326     raise URLSchemeUnknown(scheme)
    327 pool_key = pool_key_constructor(request_context)

URLSchemeUnknown: Not supported URL scheme http+docker

During handling of the above exception, another exception occurred:

InvalidURL                                Traceback (most recent call last)
File /usr/local/lib/python3.8/dist-packages/docker/api/client.py:213, in APIClient._retrieve_server_version(self)
    212 try:
--> 213     return self.version(api_version=False)["ApiVersion"]
    214 except KeyError as ke:

File /usr/local/lib/python3.8/dist-packages/docker/api/daemon.py:181, in DaemonApiMixin.version(self, api_version)
    180 url = self._url("/version", versioned_api=api_version)
--> 181 return self._result(self._get(url), json=True)

File /usr/local/lib/python3.8/dist-packages/docker/utils/decorators.py:44, in update_headers.<locals>.inner(self, *args, **kwargs)
     43         kwargs['headers'].update(self._general_configs['HttpHeaders'])
---> 44 return f(self, *args, **kwargs)

File /usr/local/lib/python3.8/dist-packages/docker/api/client.py:236, in APIClient._get(self, url, **kwargs)
    234 @update_headers
    235 def _get(self, url, **kwargs):
--> 236     return self.get(url, **self._set_request_timeout(kwargs))

File /usr/local/lib/python3.8/dist-packages/requests/sessions.py:602, in Session.get(self, url, **kwargs)
    601 kwargs.setdefault("allow_redirects", True)
--> 602 return self.request("GET", url, **kwargs)

File /usr/local/lib/python3.8/dist-packages/requests/sessions.py:589, in Session.request(self, method, url, params, data, headers, cookies, files, auth, timeout, allow_redirects, proxies, hooks, stream, verify, cert, json)
    588 send_kwargs.update(settings)
--> 589 resp = self.send(prep, **send_kwargs)
    591 return resp

File /usr/local/lib/python3.8/dist-packages/requests/sessions.py:703, in Session.send(self, request, **kwargs)
    702 # Send the request
--> 703 r = adapter.send(request, **kwargs)
    705 # Total elapsed time of the request (approximately)

File /usr/local/lib/python3.8/dist-packages/requests/adapters.py:534, in HTTPAdapter.send(self, request, stream, timeout, verify, cert, proxies)
    533 except LocationValueError as e:
--> 534     raise InvalidURL(e, request=request)
    536 self.cert_verify(conn, request.url, verify, cert)

InvalidURL: Not supported URL scheme http+docker

The above exception was the direct cause of the following exception:

DockerException                           Traceback (most recent call last)
Cell In[1], line 2
      1 import docker
----> 2 client = docker.from_env()

File /usr/local/lib/python3.8/dist-packages/docker/client.py:94, in DockerClient.from_env(cls, **kwargs)
     92 version = kwargs.pop('version', None)
     93 use_ssh_client = kwargs.pop('use_ssh_client', False)
---> 94 return cls(
     95     timeout=timeout,
     96     max_pool_size=max_pool_size,
     97     version=version,
     98     use_ssh_client=use_ssh_client,
     99     **kwargs_from_env(**kwargs)
    100 )

File /usr/local/lib/python3.8/dist-packages/docker/client.py:45, in DockerClient.__init__(self, *args, **kwargs)
     44 def __init__(self, *args, **kwargs):
---> 45     self.api = APIClient(*args, **kwargs)

File /usr/local/lib/python3.8/dist-packages/docker/api/client.py:197, in APIClient.__init__(self, base_url, version, timeout, tls, user_agent, num_pools, credstore_env, use_ssh_client, max_pool_size)
    192 # version detection needs to be after unix adapter mounting
    193 if version is None or (isinstance(
    194                         version,
    195                         str
    196                         ) and version.lower() == 'auto'):
--> 197     self._version = self._retrieve_server_version()
    198 else:
    199     self._version = version

File /usr/local/lib/python3.8/dist-packages/docker/api/client.py:220, in APIClient._retrieve_server_version(self)
    215     raise DockerException(
    216         'Invalid response from docker daemon: key "ApiVersion"'
    217         ' is missing.'
    218     ) from ke
    219 except Exception as e:
--> 220     raise DockerException(
    221         f'Error while fetching server API version: {e}'
    222     ) from e

DockerException: Error while fetching server API version: Not supported URL scheme http+docker
pip list
Package                Version             
---------------------- --------------------
asttokens              2.4.1               
attrs                  19.3.0              
Automat                0.8.0               
backcall               0.2.0               
blinker                1.4                 
certifi                2019.11.28          
chardet                3.0.4               
charset-normalizer     3.3.2               
Click                  7.0                 
cloud-init             22.2                
colorama               0.4.3               
command-not-found      0.3                 
configobj              5.0.6               
constantly             15.1.0              
cryptography           2.8                 
dbus-python            1.2.16              
decorator              5.1.1               
distro                 1.4.0               
distro-info            0.23ubuntu1         
docker                 7.0.0               
entrypoints            0.3                 
executing              2.0.1               
httplib2               0.14.0              
hyperlink              19.0.0              
idna                   2.8                 
importlib-metadata     1.5.0               
incremental            16.10.1             
ipython                8.12.3              
jedi                   0.19.1              
Jinja2                 2.10.1              
jsonpatch              1.22                
jsonpointer            2.0                 
jsonschema             3.2.0               
keyring                18.0.1              
language-selector      0.1                 
launchpadlib           1.10.13             
lazr.restfulclient     0.14.2              
lazr.uri               1.0.3               
MarkupSafe             1.1.0               
matplotlib-inline      0.1.7               
more-itertools         4.2.0               
netifaces              0.10.4              
oauthlib               3.1.0               
packaging              24.0                
parso                  0.8.4               
pexpect                4.6.0               
pickleshare            0.7.5               
pip                    20.0.2              
prompt-toolkit         3.0.43              
pure-eval              0.2.2               
pyasn1                 0.4.2               
pyasn1-modules         0.2.1               
pygments               2.18.0              
PyGObject              3.36.0              
PyHamcrest             1.9.0               
PyJWT                  1.7.1               
pymacaroons            0.13.0              
PyNaCl                 1.3.0               
pyOpenSSL              19.0.0              
pyrsistent             0.15.5              
pyserial               3.4                 
python-apt             2.0.0+ubuntu0.20.4.7
python-debian          0.1.36ubuntu1       
PyYAML                 5.3.1               
requests               2.32.0              
requests-unixsocket    0.2.0               
SecretStorage          2.3.1               
service-identity       18.1.0              
setuptools             45.2.0              
simplejson             3.16.0              
six                    1.14.0              
sos                    4.3                 
ssh-import-id          5.10                
stack-data             0.6.3               
systemd-python         234                 
traitlets              5.14.3              
Twisted                18.9.0              
typing-extensions      4.11.0              
ubuntu-advantage-tools 27.9                
ufw                    0.36                
unattended-upgrades    0.1                 
urllib3                2.2.1               
wadllib                1.3.3               
wcwidth                0.2.13              
wheel                  0.34.2              
zipp                   1.0.0               
zope.interface         4.7.1 

@danb27
Copy link

danb27 commented May 20, 2024

same!

@felixfontein
Copy link
Contributor

I'm currently debugging this a bit (ansible-collections/community.docker#860), the issue is commit psf/requests@c0813a2 in requests.

Ok, this isn't very surprising, since that commit makes send() call _get_connection() instead of get_connection(). Now the Docker SDK for Python code overwrites get_connection(), but of course doesn't magically overwrite _get_connection() as well...

See for example https://github.com/docker/docker-py/blob/main/docker/transport/unixconn.py#L66.

@felixfontein
Copy link
Contributor

A simple fix is adding

    def _get_connection(self, request, *args, proxies=None, **kwargs):
        return self.get_connection(request.url, proxies)

at the bottom of https://github.com/docker/docker-py/blob/main/docker/transport/basehttpadapter.py.

Not sure how sustainable that is, but it works for now :)

ssbarnea added a commit to ansible/molecule that referenced this issue May 20, 2024
Apparently requests==2.32.0 is incompatible with vendored versions of
docker-py from inside community.docker collection. We do not have a
direct dependency on that but one of our integration tests fails, so we
only update our test dependencies.

Related: docker/docker-py#3256
Related:
ansible-collections/community.docker#860
Related:
ansible-collections/community.docker#861
@felixfontein
Copy link
Contributor

I created #3257 with that fix.

@cardosofede
Copy link

same issue here, the fix from @felixfontein works for me!
for now I just replaced the dependency docker for git+https://github.com/felixfontein/docker-py@requests-hotfix

VietND96 added a commit to SeleniumHQ/docker-selenium that referenced this issue May 21, 2024
@geerlingguy
Copy link

Ditto here :)

Just saw lots of builds start failing, glad there's already a fix... and it seems the CI build is failing already, but unrelated to the fix above.

@AlexiaChen
Copy link

haha Seems a lot of CI fails cause this issue. me tooooooo.

@bparthas21
Copy link

Our team wasted all day today trying to triage and put a fix to this problem. After this incident, is there any plan to revisit this unpinned dependency specification:

"requests >= 2.26.0",

@felixfontein
Copy link
Contributor

Pinning dependencies, in particular very general dependencies such as requests, generally causes a lot more problems than it solves. The best place where to pin dependencies is at the end of the chain, i.e. where the venv is created that contains all the Python libraries and programs that you need.

@bparthas21
Copy link

Pinning dependencies, in particular very general dependencies such as requests, generally causes a lot more problems than it solves. The best place where to pin dependencies is at the end of the chain, i.e. where the venv is created that contains all the Python libraries and programs that you need.

In our process, we had pinned to docker==6.0.0 but did not realize we had to pin requests also.

@felixfontein
Copy link
Contributor

If you only pin direct dependencies, you're usually missing A LOT of indirect dependencies that can cause all kind of problems - such as in this case.

You might want to use pip freeze (https://pip.pypa.io/en/stable/cli/pip_freeze/) or pip-tools (https://pypi.org/project/pip-tools/) to pin all dependencies. Especially with pip-tools you can create a workflow that pins everything, and you have a CI workflow that regularly tries to upgrades dependencies and checks whether the result passes CI. Also check out Renovate (https://github.com/renovatebot/renovate) and Dependabot, which help a lot with that.

jaychia added a commit to Eventual-Inc/Daft that referenced this issue May 21, 2024
The latest version of docker-py fails on the latest version of requests

See: docker/docker-py#3256

---------

Co-authored-by: Jay Chia <[email protected]@users.noreply.github.com>
stevenengland added a commit to paperless-ngx/ansible that referenced this issue Jul 13, 2024
stevenengland added a commit to paperless-ngx/ansible that referenced this issue Jul 13, 2024
* add 2.9 support

* add v2.10 support

* Add doc for maintenance calls

* force requests==2.31.0

docker/docker-py#3256

* fix paperless_ngx_conf_trash_dir occurences

* fix none type trash dir variable
jpmat296 added a commit to matsusoft-fr/ansible-role-elastic-agent that referenced this issue Jul 13, 2024
CheckmkCI pushed a commit to Checkmk/checkmk that referenced this issue Jul 22, 2024
This version adds a bugfix for docker/docker-py#3256, which we experienced in our CI environment.

Change-Id: I4e718d8a6464577b073c3e88157c47dc155a1e2d
bagibence added a commit to BeNeuroLab/beneuro_experimental_data_organization that referenced this issue Jul 24, 2024
…g dependencies installed

We ran into this issue when trying to run kilosort through docker:
docker/docker-py#3256
Hopefully updating docker-py to 7.1.0 fixes it:
docker/docker-py#3257 (comment)
bagibence added a commit to BeNeuroLab/beneuro_experimental_data_organization that referenced this issue Jul 24, 2024
…g dependencies installed (#59)

* Update spikeinterface and docker, add workflow to test with processing dependencies installed

We ran into this issue when trying to run kilosort through docker:
docker/docker-py#3256
Hopefully updating docker-py to 7.1.0 fixes it:
docker/docker-py#3257 (comment)

* Remove numba as an explicit dependency
MikiDi added a commit to redpencilio/ldes-consumer-manager that referenced this issue Jul 29, 2024
KristoffAlaric added a commit to KristoffAlaric/ann-benchmarks that referenced this issue Aug 3, 2024
pld-gitsync pushed a commit to pld-linux/python3-docker that referenced this issue Aug 6, 2024
bestreviewmysoftware2 added a commit to bestreviewmysoftware2/ann-benchmarks that referenced this issue Aug 13, 2024
@meijiesky
Copy link

I am using requests 2.32.3 and have the same issue! Is it fixed by now and is it released?

@slonopotamus
Copy link

The fix is included in docker-py 7.1.0.

@someqst
Copy link

someqst commented Aug 16, 2024

I faced the same problem.
requests version is 2.28.2. Ubuntu 22.04. Docker 27.1.2
Any solutions?
Thank you

bestsoftwaretopappreviews3 added a commit to bestsoftwaretopappreviews3/ann-benchmarks that referenced this issue Aug 17, 2024
Latios96 added a commit to Latios96/cato that referenced this issue Aug 21, 2024
rinoradCross added a commit to rinoradCross/search_neighbour_benchamrks that referenced this issue Sep 5, 2024
0xmonsblockmans added a commit to 0xmonsblockmans/Vector_Engine that referenced this issue Sep 5, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet