Dependency resolver issue

[jancespivo]

I added the new unpinned package to a requirements.in file and I expected the pip-compile solves dependencies automatically. However, I found it is not working as I expected. Also, I found the pip-compile step depends also on the previous run.

Environment Versions

• Manjaro Linux
• Python version: Python 3.8.6
• pip version: pip 20.3.1
• pip-tools version: pip-compile --version

Steps to replicate

1. requirements.in

mypy < 0.790
django-stubs

2. pip-compile requirements.in

Could not find a version that matches mypy<0.790,>=0.790 (from -r requirements.in (line 1))
Tried: 0.1, 0.11, 0.12, 0.13, 0.14, 0.15, 0.16, 0.17, 0.18, 0.19, 0.20, 0.21, 0.221, 0.222, 0.223, 0.224, 0.225, 0.226, 0.227, 0.228, 0.229, 0.230, 0.231, 0.232, 0.233, 0.234, 0.235, 0.236, 0.237, 0.238, 0.239, 0.240, 0.241, 0.250, 0.251, 0.252, 0.253, 0.254, 0.255, 0.256, 0.470, 0.470, 0.471, 0.471, 0.501, 0.501, 0.510, 0.510, 0.511, 0.511, 0.520, 0.520, 0.521, 0.521, 0.530, 0.530, 0.540, 0.540, 0.550, 0.550, 0.560, 0.560, 0.570, 0.570, 0.580, 0.580, 0.590, 0.590, 0.600, 0.600, 0.610, 0.610, 0.620, 0.620, 0.630, 0.630, 0.641, 0.641, 0.650, 0.650, 0.660, 0.660, 0.670, 0.670, 0.700, 0.700, 0.701, 0.701, 0.710, 0.710, 0.711, 0.711, 0.720, 0.720, 0.730, 0.730, 0.740, 0.740, 0.740, 0.750, 0.750, 0.750, 0.760, 0.760, 0.761, 0.761, 0.761, 0.770, 0.770, 0.770, 0.780, 0.780, 0.780, 0.781, 0.781, 0.781, 0.782, 0.782, 0.782, 0.790, 0.790, 0.790
There are incompatible versions in the resolved dependencies:
  mypy<0.790 (from -r requirements.in (line 1))
  mypy>=0.790 (from django-stubs==1.7.0->-r requirements.in (line 2))

3. added a specific version of django-stubs to requirements.in

mypy < 0.790
django-stubs == 1.6.0

4. pip-compile

#
# This file is autogenerated by pip-compile
# To update, run:
#
#    pip-compile
#
asgiref==3.3.1            # via django
django-stubs==1.6.0       # via -r requirements.in
django==3.1.4             # via django-stubs
mypy-extensions==0.4.3    # via mypy
mypy==0.782               # via -r requirements.in, django-stubs
pytz==2020.4              # via django
sqlparse==0.4.1           # via django
typed-ast==1.4.1          # via mypy
typing-extensions==3.7.4.3  # via django-stubs, mypy

5. removed a specific version of django-stubs from requirements.in

mypy < 0.790
django-stubs

6. pip-compile

#
# This file is autogenerated by pip-compile
# To update, run:
#
#    pip-compile
#
asgiref==3.3.1            # via django
django-stubs==1.6.0       # via -r requirements.in
django==3.1.4             # via django-stubs
mypy-extensions==0.4.3    # via mypy
mypy==0.782               # via -r requirements.in, django-stubs
pytz==2020.4              # via django
sqlparse==0.4.1           # via django
typed-ast==1.4.1          # via mypy
typing-extensions==3.7.4.3  # via django-stubs, mypy

Why pip-compile in the 2nd step can't resolve there is a solution I manually added in the 3rd step?
Why does the 6th step depend on an existing requirements.txt created in the 4th step?

[webknjaz]

Does a regular pip install solve the deps properly?

[jancespivo]

Does a regular pip install solve the deps properly?

Yes:
pip install "mypy<0.790" django-stubs --no-cache-dir

Collecting django-stubs
  Downloading django_stubs-1.7.0-py3-none-any.whl (381 kB)
     |████████████████████████████████| 381 kB 1.4 MB/s 
  Downloading django_stubs-1.6.0-py3-none-any.whl (303 kB)
     |████████████████████████████████| 303 kB 27.1 MB/s 
Collecting mypy<0.790
  Downloading mypy-0.782-cp38-cp38-manylinux1_x86_64.whl (21.7 MB)
     |████████████████████████████████| 21.7 MB 15.9 MB/s 
Collecting django
  Downloading Django-3.1.4-py3-none-any.whl (7.8 MB)
     |████████████████████████████████| 7.8 MB 33.3 MB/s 
Collecting asgiref<4,>=3.2.10
  Downloading asgiref-3.3.1-py3-none-any.whl (19 kB)
Collecting mypy-extensions<0.5.0,>=0.4.3
  Downloading mypy_extensions-0.4.3-py2.py3-none-any.whl (4.5 kB)
Collecting pytz
  Downloading pytz-2020.4-py2.py3-none-any.whl (509 kB)
     |████████████████████████████████| 509 kB 18.4 MB/s 
Collecting sqlparse>=0.2.2
  Downloading sqlparse-0.4.1-py3-none-any.whl (42 kB)
     |████████████████████████████████| 42 kB 9.6 MB/s 
Collecting typed-ast<1.5.0,>=1.4.0
  Downloading typed_ast-1.4.1-cp38-cp38-manylinux1_x86_64.whl (768 kB)
     |████████████████████████████████| 768 kB 17.4 MB/s 
Collecting typing-extensions
  Downloading typing_extensions-3.7.4.3-py3-none-any.whl (22 kB)
Installing collected packages: typing-extensions, typed-ast, sqlparse, pytz, mypy-extensions, asgiref, mypy, django, django-stubs
Successfully installed asgiref-3.3.1 django-3.1.4 django-stubs-1.6.0 mypy-0.782 mypy-extensions-0.4.3 pytz-2020.4 sqlparse-0.4.1 typed-ast-1.4.1 typing-extensions-3.7.4.3

[atugushev]

That's because pip-tools doesn't have a proper resolver and always tries to install the latest versions. See how pip installs with legacy resolver:

$ pip install --use-deprecated=legacy-resolver "mypy<0.790" django-stubs
Collecting mypy<0.790
  Downloading mypy-0.782-cp38-cp38-macosx_10_9_x86_64.whl (8.8 MB)
     |████████████████████████████████| 8.8 MB 5.4 MB/s
Collecting django-stubs
  Downloading django_stubs-1.7.0-py3-none-any.whl (381 kB)
     |████████████████████████████████| 381 kB 160 kB/s
Collecting typing-extensions>=3.7.4
  Using cached typing_extensions-3.7.4.3-py3-none-any.whl (22 kB)
Collecting mypy-extensions<0.5.0,>=0.4.3
  Using cached mypy_extensions-0.4.3-py2.py3-none-any.whl (4.5 kB)
Collecting typed-ast<1.5.0,>=1.4.0
  Using cached typed_ast-1.4.1-cp38-cp38-macosx_10_15_x86_64.whl (224 kB)
Collecting django
  Using cached Django-3.1.4-py3-none-any.whl (7.8 MB)
Collecting asgiref<4,>=3.2.10
  Using cached asgiref-3.3.1-py3-none-any.whl (19 kB)
Collecting pytz
  Using cached pytz-2020.4-py2.py3-none-any.whl (509 kB)
Collecting sqlparse>=0.2.2
  Using cached sqlparse-0.4.1-py3-none-any.whl (42 kB)
Installing collected packages: typing-extensions, mypy-extensions, typed-ast, mypy, asgiref, pytz, sqlparse, django, django-stubs
ERROR: pip's legacy dependency resolver does not consider dependency conflicts when selecting packages. This behaviour is the source of the following dependency conflicts.
django-stubs 1.7.0 requires mypy>=0.790, but you'll have mypy 0.782 which is incompatible.
Successfully installed asgiref-3.3.1 django-3.1.4 django-stubs-1.7.0 mypy-0.782 mypy-extensions-0.4.3 pytz-2020.4 sqlparse-0.4.1 typed-ast-1.4.1 typing-extensions-3.7.4.3

Note:

django-stubs 1.7.0 
ERROR: pip's legacy dependency resolver does not consider dependency conflicts when selecting packages. This behaviour is the source of the following dependency conflicts.

Also:

$ pip check
django-stubs 1.7.0 has requirement mypy>=0.790, but you have mypy 0.782.

[atugushev]

pip-tools doesn't support yet new pip's resolver. See also #1190.

[atugushev]

Why does the 6th step depend on an existing requirements.txt created in the 4th step?

Because you manually helped pip-tools to resolve dependency conflict by pinning django-stubs to 1.6.0.

[jancespivo]

Why does the 6th step depend on an existing requirements.txt created in the 4th step?

Because you manually helped pip-tools to resolve dependency conflict by pinning django-stubs to 1.6.0.

Thanks, but I unpinned it in the 5th step. So I don't understand 🤷

[atugushev]

Thanks, but I unpinned it in the 5th step. So I don't understand 🤷

Yes, but it's still pinned to 1.6.0 in requirements.txt, pip-tools preserve pinned versions while pip-compile. Try pip-compile --upgrade and it'll raise a dependency conflict error again.

[jancespivo]

@atugushev Thank you for the clarification!

[atugushev]

Thanks for the issue! Don't hesitate to reach out if you have any other issues.