Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug]: Could not find a version that matches protobuf #23982

Closed
cypreess opened this issue Nov 4, 2022 · 17 comments
Closed

[Bug]: Could not find a version that matches protobuf #23982

cypreess opened this issue Nov 4, 2022 · 17 comments
Labels
gcp io P2 python

Comments

@cypreess
Copy link

cypreess commented Nov 4, 2022

What happened?

Cannot install latest apache-beam[gcp]==2.42.0 because there is a conflict of protobuf version dependencies.

Could not find a version that matches protobuf!=3.20.0,!=3.20.1,!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<4,<4.0.0dev,<5.0.0dev,>=3.12.0,>=3.12.2,>=3.15.0,>=3.19.0,>=3.19.5,>=4.21.6 (from proto-plus==1.22.1->apache-beam[gcp]==2.42.0->uni-bi-export (setup.py))
Tried: 2.0.3, 2.3.0, 2.4.1, 2.5.0, 2.6.0, 2.6.1, 3.0.0, 3.0.0, 3.1.0, 3.1.0.post1, 3.1.0.post1, 3.2.0, 3.2.0, 3.3.0, 3.4.0, 3.4.0, 3.5.0.post1, 3.5.0.post1, 3.5.1, 3.5.1, 3.5.2, 3.5.2, 3.5.2.post1, 3.5.2.post1, 3.6.0, 3.6.0, 3.6.1, 3.6.1, 3.7.0, 3.7.0, 3.7.1, 3.7.1, 3.8.0, 3.8.0, 3.9.0, 3.9.0, 3.9.1, 3.9.1, 3.9.2, 3.9.2, 3.10.0, 3.10.0, 3.11.0, 3.11.0, 3.11.1, 3.11.1, 3.11.2, 3.11.2, 3.11.3, 3.11.3, 3.12.2, 3.12.2, 3.12.4, 3.12.4, 3.13.0, 3.13.0, 3.14.0, 3.14.0, 3.15.0, 3.15.0, 3.15.1, 3.15.1, 3.15.2, 3.15.2, 3.15.3, 3.15.3, 3.15.4, 3.15.4, 3.15.5, 3.15.5, 3.15.6, 3.15.6, 3.15.7, 3.15.7, 3.15.8, 3.15.8, 3.16.0, 3.16.0, 3.17.0, 3.17.0, 3.17.0, 3.17.1, 3.17.1, 3.17.1, 3.17.2, 3.17.2, 3.17.2, 3.17.3, 3.17.3, 3.17.3, 3.18.0, 3.18.0, 3.18.0, 3.18.1, 3.18.1, 3.18.1, 3.18.3, 3.18.3, 3.18.3, 3.19.0, 3.19.0, 3.19.0, 3.19.1, 3.19.1, 3.19.1, 3.19.2, 3.19.2, 3.19.2, 3.19.3, 3.19.3, 3.19.3, 3.19.4, 3.19.4, 3.19.4, 3.19.5, 3.19.5, 3.19.5, 3.19.6, 3.19.6, 3.19.6, 3.20.0, 3.20.0, 3.20.0, 3.20.1, 3.20.1, 3.20.1, 3.20.2, 3.20.2, 3.20.2, 3.20.3, 3.20.3, 3.20.3, 4.21.0, 4.21.0, 4.21.0, 4.21.0, 4.21.1, 4.21.1, 4.21.1, 4.21.1, 4.21.2, 4.21.2, 4.21.2, 4.21.2, 4.21.3, 4.21.3, 4.21.3, 4.21.3, 4.21.4, 4.21.4, 4.21.4, 4.21.4, 4.21.5, 4.21.5, 4.21.5, 4.21.5, 4.21.6, 4.21.6, 4.21.6, 4.21.6, 4.21.7, 4.21.7, 4.21.7, 4.21.7, 4.21.8, 4.21.8, 4.21.8, 4.21.8, 4.21.9, 4.21.9, 4.21.9, 4.21.9
Skipped pre-versions: 2.0.0b0, 3.0.0a2, 3.0.0a3, 3.0.0b1, 3.0.0b1.post1, 3.0.0b1.post2, 3.0.0b2, 3.0.0b2, 3.0.0b2.post1, 3.0.0b2.post1, 3.0.0b2.post2, 3.0.0b2.post2, 3.0.0b3, 3.0.0b4, 3.0.0b4, 3.2.0rc1, 3.2.0rc1, 3.2.0rc1.post1, 3.2.0rc1.post1, 3.2.0rc2, 3.2.0rc2, 3.7.0rc2, 3.7.0rc2, 3.7.0rc3, 3.7.0rc3, 3.8.0rc1, 3.8.0rc1, 3.9.0rc1, 3.9.0rc1, 3.10.0rc1, 3.10.0rc1, 3.11.0rc1, 3.11.0rc1, 3.11.0rc2, 3.11.0rc2, 3.13.0rc3, 3.13.0rc3, 3.14.0rc1, 3.14.0rc1, 3.14.0rc2, 3.14.0rc2, 3.14.0rc3, 3.14.0rc3, 3.15.0rc1, 3.15.0rc1, 3.15.0rc2, 3.15.0rc2, 3.16.0rc1, 3.16.0rc1, 3.16.0rc2, 3.16.0rc2, 3.17.0rc1, 3.17.0rc1, 3.17.0rc1, 3.17.0rc2, 3.17.0rc2, 3.17.0rc2, 3.18.0rc1, 3.18.0rc1, 3.18.0rc1, 3.18.0rc2, 3.18.0rc2, 3.18.0rc2, 3.19.0rc1, 3.19.0rc1, 3.19.0rc1, 3.19.0rc2, 3.19.0rc2, 3.19.0rc2, 3.20.0rc1, 3.20.0rc1, 3.20.0rc1, 3.20.0rc2, 3.20.0rc2, 3.20.0rc2, 3.20.1rc1, 3.20.1rc1, 3.20.1rc1, 4.0.0rc1, 4.0.0rc1, 4.0.0rc2, 4.0.0rc2, 4.21.0rc1, 4.21.0rc1, 4.21.0rc1, 4.21.0rc1, 4.21.0rc2, 4.21.0rc2, 4.21.0rc2, 4.21.0rc2
There are incompatible versions in the resolved dependencies:
  protobuf!=3.20.0,!=3.20.1,!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.19.5 (from google-api-core[grpc,grpcgcp]==2.10.2->apache-beam[gcp]==2.42.0->uni-bi-export (setup.py))
  protobuf<5.0.0dev,>=3.19.0 (from google-cloud-recommendations-ai==0.7.1->apache-beam[gcp]==2.42.0->uni-bi-export (setup.py))
  protobuf!=3.20.0,!=3.20.1,!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.19.5 (from google-cloud-dlp==3.9.2->apache-beam[gcp]==2.42.0->uni-bi-export (setup.py))
  protobuf>=4.21.6 (from grpcio-status==1.50.0->google-cloud-pubsub==2.13.10->apache-beam[gcp]==2.42.0->uni-bi-export (setup.py))
  protobuf<4.0.0dev (from google-cloud-bigtable==1.7.2->apache-beam[gcp]==2.42.0->uni-bi-export (setup.py))
  protobuf<4.0.0dev,>=3.12.0 (from google-cloud-bigquery==2.34.4->apache-beam[gcp]==2.42.0->uni-bi-export (setup.py))
  protobuf<5.0.0dev,>=3.15.0 (from googleapis-common-protos==1.56.4->google-api-core==2.10.2->apache-beam[gcp]==2.42.0->uni-bi-export (setup.py))
  protobuf<4.0.0dev (from google-cloud-datastore==1.15.5->apache-beam[gcp]==2.42.0->uni-bi-export (setup.py))
  protobuf<4.0.0dev,>=3.19.0 (from google-cloud-bigquery-storage==2.13.2->apache-beam[gcp]==2.42.0->uni-bi-export (setup.py))
  protobuf<4.0.0dev (from google-cloud-language==1.3.2->apache-beam[gcp]==2.42.0->uni-bi-export (setup.py))
  protobuf<4.0.0dev (from google-cloud-videointelligence==1.16.3->apache-beam[gcp]==2.42.0->uni-bi-export (setup.py))
  protobuf!=3.20.0,!=3.20.1,!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.19.5 (from google-cloud-pubsub==2.13.10->apache-beam[gcp]==2.42.0->uni-bi-export (setup.py))
  protobuf<4.0.0dev (from google-cloud-spanner==1.19.3->apache-beam[gcp]==2.42.0->uni-bi-export (setup.py))
  protobuf<5.0.0dev,>=3.19.0 (from proto-plus==1.22.1->apache-beam[gcp]==2.42.0->uni-bi-export (setup.py))
  protobuf<4.0.0dev (from google-cloud-vision==1.0.2->apache-beam[gcp]==2.42.0->uni-bi-export (setup.py))
  protobuf<4,>=3.12.2 (from apache-beam[gcp]==2.42.0->uni-bi-export (setup.py))

Issue Priority

Priority: 1

Issue Component

Component: io-py-gcp
@kennknowles
Copy link
Member

This appears to be caused by other dependencies in your project, perhaps proto-plus. Or if there is an incompatibility in the GCP dependencies, we would certainly like to have a set of dependencies that works together. And of course it would be useful for Beam to be compatible with as many things as possible.

CC @tvalentyn

@tvalentyn
Copy link
Contributor

@cypreess I don't reproduce the issue in a clean virutal environment. Please add relevant details if you have and reopen the issue if still present.

@tvalentyn tvalentyn closed this as not planned Won't fix, can't repro, duplicate, stale Nov 17, 2022
@LaurenzReitsam
Copy link

LaurenzReitsam commented Dec 7, 2022 •
edited
Loading

I have a similar issue. Seems like apache-beam[gcp] dependencies (protobuf==3.*) are not up to date with other google packages (protobuf==4.*).

python -m venv .venv
source .venv/bin/activate
pip install pip-tools
echo -e "apache-beam[gcp]\ngoogle-cloud-secret-manager" > requirements.in
pip-compile

Out:


There are incompatible versions in the resolved dependencies:
  protobuf<5.0.0dev,>=3.19.0 (from google-cloud-recommendations-ai==0.7.1->apache-beam[gcp]==2.43.0->-r requirements.in (line 5))
  protobuf<4.0.0dev (from google-cloud-bigtable==1.7.3->apache-beam[gcp]==2.43.0->-r requirements.in (line 5))
  protobuf!=3.20.0,!=3.20.1,!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.19.5 (from google-cloud-dlp==3.9.2->apache-beam[gcp]==2.43.0->-r requirements.in (line 5))
  protobuf<4.0.0dev (from google-cloud-language==1.3.2->apache-beam[gcp]==2.43.0->-r requirements.in (line 5))
  protobuf<4,>3.12.2 (from apache-beam[gcp]==2.43.0->-r requirements.in (line 5))
  protobuf!=3.20.0,!=3.20.1,!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.19.5 (from google-api-core[grpc]==2.11.0->google-cloud-secret-manager==2.12.6->-r requirements.in (line 6))
  protobuf!=3.20.0,!=3.20.1,!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.19.5 (from google-cloud-secret-manager==2.12.6->-r requirements.in (line 6))
  protobuf!=3.20.0,!=3.20.1,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.19.5 (from googleapis-common-protos[grpc]==1.57.0->grpc-google-iam-v1==0.12.4->google-cloud-secret-manager==2.12.6->-r requirements.in (line 6))
  protobuf!=3.20.0,!=3.20.1,!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.19.5 (from google-cloud-spanner==3.24.0->apache-beam[gcp]==2.43.0->-r requirements.in (line 5))
  protobuf<4.0.0dev (from google-cloud-vision==1.0.2->apache-beam[gcp]==2.43.0->-r requirements.in (line 5))
  protobuf>=4.21.6 (from grpcio-status==1.51.1->google-cloud-pubsub==2.13.11->apache-beam[gcp]==2.43.0->-r requirements.in (line 5))
  protobuf!=3.20.0,!=3.20.1,!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.19.5 (from google-cloud-pubsub==2.13.11->apache-beam[gcp]==2.43.0->-r requirements.in (line 5))

Can we reopen this?

@tvalentyn
Copy link
Contributor

It seems like a bug in dependency resolution in piptools. But it may be also time to try to upgrade our protobuf depdendency. Opened #24569

@LaurenzReitsam
Copy link

@tvalentyn yes it is a problem with pip-tools.
Problem can be fixed with adding grpcio-status==1.48.2 as dependency... 🙄

@tvalentyn
Copy link
Contributor

We prefer not to add indirect dependencies, let alone pin versions thereof. Can this be worked around by preinstalling protobuf v3, or explicitly adding it in the requirements.in ?

@LaurenzReitsam
Copy link

grpcio-status is not an apache-beam dependency. This is just a workaround for me to fix issues with pipenv.

Just pinning protobuf<4.0.0 does not solve the issue.

Just tried the same procedure with poetry as an alternative dependency solver. Seems like the same error:

pip install poetry
mkdir foo && cd ./foo
poetry init
poetry add apache-beam[gcp] google-cloud-secret-manager
Because no versions of google-cloud-secret-manager match >2.12.6,<3.0.0
 and google-cloud-secret-manager (2.12.6) depends on protobuf (>=3.19.5,<3.20.0 || >3.20.0,<3.20.1 || >3.20.1,<4.21.0 || >4.21.0,<4.21.1 || >4.21.1,<4.21.2 || >4.21.2,<4.21.3 || >4.21.3,<4.21.4 || >4.21.4,<4.21.5 || >4.21.5,<5.0.0dev), google-cloud-secret-manager (>=2.12.6,<3.0.0) requires protobuf (>=3.19.5,<3.20.0 || >3.20.0,<3.20.1 || >3.20.1,<4.21.0 || >4.21.0,<4.21.1 || >4.21.1,<4.21.2 || >4.21.2,<4.21.3 || >4.21.3,<4.21.4 || >4.21.4,<4.21.5 || >4.21.5,<5.0.0dev).
And because apache-beam (2.43.0) depends on protobuf (>3.12.2,<3.19.5)
 and no versions of apache-beam match >2.43.0,<3.0.0, google-cloud-secret-manager (>=2.12.6,<3.0.0) is incompatible with apache-beam (>=2.43.0,<3.0.0).
So, because foo depends on both apache-beam (^2.43.0) and google-cloud-secret-manager (^2.12.6), version solving failed.

Adding protobuf<4.0.0 does not change result.

@tvalentyn
Copy link
Contributor

@AnandInguva do we still need this mac-os only restriction?

beam/sdks/python/setup.py

Line 189 in da100f9

protobuf_dependency = ['protobuf>3.12.2,<3.19.5']

IIRC we added b/c MacOS github action tests were failing. but if that not an issue anymore, it would be good to relax that.

@AnandInguva
Copy link
Contributor 

echo -e "apache-beam[gcp]\ngoogle-cloud-secret-manager" > requirements.in
pip-compile

It is due to pip-compile resolver.

You can try using

pip-compile --resolver backtracking with latest version of pip-tools. You can find more details at jazzband/pip-tools#1190

@AnandInguva
Copy link
Contributor 

#
# This file is autogenerated by pip-compile with Python 3.8
# by the following command:
#
#    pip-compile --resolver=backtracking
#
--extra-index-url https://pypi.ngc.nvidia.com
--trusted-host pypi.ngc.nvidia.com

apache-beam[gcp]==2.43.0
    # via -r requirements.in
cachetools==4.2.4
    # via
    #   apache-beam
    #   google-auth
certifi==2022.9.24
    # via requests
charset-normalizer==2.1.1
    # via requests
cloudpickle==2.2.0
    # via apache-beam
crcmod==1.7
    # via apache-beam
dill==0.3.1.1
    # via apache-beam
docopt==0.6.2
    # via hdfs
fastavro==1.7.0
    # via apache-beam
fasteners==0.18
    # via
    #   apache-beam
    #   google-apitools
google-api-core[grpc]==2.11.0
    # via
    #   google-cloud-bigquery
    #   google-cloud-bigquery-storage
    #   google-cloud-bigtable
    #   google-cloud-core
    #   google-cloud-datastore
    #   google-cloud-dlp
    #   google-cloud-language
    #   google-cloud-pubsub
    #   google-cloud-pubsublite
    #   google-cloud-recommendations-ai
    #   google-cloud-secret-manager
    #   google-cloud-spanner
    #   google-cloud-videointelligence
    #   google-cloud-vision
google-apitools==0.5.31
    # via apache-beam
google-auth==2.15.0
    # via
    #   apache-beam
    #   google-api-core
    #   google-auth-httplib2
    #   google-cloud-core
google-auth-httplib2==0.1.0
    # via apache-beam
google-cloud-bigquery==3.4.0
    # via apache-beam
google-cloud-bigquery-storage==2.13.2
    # via
    #   apache-beam
    #   google-cloud-bigquery
google-cloud-bigtable==1.7.3
    # via apache-beam
google-cloud-core==2.3.2
    # via
    #   apache-beam
    #   google-cloud-bigquery
    #   google-cloud-bigtable
    #   google-cloud-datastore
    #   google-cloud-spanner
google-cloud-datastore==1.15.5
    # via apache-beam
google-cloud-dlp==3.9.2
    # via apache-beam
google-cloud-language==1.3.2
    # via apache-beam
google-cloud-pubsub==2.13.11
    # via
    #   apache-beam
    #   google-cloud-pubsublite
google-cloud-pubsublite==1.6.0
    # via apache-beam
google-cloud-recommendations-ai==0.7.1
    # via apache-beam
google-cloud-secret-manager==2.12.6
    # via -r requirements.in
google-cloud-spanner==3.24.0
    # via apache-beam
google-cloud-videointelligence==1.16.3
    # via apache-beam
google-cloud-vision==1.0.2
    # via apache-beam
google-crc32c==1.5.0
    # via google-resumable-media
google-resumable-media==2.4.0
    # via google-cloud-bigquery
googleapis-common-protos[grpc]==1.57.0
    # via
    #   google-api-core
    #   grpc-google-iam-v1
    #   grpcio-status
grpc-google-iam-v1==0.12.4
    # via
    #   google-cloud-bigtable
    #   google-cloud-pubsub
    #   google-cloud-secret-manager
    #   google-cloud-spanner
grpcio==1.51.1
    # via
    #   apache-beam
    #   google-api-core
    #   google-cloud-bigquery
    #   google-cloud-pubsub
    #   google-cloud-pubsublite
    #   googleapis-common-protos
    #   grpc-google-iam-v1
    #   grpcio-status
grpcio-status==1.48.2
    # via
    #   google-api-core
    #   google-cloud-pubsub
    #   google-cloud-pubsublite
hdfs==2.7.0
    # via apache-beam
httplib2==0.20.4
    # via
    #   apache-beam
    #   google-apitools
    #   google-auth-httplib2
    #   oauth2client
idna==3.4
    # via requests
numpy==1.22.4
    # via
    #   apache-beam
    #   pyarrow
oauth2client==4.1.3
    # via google-apitools
objsize==0.5.2
    # via apache-beam
orjson==3.8.3
    # via apache-beam
overrides==6.5.0
    # via google-cloud-pubsublite
packaging==21.3
    # via
    #   google-cloud-bigquery
    #   google-cloud-spanner
proto-plus==1.22.1
    # via
    #   apache-beam
    #   google-cloud-bigquery
    #   google-cloud-bigquery-storage
    #   google-cloud-dlp
    #   google-cloud-pubsub
    #   google-cloud-recommendations-ai
    #   google-cloud-secret-manager
    #   google-cloud-spanner
protobuf==3.20.3
    # via
    #   apache-beam
    #   google-api-core
    #   google-cloud-bigquery
    #   google-cloud-bigquery-storage
    #   google-cloud-bigtable
    #   google-cloud-datastore
    #   google-cloud-dlp
    #   google-cloud-language
    #   google-cloud-pubsub
    #   google-cloud-recommendations-ai
    #   google-cloud-secret-manager
    #   google-cloud-spanner
    #   google-cloud-videointelligence
    #   google-cloud-vision
    #   googleapis-common-protos
    #   grpcio-status
    #   proto-plus
pyarrow==9.0.0
    # via
    #   apache-beam
    #   google-cloud-bigquery
pyasn1==0.4.8
    # via
    #   oauth2client
    #   pyasn1-modules
    #   rsa
pyasn1-modules==0.2.8
    # via
    #   google-auth
    #   oauth2client
pydot==1.4.2
    # via apache-beam
pymongo==3.13.0
    # via apache-beam
pyparsing==3.0.9
    # via
    #   httplib2
    #   packaging
    #   pydot
python-dateutil==2.8.2
    # via
    #   apache-beam
    #   google-cloud-bigquery
pytz==2022.6
    # via apache-beam
regex==2022.10.31
    # via apache-beam
requests==2.28.1
    # via
    #   apache-beam
    #   google-api-core
    #   google-cloud-bigquery
    #   hdfs
rsa==4.9
    # via
    #   google-auth
    #   oauth2client
six==1.16.0
    # via
    #   google-apitools
    #   google-auth
    #   google-auth-httplib2
    #   hdfs
    #   oauth2client
    #   python-dateutil
sqlparse==0.4.3
    # via google-cloud-spanner
typing-extensions==4.4.0
    # via apache-beam
urllib3==1.26.13
    # via requests
zstandard==0.19.0
    # via apache-beam

Output with pip-compile --resolver backtracking

@AnandInguva
Copy link
Contributor

@AnandInguva do we still need this mac-os only restriction?

beam/sdks/python/setup.py

Line 189 in da100f9

protobuf_dependency = ['protobuf>3.12.2,<3.19.5']

IIRC we added b/c MacOS github action tests were failing. but if that not an issue anymore, it would be good to relax that.

Yes, I will take a look at this.

@LaurenzReitsam
Copy link

@AnandInguva you are right pip-compile --resolver backtracking works for me on Linux + Python3.9.
Just tried this on Windows + Python3.10 until now what resulted in several errors...

...
numpy_linalg_lapack_lite:
  FOUND:
   language = c
  define_macros = [('HAVE_BLAS_ILP64', None), ('BLAS_SYMBOL_SUFFIX', '64_')]

C:\Users\laurenz.reitsam\.pyenv\pyenv-win\versions\3.10.4\lib\distutils\dist.py:274: UserWarning: Unknown distribution option: 'define_macros'
 warnings.warn(msg)
running dist_info
running build_src
build_src
building py_modules sources
creating build
creating build\src.win-amd64-3.10
creating build\src.win-amd64-3.10\numpy
creating build\src.win-amd64-3.10\numpy\distutils
building library "npymath" sources
error: Microsoft Visual C++ 14.0 is required. Get it with "Build Tools for Visual Studio": https://visualstudio.microsoft.com/downloads/
...

Concerning poetry I had the issue in both environments.

@jhoareau
Copy link

Hi @AnandInguva we are blocked because of the above Protobuf restriction ('protobuf>3.12.2,<3.19.5') applies to all environments (also non-MacOS) when using Poetry.
The reason is that the PyPI repository has this quite restrictive requirement in place: https://pypi.org/pypi/apache-beam/2.43.0/json instead of the looser "<4" for non-macOS environments.

It'd be great if you could either remove the condition from setup.py or fix the PyPI API requirements!

@AnandInguva
Copy link
Contributor

Yes, protobuf is tightly bounded and as of now, there are efforts going on for upgrading protobuf to 4.x.x version in Apache Beam.

We need to have a tight restriction in protobuf because protobuf is a build dependency of beam and minor changes on protobuf version needs to be validated so we couldn't have a loose bound.

@jhoareau
Copy link

I understand that. However, the setup.py file seems to have the loose bound (protobuf>3.12.2,<4), while the PyPI bound is tighter, like the macOS build one.
Since PyPI usecases usually do not include building but simply installing from wheel, I think it would make sense to relax the PyPI bound to be the same as the loose setup.py bound. Please correct me if I'm wrong!

@tvalentyn
Copy link
Contributor

intresting. is poetry using : the metadata from https://packaging.python.org/en/latest/specifications/core-metadata/#requires-dist-multiple-use / https://pypi.org/pypi/apache-beam/2.43.0/json as SOT?

it appears to be generated incorrectly. @AnandInguva I think we could fix it by adding a platform constraint (see https://peps.python.org/pep-0508/) instead of evaluating the platform dynamically in

beam/sdks/python/setup.py

Line 183 in f40057b

if sys.platform == 'darwin' and (

@jhoareau
Copy link

Indeed @tvalentyn @AnandInguva https://pypi.org/pypi/apache-beam/2.43.0/json is the source of truth for Poetry, and this source of truth seems to be overly restrictive. Platform constraints sound like a reasonable approach.

@jhoareau jhoareau mentioned this issue Feb 6, 2023
Closed
15 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
gcp io P2 python
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@kennknowles @cypreess @tvalentyn @jhoareau @AnandInguva @LaurenzReitsam