Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

typing_extensions >= 4.6.0 causes pip unit test failure #60687

Open
elfringham opened this issue May 24, 2023 · 11 comments
Open

typing_extensions >= 4.6.0 causes pip unit test failure #60687

elfringham opened this issue May 24, 2023 · 11 comments
Assignees
Labels
stat:awaiting tensorflower Status - Awaiting response from tensorflower type:bug Bug type:support Support issues

Comments

@elfringham
Copy link
Contributor

elfringham commented May 24, 2023

Click to expand!

Issue Type

Bug

Have you reproduced the bug with TF nightly?

Yes

Source

source

Tensorflow Version

git HEAD

Custom Code

No

OS Platform and Distribution

Ubuntu 20.04

Mobile device

n/a

Python version

3.10

Bazel version

5.3.0

GCC/Compiler version

10.2.1

CUDA/cuDNN version

n/a

GPU model and memory

n/a

Current Behaviour?

//bazel_pip/tensorflow/python/trackable:data_structures_test will fail with typing_extensions >= 4.6.0 installed when run as a pip test against an installed TensorFlow wheel.

Standalone code to reproduce the issue

bazel test --build_tests_only --cache_test_results=no --config=mkl_aarch64_threadpool --copt=-flax-vector-conversions --jobs=75 --test_env=TF_ENABLE_ONEDNN_OPTS=1 --test_env=TF2_BEHAVIOR=1 --define=no_tensorflow_py_deps=true --test_lang_filters=py --test_size_filters=small,medium --test_output=errors --verbose_failures=true //bazel_pip/tensorflow/python/trackable:data_structures_test

Relevant log output

======================================================================
ERROR: testFunctionCaching (__main__.MappingTests)
MappingTests.testFunctionCaching
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/root/.cache/bazel/_bazel_root/eab0d61a99b6696edb3d2aff87b585e8/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/bazel_pip/tensorflow/python/trackable/data_structures_test.runfiles/org_tensorflow/bazel_pip/tensorflow/python/trackable/data_structures_test.py", line 507, in testFunctionCaching
    second_trace = f.get_concrete_function(
  File "/workspace/pip_test/venv_clean/lib/python3.8/site-packages/tensorflow/python/eager/polymorphic_function/polymorphic_function.py", line 1198, in get_concrete_function
    concrete = self._get_concrete_function_garbage_collected(*args, **kwargs)
  File "/workspace/pip_test/venv_clean/lib/python3.8/site-packages/tensorflow/python/eager/polymorphic_function/polymorphic_function.py", line 1189, in _get_concrete_function_garbage_collected
    concrete = self._variable_creation_fn.get_concrete_function(
  File "/workspace/pip_test/venv_clean/lib/python3.8/site-packages/tensorflow/python/eager/polymorphic_function/tracing_compiler.py", line 197, in get_concrete_function
    concrete_function, _ = self._maybe_define_concrete_function(args, kwargs)
  File "/workspace/pip_test/venv_clean/lib/python3.8/site-packages/tensorflow/python/eager/polymorphic_function/tracing_compiler.py", line 172, in _maybe_define_concrete_function
    return self._maybe_define_function(args, kwargs)
  File "/workspace/pip_test/venv_clean/lib/python3.8/site-packages/tensorflow/python/eager/polymorphic_function/tracing_compiler.py", line 294, in _maybe_define_function
    function_type_utils.make_canonicalized_monomorphic_type(
  File "/workspace/pip_test/venv_clean/lib/python3.8/site-packages/tensorflow/python/eager/polymorphic_function/function_type_utils.py", line 378, in make_canonicalized_monomorphic_type
    function_type_lib.canonicalize_to_monomorphic(
  File "/workspace/pip_test/venv_clean/lib/python3.8/site-packages/tensorflow/core/function/polymorphism/function_type.py", line 481, in canonicalize_to_monomorphic
    _make_validated_mono_param(name, arg, poly_parameter.kind,
  File "/workspace/pip_test/venv_clean/lib/python3.8/site-packages/tensorflow/core/function/polymorphism/function_type.py", line 421, in _make_validated_mono_param
    mono_type = trace_type.from_value(value, type_context)
  File "/workspace/pip_test/venv_clean/lib/python3.8/site-packages/tensorflow/core/function/trace_type/trace_type_builder.py", line 142, in from_value
    elif isinstance(value, trace.SupportsTracingProtocol):
  File "/workspace/pip_test/venv_clean/lib/python3.8/site-packages/typing_extensions.py", line 605, in __instancecheck__
    val = inspect.getattr_static(instance, attr)
  File "/usr/lib/python3.8/inspect.py", line 1596, in getattr_static
    instance_result = _check_instance(obj, attr)
  File "/usr/lib/python3.8/inspect.py", line 1543, in _check_instance
    instance_dict = object.__getattribute__(obj, "__dict__")
TypeError: this __dict__ descriptor does not support '_DictWrapper' objects

======================================================================
ERROR: testFunctionCaching (__main__.TupleTests)
TupleTests.testFunctionCaching
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/root/.cache/bazel/_bazel_root/eab0d61a99b6696edb3d2aff87b585e8/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/bazel_pip/tensorflow/python/trackable/data_structures_test.runfiles/org_tensorflow/bazel_pip/tensorflow/python/trackable/data_structures_test.py", line 716, in testFunctionCaching
    second_trace = f.get_concrete_function(
  File "/workspace/pip_test/venv_clean/lib/python3.8/site-packages/tensorflow/python/eager/polymorphic_function/polymorphic_function.py", line 1198, in get_concrete_function
    concrete = self._get_concrete_function_garbage_collected(*args, **kwargs)
  File "/workspace/pip_test/venv_clean/lib/python3.8/site-packages/tensorflow/python/eager/polymorphic_function/polymorphic_function.py", line 1189, in _get_concrete_function_garbage_collected
    concrete = self._variable_creation_fn.get_concrete_function(
  File "/workspace/pip_test/venv_clean/lib/python3.8/site-packages/tensorflow/python/eager/polymorphic_function/tracing_compiler.py", line 197, in get_concrete_function
    concrete_function, _ = self._maybe_define_concrete_function(args, kwargs)
  File "/workspace/pip_test/venv_clean/lib/python3.8/site-packages/tensorflow/python/eager/polymorphic_function/tracing_compiler.py", line 172, in _maybe_define_concrete_function
    return self._maybe_define_function(args, kwargs)
  File "/workspace/pip_test/venv_clean/lib/python3.8/site-packages/tensorflow/python/eager/polymorphic_function/tracing_compiler.py", line 294, in _maybe_define_function
    function_type_utils.make_canonicalized_monomorphic_type(
  File "/workspace/pip_test/venv_clean/lib/python3.8/site-packages/tensorflow/python/eager/polymorphic_function/function_type_utils.py", line 378, in make_canonicalized_monomorphic_type
    function_type_lib.canonicalize_to_monomorphic(
  File "/workspace/pip_test/venv_clean/lib/python3.8/site-packages/tensorflow/core/function/polymorphism/function_type.py", line 481, in canonicalize_to_monomorphic
    _make_validated_mono_param(name, arg, poly_parameter.kind,
  File "/workspace/pip_test/venv_clean/lib/python3.8/site-packages/tensorflow/core/function/polymorphism/function_type.py", line 421, in _make_validated_mono_param
    mono_type = trace_type.from_value(value, type_context)
  File "/workspace/pip_test/venv_clean/lib/python3.8/site-packages/tensorflow/core/function/trace_type/trace_type_builder.py", line 142, in from_value
    elif isinstance(value, trace.SupportsTracingProtocol):
  File "/workspace/pip_test/venv_clean/lib/python3.8/site-packages/typing_extensions.py", line 605, in __instancecheck__
    val = inspect.getattr_static(instance, attr)
  File "/usr/lib/python3.8/inspect.py", line 1596, in getattr_static
    instance_result = _check_instance(obj, attr)
  File "/usr/lib/python3.8/inspect.py", line 1543, in _check_instance
    instance_dict = object.__getattribute__(obj, "__dict__")
TypeError: this __dict__ descriptor does not support '_TupleWrapper' objects

----------------------------------------------------------------------
Ran 74 tests in 1.021s

FAILED (errors=2, skipped=4)
================================================================================
@tilakrayal
Copy link
Contributor

@elfringham,
The related PR #60688 was merged and as requested the 'typing_extensions >= 3.6.6', are changed to
'typing_extensions>=3.6.6,<4.6.0' with the mentioned merged PR.

https://github.com/tensorflow/tensorflow/blob/master/tensorflow/tools/pip_package/setup.py#L100.

Could you please confirm whether changes are reflected as mentioned. Thank you!

@tilakrayal tilakrayal added the stat:awaiting response Status - Awaiting response from author label May 29, 2023
@hmc-cs-mdrissi
Copy link

hmc-cs-mdrissi commented May 31, 2023

The rough cause of error is here. The current getattribute for dictwrapper can return TypeError. When an attribute is not found it's instead expected to return AttributeError. Finding place that produces TypeError and replacing it with AttributeError should fix typing extensions incompatibility. typing 4.6 uses inspect.getattr_static which expects that failed attribute access using getattribute only triggers attribute error.

This particular error will also be an error if you try to run tensorflow on python 3.12. The typing extensions change was motivated from here.

edit: Minimal version of this error is,

from tensorflow.python.trackable.data_structures import _DictWrapper

t = _DictWrapper()
object.__getattribute__(t, "__dict__")

The last line raises TypeError. On other hand t.__dict__ properly raises AttributeError. DictWrapper's custom getattribute is rough area that needs bug fix.

The error message comes from here. I've cross posted in cpython side as unsure whether this tensorflow bug or cpython bug.

@stewartmiles
Copy link
Contributor

FYI folks @AlexWaygood over on typing-extensions thinks this is an issue with wrapt, from python/typing_extensions#216 (comment) here's @AlexWaygood's message:

Looks to me like it's probably an issue with wrapt rather than typing_extensions or TensorFlow, actually. We discussed this a bit in python/cpython#105134.

Wrapt raises TypeError if you do object.getattribute(<some_wrapt_ObjectProxy_instance, 'dict'). I know of no way to construct a class using pure Python that has this behaviour (wrapt achieves this behaviour by using a C extension), and it breaks some fundamental assumptions that the stdlib function inspect.getattr_static makes. That, in turn, breaks typing_extensions.

@github-actions
Copy link

This issue is stale because it has been open for 7 days with no activity. It will be closed if no further activity occurs. Thank you.

@github-actions github-actions bot added the stale This label marks the issue/pr stale - to be closed automatically if no activity label Jun 10, 2023
@MichaelHudgins MichaelHudgins removed the stale This label marks the issue/pr stale - to be closed automatically if no activity label Jun 12, 2023
@github-actions
Copy link

This issue is stale because it has been open for 7 days with no activity. It will be closed if no further activity occurs. Thank you.

@github-actions github-actions bot added the stale This label marks the issue/pr stale - to be closed automatically if no activity label Jun 20, 2023
@tilakrayal tilakrayal removed the stale This label marks the issue/pr stale - to be closed automatically if no activity label Jun 20, 2023
@github-actions
Copy link

This issue is stale because it has been open for 7 days with no activity. It will be closed if no further activity occurs. Thank you.

@github-actions github-actions bot added the stale This label marks the issue/pr stale - to be closed automatically if no activity label Jun 28, 2023
@tilakrayal tilakrayal removed stat:awaiting response Status - Awaiting response from author stale This label marks the issue/pr stale - to be closed automatically if no activity labels Jun 28, 2023
@tilakrayal tilakrayal added the stale This label marks the issue/pr stale - to be closed automatically if no activity label Jul 14, 2023
@fergusdixon
Copy link

A consequence of this is that Tensorflow is not compatible with Pydantic V2 due to the restraints on typing_extensions
Is there any roadmap to loosen this?

@hmc-cs-mdrissi
Copy link

hmc-cs-mdrissi commented Jul 24, 2023

The root issue comes more from wrapt then tensorflow and is this one. I think downgrading wrapt version should be enough to work around this until that issue is closed. Specifically I think only wrapt 1.15 (or maybe 1.14+) causes an issue while tensorflow requirement is >1.11. Add an upper bound and we should be fine.

This is not ideal long term fix and that is still resolve wrapt issue (or remove wrapt dependency).

Second solution is set WRAPT_DISABLE_EXTENSIONS=true. wrapt has both c extension and pure python implementation. The former causes the error, while latter works fine.

elfringham added a commit to elfringham/tensorflow that referenced this issue Jul 25, 2023
A change in the behaviour of wrapt ends up in a unit test failure
for TensorFlow.
This can be removed once mitigation in TensorFlow is in place.
tensorflow#60687
and
GrahamDumpleton/wrapt#231
This is an alternative and less intrusive fix to
tensorflow#60688 which was
inadvertently reverted by a mis-merge in another commit.
@tommyzieba
Copy link

tommyzieba commented Jul 28, 2023

Is anything being done to loosen the upper bound on typing-extensions here? This continues to cause issues on macOS.

I have a Pipfile which includes pydantic, pydantic-settings, and tensorflow as packages. Locking packages with a pre-release flag using pipenv works fine with pre-release tag 2.13.0-rc1. However I cannot include tensorflow in this project without using the pre-release. Here is the output from pipenv when manually installing the latest pydantic-settings, which indicates that tensorflow-macos is the problem which seems to be related to the upperbound of typing-extensions versions here. Everything installs just fine on a Windows OS with the most recent versions of packages and without a pre-release flag.

% pipenv run pip show pydantic-settings 

Name: pydantic-settings
Version: 1.99
Summary: Settings management using Pydantic
Home-page: 
Author: 
Author-email: Samuel Colvin <[email protected]>, Eric Jolibois <[email protected]>, Hasan Ramezani <[email protected]>
License: The MIT License (MIT)

Copyright (c) 2022 Samuel Colvin and other contributors

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
Location: /Users/user/.local/share/virtualenvs/app-B-pslT81/lib/python3.11/site-packages
Requires: pydantic, python-dotenv
Required-by: 
% pipenv run pip install -v pydantic-settings==2.0.2

Using pip 23.2.1 from /Users/user/.local/share/virtualenvs/app-B-pslT81/lib/python3.11/site-packages/pip (python 3.11)
Collecting pydantic-settings==2.0.2
  Obtaining dependency information for pydantic-settings==2.0.2 from https://files.pythonhosted.org/packages/9c/3e/3311eeab406db6116e7f2d4ed9b05c2811282143efa55feb12cc513a8b84/pydantic_settings-2.0.2-py3-none-any.whl.metadata
  Downloading pydantic_settings-2.0.2-py3-none-any.whl.metadata (2.9 kB)
Collecting pydantic>=2.0.1 (from pydantic-settings==2.0.2)
  Obtaining dependency information for pydantic>=2.0.1 from https://files.pythonhosted.org/packages/87/80/52770e747e4bee5012e60b2684db36c8fdf010f8dadb4ded0efec808b07d/pydantic-2.1.1-py3-none-any.whl.metadata
  Downloading pydantic-2.1.1-py3-none-any.whl.metadata (136 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 136.5/136.5 kB 3.1 MB/s eta 0:00:00
Requirement already satisfied: python-dotenv>=0.21.0 in /Users/user/.local/share/virtualenvs/app-B-pslT81/lib/python3.11/site-packages (from pydantic-settings==2.0.2) (1.0.0)
Requirement already satisfied: annotated-types>=0.4.0 in /Users/user/.local/share/virtualenvs/app-B-pslT81/lib/python3.11/site-packages (from pydantic>=2.0.1->pydantic-settings==2.0.2) (0.5.0)
Collecting pydantic-core==2.4.0 (from pydantic>=2.0.1->pydantic-settings==2.0.2)
  Obtaining dependency information for pydantic-core==2.4.0 from https://files.pythonhosted.org/packages/32/68/324013cf826ad1f09bce46b01259c2d88607d137e297d213fe7ef225a91f/pydantic_core-2.4.0-cp311-cp311-macosx_11_0_arm64.whl.metadata
  Downloading pydantic_core-2.4.0-cp311-cp311-macosx_11_0_arm64.whl.metadata (6.5 kB)
Collecting typing-extensions>=4.6.1 (from pydantic>=2.0.1->pydantic-settings==2.0.2)
  Obtaining dependency information for typing-extensions>=4.6.1 from https://files.pythonhosted.org/packages/ec/6b/63cc3df74987c36fe26157ee12e09e8f9db4de771e0f3404263117e75b95/typing_extensions-4.7.1-py3-none-any.whl.metadata
  Downloading typing_extensions-4.7.1-py3-none-any.whl.metadata (3.1 kB)
Downloading pydantic_settings-2.0.2-py3-none-any.whl (11 kB)
Downloading pydantic-2.1.1-py3-none-any.whl (370 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 370.9/370.9 kB 11.4 MB/s eta 0:00:00
Downloading pydantic_core-2.4.0-cp311-cp311-macosx_11_0_arm64.whl (1.6 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.6/1.6 MB 31.1 MB/s eta 0:00:00
Downloading typing_extensions-4.7.1-py3-none-any.whl (33 kB)
Installing collected packages: typing-extensions, pydantic-core, pydantic, pydantic-settings
  Attempting uninstall: typing-extensions
    Found existing installation: typing_extensions 4.5.0
    Uninstalling typing_extensions-4.5.0:
      Removing file or directory /Users/user/.local/share/virtualenvs/app-B-pslT81/lib/python3.11/site-packages/__pycache__/typing_extensions.cpython-311.pyc
      Removing file or directory /Users/user/.local/share/virtualenvs/app-B-pslT81/lib/python3.11/site-packages/typing_extensions-4.5.0.dist-info/
      Removing file or directory /Users/user/.local/share/virtualenvs/app-B-pslT81/lib/python3.11/site-packages/typing_extensions.py
      Successfully uninstalled typing_extensions-4.5.0
  Attempting uninstall: pydantic-core
    Found existing installation: pydantic_core 0.25.0
    Uninstalling pydantic_core-0.25.0:
      Removing file or directory /Users/user/.local/share/virtualenvs/app-B-pslT81/lib/python3.11/site-packages/pydantic_core-0.25.0.dist-info/
      Removing file or directory /Users/user/.local/share/virtualenvs/app-B-pslT81/lib/python3.11/site-packages/pydantic_core/
      Successfully uninstalled pydantic_core-0.25.0
  Attempting uninstall: pydantic
    Found existing installation: pydantic 2.0a3
    Uninstalling pydantic-2.0a3:
      Removing file or directory /Users/user/.local/share/virtualenvs/app-B-pslT81/lib/python3.11/site-packages/pydantic-2.0a3.dist-info/
      Removing file or directory /Users/user/.local/share/virtualenvs/app-B-pslT81/lib/python3.11/site-packages/pydantic/
      Successfully uninstalled pydantic-2.0a3
  Attempting uninstall: pydantic-settings
    Found existing installation: pydantic-settings 1.99
    Uninstalling pydantic-settings-1.99:
      Removing file or directory /Users/user/.local/share/virtualenvs/app-B-pslT81/lib/python3.11/site-packages/pydantic_settings-1.99.dist-info/
      Removing file or directory /Users/user/.local/share/virtualenvs/app-B-pslT81/lib/python3.11/site-packages/pydantic_settings/
      Successfully uninstalled pydantic-settings-1.99
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
tensorflow-macos 2.13.0 requires typing-extensions<4.6.0,>=3.6.6, but you have typing-extensions 4.7.1 which is incompatible.
Successfully installed pydantic-2.1.1 pydantic-core-2.4.0 pydantic-settings-2.0.2 typing-extensions-4.7.1

@elfringham
Copy link
Contributor Author

@tommyzieba You can see from #61387 that there is no longer an upper limit on typing-extensions in git HEAD and that this will be picked up by the next release of TensorFlow which will be 2.14.0.

@google-ml-butler google-ml-butler bot removed the stale This label marks the issue/pr stale - to be closed automatically if no activity label Jul 28, 2023
@mattdangerw
Copy link
Member

mattdangerw commented Sep 20, 2023

Is this issue still active? It looks like it is about to cause an issue for keras-core with the 2.14 release, probably other things affected as well.

https://colab.research.google.com/gist/mattdangerw/019d1ebcec746d6f7424248911cf16ae/tf-2-14-bug.ipynb

More directly, with keras out of the picture, I think the minimal repro below will cause failures in the tensorflow 2.14 docker images (which have a more recent typing_extensions).

from tensorflow.python.trackable.data_structures import _DictWrapper

t = _DictWrapper()
object.__getattribute__(t, "__dict__")

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
stat:awaiting tensorflower Status - Awaiting response from tensorflower type:bug Bug type:support Support issues
Projects
None yet
Development

No branches or pull requests

9 participants