-
-
Notifications
You must be signed in to change notification settings - Fork 29.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
bpo-44863: In TypedDict allow inherit from Generic and preserve bases #27663
bpo-44863: In TypedDict allow inherit from Generic and preserve bases #27663
Conversation
Hello, and thanks for your contribution! I'm a bot set up to make sure that the project can legally accept this contribution by verifying everyone involved has signed the PSF contributor agreement (CLA). Recognized GitHub usernameWe couldn't find a bugs.python.org (b.p.o) account corresponding to the following GitHub usernames: This might be simply due to a missing "GitHub Name" entry in one's b.p.o account settings. This is necessary for legal reasons before we can look at this contribution. Please follow the steps outlined in the CPython devguide to rectify this issue. You can check yourself to see if the CLA has been received. Thanks again for the contribution, we look forward to reviewing it! |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Could you please add News
file using blurb
? You can find more information here.
I have one issue that I'm not sure if needs to be resolved.
I'm just mulling over if the Any opinions or ideas? Edit 1: In a following commit, I chose to override |
Lib/typing.py
Outdated
params = tuple(_type_check(p, msg) for p in params) | ||
nparams = len(cls.__dict__.get("__parameters__", ())) | ||
_check_generic(cls, params, nparams) | ||
return types.GenericAlias(cls, params) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think there are no need to define __getitem__
method as far as Generic
class has __class_getitem__
method.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I had two issues by not overriding the default:
- Dunder attributes are not proxied
class A(TypedDict, Generic[T]):
foo: T
assert(A[str].__required_keys__ == frozenset(['foo']) # Raises Attribute error
assert(A[str].__origin__.__required_keys__ == frozenset(['foo']) # Works
- May be a bug in the original code. But even without the changes in this PR, following is valid at runtime (digging in what I think happens is that it inherits a
__getitem__
fromdict
meta):
class G(TypedDict):
g: str
print(G[int]) # G is subscriptable although not a Generic
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@Fidget-Spinner What is your opinion about issue mentioned by Samodya?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
For 1st issue:
That happens because _BaseGenericAlias.__getattr__
intentionally doesn't proxy dunders to self.__origin__
https://github.com/python/cpython/blob/main/Lib/typing.py#L966. I don't know what's the correct behavior until I read the final version of the proposed PEP update/ PEP. Sorry.
For the 2nd behavior:
Since python 3.9, builtin generics support subscripting thanks to PEP 585. E.g. dict[str, int]
. Since subclassing TypedDict subclasses dict
, it also pulls dict.__class_getitem__
(and hence the type supports arbitrary subscripting). A quick way to verify this:
>>> type(G[int])
<class 'types.GenericAlias'>
>>> G.__mro__
(<class '__main__.G'>, <class 'dict'>, <class 'object'>)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Re 1: Dunder proxying behavior isn't specified in the PEP draft either :) I switched gears to add this feature through a bpo and implementation as I am not sure whether this change needed a PEP/PEP update (my reasoning in the ticket description). By any chance do you know a PEP that specifies the behavior of GenericAlias
for another type?
Re 2: I see. Thank you. But that does seem like something that shouldn't be allowed for a non generic Typeddict (Not a big deal though since it doesn't really do anything).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Considering following is the behavior for normal Generic class, I will revert the overriding of __getitem__
and not proxy the dunder attributes.
class A(Generic[T]):
a: T
print(A.__annotations__) # {'a': ~T}
print(A[str].__annotations__) # raises AttributeError
print(A[str].__origin__.__annotations__) # {'a': ~T}
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Awesome feature! I've been waiting for it for a long time 🎉
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I left some small notes below. I'll try to do a full and more thorough review eventually when the PEP solidifies. Thanks for moving this change forward!
Lib/typing.py
Outdated
params = tuple(_type_check(p, msg) for p in params) | ||
nparams = len(cls.__dict__.get("__parameters__", ())) | ||
_check_generic(cls, params, nparams) | ||
return types.GenericAlias(cls, params) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is there a reason why you're using the builtin types.GenericAlias
vs the typing typing._GenericAlias
version?
IMO, I'd prefer you use the typing version. It's much easier to subclass and customize. It also supports older Python versions if you're porting this to typing_extensions
. types.GenericAlias
only works for Python 3.9 onwards. (Also, I think there isn't a precedent for things in typing returning a types.GenericAlias
.)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
types.GenericAlias
was chosen just to circumvent the fact that typing._GenericAlias
doesn't proxy the dunder attributes (specifically __required_keys__
, __optional_keys__
etc.). Given your reasoning we can make the change to override __getattr__
of typing._GenericAlias
. That is assuming we want to proxy those attributes.
Lib/typing.py
Outdated
params = tuple(_type_check(p, msg) for p in params) | ||
nparams = len(cls.__dict__.get("__parameters__", ())) | ||
_check_generic(cls, params, nparams) | ||
return types.GenericAlias(cls, params) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
For 1st issue:
That happens because _BaseGenericAlias.__getattr__
intentionally doesn't proxy dunders to self.__origin__
https://github.com/python/cpython/blob/main/Lib/typing.py#L966. I don't know what's the correct behavior until I read the final version of the proposed PEP update/ PEP. Sorry.
For the 2nd behavior:
Since python 3.9, builtin generics support subscripting thanks to PEP 585. E.g. dict[str, int]
. Since subclassing TypedDict subclasses dict
, it also pulls dict.__class_getitem__
(and hence the type supports arbitrary subscripting). A quick way to verify this:
>>> type(G[int])
<class 'types.GenericAlias'>
>>> G.__mro__
(<class '__main__.G'>, <class 'dict'>, <class 'object'>)
Misc/NEWS.d/next/Library/2021-09-03-07-56-48.bpo-44863.udgz95.rst
Outdated
Show resolved
Hide resolved
@serhiy-storchaka Agreed that this PR shouldn't try to meddle with previously allowed behavior in non generic TypedDicts. Using the suggestion from your comment, the commit 80e9104 reverts forbidding of parametrization of non-generic TypedDict types. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We're missing one big thing: documentation. We should add a versionchanged: 3.11 note in https://docs.python.org/3.11/library/typing.html#typing.TypedDict saying that TypedDicts can now be generic.
@@ -1775,7 +1775,9 @@ def __init_subclass__(cls, *args, **kwargs): | |||
if '__orig_bases__' in cls.__dict__: | |||
error = Generic in cls.__orig_bases__ | |||
else: | |||
error = Generic in cls.__bases__ and cls.__name__ != 'Protocol' | |||
error = (Generic in cls.__bases__ and | |||
cls.__name__ != 'Protocol' and |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I see the existing code did this already, but checking the class name only seems wrong. Let me see if I can find a user-visible bug based on this.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It could be written as type(cls) != _ProtocolMeta
, but let left this to other issue.
@@ -1775,7 +1775,9 @@ def __init_subclass__(cls, *args, **kwargs): | |||
if '__orig_bases__' in cls.__dict__: | |||
error = Generic in cls.__orig_bases__ | |||
else: | |||
error = Generic in cls.__bases__ and cls.__name__ != 'Protocol' | |||
error = (Generic in cls.__bases__ and | |||
cls.__name__ != 'Protocol' and |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It could be written as type(cls) != _ProtocolMeta
, but let left this to other issue.
I'll update the branch to apply Serhyi's suggestions and add docs. @serhiy-storchaka are you OK with merging this and the generic namedtuple PR once your comments are resolved? |
Yes, I am OK with this. I just want to write tests that subscription works for arbitrary TypedDicts and named tuples (since the idea of forbidding it was rejected). |
At least on my end I was rejecting subscription arbitrary TypedDicts in previous Python versions that happened to be forward-compatible with the use of generic TypeVars proposed in this thread. I think it is questionable to allow subscripting TypedDicts in a way incompatible with this thread since then the semantics of that subscripting isn't clear. |
Subscripting TypedDict classes worked in 3.9 and 3.10, but not 3.8. I assume that was accidental. But now that the cat is out of the bag, I'd rather not change the user-visible behavior. |
@JelleZijlstra just to clarify, there are currently no plans to support generic typeddicts in 3.8? That would be a bummer |
Python 3.8 is in security-only mode (and has been for a while). This is a new feature, so being able to create generic
|
Changes: # Release 4.3.0 (July 1, 2022) - Add `typing_extensions.NamedTuple`, allowing for generic `NamedTuple`s on Python <3.11 (backport from python/cpython#92027, by Serhiy Storchaka). Patch by Alex Waygood (@AlexWaygood). - Adjust `typing_extensions.TypedDict` to allow for generic `TypedDict`s on Python <3.11 (backport from python/cpython#27663, by Samodya Abey). Patch by Alex Waygood (@AlexWaygood). # Release 4.2.0 (April 17, 2022) - Re-export `typing.Unpack` and `typing.TypeVarTuple` on Python 3.11. - Add `ParamSpecArgs` and `ParamSpecKwargs` to `__all__`. - Improve "accepts only single type" error messages. - Improve the distributed package. Patch by Marc Mueller (@cdce8p). - Update `typing_extensions.dataclass_transform` to rename the `field_descriptors` parameter to `field_specifiers` and accept arbitrary keyword arguments. - Add `typing_extensions.get_overloads` and `typing_extensions.clear_overloads`, and add registry support to `typing_extensions.overload`. Backport from python/cpython#89263. - Add `typing_extensions.assert_type`. Backport from bpo-46480. - Drop support for Python 3.6. Original patch by Adam Turner (@AA-Turner). # Release 4.1.1 (February 13, 2022) - Fix importing `typing_extensions` on Python 3.7.0 and 3.7.1. Original patch by Nikita Sobolev (@sobolevn). # Release 4.1.0 (February 12, 2022) - Runtime support for PEP 646, adding `typing_extensions.TypeVarTuple` and `typing_extensions.Unpack`. - Add interaction of `Required` and `NotRequired` with `__required_keys__`, `__optional_keys__` and `get_type_hints()`. Patch by David Cabot (@d-k-bo). - Runtime support for PEP 675 and `typing_extensions.LiteralString`. - Add `Never` and `assert_never`. Backport from bpo-46475. - `ParamSpec` args and kwargs are now equal to themselves. Backport from bpo-46676. Patch by Gregory Beauregard (@GBeauregard). - Add `reveal_type`. Backport from bpo-46414. - Runtime support for PEP 681 and `typing_extensions.dataclass_transform`. - `Annotated` can now wrap `ClassVar` and `Final`. Backport from bpo-46491. Patch by Gregory Beauregard (@GBeauregard). - Add missed `Required` and `NotRequired` to `__all__`. Patch by Yuri Karabas (@uriyyo). - The `@final` decorator now sets the `__final__` attribute on the decorated object to allow runtime introspection. Backport from bpo-46342. - Add `is_typeddict`. Patch by Chris Moradi (@chrismoradi) and James Hilton-Balfe (@Gobot1234). # Release 4.0.1 (November 30, 2021) - Fix broken sdist in release 4.0.0. Patch by Adam Turner (@AA-Turner). - Fix equality comparison for `Required` and `NotRequired`. Patch by Jelle Zijlstra (@JelleZijlstra). - Fix usage of `Self` as a type argument. Patch by Chris Wesseling (@CharString) and James Hilton-Balfe (@Gobot1234). # Release 4.0.0 (November 14, 2021) - Starting with version 4.0.0, typing_extensions uses Semantic Versioning. See the README for more information. - Dropped support for Python versions 3.5 and older, including Python 2.7. - Simplified backports for Python 3.6.0 and newer. Patch by Adam Turner (@AA-Turner). ## Added in version 4.0.0 - Runtime support for PEP 673 and `typing_extensions.Self`. Patch by James Hilton-Balfe (@Gobot1234). - Runtime support for PEP 655 and `typing_extensions.Required` and `NotRequired`. Patch by David Foster (@davidfstr). ## Removed in version 4.0.0 The following non-exported but non-private names have been removed as they are unneeded for supporting Python 3.6 and newer. - TypingMeta - OLD_GENERICS - SUBS_TREE - HAVE_ANNOTATED - HAVE_PROTOCOLS - V_co - VT_co # Previous releases Prior to release 4.0.0 we did not provide a changelog. Please check the Git history for details.
…thon-3.10.8 - Updated from version 4.1.1 to 4.4.0 - Update of rootfile - Changelog # Release 4.4.0 (October 6, 2022) - Add `typing_extensions.Any` a backport of python 3.11's Any class which is subclassable at runtime. (backport from python/cpython#31841, by Shantanu and Jelle Zijlstra). Patch by James Hilton-Balfe (@Gobot1234). - Add initial support for TypeVarLike `default` parameter, PEP 696. Patch by Marc Mueller (@cdce8p). - Runtime support for PEP 698, adding `typing_extensions.override`. Patch by Jelle Zijlstra. - Add the `infer_variance` parameter to `TypeVar`, as specified in PEP 695. Patch by Jelle Zijlstra. # Release 4.3.0 (July 1, 2022) - Add `typing_extensions.NamedTuple`, allowing for generic `NamedTuple`s on Python <3.11 (backport from python/cpython#92027, by Serhiy Storchaka). Patch by Alex Waygood (@AlexWaygood). - Adjust `typing_extensions.TypedDict` to allow for generic `TypedDict`s on Python <3.11 (backport from python/cpython#27663, by Samodya Abey). Patch by Alex Waygood (@AlexWaygood). # Release 4.2.0 (April 17, 2022) - Re-export `typing.Unpack` and `typing.TypeVarTuple` on Python 3.11. - Add `ParamSpecArgs` and `ParamSpecKwargs` to `__all__`. - Improve "accepts only single type" error messages. - Improve the distributed package. Patch by Marc Mueller (@cdce8p). - Update `typing_extensions.dataclass_transform` to rename the `field_descriptors` parameter to `field_specifiers` and accept arbitrary keyword arguments. - Add `typing_extensions.get_overloads` and `typing_extensions.clear_overloads`, and add registry support to `typing_extensions.overload`. Backport from python/cpython#89263. - Add `typing_extensions.assert_type`. Backport from bpo-46480. - Drop support for Python 3.6. Original patch by Adam Turner (@AA-Turner). Tested-by: Adolf Belka <[email protected]> Signed-off-by: Adolf Belka <[email protected]>
Copied from: https://bugs.python.org/issue44863
TypedDict PEP-589 says:
A TypedDict cannot inherit from both a TypedDict type and a non-TypedDict base class.
So the current implementation has:
if type(base) is not _TypedDictMeta: raise TypeError(...)
This restricts the user from defining generic TypedDicts in the natural class based syntax:
class Pager(TypedDict, Generic[T]): ...
Although PEP 589 doesn't explicitly state generic support, I believe it is complete in covering the specification even if generics were involved (at least for the class based syntax).
I have tried putting together a PEP from guidance of typing-sig https://github.com/sransara/py-generic-typeddict/blob/master/pep-9999.rst. There is not much new contributions by that draft, except for specifying the alternative syntax and being more explicit about Generics.
So I'm wondering if it would be possible to relax the constraint: TypedDict inheritance to include Generic. In my point of view
Generic
is more of a mixin, so it doesn't go against the PEP 589. Or is this change big enough to warrant a PEP?https://bugs.python.org/issue44863