-
-
Notifications
You must be signed in to change notification settings - Fork 2.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support function decorators excellently #3157
Comments
Great to see this may be really happening! The Twitter discussion gave me a new suggestion for what to name "second-order decorators" -- people seemed to like "decorator factory" best. |
"decoratory" :) |
Here are some thoughts about decorators that tweak arguments/return types. This proposal combines features from @sixolet's proposal and other sources, such as #1927 and python/typing#193. Argspec type variablesWe'd add a new kinds of type variable: argspec. An argspec type variable represents an arbitrary sequence of callable argument types/kinds, such as This is how we'd define one: from typing import TypeVar
Args = TypeVar('Args', argspec=True) For example, we could use this in a stub for RT = TypeVar('RT') # Regular type variable
def contextmanager(
func: Callable[Args, Iterator[RT]]) -> Callable[Args, GeneratorContextManager[RT]]: ... Decorator implementationsOutside a stub, we hit a problem: it's going to be tricky to type check implementations of functions that use an argspec type variable in their signatures. We can sidestep this issue by not supporting this at all -- instead we'd require a separate external signature declaration and an implementation for such functions, and the implementation can't use argspecs. This is analogous to how overloaded functions with implementations work. Sketch of how to implement from typing import declared_type
...
@declared_type
def contextmanager(
func: Callable[Args, Iterator[RT]]) -> Callable[Args, GeneratorContextManager[RT]]: ...
def contextmanager(
func: Callable[..., Iterator[RT]]) -> Callable[..., GeneratorContextManager[RT]]:
<implementation of context manager>
return <something> The name
|
Oh and the updated |
I have two clarifying questions/proposals:
|
How so? I'm aware that it's difficult for general variadic functions (as mentioned in python/typing#193), but I'd expect that most decorators will just treat values of |
@JukkaL I have been pondering this problem for the last two weeks, and haven't started an implementation of any solution because I hadn't solved it yet in my head. I like a whole lot about your suggestions, and I think they get us a lot further. Some wiggly bits I have:
from typing import declared_type
@declared_type
def contextmanager(
func: Callable[Args, Iterator[RT]]) -> Callable[Args, GeneratorContextManager[RT]]: ...
def contextmanager(
func: Callable[..., Iterator[RT]]) -> Callable[..., GeneratorContextManager[RT]]:
<implementation of context manager>
return <something> Can we consider this instead:
This uses a more direct method than redefinition to declare the type of the function aside from its implementation. It also neatly sidesteps the problem of how to define an argspec in the
... I don't think these weird signatures are important enough to specifically jump through hoops on their own. Should do this form if we feel like it's cleaner. The following is speculation and musing and I'm not particularly strongly attached to it. I have some kind of sneaking suspicion that what we're calling "argspecs" here and variadic type variables are more closely related than (at least I) suspected. I'm looking for a way to relate these two concepts -- for example, a way to cleanly and pleasantly write the signature of Maybe something like: Args = ArgSpec('ArgSpec', keyword=False) # matches only positional arguments
R = TypeVar('R')
@declared_type(Callable[[Callable[Args, R], Expand[Iterable[Args], Args]], Iterable[R]])
def map(f: Callable[..., R], *args) -> Iterable[R]: ... (Here
|
That's fair. Anyway, we don't need to decide this very early, since this is only a matter of syntax.
I have no strong opinion either way right now, but I have a feeling that the flexible callable syntax can make more complex signatures hard to read. Again, this is mostly a matter of syntax and we can bikeshed it later after we've agreed on basic principles. And I agree that making this consistent with how we declare the decorated signature of a function would be nice.
Yes, I think that a variadic type variables would mostly behave like an argspec, but it wouldn't have any argument kinds (just a sequence of types). I left this out from my proposal because it's not very directly related to the current issue. A variadic type variable would also allow things like
Yes, most decorators probably are trivial in that respect. However, any non-trivial decorators could be very tricky to type check. I'm worried that users have a significant number of those (though still a minority of all decorators) and if we don't type check them properly it will be a never-ending stream of bug reports and ideas about handling various edge cases. Also, as these are otherwise quite similar to variadic type variables, it might feel a little odd if we could type check one but not the other.
Unfortunately that doesn't work with variadic type variables for things like
Yes, though changing the number of arguments might be something we implement only a bit later as the feature would be quite useful even without it. |
It's been over a year w/o updates to this thread, but it's still the first one that comes up for me in a google search for mypy and decorators. Is this still the right place to look for updates/status/plans for decorator typing improvements? |
Just for the record: if someone needs to change the return type of the function inside the decorator and still have typed parameters, you can use a custom Saved me a lot of time! |
Thanks for the heads-up and detailed explanation! This looks very useful indeed as well. What I'm actually looking for currently is a way to let decorators |
@rggjan , this is actually in scope for ListVariadics, and in fact is already implemented (but unfortunately only documented at this point in https://github.com/facebook/pyre-check/blob/master/docs/Variadic_Type_Variables_for_Decorators_and_Tensors.pdf). The addition you'll need here is For your example, you'll need this:
Note that again you're losing the names of those parameters by passing through a ListVariadic, but the transformation you're looking for is there. Does that work for your use case? |
@mrkmndz Thanks a lot, that seems to be exactly what I was looking for (except the keeping parameter names would be nice, of course). What's |
Which parts of @mrkmndz's pdf are on a standards track? Is this a strawman implementation for a PEP? (Otherwise, very interesting ideas in this document.) |
@rggjan Pyre (https://pyre-check.org/) is another implementation of PEP484 type checking, and pyre_extensions (https://pypi.org/project/pyre-extensions/) is our pip package for the runtime components of our extensions that are not yet standardized. As for MyPy, from my conversations with @ilevkivskyi and @JukkaL I believe that they are planning on building out a compatible implementation on the MyPy side in near future with the help of @theodoretliu . With regards to mapping a ParameterSpecification instead of a ListVariadic, one could definitely imagine an analogous @kaste I will be working on drafting a PEP on ListVariadic, Map, and Concatenate in the next month. I am planning on deferring the IntVar related stuff (e.g. Index etc.) into another one once that has an example implementation. Syntax is definitely subject to change, but I would anticipate that most of the core ideas there are planned to be headed for a standards-track PEP. |
Clear, thanks @mrkmndz for the information! |
I think the initial post of this issue should be updated to briefly mention the current state of decorator support. A while ago, I skimmed through this issue and got the impression that "second-order decorators" (without modifying arguments) are not supported by Mypy. Only later I discovered that in fact they are supported. This is also a documentation issue, the current documentation only explanis bare decorators. I will look into submitting a PR for that. |
Can this issue be closed now?
Now it can: from typing import TypeVar, Callable, cast, ParamSpec
T = TypeVar('T')
P = ParamSpec('P')
def print_callcount(f: Callable[P, T]) -> Callable[P, T]:
x = 0
def ret(*args: P.args, **kwargs: P.kwargs) -> T:
nonlocal x
x += 1
print("%d calls so far" % x)
return f(*args, **kwargs)
return ret # No cast!
@print_callcount
def lol(x: int) -> str:
return "lol"*x
reveal_type(lol) # E: Revealed type is 'def (x: builtins.int) -> builtins.str' The other broken example depends on completion of #8645 |
Decorators currently stress mypy's support for functional programming to the point that many decorators are impossible to type. I'm intending this issue as a project plan and exploration of the issue of decorators and how to type them. It's a collection point for problems or incompletenesses that keep decorators from being fully supported, plans for solutions to those problems, and fully-implemented solutions to those problems.
Decorators that can only decorate a function of a fixed signature
You can define your decorator's signature explicitly, and it's completely supported:
Notes:
Callable[[Arg('x', int), VarArg(str)], int]
now a thing you can do #2607 enables thatDecorators that do not change the signature of the function
Nearly fully supported. Here's how you do it:
Notes:
f
. You can set a bound onT
to beCallable
, but you don't need to for it to typecheck.print_callcount
, and there's no way to declare that argument as aCallable
explicitly without losing the argument type oflol
later. We'd like to minimize the places where casts are required. Doing so requires something along the lines of "variadic argument variables", discussed below.Decorators that take arguments ("second-order"?)
Plenty of decorators "take arguments" by actually being functions that return a decorator. For example, we'd like to be able to do this:
Notes:
lol
ends up typed asNone
Mess with the return type or with arguments
For an arbitrary function you can't do this at all yet -- there isn't even a syntax. Here's me making up some syntax for it.
Messing with the return type
Messing with the arguments
The syntax here is fungible, but we would need a way to do approximately this thing -- capture the types and kinds of all a function's arguments in some kind of variation on a type variable.
Relevant issues and discussions:
Variadic type variables alone (python/typing#193) get you some of the way there, but lose all keyword arguments of the decorated function.
Things to do:
*args
and**kwargs
SomeArguments
-style alias, so nobody has to actually engage with the details of the above when writing normal decorators.mypy_extensions
nottyping
at first -- this can be fodder for the future of PEP484, but while we're playing in such experimental land, not yet.SomeArguments
thing.The text was updated successfully, but these errors were encountered: