diff --git a/peps/pep-0821.rst b/peps/pep-0821.rst index 554887b4cf8..f597e8f58d6 100644 --- a/peps/pep-0821.rst +++ b/peps/pep-0821.rst @@ -77,7 +77,7 @@ Allowing ``Unpack[TypedDict]`` inside ``Callable`` achieves the following: with existing semantics from :pep:`692` (``Unpack`` for ``**kwargs``) and :pep:`728` (``extra_items`` and ``closed``). * Keeps the feature additive, backwards compatible, and stays mostly a - typing-specification change only. + typing-specification change only, see `Backwards Compatibility`_. Alternatives considered ----------------------- @@ -105,9 +105,16 @@ Design trade-offs and decisions * Positional parameters: Allowing positional parameters to precede ``Unpack[TD]`` retains existing ``Callable`` semantics and mirrors real Python functions where positional and keyword-only parameters coexist. -* ``Concatenate``: Combining ``Unpack[TD]`` with ``Concatenate`` would enable - interspersed keyword-only parameters among ``*args`` and ``**kwargs``. This - increases complexity and is not proposed here. + +* ``Unpack[TD]`` cannot appear within ``Concatenate`` + (e.g. ``Concatenate[int, Unpack[TD], P]``), as this would enable + interspersed keyword-only parameters among ``*args`` and ``**kwargs``. + This increases complexity and is not proposed here. + +* Abbreviation with ``**``: This PEP does not add a suggestion to add a ``**`` + shorthand (e.g. ``**TD``) for ``Unpack[TD]`` in ``Callable``. The proposal + is intentionally minimal, reusing the existing ``Unpack[TD]`` form. Adding + such an abbreviation would require a grammar change and is out of scope. Specification ============= @@ -119,35 +126,24 @@ It becomes valid to write:: Callable[[Unpack[TD]], R] -where ``TD`` is a ``TypedDict``. A shorter form is also allowed:: - - Callable[Unpack[TD], R] - -Additionally, positional parameters may be combined with an unpacked -``TypedDict``:: +where ``TD`` is a ``TypedDict``. Additionally, positional parameters may be +combined with an unpacked ``TypedDict``:: Callable[[int, str, Unpack[TD]], R] Semantics --------- -For type-checking purposes, ``Callable[[Unpack[TD]], R]`` behaves as if it were -specified via a callback protocol whose ``__call__`` method has -``**kwargs: Unpack[TD]``. -The semantics of ``Unpack`` itself are exactly those described in the typing -specification's `Unpack for keyword arguments -`_ -section and :pep:`692`, together with :pep:`728` for ``extra_items`` and -``closed``. - -This PEP only adds the following Callable-specific rules: - -* ``Unpack[TD]`` may appear inside the parameter list of - ``Callable``. -* Positional parameters may appear in ``Callable`` before ``Unpack[TD]`` and - follow existing ``Callable`` semantics. -* Only a ``ParamSpec`` may be substituted by an unpacked ``TypedDict`` within a - ``Callable``. +``Callable[[Unpack[TD]], R]`` is, for type-checking purposes, equivalent to a +callback protocol whose ``__call__`` method is defined as +``def __call__(self, **kwargs: Unpack[TD]) -> R: ...``. +All rules — handling of required and optional keys, ``extra_items``, +``closed``, assignment compatibility — follow from this equivalence per +:pep:`692`, :pep:`728`, and other existing typing specifications related to +``Protocol`` and ``**kwargs`` unpacking through ``TypedDict``. +When positional parameters precede ``Unpack[TD]``, as in +``Callable[[int, str, Unpack[TD]], R]``, the same equivalence applies with +those parameters prepended to ``__call__`` before ``**kwargs``. Examples -------- @@ -158,7 +154,7 @@ compatible if it can be called with the required keywords (even if they are also accepted positionally); positional-only parameters for those keys are rejected:: - from typing import TypedDict, Callable, Unpack, Any, NotRequired + from typing import TypedDict, Callable, Unpack, Any class KeywordTD(TypedDict): a: int @@ -175,143 +171,31 @@ rejected:: f3: IntKwCallable = pos_only # Rejected f4: IntKwCallable = different # Rejected -Optional arguments ------------------- - -Keys marked ``NotRequired`` in the ``TypedDict`` correspond to optional -keyword arguments. -This means that the callable must accept them, but callers may omit them. -Functions that accept the keyword argument must also provide a default value -that is compatible; functions that omit the parameter entirely are rejected:: - - class OptionalKws(TypedDict): - a: NotRequired[int] - - type OptCallable = Callable[[Unpack[OptionalKws]], Any] - - def defaulted(a: int = 1): ... - def kw_default(*, a: int = 1): ... - def no_params(): ... - def required(a: int): ... - - g1: OptCallable = defaulted # Accepted - g2: OptCallable = kw_default # Accepted - g3: OptCallable = no_params # Rejected - g4: OptCallable = required # Rejected - -Additional keyword arguments ----------------------------- - -Default Behavior (no ``extra_items`` or ``closed``) -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -If the ``TypedDict`` does not specify ``extra_items`` or ``closed``, additional -keyword arguments are permitted with type ``object``. -This is the default behavior:: - - # implies extra_items=object - class DefaultTD(TypedDict): - a: int - - type DefaultCallable = Callable[[Unpack[DefaultTD]], Any] - - def v_any(**kwargs: object): ... - def v_ints(a: int, b: int=2): ... - - d1: DefaultCallable = v_any # Accepted (implicit object for extras) - d1(a=1, c="more") # Accepted (extras allowed) - d2: DefaultCallable = v_ints # Rejected (b: int is not a supertype of object) - -``closed`` behavior (PEP 728) -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -If ``closed=True`` is specified on the ``TypedDict``, no additional keyword -arguments beyond those declared are expected:: - - class ClosedTD(TypedDict, closed=True): - a: int - - type ClosedCallable = Callable[[Unpack[ClosedTD]], Any] - - def v_any(**kwargs: object): ... - def v_ints(a: int, b: int=2): ... - - c1: ClosedCallable = v_any # Accepted - c1(a=1, c="more") # Rejected (extra c not allowed) - c2: ClosedCallable = v_ints # Accepted - c2(a=1, b=2) # Rejected (extra b not allowed) - -Interaction with ``extra_items`` (PEP 728) -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -If a ``TypedDict`` specifies the ``extra_items`` parameter (with the exception -of ``extra_items=Never``), the corresponding ``Callable`` -must accept additional keyword arguments of the specified type. - -For example:: - - class ExtraTD(TypedDict, extra_items=str): - a: int - - type ExtraCallable = Callable[[Unpack[ExtraTD]], Any] - - def accepts_str(**kwargs: str): ... - def accepts_object(**kwargs: object): ... - def accepts_int(**kwargs: int): ... +Interaction with ``ParamSpec``, ``TypeVarTuple``, and ``Concatenate`` +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - e1: ExtraCallable = accepts_str # Accepted (matches extra_items type) - e2: ExtraCallable = accepts_object # Accepted (object is a supertype of str) - e3: ExtraCallable = accepts_int # Rejected (int is not a supertype of str) +A ``TypeVarTuple`` may coexist with ``Unpack[TD]`` in the parameter list, +because ``TypeVarTuple`` corresponds to ``*args`` and does not interfere +with the ``**kwargs``-equivalent role of ``Unpack[TD]``:: - e1(a=1, b="foo") # Accepted - e1(a=1, b=2) # Rejected (b must be str) + # With Tv a TypeVarTuple + Callable[[Unpack[Tv], Unpack[KeywordTD]], R] # Allowed +``Unpack[TD]`` cannot appear within ``Concatenate`` +(e.g. ``Concatenate[int, Unpack[TD]]``); use +``Callable[[int, Unpack[TD]], R]`` instead. +See `Design trade-offs and decisions`_ for the rationale. -Interaction with ``ParamSpec`` and ``Concatenate`` -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +A ``ParamSpec`` may not appear alongside ``Unpack[TD]`` in the same +``Callable`` parameter list, because ``ParamSpec`` covers both ``*args`` +and ``**kwargs``, which conflicts with ``Unpack[TD]``:: -A ``ParamSpec`` can be substituted by ``Unpack[KeywordTD]`` to define a -parameterized callable alias. Substituting ``Unpack[KeywordTD]`` produces the -same effect as writing the callable with an unpacked ``TypedDict`` directly. -Using a ``TypedDict`` within ``Concatenate`` is not allowed. :: - - type CallableP[**P] = Callable[P, Any] - - h: CallableP[Unpack[KeywordTD]] = normal # Accepted - h2: CallableP[Unpack[KeywordTD]] = kw_only # Accepted - h3: CallableP[Unpack[KeywordTD]] = pos_only # Rejected - -The current implementation needs to be updated to allow subscripting with a -generic ``Unpack[TypedDict]`` without extra brackets; -see `Backwards Compatibility`_. - -Combined positional parameters and ``Unpack`` -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -Positional parameters may precede an unpacked ``TypedDict`` inside ``Callable``. -Functions that accept the required positional arguments and can be called with -the specified keyword(s) are compatible; making the keyword positional-only is -rejected:: - - from typing import TypedDict, Callable, Unpack, Any - - class KeywordTD(TypedDict): - a: int - - type IntKwPosCallable = Callable[[int, str, Unpack[KeywordTD]], Any] - - def mixed_kwonly(x: int, y: str, *, a: int): ... - def mixed_poskw(x: int, y: str, a: int): ... - def mixed_posonly(x: int, y: str, a: int, /): ... - - m1: IntKwPosCallable = mixed_kwonly # Accepted - m2: IntKwPosCallable = mixed_poskw # Accepted - m3: IntKwPosCallable = mixed_posonly # Rejected + Callable[[P, Unpack[KeywordTD]], R] # Not allowed Backwards Compatibility ======================= -This feature is mostly an additive typing-only feature. It does not affect +This feature is an additive typing-only feature. It does not affect existing code. Subscripting a ``ParamSpec`` with a generic ``Unpack`` of a ``TypedDict`` is only backwards compatible when placed inside extra brackets; a ``TypeAliasType`` @@ -363,23 +247,69 @@ A prototype exists in mypy: Rejected Ideas ============== -- Combining ``Unpack[TD]`` with ``Concatenate``. With such support, one could - write ``Callable[Concatenate[int, Unpack[TD], P], R]`` which in turn would - allow a keyword-only parameter between ``*args`` and ``**kwargs``, i.e. - ``def func(*args: Any, a: int, **kwargs: Any) -> R: ...`` +- Combining ``Unpack[TD]`` with ``Concatenate`` in the form + ``Callable[Concatenate[int, Unpack[TD], P], R]`` would allow a + keyword-only parameter between ``*args`` and ``**kwargs``, i.e. + ``def func(*args: Any, a: int, **kwargs: Any) -> R: ...``, which is currently not allowed per :pep:`612`. To keep the initial implementation simple, this PEP does not propose such support. +- Allowing ``Callable[Unpack[TD], R]`` (without inner brackets) as a short + form was considered for brevity. It mirrors a ``ParamSpec`` shortcut that + has proven confusing in practice; discussion consensus is against + introducing a similar pattern here. + +- Supporting multiple ``TypedDict`` unpacks (e.g. + ``Callable[[Unpack[TD1], Unpack[TD2]], R]``) was considered. + For example:: + + class TD1(TypedDict): + a: int + b: int + + class TD2(TypedDict): + b: str + + f: Callable[[Unpack[TD1], Unpack[TD2]], R] + + could be written with the following ``Protocol`` approach, but with no clear + resolution for the conflict on key ``b``:: + + class CallWithTwoTDs(Protocol): + def __call__(self, a: int, b: ???) -> R: ... + + This would need a new concept that has no precedent in the existing typing + system and would require further design work and consensus. A resolution, + similar to runtime dict merging with ``**``, could be to have the last unpack + take precedence and overwrite previous keys. + Open Questions ============== -* Should multiple ``TypedDict`` unpacks be allowed to form a union, and if so, - how to handle overlapping keys of non-identical types? Which restrictions - should apply in such a case? Should the order matter? -* Should we allow the shorter form ``Callable[Unpack[TD], R]`` in addition to - ``Callable[[Unpack[TD]], R]``? -* Is there a necessity to differentiate between normal and ``ReadOnly`` keys? +* Should this PEP explicitly allow a ``ParamSpec`` to be substituted by an + unpacked ``TypedDict`` in generic type arguments, independent of + ``Callable``? This would broaden the scope beyond ``Callable`` and would + apply to any generic parameterized by a ``ParamSpec``. For example:: + + type CallableP[**P] = Callable[P, Any] + + h: CallableP[Unpack[KeywordTD]] = normal # Accepted + h2: CallableP[Unpack[KeywordTD]] = kw_only # Accepted + h3: CallableP[Unpack[KeywordTD]] = pos_only # Rejected + +* Should ``Unpack`` of a generic ``TypedDict`` be permitted inside + ``Callable``? For example:: + + class TD1[T](TypedDict): + a: T + + def func2[T](a: T, cb: Callable[[Unpack[TD1[T]]], None]) -> None: ... + + :pep:`692` did not address generic ``TypedDict`` in ``**kwargs``, and + the rules remain an open ambiguity in the typing spec. If the equivalence + principle applies, the rules should follow from the callback protocol + equivalent. Acknowledgements @@ -388,9 +318,9 @@ Acknowledgements Thanks to Jelle Zijlstra for sponsoring this PEP and his valuable review feedback. -Hugo van Kemenade, for helpful feedback on the draft and PR of this PEP. +Eric Traut, for careful review, detailed feedback, and valuable input. -Eric Traut, for feedback on the initial idea and discussions. +Hugo van Kemenade, for helpful feedback on the draft and PR of this PEP. References