PEP 821: Refinement from Feedback

peps/pep-0821.rst

@@ -77,7 +77,7 @@ Allowing ``Unpack[TypedDict]`` inside ``Callable`` achieves the following:
   with existing semantics from :pep:`692` (``Unpack`` for ``**kwargs``) and
   :pep:`728` (``extra_items`` and ``closed``).
 * Keeps the feature additive, backwards compatible, and stays mostly a
-  typing-specification change only.
+  typing-specification change only, see `Backwards Compatibility`_.
 
 Alternatives considered
 -----------------------
@@ -105,9 +105,16 @@ Design trade-offs and decisions
 * Positional parameters: Allowing positional parameters to precede
   ``Unpack[TD]`` retains existing ``Callable`` semantics and mirrors real
   Python functions where positional and keyword-only parameters coexist.
-* ``Concatenate``: Combining ``Unpack[TD]`` with ``Concatenate`` would enable
-  interspersed keyword-only parameters among ``*args`` and ``**kwargs``. This
-  increases complexity and is not proposed here.
+
+* ``Unpack[TD]`` cannot appear within ``Concatenate``
+  (e.g. ``Concatenate[int, Unpack[TD], P]``), as this would enable
+  interspersed keyword-only parameters among ``*args`` and ``**kwargs``.
+  This increases complexity and is not proposed here.
+
+* Abbreviation with ``**``: This PEP does not add a suggestion to add a ``**``
+  shorthand (e.g. ``**TD``) for ``Unpack[TD]`` in ``Callable``. The proposal
+  is intentionally minimal, reusing the existing ``Unpack[TD]`` form. Adding
+  such an abbreviation would require a grammar change and is out of scope.
 
 Specification
 =============
@@ -119,35 +126,24 @@ It becomes valid to write::
 
     Callable[[Unpack[TD]], R]
 
-where ``TD`` is a ``TypedDict``. A shorter form is also allowed::
-
-    Callable[Unpack[TD], R]
-
-Additionally, positional parameters may be combined with an unpacked
-``TypedDict``::
+where ``TD`` is a ``TypedDict``. Additionally, positional parameters may be
+combined with an unpacked ``TypedDict``::
 
     Callable[[int, str, Unpack[TD]], R]
 
 Semantics
 ---------
 
-For type-checking purposes, ``Callable[[Unpack[TD]], R]`` behaves as if it were
-specified via a callback protocol whose ``__call__`` method has
-``**kwargs: Unpack[TD]``.
-The semantics of ``Unpack`` itself are exactly those described in the typing
-specification's `Unpack for keyword arguments
-<https://typing.python.org/en/latest/spec/callables.html#unpack-for-keyword-arguments>`_
-section and :pep:`692`, together with :pep:`728` for ``extra_items`` and
-``closed``.
-
-This PEP only adds the following Callable-specific rules:
-
-* ``Unpack[TD]`` may appear inside the parameter list of
-  ``Callable``.
-* Positional parameters may appear in ``Callable`` before ``Unpack[TD]`` and
-  follow existing ``Callable`` semantics.
-* Only a ``ParamSpec`` may be substituted by an unpacked ``TypedDict`` within a
-  ``Callable``.
+``Callable[[Unpack[TD]], R]`` is, for type-checking purposes, equivalent to a
+callback protocol whose ``__call__`` method is defined as
+``def __call__(self, **kwargs: Unpack[TD]) -> R: ...``.
+All rules — handling of required and optional keys, ``extra_items``,
+``closed``, assignment compatibility — follow from this equivalence per
+:pep:`692`, :pep:`728`, and other existing typing specifications related to
+``Protocol`` and ``**kwargs`` unpacking through ``TypedDict``.
+When positional parameters precede ``Unpack[TD]``, as in
+``Callable[[int, str, Unpack[TD]], R]``, the same equivalence applies with
+those parameters prepended to ``__call__`` before ``**kwargs``.
 
 Examples
 --------
@@ -158,7 +154,7 @@ compatible if it can be called with the required keywords (even if they are
 also accepted positionally); positional-only parameters for those keys are
 rejected::
 
-    from typing import TypedDict, Callable, Unpack, Any, NotRequired
+    from typing import TypedDict, Callable, Unpack, Any
 
     class KeywordTD(TypedDict):
         a: int
@@ -175,143 +171,31 @@ rejected::
     f3: IntKwCallable = pos_only   # Rejected
     f4: IntKwCallable = different  # Rejected
 
-Optional arguments
-------------------
-
-Keys marked ``NotRequired`` in the ``TypedDict`` correspond to optional
-keyword arguments.
-This means that the callable must accept them, but callers may omit them.
-Functions that accept the keyword argument must also provide a default value
-that is compatible; functions that omit the parameter entirely are rejected::
-
-    class OptionalKws(TypedDict):
-        a: NotRequired[int]
-
-    type OptCallable = Callable[[Unpack[OptionalKws]], Any]
-
-    def defaulted(a: int = 1): ...
-    def kw_default(*, a: int = 1): ...
-    def no_params(): ...
-    def required(a: int): ...
-
-    g1: OptCallable = defaulted    # Accepted
-    g2: OptCallable = kw_default   # Accepted
-    g3: OptCallable = no_params    # Rejected
-    g4: OptCallable = required     # Rejected
-
-Additional keyword arguments
-----------------------------
-
-Default Behavior (no ``extra_items`` or ``closed``)
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-If the ``TypedDict`` does not specify ``extra_items`` or ``closed``, additional
-keyword arguments are permitted with type ``object``.
-This is the default behavior::
-
-  # implies extra_items=object
-  class DefaultTD(TypedDict):
-    a: int
-
-  type DefaultCallable = Callable[[Unpack[DefaultTD]], Any]
-
-  def v_any(**kwargs: object): ...
-  def v_ints(a: int, b: int=2): ...
-
-  d1: DefaultCallable = v_any   # Accepted (implicit object for extras)
-  d1(a=1, c="more")             # Accepted (extras allowed)
-  d2: DefaultCallable = v_ints  # Rejected (b: int is not a supertype of object)
-
-``closed`` behavior (PEP 728)
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-If ``closed=True`` is specified on the ``TypedDict``, no additional keyword
-arguments beyond those declared are expected::
-
-  class ClosedTD(TypedDict, closed=True):
-    a: int
-
-  type ClosedCallable = Callable[[Unpack[ClosedTD]], Any]
-
-  def v_any(**kwargs: object): ...
-  def v_ints(a: int, b: int=2): ...
-
-  c1: ClosedCallable = v_any   # Accepted
-  c1(a=1, c="more")            # Rejected (extra c not allowed)
-  c2: ClosedCallable = v_ints  # Accepted
-  c2(a=1, b=2)                 # Rejected (extra b not allowed)
-
-Interaction with ``extra_items`` (PEP 728)
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-If a ``TypedDict`` specifies the ``extra_items`` parameter (with the exception
-of ``extra_items=Never``), the corresponding ``Callable``
-must accept additional keyword arguments of the specified type.
-
-For example::
-
-  class ExtraTD(TypedDict, extra_items=str):
-    a: int
-
-  type ExtraCallable = Callable[[Unpack[ExtraTD]], Any]
-
-  def accepts_str(**kwargs: str): ...
-  def accepts_object(**kwargs: object): ...
-  def accepts_int(**kwargs: int): ...
+Interaction with ``ParamSpec``, ``TypeVarTuple``, and ``Concatenate``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-  e1: ExtraCallable = accepts_str     # Accepted (matches extra_items type)
-  e2: ExtraCallable = accepts_object  # Accepted (object is a supertype of str)
-  e3: ExtraCallable = accepts_int     # Rejected (int is not a supertype of str)
+A ``TypeVarTuple`` may coexist with ``Unpack[TD]`` in the parameter list,
+because ``TypeVarTuple`` corresponds to ``*args`` and does not interfere
+with the ``**kwargs``-equivalent role of ``Unpack[TD]``::
 
-  e1(a=1, b="foo")   # Accepted
-  e1(a=1, b=2)       # Rejected (b must be str)
+    # With Tv a TypeVarTuple
+    Callable[[Unpack[Tv], Unpack[KeywordTD]], R]  # Allowed
 
+``Unpack[TD]`` cannot appear within ``Concatenate``
+(e.g. ``Concatenate[int, Unpack[TD]]``); use
+``Callable[[int, Unpack[TD]], R]`` instead.
+See `Design trade-offs and decisions`_ for the rationale.
 
-Interaction with ``ParamSpec`` and ``Concatenate``
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+A ``ParamSpec`` may not appear alongside ``Unpack[TD]`` in the same
+``Callable`` parameter list, because ``ParamSpec`` covers both ``*args``
+and ``**kwargs``, which conflicts with ``Unpack[TD]``::
 
-A ``ParamSpec`` can be substituted by ``Unpack[KeywordTD]`` to define a
-parameterized callable alias. Substituting ``Unpack[KeywordTD]`` produces the
-same effect as writing the callable with an unpacked ``TypedDict`` directly.
-Using a ``TypedDict`` within ``Concatenate`` is not allowed. ::
-
-    type CallableP[**P] = Callable[P, Any]
-
-    h: CallableP[Unpack[KeywordTD]] = normal   # Accepted
-    h2: CallableP[Unpack[KeywordTD]] = kw_only # Accepted
-    h3: CallableP[Unpack[KeywordTD]] = pos_only # Rejected
-
-The current implementation needs to be updated to allow subscripting with a
-generic ``Unpack[TypedDict]`` without extra brackets;
-see `Backwards Compatibility`_.
-
-Combined positional parameters and ``Unpack``
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-Positional parameters may precede an unpacked ``TypedDict`` inside ``Callable``.
-Functions that accept the required positional arguments and can be called with
-the specified keyword(s) are compatible; making the keyword positional-only is
-rejected::
-
-    from typing import TypedDict, Callable, Unpack, Any
-
-    class KeywordTD(TypedDict):
-        a: int
-
-    type IntKwPosCallable = Callable[[int, str, Unpack[KeywordTD]], Any]
-
-    def mixed_kwonly(x: int, y: str, *, a: int): ...
-    def mixed_poskw(x: int, y: str, a: int): ...
-    def mixed_posonly(x: int, y: str, a: int, /): ...
-
-    m1: IntKwPosCallable = mixed_kwonly  # Accepted
-    m2: IntKwPosCallable = mixed_poskw   # Accepted
-    m3: IntKwPosCallable = mixed_posonly # Rejected
+    Callable[[P, Unpack[KeywordTD]], R]  # Not allowed
 
 Backwards Compatibility
 =======================
 
-This feature is mostly an additive typing-only feature. It does not affect
+This feature is an additive typing-only feature. It does not affect
 existing code.
 Subscripting a ``ParamSpec`` with a generic ``Unpack`` of a ``TypedDict`` is
 only backwards compatible when placed inside extra brackets; a ``TypeAliasType``
@@ -363,23 +247,69 @@ A prototype exists in mypy:
 Rejected Ideas
 ==============
 
-- Combining ``Unpack[TD]`` with ``Concatenate``. With such support, one could
-  write ``Callable[Concatenate[int, Unpack[TD], P], R]`` which in turn would
-  allow a keyword-only parameter between ``*args`` and ``**kwargs``, i.e.
-  ``def func(*args: Any, a: int, **kwargs: Any) -> R: ...``
+- Combining ``Unpack[TD]`` with ``Concatenate`` in the form
+  ``Callable[Concatenate[int, Unpack[TD], P], R]`` would allow a
+  keyword-only parameter between ``*args`` and ``**kwargs``, i.e.
+  ``def func(*args: Any, a: int, **kwargs: Any) -> R: ...``,
   which is currently not allowed per :pep:`612`.
   To keep the initial implementation simple, this PEP does not propose such
   support.
 
+- Allowing ``Callable[Unpack[TD], R]`` (without inner brackets) as a short
+  form was considered for brevity. It mirrors a ``ParamSpec`` shortcut that
+  has proven confusing in practice; discussion consensus is against
+  introducing a similar pattern here.
+
+- Supporting multiple ``TypedDict`` unpacks (e.g.
+  ``Callable[[Unpack[TD1], Unpack[TD2]], R]``) was considered.
+  For example::
+
+      class TD1(TypedDict):
+          a: int
+          b: int
+
+      class TD2(TypedDict):
+          b: str
+
+      f: Callable[[Unpack[TD1], Unpack[TD2]], R]
+
+  could be written with the following ``Protocol`` approach, but with no clear
+  resolution for the conflict on key ``b``::
+
+      class CallWithTwoTDs(Protocol):
+          def __call__(self, a: int, b: ???) -> R: ...
+
+  This would need a new concept that has no precedent in the existing typing
+  system and would require further design work and consensus. A resolution,
+  similar to runtime dict merging with ``**``, could be to have the last unpack
+  take precedence and overwrite previous keys.
+
 Open Questions
 ==============
 
-* Should multiple ``TypedDict`` unpacks be allowed to form a union, and if so,
-  how to handle overlapping keys of non-identical types? Which restrictions
-  should apply in such a case? Should the order matter?
-* Should we allow the shorter form ``Callable[Unpack[TD], R]`` in addition to
-  ``Callable[[Unpack[TD]], R]``?
-* Is there a necessity to differentiate between normal and ``ReadOnly`` keys?
+* Should this PEP explicitly allow a ``ParamSpec`` to be substituted by an
+  unpacked ``TypedDict`` in generic type arguments, independent of
+  ``Callable``? This would broaden the scope beyond ``Callable`` and would
+  apply to any generic parameterized by a ``ParamSpec``. For example::
+
+      type CallableP[**P] = Callable[P, Any]
+
+      h: CallableP[Unpack[KeywordTD]] = normal    # Accepted
+      h2: CallableP[Unpack[KeywordTD]] = kw_only  # Accepted
+      h3: CallableP[Unpack[KeywordTD]] = pos_only # Rejected
+
+* Should ``Unpack`` of a generic ``TypedDict`` be permitted inside
+  ``Callable``? For example::
+
+      class TD1[T](TypedDict):
+          a: T
+
+      def func2[T](a: T, cb: Callable[[Unpack[TD1[T]]], None]) -> None: ...
+
+  :pep:`692` did not address generic ``TypedDict`` in ``**kwargs``, and
+  the rules remain an open ambiguity in the typing spec. If the equivalence
+  principle applies, the rules should follow from the callback protocol
+  equivalent.
 
 
 Acknowledgements
@@ -388,9 +318,9 @@ Acknowledgements
 Thanks to Jelle Zijlstra for sponsoring this PEP and his valuable review
 feedback.
 
-Hugo van Kemenade, for helpful feedback on the draft and PR of this PEP.
+Eric Traut, for careful review, detailed feedback, and valuable input.
 
-Eric Traut, for feedback on the initial idea and discussions.
+Hugo van Kemenade, for helpful feedback on the draft and PR of this PEP.
 
 
 References