PEP: 655
Title: Marking individual TypedDict items as required or potentially-missing
Author: David Foster <david at dafoster.net>
Sponsor: Guido van Rossum <guido at python.org>
Discussions-To: https://mail.python.org/archives/list/typing-sig@python.org/thread/53XVOD5ZUKJ263MWA6AUPEA6J7LBBLNV/
Status: Final
Type: Standards Track
Topic: Typing
Created: 30-Jan-2021
Python-Version: 3.11
Post-History: 31-Jan-2021, 11-Feb-2021, 20-Feb-2021, 26-Feb-2021, 17-Jan-2022, 28-Jan-2022
Resolution: https://mail.python.org/archives/list/python-dev@python.org/message/AJEDNVC3FXM5QXNNW5CR4UCT4KI5XVUE/

. canonical-typing-spec:: :ref:`typing:required-notrequired`,
                          :py:data:`typing.Required` and
                          :py:data:`typing.NotRequired`

Abstract
========

:pep:`589` defines notation
for declaring a TypedDict with all required keys and notation for defining
a TypedDict with :pep:`all potentially-missing keys <589#totality>`, however it
does not provide a mechanism to declare some keys as required and others
as potentially-missing. This PEP introduces two new notations:
``Required[]``, which can be used on individual items of a
TypedDict to mark them as required, and
``NotRequired[]``, which can be used on individual items
to mark them as potentially-missing.

This PEP makes no Python grammar changes. Correct usage
of required and potentially-missing keys of TypedDicts is intended to be
enforced only by static type checkers and need not be enforced by
Python itself at runtime.


Motivation
==========

It is not uncommon to want to define a TypedDict with some keys that are
required and others that are potentially-missing. Currently the only way
to define such a TypedDict is to declare one TypedDict with one value
for ``total`` and then inherit it from another TypedDict with a
different value for ``total``:

::

  class _MovieBase(TypedDict):  # implicitly total=True
      title: str

  class Movie(_MovieBase, total=False):
      year: int

Having to declare two different TypedDict types for this purpose is
cumbersome.

This PEP introduces two new type qualifiers, ``typing.Required`` and
``typing.NotRequired``, which allow defining a *single* TypedDict with
a mix of both required and potentially-missing keys:

::

  class Movie(TypedDict):
      title: str
      year: NotRequired[int]

This PEP also makes it possible to define TypedDicts in the
:pep:`alternative functional syntax <589#alternative-syntax>`
with a mix of required and potentially-missing keys,
which is not currently possible at all because the alternative syntax does
not support inheritance:

::

  Actor = TypedDict('Actor', {
      'name': str,
      # "in" is a keyword, so the functional syntax is necessary
      'in': NotRequired[List[str]],
  })


Rationale
=========

One might think it unusual to propose notation that prioritizes marking
*required* keys rather than *potentially-missing* keys, as is
customary in other languages like TypeScript:

. code-block:: typescript

  interface Movie {
      title: string;
      year?: number;  // ? marks potentially-missing keys
  }

The difficulty is that the best word for marking a potentially-missing
key, ``Optional[]``, is already used in Python for a completely
different purpose: marking values that could be either of a particular
type or ``None``. In particular the following does not work:

::

  class Movie(TypedDict):
      ...
      year: Optional[int]  # means int|None, not potentially-missing!

Attempting to use any synonym of “optional” to mark potentially-missing
keys (like ``Missing[]``) would be too similar to ``Optional[]``
and be easy to confuse with it.

Thus it was decided to focus on positive-form phrasing for required keys
instead, which is straightforward to spell as ``Required[]``.

Nevertheless it is common for folks wanting to extend a regular
(``total=True``) TypedDict to only want to add a small number of
potentially-missing keys, which necessitates a way to mark keys that are
*not* required and potentially-missing, and so we also allow the
``NotRequired[]`` form for that case.


Specification
=============

The ``typing.Required`` type qualifier is used to indicate that a
variable declared in a TypedDict definition is a required key:

::

  class Movie(TypedDict, total=False):
      title: Required[str]
      year: int

Additionally the ``typing.NotRequired`` type qualifier is used to
indicate that a variable declared in a TypedDict definition is a
potentially-missing key:

::

  class Movie(TypedDict):  # implicitly total=True
      title: str
      year: NotRequired[int]

It is an error to use ``Required[]`` or ``NotRequired[]`` in any
location that is not an item of a TypedDict.
Type checkers must enforce this restriction.

It is valid to use ``Required[]`` and ``NotRequired[]`` even for
items where it is redundant, to enable additional explicitness if desired:

::

  class Movie(TypedDict):
      title: Required[str]  # redundant
      year: NotRequired[int]

It is an error to use both ``Required[]`` and ``NotRequired[]`` at the
same time:

::

  class Movie(TypedDict):
      title: str
      year: NotRequired[Required[int]]  # ERROR

Type checkers must enforce this restriction.
The runtime implementations of ``Required[]`` and ``NotRequired[]``
may also enforce this restriction.

The :pep:`alternative functional syntax <589#alternative-syntax>`
for TypedDict also supports
``Required[]`` and ``NotRequired[]``:

::

  Movie = TypedDict('Movie', {'name': str, 'year': NotRequired[int]})


Interaction with ``total=False``
--------------------------------

Any :pep:`589`-style TypedDict declared with ``total=False`` is equivalent
to a TypedDict with an implicit ``total=True`` definition with all of its
keys marked as ``NotRequired[]``.

Therefore:

::

  class _MovieBase(TypedDict):  # implicitly total=True
      title: str

  class Movie(_MovieBase, total=False):
      year: int


is equivalent to:

::

  class _MovieBase(TypedDict):
      title: str

  class Movie(_MovieBase):
      year: NotRequired[int]


Interaction with ``Annotated[]``
-----------------------------------

``Required[]`` and ``NotRequired[]`` can be used with ``Annotated[]``,
in any nesting order:

::

  class Movie(TypedDict):
      title: str
      year: NotRequired[Annotated[int, ValueRange(-9999, 9999)]]  # ok

::

  class Movie(TypedDict):
      title: str
      year: Annotated[NotRequired[int], ValueRange(-9999, 9999)]  # ok

In particular allowing ``Annotated[]`` to be the outermost annotation
for an item allows better interoperability with non-typing uses of
annotations, which may always want ``Annotated[]`` as the outermost annotation.
[3]_


Runtime behavior
----------------


Interaction with ``get_type_hints()``
'''''''''''''''''''''''''''''''''''''

``typing.get_type_hints(...)`` applied to a TypedDict will by default
strip out any ``Required[]`` or ``NotRequired[]`` type qualifiers,
since these qualifiers are expected to be inconvenient for code
casually introspecting type annotations.

``typing.get_type_hints(..., include_extras=True)`` however
*will* retain ``Required[]`` and ``NotRequired[]`` type qualifiers,
for advanced code introspecting type annotations that
wishes to preserve *all* annotations in the original source:

::

  class Movie(TypedDict):
      title: str
      year: NotRequired[int]

  assert get_type_hints(Movie) == \
      {'title': str, 'year': int}
  assert get_type_hints(Movie, include_extras=True) == \
      {'title': str, 'year': NotRequired[int]}


Interaction with ``get_origin()`` and ``get_args()``
''''''''''''''''''''''''''''''''''''''''''''''''''''

``typing.get_origin()`` and ``typing.get_args()`` will be updated to
recognize ``Required[]`` and ``NotRequired[]``:

::

  assert get_origin(Required[int]) is Required
  assert get_args(Required[int]) == (int,)

  assert get_origin(NotRequired[int]) is NotRequired
  assert get_args(NotRequired[int]) == (int,)


Interaction with ``__required_keys__`` and ``__optional_keys__``
''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''

An item marked with ``Required[]`` will always appear
in the ``__required_keys__`` for its enclosing TypedDict. Similarly an item
marked with ``NotRequired[]`` will always appear in ``__optional_keys__``.

::

  assert Movie.__required_keys__ == frozenset({'title'})
  assert Movie.__optional_keys__ == frozenset({'year'})


Backwards Compatibility
=======================

No backward incompatible changes are made by this PEP.


How to Teach This
=================

To define a TypedDict where most keys are required and some are
potentially-missing, define a single TypedDict as normal
(without the ``total`` keyword)
and mark those few keys that are potentially-missing with ``NotRequired[]``.

To define a TypedDict where most keys are potentially-missing and a few are
required, define a ``total=False`` TypedDict
and mark those few keys that are required with ``Required[]``.

If some items accept ``None`` in addition to a regular value, it is
recommended that the ``TYPE|None`` notation be preferred over
``Optional[TYPE]`` for marking such item values, to avoid using
``Required[]`` or ``NotRequired[]`` alongside ``Optional[]``
within the same TypedDict definition:

Yes:

. code-block::
  :class: good

  from __future__ import annotations  # for Python 3.7-3.9

  class Dog(TypedDict):
      name: str
      owner: NotRequired[str|None]

Okay (required for Python 3.5.3-3.6):

. code-block::
  :class: maybe

  class Dog(TypedDict):
      name: str
      owner: 'NotRequired[str|None]'

No:

. code-block::
  :class: bad

  class Dog(TypedDict):
      name: str
      # ick; avoid using both Optional and NotRequired
      owner: NotRequired[Optional[str]]

Usage in Python <3.11
---------------------

If your code supports Python <3.11 and wishes to use ``Required[]`` or
``NotRequired[]`` then it should use ``typing_extensions.TypedDict`` rather
than ``typing.TypedDict`` because the latter will not understand
``(Not)Required[]``. In particular ``__required_keys__`` and
``__optional_keys__`` on the resulting TypedDict type will not be correct:

Yes (Python 3.11+ only):

. code-block::
  :class: good

  from typing import NotRequired, TypedDict

  class Dog(TypedDict):
      name: str
      owner: NotRequired[str|None]

Yes (Python <3.11 and 3.11+):

. code-block::
  :class: good

  from __future__ import annotations  # for Python 3.7-3.9

  from typing_extensions import NotRequired, TypedDict  # for Python <3.11 with (Not)Required

  class Dog(TypedDict):
      name: str
      owner: NotRequired[str|None]

No (Python <3.11 and 3.11+):

. code-block::
  :class: bad

  from typing import TypedDict  # oops: should import from typing_extensions instead
  from typing_extensions import NotRequired

  class Movie(TypedDict):
      title: str
      year: NotRequired[int]

  assert Movie.__required_keys__ == frozenset({'title', 'year'})  # yikes
  assert Movie.__optional_keys__ == frozenset()  # yikes


Reference Implementation
========================

The `mypy <http://www.mypy-lang.org/>`__
`0.930 <https://mypy-lang.blogspot.com/2021/12/mypy-0930-released.html>`__,
`pyright <https://github.com/Microsoft/pyright>`__
`1.1.117 <https://github.com/microsoft/pyright/commit/7ed245b1845173090c6404e49912e8cbfb3417c8>`__,
and `pyanalyze <https://github.com/quora/pyanalyze>`__
`0.4.0 <https://pyanalyze.readthedocs.io/en/latest/changelog.html#version-0-4-0-november-18-2021>`__
type checkers support ``Required`` and ``NotRequired``.

A reference implementation of the runtime component is provided in the
`typing_extensions <https://github.com/python/typing/tree/master/typing_extensions>`__
module.


Rejected Ideas
==============

Special syntax around the *key* of a TypedDict item
---------------------------------------------------

::

  class MyThing(TypedDict):
      opt1?: str  # may not exist, but if exists, value is string
      opt2: Optional[str]  # always exists, but may have None value

This notation would require Python grammar changes and it is not
believed that marking TypedDict items as required or potentially-missing
would meet the high bar required to make such grammar changes.

::

  class MyThing(TypedDict):
      Optional[opt1]: str  # may not exist, but if exists, value is string
      opt2: Optional[str]  # always exists, but may have None value

This notation causes ``Optional[]`` to take on different meanings depending
on where it is positioned, which is inconsistent and confusing.

Also, “let’s just not put funny syntax before the colon.” [1]_


Marking required or potentially-missing keys with an operator
-------------------------------------------------------------

We could use unary ``+`` as shorthand to mark a required key, unary
``-`` to mark a potentially-missing key, or unary ``~`` to mark a key
with opposite-of-normal totality:

::

  class MyThing(TypedDict, total=False):
      req1: +int    # + means a required key, or Required[]
      opt1: str
      req2: +float

  class MyThing(TypedDict):
      req1: int
      opt1: -str    # - means a potentially-missing key, or NotRequired[]
      req2: float

  class MyThing(TypedDict):
      req1: int
      opt1: ~str    # ~ means a opposite-of-normal-totality key
      req2: float

Such operators could be implemented on ``type`` via the ``__pos__``,
``__neg__`` and ``__invert__`` special methods without modifying the
grammar.

It was decided that it would be prudent to introduce long-form notation
(i.e. ``Required[]`` and ``NotRequired[]``) before introducing
any short-form notation. Future PEPs may reconsider introducing this
or other short-form notation options.

Note when reconsidering introducing this short-form notation that
``+``, ``-``, and ``~`` already have existing meanings in the Python
typing world: covariant, contravariant, and invariant:

::

  >>> from typing import TypeVar
  >>> (TypeVar('T', covariant=True), TypeVar('U', contravariant=True), TypeVar('V'))
  (+T, -U, ~V)


Marking absence of a value with a special constant
--------------------------------------------------

We could introduce a new type-level constant which signals the absence
of a value when used as a union member, similar to JavaScript’s
``undefined`` type, perhaps called ``Missing``:

::

  class MyThing(TypedDict):
      req1: int
      opt1: str|Missing
      req2: float

Such a ``Missing`` constant could also be used for other scenarios such
as the type of a variable which is only conditionally defined:

::

  class MyClass:
      attr: int|Missing

      def __init__(self, set_attr: bool) -> None:
          if set_attr:
              self.attr = 10

::

  def foo(set_attr: bool) -> None:
      if set_attr:
          attr = 10
      reveal_type(attr)  # int|Missing

Misalignment with how unions apply to values
''''''''''''''''''''''''''''''''''''''''''''

However this use of ``...|Missing``, equivalent to
``Union[..., Missing]``, doesn’t align well with what a union normally
means: ``Union[...]`` always describes the type of a *value* that is
present. By contrast missingness or non-totality is a property of a
*variable* instead. Current precedent for marking properties of a
variable include ``Final[...]`` and ``ClassVar[...]``, which the
proposal for ``Required[...]`` is aligned with.

Misalignment with how unions are subdivided
'''''''''''''''''''''''''''''''''''''''''''

Furthermore the use of ``Union[..., Missing]`` doesn’t align with the
usual ways that union values are broken down: Normally you can eliminate
components of a union type using ``isinstance`` checks:

::

  class Packet:
      data: Union[str, bytes]

  def send_data(packet: Packet) -> None:
      if isinstance(packet.data, str):
          reveal_type(packet.data)  # str
          packet_bytes = packet.data.encode('utf-8')
      else:
          reveal_type(packet.data)  # bytes
          packet_bytes = packet.data
      socket.send(packet_bytes)

However if we were to allow ``Union[..., Missing]`` you’d either have to
eliminate the ``Missing`` case with ``hasattr`` for object attributes:

::

  class Packet:
      data: Union[str, Missing]

  def send_data(packet: Packet) -> None:
      if hasattr(packet, 'data'):
          reveal_type(packet.data)  # str
          packet_bytes = packet.data.encode('utf-8')
      else:
          reveal_type(packet.data)  # Missing? error?
          packet_bytes = b''
      socket.send(packet_bytes)

or a check against ``locals()`` for local variables:

::

  def send_data(packet_data: Optional[str]) -> None:
      packet_bytes: Union[str, Missing]
      if packet_data is not None:
          packet_bytes = packet.data.encode('utf-8')

      if 'packet_bytes' in locals():
          reveal_type(packet_bytes)  # bytes
          socket.send(packet_bytes)
      else:
          reveal_type(packet_bytes)  # Missing? error?

or a check via other means, such as against ``globals()`` for global
variables:

::

  warning: Union[str, Missing]
  import sys
  if sys.version_info < (3, 6):
      warning = 'Your version of Python is unsupported!'

  if 'warning' in globals():
      reveal_type(warning)  # str
      print(warning)
  else:
      reveal_type(warning)  # Missing? error?

Weird and inconsistent. ``Missing`` is not really a value at all; it’s
an absence of definition and such an absence should be treated
specially.

Difficult to implement
''''''''''''''''''''''

Eric Traut from the Pyright type checker team has stated that
implementing a ``Union[..., Missing]``-style notation would be
difficult. [2]_

Introduces a second null-like value into Python
'''''''''''''''''''''''''''''''''''''''''''''''

Defining a new ``Missing`` type-level constant would be very close to
introducing a new ``Missing`` value-level constant at runtime, creating
a second null-like runtime value in addition to ``None``. Having two
different null-like constants in Python (``None`` and ``Missing``) would
be confusing. Many newcomers to JavaScript already have difficulty
distinguishing between its analogous constants ``null`` and
``undefined``.


Replace Optional with Nullable. Repurpose Optional to mean “optional item”.
---------------------------------------------------------------------------

``Optional[]`` is too ubiquitous to deprecate, although use of it
*may* fade over time in favor of the ``T|None`` notation specified by :pep:`604`.


Change Optional to mean “optional item” in certain contexts instead of “nullable”
---------------------------------------------------------------------------------

Consider the use of a special flag on a TypedDict definition to alter
the interpretation of ``Optional`` inside the TypedDict to mean
“optional item” rather than its usual meaning of “nullable”:

::

  class MyThing(TypedDict, optional_as_missing=True):
      req1: int
      opt1: Optional[str]

or:

::

  class MyThing(TypedDict, optional_as_nullable=False):
      req1: int
      opt1: Optional[str]

This would add more confusion for users because it would mean that in
*some* contexts the meaning of ``Optional[]`` is different than in
other contexts, and it would be easy to overlook the flag.


Various synonyms for “potentially-missing item”
-----------------------------------------------

-  Omittable – too easy to confuse with optional
-  OptionalItem, OptionalKey – two words; too easy to confuse with
  optional
-  MayExist, MissingOk – two words
-  Droppable – too similar to Rust’s ``Drop``, which means something
  different
-  Potential – too vague
-  Open – sounds like applies to an entire structure rather then to an
  item
-  Excludable
-  Checked


References
==========

. [1] https://mail.python.org/archives/list/typing-sig@python.org/message/4I3GPIWDUKV6GUCHDMORGUGRE4F4SXGR/

. [2] https://mail.python.org/archives/list/typing-sig@python.org/message/S2VJSVG6WCIWPBZ54BOJPG56KXVSLZK6/

. [3] https://bugs.python.org/issue46491

Copyright
=========

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.

You are viewing proxied material from gopher.fnord.one. The copyright of proxied material belongs to its original authors. Any comments or complaints in relation to proxied material should be directed to the original authors of the content concerned. Please see the disclaimer for more details.