Merge branch 'main' into fix-module-file-docs

Author: RaghunandanKumar

.github/CODEOWNERS

@@ -63,7 +63,7 @@
 .azure-pipelines/             @AA-Turner
 
 # GitHub & related scripts
-.github/                                    @ezio-melotti @hugovk @AA-Turner @webknjaz
+.github/                                    @ezio-melotti @hugovk @AA-Turner @webknjaz @itamaro
 Tools/build/compute-changes.py              @AA-Turner @hugovk @webknjaz
 Lib/test/test_tools/test_compute_changes.py @AA-Turner @hugovk @webknjaz
 Tools/build/verify_ensurepip_wheels.py      @AA-Turner @pfmoore @pradyunsg
@@ -73,19 +73,19 @@ Tools/build/verify_ensurepip_wheels.py      @AA-Turner @pfmoore @pradyunsg
 .ruff.toml                    @hugovk @AlexWaygood @AA-Turner
 
 # Patchcheck
-Tools/patchcheck/             @AA-Turner
+Tools/patchcheck/             @AA-Turner @itamaro
 
 
 # ----------------------------------------------------------------------------
 # Build System
 # ----------------------------------------------------------------------------
 
 # Autotools
-configure*                       @erlend-aasland @corona10 @AA-Turner @emmatyping
-Makefile.pre.in                  @erlend-aasland @AA-Turner @emmatyping
-Modules/makesetup                @erlend-aasland @AA-Turner @emmatyping
-Modules/Setup*                   @erlend-aasland @AA-Turner @emmatyping
-Tools/build/regen-configure.sh   @AA-Turner
+configure*                       @erlend-aasland @corona10 @AA-Turner @emmatyping @itamaro
+Makefile.pre.in                  @erlend-aasland @AA-Turner @emmatyping @itamaro
+Modules/makesetup                @erlend-aasland @AA-Turner @emmatyping @itamaro
+Modules/Setup*                   @erlend-aasland @AA-Turner @emmatyping @itamaro
+Tools/build/regen-configure.sh   @AA-Turner @itamaro
 
 # generate-build-details
 Tools/build/generate-build-details.py   @FFY00

.github/SECURITY.md

@@ -1,17 +1,24 @@
 # Security Policy
 
-## Supported Versions
+Python [provides a security policy and threat model](https://devguide.python.org/security/policy/)
+in the Python Development Guide documenting what bugs are vulnerabilities,
+how to structure reports, and what versions of Python accept reports.
 
-The Python team applies security fixes according to the table
-in [the devguide](
-https://devguide.python.org/versions/#supported-versions
-).
+Python Security Response Team (PSRT) members
+balance security work against many other responsibilities. Please be thoughtful
+about the time and attention your report requires. Repeated failure to respect
+the security policy will result in future reports being rejected, or the
+reporter being banned from the ``python`` GitHub organization, regardless of
+technical merit.
 
 ## Reporting a Vulnerability
 
-Please read the guidelines on reporting security issues [on the
-official website](https://www.python.org/dev/security/) for
-instructions on how to report a security-related problem to
-the Python team responsibly.
+The [Python security policy](https://devguide.python.org/security/policy/)
+documents [how to submit a vulnerability report](https://devguide.python.org/security/policy/#how-to-submit-a-vulnerability-report)
+using GitHub Security Advisories. Please read the security policy
+prior to filing a vulnerability report, especially the section on [what information to
+include and exclude](https://devguide.python.org/security/policy/#what-to-include-and-how-to-structure-a-vulnerability-report)
+in vulnerability reports. Following the security policy means the PSRT can
+quickly and efficiently triage your report, not following the security policy
+will only delay triaging your report.
 
-To reach the response team, email `security at python dot org`.

Doc/c-api/exceptions.rst

@@ -412,7 +412,7 @@ an error value).
 
 .. c:function:: int PyErr_WarnFormat(PyObject *category, Py_ssize_t stack_level, const char *format, ...)
 
-   Function similar to :c:func:`PyErr_WarnEx`, but use
+   Function similar to :c:func:`PyErr_WarnEx`, but uses
    :c:func:`PyUnicode_FromFormat` to format the warning message.  *format* is
    an ASCII-encoded string.
 
@@ -1392,7 +1392,7 @@ Tracebacks
 
    This function will return ``NULL`` on success, or an error message on error.
 
-   This function is meant to debug debug situations such as segfaults, fatal
+   This function is meant to debug situations such as segfaults, fatal
    errors, and similar. It calls :c:func:`PyUnstable_DumpTraceback` for each
    thread. It only writes the tracebacks of the first *max_threads* threads,
    further output is truncated with the line ``...``. If *max_threads* is 0, the

Doc/c-api/threads.rst

@@ -736,10 +736,10 @@ Low-level APIs
 .. c:function:: PyObject* PyThreadState_GetDict()
 
    Return a dictionary in which extensions can store thread-specific state
-   information.  Each extension should use a unique key to use to store state in
+   information.  Each extension should use a unique key to store a state in
    the dictionary.  It is okay to call this function when no :term:`thread state`
    is :term:`attached <attached thread state>`. If this function returns
-   ``NULL``, no exception has been raised and the caller should assume no
+   ``NULL`` and no exception has been raised, then the caller should assume no
    thread state is attached.
 
 

Doc/c-api/typehints.rst

@@ -31,7 +31,7 @@ two types exist -- :ref:`GenericAlias <types-genericalias>` and
       static PyMethodDef my_obj_methods[] = {
           // Other methods.
           ...
-          {"__class_getitem__", Py_GenericAlias, METH_O|METH_CLASS, "See PEP 585"}
+          {"__class_getitem__", Py_GenericAlias, METH_O|METH_CLASS, "my_obj is generic over its contained type"}
           ...
       }
 

Doc/conf.py

@@ -349,8 +349,6 @@
   \sphinxstrong{Python Software Foundation}\\
   Email: \sphinxemail{docs@python.org}
 }
-\let\Verbatim=\OriginalVerbatim
-\let\endVerbatim=\endOriginalVerbatim
 \setcounter{tocdepth}{2}
 ''',
     # The paper size ('letterpaper' or 'a4paper').

Doc/deprecations/pending-removal-in-3.21.rst

@@ -1,6 +1,15 @@
 Pending removal in Python 3.21
 ------------------------------
 
+* :mod:`abc`
+
+   * Soft-deprecated since Python 3.3 :class:`abc.abstractclassmethod`,
+     :class:`abc.abstractstaticmethod`, and :class:`abc.abstractproperty`
+     now raise a :exc:`DeprecationWarning`.
+     These classes will be removed in Python 3.21, instead
+     use :func:`abc.abstractmethod` with :func:`classmethod`,
+     :func:`staticmethod`, and :class:`property` respectively.
+
 * :mod:`ast`:
 
   * Classes ``slice``, ``Index``, ``ExtSlice``, ``Suite``, ``Param``,

Doc/howto/a-conceptual-overview-of-asyncio.rst

@@ -115,7 +115,7 @@ The terms "coroutine function" and "coroutine object" are often conflated
 as coroutine.
 That can be confusing!
 In this article, coroutine specifically refers to a coroutine object, or more
-precisely, an instance of :data:`types.CoroutineType` (native coroutine).
+precisely, an instance of :class:`types.CoroutineType` (native coroutine).
 Note that coroutines can also exist as instances of
 :class:`collections.abc.Coroutine` -- a distinction that matters for type
 checking.

Doc/howto/free-threading-python.rst

@@ -165,3 +165,132 @@ to false.  If the flag is true then the :class:`warnings.catch_warnings`
 context manager uses a context variable for warning filters.  If the flag is
 false then :class:`~warnings.catch_warnings` modifies the global filters list,
 which is not thread-safe.  See the :mod:`warnings` module for more details.
+
+
+Increased memory usage
+----------------------
+
+The free-threaded build will typically use more memory compared to the default
+build.  There are multiple reasons for this, mostly due to design decisions.
+
+
+All interned strings are immortal
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+For modern Python versions (since version 2.3), interning a string (e.g. with
+:func:`sys.intern`) does not cause it to become immortal.  Instead, if the last
+reference to that string disappears, it will be removed from the interned
+string table.  This is not the case for the free-threaded build and any interned
+string will become immortal, surviving until interpreter shutdown.
+
+
+Non-GC objects have a larger object header
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The free-threaded build uses a different :c:type:`PyObject` structure.  Instead
+of having the GC related information allocated before the :c:type:`PyObject`
+structure, like in the default build, the GC related info is part of the normal
+object header.  For example, on the AMD64 platform, ``None`` uses 32 bytes on
+the free-threaded build vs 16 bytes for the default build.  GC objects (such as
+dicts and lists) are the same size for both builds since the free-threaded
+build does not use additional space for the GC info.
+
+
+QSBR can delay freeing of memory
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+In order to safely implement lock-free data structures, a safe memory
+reclamation (SMR) scheme is used, known as quiescent state-based reclamation
+(QSBR).  This means that the memory backing data structures allowing lock-free
+access will use QSBR, which defers the free operation, rather than immediately
+freeing the memory.  Two examples of these data structures are the list object
+and the dictionary keys object.  See ``InternalDocs/qsbr.md`` in the CPython
+source tree for more details on how QSBR is implemented.  Running
+:func:`gc.collect` should cause all memory being held by QSBR to be actually
+freed.  Note that even when QSBR frees the memory, the underlying memory
+allocator may not immediately return that memory to the OS and so the resident
+set size (RSS) of the process might not decrease.
+
+
+mimalloc allocator vs pymalloc
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The default build will normally use the "pymalloc" memory allocator for small
+allocations (512 bytes or smaller).  The free-threaded build does not use
+pymalloc and allocates all Python objects using the "mimalloc" allocator.  The
+pymalloc allocator has the following properties that help keep memory usage
+low: small per-allocated-block overhead, effective memory fragmentation
+prevention, and quick return of free memory to the operating system.  The
+mimalloc allocator does quite well in these respects as well but can have some
+more overhead.
+
+In the free-threaded build, mimalloc manages memory in a number of separate
+heaps (currently four).  For example, all GC supporting objects are allocated
+from their own heap.  Using separate heaps means that free memory in one heap
+cannot be used for an allocation that uses another heap.  Also, some heaps are
+configured to use QSBR (quiescent-state based reclamation) when freeing the
+memory that backs up the heap (known as "pages" in mimalloc terminology).  The
+use of QSBR creates a delay between all memory blocks for a page being freed
+and the memory page being released, either for new allocations or back to the
+OS.
+
+The mimalloc allocator also defers returning freed memory back to the OS.  You
+can reduce that delay by setting the environment variable
+:envvar:`!MIMALLOC_PURGE_DELAY` to ``0``.  Note that this will likely reduce
+the performance of the allocator.
+
+
+Free-threaded reference counting can cause objects to live longer
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+In the default build, when an object's reference count reaches zero, it is
+normally deallocated.  The free-threaded build uses "biased reference
+counting", with a fast-path for objects "owned" by the current thread and a
+slow path for other objects.  See :pep:`703` for additional details.  Any time
+an object's reference count ends up in a "queued" state, deallocation can be
+deferred.  The queued state is cleared from the "eval breaker" section of the
+bytecode evaluator.
+
+The free-threaded build also allows a different mode of reference counting,
+known as "deferred reference counting".  This mode is enabled by setting a flag
+on a per-object basis.  Deferred reference counting is enabled for the
+following types:
+
+* module objects
+* module top-level functions
+* class methods defined in the class scope
+* descriptor objects
+* thread-local objects, created by :class:`threading.local`
+
+When deferred reference counting is enabled, references from Python function
+stacks are not added to the reference count.  This scheme reduces the overhead
+of reference counting, especially for objects used from multiple threads.
+Because the stack references are not counted, objects with deferred reference
+counting are not immediately freed when their internal reference count goes to
+zero.  Instead, they are examined by the next GC run and, if no stack
+references to them are found, they are freed.  This means these objects are
+freed by the GC and not when their reference count goes to zero, as is typical.
+
+
+Per-thread reference counting can delay freeing objects
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+To avoid contention on the reference count fields of frequently shared
+objects, the free-threaded build also uses "per-thread reference counting"
+for a few selected object types.  Rather than updating a single shared
+reference count, each thread maintains its own local reference count array,
+indexed by a unique id assigned to the object.  The true reference count is
+only computed by summing the per-thread counts when the object's local
+count drops to zero.  Per-thread reference counting is currently used for:
+
+* heap type objects (classes created in Python)
+* code objects
+* the ``__dict__`` of module objects
+
+Because the per-thread counts must be merged back to the object before it
+can be deallocated, objects using per-thread reference counting are
+typically freed later than they would be in the default build.  In
+particular, such an object is usually not freed until the thread that
+referenced it reaches a safe point (for example, in the "eval breaker"
+section of the bytecode evaluator) or exits.  Running :func:`gc.collect`
+will merge the per-thread counts and allow these objects to be freed.

Doc/library/_thread.rst

@@ -36,11 +36,6 @@ This module defines the following constants and functions:
       This is now a synonym of the built-in :exc:`RuntimeError`.
 
 
-.. data:: LockType
-
-   This is the type of lock objects.
-
-
 .. function:: start_new_thread(function, args[, kwargs])
 
    Start a new thread and return its identifier.  The thread executes the
@@ -162,58 +157,66 @@ This module defines the following constants and functions:
    .. versionadded:: 3.2
 
 
-Lock objects have the following methods:
+.. raw:: html
+
+   <!-- Keep the old URL fragments working (see gh-89554) -->
+   <span id='thread.lock.acquire'></span>
+   <span id='thread.lock.release'></span>
+   <span id='thread.lock.locked'></span>
 
+.. class:: LockType
 
-.. method:: lock.acquire(blocking=True, timeout=-1)
+   This is the type of lock objects.
 
-   Without any optional argument, this method acquires the lock unconditionally, if
-   necessary waiting until it is released by another thread (only one thread at a
-   time can acquire a lock --- that's their reason for existence).
+   Lock objects have the following methods:
 
-   If the *blocking* argument is present, the action depends on its
-   value: if it is false, the lock is only acquired if it can be acquired
-   immediately without waiting, while if it is true, the lock is acquired
-   unconditionally as above.
+   .. method:: acquire(blocking=True, timeout=-1)
 
-   If the floating-point *timeout* argument is present and positive, it
-   specifies the maximum wait time in seconds before returning.  A negative
-   *timeout* argument specifies an unbounded wait.  You cannot specify
-   a *timeout* if *blocking* is false.
+      Without any optional argument, this method acquires the lock unconditionally, if
+      necessary waiting until it is released by another thread (only one thread at a
+      time can acquire a lock --- that's their reason for existence).
 
-   The return value is ``True`` if the lock is acquired successfully,
-   ``False`` if not.
+      If the *blocking* argument is present, the action depends on its
+      value: if it is false, the lock is only acquired if it can be acquired
+      immediately without waiting, while if it is true, the lock is acquired
+      unconditionally as above.
 
-   .. versionchanged:: 3.2
-      The *timeout* parameter is new.
+      If the floating-point *timeout* argument is present and positive, it
+      specifies the maximum wait time in seconds before returning.  A negative
+      *timeout* argument specifies an unbounded wait.  You cannot specify
+      a *timeout* if *blocking* is false.
 
-   .. versionchanged:: 3.2
-      Lock acquires can now be interrupted by signals on POSIX.
+      The return value is ``True`` if the lock is acquired successfully,
+      ``False`` if not.
 
-   .. versionchanged:: 3.14
-      Lock acquires can now be interrupted by signals on Windows.
+      .. versionchanged:: 3.2
+         The *timeout* parameter is new.
 
+      .. versionchanged:: 3.2
+         Lock acquires can now be interrupted by signals on POSIX.
 
-.. method:: lock.release()
+      .. versionchanged:: 3.14
+         Lock acquires can now be interrupted by signals on Windows.
 
-   Releases the lock.  The lock must have been acquired earlier, but not
-   necessarily by the same thread.
+   .. method:: release()
 
+      Releases the lock.  The lock must have been acquired earlier, but not
+      necessarily by the same thread.
 
-.. method:: lock.locked()
+   .. method:: locked()
 
-   Return the status of the lock: ``True`` if it has been acquired by some thread,
-   ``False`` if not.
+      Return the status of the lock: ``True`` if it has been acquired by some thread,
+      ``False`` if not.
 
-In addition to these methods, lock objects can also be used via the
-:keyword:`with` statement, e.g.::
+   In addition to these methods, lock objects can also be used via the
+   :keyword:`with` statement, e.g.::
 
-   import _thread
+      import _thread
 
-   a_lock = _thread.allocate_lock()
+      a_lock = _thread.allocate_lock()
 
-   with a_lock:
-       print("a_lock is locked while this executes")
+      with a_lock:
+          print("a_lock is locked while this executes")
 
 **Caveats:**