gh-148596: Fix callback return handling for subclassed ctypes types

[dplusj]

PR Summary: Fix handling of callback return values in ctypes

Fixes the handling of callback return values in ctypes when a callback returns an instance of a subclass of a simple ctypes type (e.g., a subclass of c_int).

Closes: gh-148596

---

Problem

When a Python callback created with CFUNCTYPE returns an instance of a subclassed simple ctypes type, the return value is passed directly to the corresponding setfunc in _ctypes.

Example Scenario:

class MyInt(ctypes.c_int):
    pass

The Issue:

1. setfunc expects a native Python value (such as int or float).
2. It does not accept ctypes instances.
3. This results in an internal TypeError, reported as an unraisable exception.
4. The C result buffer remains uninitialized, leading the caller to observe undefined or garbage values.

---

Cause

In Modules/_ctypes/callbacks.c::_CallPythonObject, the callback return value is written into the C result buffer via:

setfunc(mem, result, restype->size);

Unlike argument handling—which uses ConvParam() in callproc.c to unwrap ctypes instances into underlying Python values—this callback return path lacks the unwrapping step. This creates an inconsistency between how arguments and return values are processed.

---

Solution

The fix ensures that ctypes instances are unwrapped by accessing their .value attribute before being passed to setfunc.

Key Improvements:

• Correct Conversion: Subclassed ctypes simple types are converted to native Python values.
• Compatibility: setfunc receives input it can process.
• Stability: The result buffer is properly initialized.
• Consistency: Native Python return values remain unaffected, preserving existing behavior.

---

Behavior After Fix

import ctypes

class MyInt(ctypes.c_int):
    pass

@ctypes.CFUNCTYPE(MyInt, MyInt)
def identity(x):
    # Now correctly unwraps the MyInt instance
    return x

result = identity(MyInt(42))

assert isinstance(result, MyInt)
assert result.value == 42

---

Implementation Details

• Tests: Added a regression test to test_ctypes to verify that callbacks can safely return subclassed simple ctypes types.
• Alignment: This change aligns callback return value handling with standard argument conversion logic.

[python-cla-bot]

All commit authors signed the Contributor License Agreement.

[bedevere-app]

Most changes to Python require a NEWS entry. Add one using the blurb_it web app or the blurb command-line tool.

If this change has little impact on Python users, wait for a maintainer to apply the skip news label instead.

[bedevere-app]

Most changes to Python require a NEWS entry. Add one using the blurb_it web app or the blurb command-line tool.

If this change has little impact on Python users, wait for a maintainer to apply the skip news label instead.