Skip to content

fix(python): Avoid exceptions when decoding non-UTF-8 data from string attrs - #5485

Draft
nrusch wants to merge 1 commit into
AcademySoftwareFoundation:mainfrom
nrusch:py_string_safe_decode
Draft

nrusch wants to merge 1 commit into
AcademySoftwareFoundation:mainfrom
nrusch:py_string_safe_decode

Conversation

@nrusch

@nrusch nrusch commented Sep 19, 2026

Copy link
Copy Markdown
Contributor

Add a new py_str_escaped helper function to handle conversion of string-like C++ types to py:str without raising a UnicodeDecodeError for non-UTF-8 input data.

This function is a fairly simple wrapper that calls the PyUnicode_DecodeUTF8 with the 'surrogateescape' error handling scheme (described at https://docs.python.org/3/library/codecs.html#error-handlers) and transfers ownership to the binding-specific object type.

Fixes #3856

Tests

Only manual tests so far, but I plan to add some test cases to the harness.

Checklist:

  • I have read the guidelines on contributions and code review procedures.
  • I have read the Policy on AI Coding Assistants
    and if I used AI coding assistants, I have an Assisted-by: TOOL / MODEL
    line in the pull request description above.
  • I have updated the documentation if my PR adds features or changes
    behavior.
  • I am sure that this PR's changes are tested in the testsuite.
  • I have run and passed the testsuite in CI before submitting the
    PR, by pushing the changes to my fork and seeing that the automated CI
    passed there. (Exceptions: If most tests pass and you can't figure out why
    the remaining ones fail, it's ok to submit the PR and ask for help. Or if
    any failures seem entirely unrelated to your change; sometimes things break
    on the GitHub runners.)
  • My code follows the prevailing code style of this project and I
    fixed any problems reported by the clang-format CI test.
  • If I added or modified a public C++ API call, I have also amended the
    corresponding Python bindings. If altering ImageBufAlgo functions, I also
    exposed the new functionality as oiiotool options.

Signed-off-by: Nathan Rusch <nrusch@users.noreply.github.com>
Comment thread src/python/py_oiio.h
Comment on lines +481 to +491
#if defined(OIIO_PY_BACKEND_NANOBIND)
if (!py_str) {
py::raise_python_error();
}
return py::steal<py::str>(py_str);
#else
if (!py_str) {
throw py::error_already_set();
}
return py::reinterpret_steal<py::str>(py_str);
#endif

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I went with a single function definition containing implementation details for both backends, but it would be easy enough to just have two definitions in the oiio_py namespace with the Python API call duplicated if you prefer. Or we could split the difference and alias the exception type and handle stealing functions in py_backend.h in order to collapse this into one code path.

.OIIO_PY_PROP_RO("name",
[](const ParamValue& self) {
return oiio_py::str(self.name().string());
return py_str_escaped(self.name().string());

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Flagging that I've also wrapped the ParamValue.name accessor, because there's nothing that normalizes/sanitizes attribute names in the C++ API.

@nrusch nrusch changed the title python: Avoid exceptions when decoding non-UTF-8 data from string attrs fix(python): Avoid exceptions when decoding non-UTF-8 data from string attrs Sep 19, 2026
@lgritz

lgritz commented Sep 19, 2026

Copy link
Copy Markdown
Collaborator

Those wheel failures are unrelated. I just merged a fix that should take care of them. If you rebase on the current main, they should go away.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] python3.10 spec.get_string_attribute() throws a runtime exception with non-utf-8 string headers

2 participants