Skip to content

fix(python/adbc_driver_manager): reimport PyArrow if installed mid-session - #4641

Merged
lidavidm merged 4 commits into
apache:mainfrom
amoeba:fix/lazy-pyarrow-import
Aug 2, 2026
Merged

fix(python/adbc_driver_manager): reimport PyArrow if installed mid-session#4641
lidavidm merged 4 commits into
apache:mainfrom
amoeba:fix/lazy-pyarrow-import

Conversation

@amoeba

@amoeba amoeba commented Jul 31, 2026

Copy link
Copy Markdown
Member

Currently, with the dbapi module in the Python driver manager, if I am using a REPL or notebook session started without PyArrow installed, I have to restart my session to pick PyArrow up so I can use PyArrow-enabled methods. This is because the module attempts to import PyArrow at startup and sets a persistent flag that never changes after import.

It would be nicer if we could rescue the user so they can install PyArrow in their environment while a REPL or notebook is running, make the call again, and have it work.

Steps to reproduce:

  1. Create an environment without PyArrow:
python3 -m venv .venv
source .venv/bin/activate
pip install adbc_driver_manager
  1. See we get the expected error when we try to use a PyArrow-enabled routine:
.venv $ python
Python 3.14.0 (main, Oct 14 2025, 21:10:22) [Clang 20.1.4 ] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from adbc_driver_manager import dbapi
>>> con = dbapi.connect("sqlite://:memory:")
>>> cur = con.cursor()
>>> cur.execute("SELECT 1").fetch_arrow_table()
Traceback (most recent call last):
  File "<python-input-3>", line 1, in <module>
    cur.execute("SELECT 1").fetch_arrow_table()
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/Users/bryce/src/apache/arrow-adbc/test_env/.venv/lib/python3.14/site-packages/adbc_driver_manager/dbapi.py", line 1355, in fetch_arrow_table
    return self._results.fetch_arrow_table()
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/Users/bryce/src/apache/arrow-adbc/test_env/.venv/lib/python3.14/site-packages/adbc_driver_manager/dbapi.py", line 1532, in fetch_arrow_table
    return _blocking_call(self.reader.read_all, (), {}, self._stmt.cancel)
                          ^^^^^^^^^^^
  File "/Users/bryce/src/apache/arrow-adbc/test_env/.venv/lib/python3.14/site-packages/adbc_driver_manager/dbapi.py", line 1466, in reader
    _requires_pyarrow()
    ~~~~~~~~~~~~~~~~~^^
  File "/Users/bryce/src/apache/arrow-adbc/test_env/.venv/lib/python3.14/site-packages/adbc_driver_manager/dbapi.py", line 1588, in _requires_pyarrow
    raise ProgrammingError(
    ...<2 lines>...
    )
adbc_driver_manager.ProgrammingError: This API requires PyArrow to be installed
  1. Try to fix it in another shell
source .venv/bin/activate
pip install pyarrow
  1. Run (2) again and get the same error

The Python driver manager's dbapi module guards PyArrow-enabled routines with _requires_pyarrow(). This PR tweaks that helper and lets it attempt to re-import PyArrow but only if PyArrow wasn't enabled at import time.

This PR was generated using an LLM.

@amoeba
amoeba requested a review from lidavidm as a code owner July 31, 2026 21:33
@amoeba amoeba changed the title fix(python,dbapi): reimport PyArrow if installed mid-session fix(python/adbc_driver_manager): reimport PyArrow if installed mid-session Jul 31, 2026
@lidavidm
lidavidm merged commit b78ca94 into apache:main Aug 2, 2026
76 of 77 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants