RuntimeError: Event loop is closed / unable to perform operation on `<TCPTransport closed=True reading=False>`; the handler is closed

[eigenein]

Hi,

I'm splitting the issue from here: Kludex/uvicorn#1251.

Environment

• Python 3.10.2
• httpcore==0.14.7

Example

import asyncio
from time import sleep

from httpcore._async import AsyncConnectionPool

pool = AsyncConnectionPool(keepalive_expiry=1.0)


async def main():
    response = await pool.request("GET", "https://httpbin.org/get")
    assert response.status == 200


# First request creates a new connection within the event loop.
asyncio.run(main())

# Sleep for longer than the `keepalive_expiry` to make the pool close expired connections.
sleep(2.0)

# Sending another request from a different event loop.
# Why different? For me that's the case in an ASGI application (Gunicorn + Django),
# where ingoing requests may be handled in different event loops.
# I'm trying to keep a global connection pool in the app which results in calling
# `pool.request` from different event loops.
asyncio.run(main())

Expected

I assume that httpcore should silently ignore the error since we want the connection closed and it just isn't valid anymore.

Actual

Traceback (most recent call last):
  File "/Users/eigenein/GitHub/blitz-lab/src/mre.py", line 16, in <module>
    asyncio.run(main())
  File "/usr/local/Cellar/python@3.10/3.10.2/Frameworks/Python.framework/Versions/3.10/lib/python3.10/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/Cellar/python@3.10/3.10.2/Frameworks/Python.framework/Versions/3.10/lib/python3.10/asyncio/base_events.py", line 641, in run_until_complete
    return future.result()
  File "/Users/eigenein/GitHub/blitz-lab/src/mre.py", line 10, in main
    response = await pool.request("GET", "https://httpbin.org/get")
  File "/Users/eigenein/GitHub/blitz-lab/venv/lib/python3.10/site-packages/httpcore/_async/interfaces.py", line 41, in request
    response = await self.handle_async_request(request)
  File "/Users/eigenein/GitHub/blitz-lab/venv/lib/python3.10/site-packages/httpcore/_async/connection_pool.py", line 220, in handle_async_request
    await self._close_expired_connections()
  File "/Users/eigenein/GitHub/blitz-lab/venv/lib/python3.10/site-packages/httpcore/_async/connection_pool.py", line 188, in _close_expired_connections
    await connection.aclose()
  File "/Users/eigenein/GitHub/blitz-lab/venv/lib/python3.10/site-packages/httpcore/_async/connection.py", line 159, in aclose
    await self._connection.aclose()
  File "/Users/eigenein/GitHub/blitz-lab/venv/lib/python3.10/site-packages/httpcore/_async/http11.py", line 212, in aclose
    await self._network_stream.aclose()
  File "/Users/eigenein/GitHub/blitz-lab/venv/lib/python3.10/site-packages/httpcore/backends/asyncio.py", line 48, in aclose
    await self._stream.aclose()
  File "/Users/eigenein/GitHub/blitz-lab/venv/lib/python3.10/site-packages/anyio/streams/tls.py", line 168, in aclose
    await self.transport_stream.aclose()
  File "/Users/eigenein/GitHub/blitz-lab/venv/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 1163, in aclose
    self._transport.close()
  File "/usr/local/Cellar/python@3.10/3.10.2/Frameworks/Python.framework/Versions/3.10/lib/python3.10/asyncio/selector_events.py", line 700, in close
    self._loop.call_soon(self._call_connection_lost, None)
  File "/usr/local/Cellar/python@3.10/3.10.2/Frameworks/Python.framework/Versions/3.10/lib/python3.10/asyncio/base_events.py", line 745, in call_soon
    self._check_closed()
  File "/usr/local/Cellar/python@3.10/3.10.2/Frameworks/Python.framework/Versions/3.10/lib/python3.10/asyncio/base_events.py", line 510, in _check_closed
    raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed

With uvloop you get something like (quoting the differing part at the bottom):

  File "anyio/streams/tls.py", line 168, in aclose
    await self.transport_stream.aclose()
  File "anyio/_backends/_asyncio.py", line 1159, in aclose
    self._transport.write_eof()
  File "uvloop/handles/stream.pyx", line 696, in uvloop.loop.UVStream.write_eof
    self._ensure_alive()
  File "uvloop/handles/handle.pyx", line 159, in uvloop.loop.UVHandle._ensure_alive
    raise RuntimeError(
RuntimeError: unable to perform operation on <TCPTransport closed=True reading=False 0x55ee6d824e80>; the handler is closed

[lovelydinosaur]

👀 - I really wouldn't know what behaviour to expect with that usage.

I'd strongly recommend that you instantiate the pool within the async context, rather than outside it...

async def main():
    async with AsyncConnectionPool(keepalive_expiry=1.0) as pool:
        response = await pool.request("GET", "https://httpbin.org/get")
        assert response.status == 200

Why different? For me that's the case in an ASGI application (Gunicorn + Django),
where ingoing requests may be handled in different event loops.

Right. I don't know enough about Django under ASGI to figure out if there's some top-level async context where you can instantiate an httpx connection pool, that'll be properly shared. Might be something to figure out on one of their discussion groups.