Batch INSERT statements in executemany to reduce round trips by srchilukoori · Pull Request #605 · trinodb/trino-python-client

srchilukoori · 2026-05-25T15:43:42Z

Description

Fixes #602.

Cursor.executemany() executes INSERT...VALUES one row at a time.
For 1,000 rows that means 1,000 HTTP round trips to Trino.

This adds a regex check for INSERT...VALUES patterns and batches
rows into multi-row VALUES clauses, 100 per request. Non-INSERT
statements still use the existing row-by-row path. Parameters are
formatted with the existing _format_prepared_param() method.

Scenario	Row-by-row	Batched	Speedup
500 rows x 2 cols	8.9s	0.23s	~39x
1,000 rows x 25 cols (mixed types)	timeout	3.1s (322 rows/s)	—

Non-technical explanation

Bulk inserts via SQLAlchemy text() or cursor.executemany() now
send fewer queries to Trino instead of one per row.

Release notes

( ) This is not user-visible or docs only and no release notes are required.
( ) Release notes are required, please propose a release note for me.
(x) Release notes are required, with the following suggested text:

* Batch INSERT...VALUES in `executemany()` into multi-row statements to reduce HTTP round trips. ({issue}`602`)

cla-bot · 2026-05-25T15:43:45Z

Thank you for your pull request and welcome to our community. We could not parse the GitHub identity of the following contributors: Sadha Chilukoori.
This is most likely caused by a git client misconfiguration; please make sure to:

check if your git client is configured with an email to sign commits git config --list | grep email
If not, set it up using git config --global user.email email@example.com
Make sure that the git commit email is configured in your GitHub account settings, see https://github.com/settings/emails

cla-bot · 2026-05-26T15:52:55Z

Thank you for your pull request and welcome to the Trino community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. Continue to work with us on the review and improvements in this PR, and submit the signed CLA to cla@trino.io. Photos, scans, or digitally-signed PDF files are all suitable. Processing may take a few days. The CLA needs to be on file before we merge your changes. For more information, see https://github.com/trinodb/cla

srchilukoori · 2026-06-03T15:21:11Z

@hashhar need your review on this PR

hashhar

Excellent feature and thanks for the tests.
I need to think a bit about if there are cases where this may do the wrong thing and hence should require an opt-in behaviour for a few releases maybe.

srchilukoori · 2026-06-17T17:00:15Z

Excellent feature and thanks for the tests. I need to think a bit about if there are cases where this may do the wrong thing and hence should require an opt-in behaviour for a few releases maybe.

@hashhar I added it as opt-in, with a default of false, as suggested. I'll also put together a plan for the edge cases.

Add opt-in experimental_batch_executemany connection parameter that batches INSERT...VALUES in executemany() into multi-row statements (100 rows per request) instead of one HTTP round trip per row. Default is False to preserve existing behavior. Batching changes partial failure semantics and rowcount reporting, so it is opt-in for now.

srchilukoori · 2026-07-08T16:16:53Z

@hashhar please review the updated PR

hashhar · 2026-07-10T18:03:05Z

+    re.IGNORECASE | re.DOTALL,
+)
+
+_EXECUTEMANY_BATCH_SIZE = 100


this needs to be configurable. The actual limit is the query text size configured on the server. Max on the server is 16MB IIRC but default is lower.

hashhar · 2026-07-10T18:22:29Z

 logger = trino.logging.get_logger(__name__)

+_INSERT_VALUES_RE = re.compile(
+    r"\A(\s*INSERT\s+INTO\s+.+\bVALUES)\s*\([^)]+\)\s*\Z",


seems to match all parens grouped things not just things with placeholders? This is only safe to do if ONLY placeholders are present in the VALUES.

Also tests for the false positives we want to avoid (e.g. placeholder + literal) are missing which would've caught this

hashhar · 2026-07-10T18:26:25Z

+                formatted = ", ".join(self._format_prepared_param(p) for p in params)
+                value_rows.append("(%s)" % formatted)
+            sql = "%s %s" % (prefix, ", ".join(value_rows))
+            self._query = trino.client.TrinoQuery(


this bypasses the SegmentCursor.

hashhar · 2026-07-10T18:27:00Z

        self._transaction = None
        self.legacy_primitive_types = legacy_primitive_types
        self.legacy_prepared_statements = legacy_prepared_statements
+        self.experimental_batch_executemany = experimental_batch_executemany


also update sqlalchemy _url()? Also add test for sqlalchemy.

hashhar · 2026-07-10T18:27:18Z

+                legacy_primitive_types=self._legacy_primitive_types)
+            self._iterator = iter(self._query.execute())
+            if self._query.update_type is None:
+                raise NotSupportedError("Query must return update type")


this is not tested

hashhar · 2026-07-10T18:31:28Z

+        finally:
+            trino.client.TrinoQuery = original
+
+    def test_connection_threads_flag_to_cursor(self):


weird test since we assign value ourself and then we assert. Should it not instead be testing by creating the connection with kwargs and verifying connection state?

hashhar · 2026-07-10T18:52:22Z

+
+        match = _INSERT_VALUES_RE.match(operation.strip())
+        if match and self._experimental_batch_executemany:
+            return self._executemany_batch_insert(match.group(1), seq_of_params)


shouldn't the rowcount be accumulated across all batches and returned? this just returns value from last batch?

srchilukoori force-pushed the batch-executemany-insert branch from 35385b1 to 92069b8 Compare May 26, 2026 15:52

srchilukoori closed this Jun 3, 2026

srchilukoori reopened this Jun 3, 2026

srchilukoori force-pushed the batch-executemany-insert branch from 92069b8 to c8a8722 Compare June 3, 2026 15:17

cla-bot Bot added the cla-signed label Jun 3, 2026

hashhar requested a review from damian3031 June 15, 2026 11:06

hashhar reviewed Jun 15, 2026

View reviewed changes

srchilukoori force-pushed the batch-executemany-insert branch from c8a8722 to e189b92 Compare June 17, 2026 16:40

srchilukoori force-pushed the batch-executemany-insert branch from e189b92 to 6f0c86a Compare June 17, 2026 17:34

Fix flake8 line-length violation in SQLAlchemy dialect

d0fcd2d

srchilukoori requested a review from hashhar July 8, 2026 16:16

hashhar requested changes Jul 10, 2026

View reviewed changes

hashhar reviewed Jul 10, 2026

View reviewed changes

hashhar requested a review from azawlocki-sbdt July 10, 2026 18:45

hashhar reviewed Jul 10, 2026

View reviewed changes

hashhar assigned srchilukoori Jul 10, 2026

Uh oh!

Conversation

srchilukoori commented May 25, 2026

Description

Non-technical explanation

Release notes

Uh oh!

cla-bot Bot commented May 25, 2026

Uh oh!

cla-bot Bot commented May 26, 2026

Uh oh!

srchilukoori commented Jun 3, 2026

Uh oh!

hashhar left a comment

Choose a reason for hiding this comment

Uh oh!

srchilukoori commented Jun 17, 2026

Uh oh!

srchilukoori commented Jul 8, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

2 participants