refactor: rest client by NguyenHoangSon96 · Pull Request #217 · InfluxCommunity/influxdb3-python

NguyenHoangSon96 · 2026-06-12T15:16:42Z

Closes #216

Proposed Changes

I'm trying to make python3 similar to other v3 clients as possible, so I removed some functionalities that currently don't exist in other clients.

Summarize

RestClient will be created in InfluxDBClient3 and injected to WriteApi; WriteApi will have all functions to write to InfluxDB, so their relations are something like this InfluxDBClient3 -> WriteApi -> RestClient.

Removed features:

_return_http_data_only param.
zap-trace-span param.
_preload_content

Checklist

Copilot

Pull request overview

This PR refactors the write path to use a new synchronous RestClient (urllib3-based) instead of the previous generated client/write service wiring, and updates InfluxDBClient3 construction accordingly.

Changes:

Refactors WriteApi to issue HTTP requests via a new RestClient, adds gzip decision/compression utilities, and adds endpoint/exception translation logic.
Introduces influxdb_client_3/write_client/_sync/rest_client.py to encapsulate urllib3 request/response handling.
Updates InfluxDBClient3 initialization to pass write connection details (base_url/auth/gzip) directly into WriteApi.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 15 comments.

File	Description
`influxdb_client_3/write_client/client/write_api.py`	Major refactor of write implementation to use a new REST client and custom request building/serialization paths.
`influxdb_client_3/write_client/_sync/rest_client.py`	Adds a new urllib3-based synchronous REST client abstraction used by `WriteApi`.
`influxdb_client_3/__init__.py`	Wires new write API constructor parameters (auth/gzip/base_url) and updates public kwargs documentation.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot

Pull request overview

Copilot reviewed 19 out of 19 changed files in this pull request and generated 12 comments.

Comments suppressed due to low confidence (2)

tests/test_write_api.py:251

WriteApi.write() takes parameters in (bucket, org, record, ...) order, but this test passes (org, bucket, record). This makes the test less representative of real usage and can mask issues in how query parameters are constructed.
tests/test_write_api.py:293
This call passes (org, bucket, record) instead of (bucket, org, record), which makes the timeout test less representative of real usage and inconsistent with other tests in this file.

Copilot

Pull request overview

Copilot reviewed 19 out of 19 changed files in this pull request and generated 2 comments.

Comments suppressed due to low confidence (4)

influxdb_client_3/write_client/client/write_api.py:1171

__getstate__ deletes _write_service, but WriteApi no longer defines that attribute. This will raise KeyError during pickling (and del state['_subject'] / del state['_disposable'] can also KeyError depending on instance state). Use state.pop(..., None) and remove the stale _write_service reference.

    def __getstate__(self):
        """Return a dict of attributes that you want to pickle."""
        state = self.__dict__.copy()
        # Remove rx
        del state['_subject']
        del state['_disposable']
        del state['_write_service']
        return state

influxdb_client_3/write_client/client/write_api.py:1181

__setstate__ calls self.__init__(self._write_options, self._point_settings, ...), but WriteApi.__init__ requires token, bucket, and org as the first arguments. As written, unpickling will raise TypeError. Recreate the Rx batching pipeline based on the restored _write_options instead of re-calling __init__ with the wrong signature.

    def __setstate__(self, state):
        """Set your object with the provided dict."""
        self.__dict__.update(state)
        # Init Rx
        self.__init__(self._write_options,
                      self._point_settings,
                      success_callback=self._success_callback,
                      error_callback=self._error_callback,
                      retry_callback=self._retry_callback)

tests/test_write_api.py:252

WriteApi.write() expects positional arguments in the order (bucket, org, record). This call passes org first, which swaps the query parameters and makes the test less representative of real usage. Use keyword arguments (or correct positional order) to avoid accidental swaps.
tests/test_write_api.py:293
WriteApi.write() positional argument order is (bucket, org, record). This test passes org then bucket, which is easy to get wrong and can hide routing/query-param bugs. Prefer keyword arguments to make the intent explicit (and keep _request_timeout behavior under test).

Copilot

Pull request overview

Copilot reviewed 19 out of 19 changed files in this pull request and generated 2 comments.

Comments suppressed due to low confidence (5)

tests/test_write_api.py:29

mock_urllib3_timeout_request is used as a side_effect for urllib3._request_methods.RequestMethods.request, which passes the RequestMethods instance as the first positional arg. The helper currently omits that parameter, so method/url are shifted and the test can behave incorrectly.
tests/test_write_api.py:252
WriteApi.write() takes bucket as the first positional argument and org as the second. This call passes them in the opposite order, so the request routing/query params will be wrong.
tests/test_write_api.py:293
WriteApi.write() takes bucket as the first positional argument and org as the second. This call passes them in the opposite order, so it won't exercise the intended timeout behavior for the right endpoint params.
influxdb_client_3/write_client/client/write_api.py:1171
__getstate__ deletes state['_write_service'], but this class no longer defines _write_service. Pickling will raise KeyError here.

        state = self.__dict__.copy()
        # Remove rx
        del state['_subject']
        del state['_disposable']
        del state['_write_service']
        return state

influxdb_client_3/write_client/client/write_api.py:1181

__setstate__ calls self.__init__(self._write_options, self._point_settings, ...), but __init__ now requires token, bucket, org, and other constructor args. This will break unpickling; it should just recreate the Rx batching pipeline based on the restored _write_options.

    def __setstate__(self, state):
        """Set your object with the provided dict."""
        self.__dict__.update(state)
        # Init Rx
        self.__init__(self._write_options,
                      self._point_settings,
                      success_callback=self._success_callback,
                      error_callback=self._error_callback,
                      retry_callback=self._retry_callback)

Copilot

Pull request overview

Copilot reviewed 19 out of 19 changed files in this pull request and generated 2 comments.

Comments suppressed due to low confidence (3)

tests/test_write_api.py:252

WriteApi.write() takes positional arguments in the order (bucket, org, record). This call passes org first and bucket second, which swaps routing parameters and can hide bugs in header/query handling. Prefer keyword arguments (or swap the positionals) to make the intent unambiguous.
tests/test_write_api.py:293
Same positional-argument issue as above: this passes org first and bucket second to WriteApi.write(bucket, org, record), which inverts query parameters. Using keywords also makes the _request_timeout intent clearer.
influxdb_client_3/write_client/client/write_api.py:1181
__getstate__ / __setstate__ are broken after the refactor: _write_service is no longer an attribute, so del state['_write_service'] will raise KeyError during pickling. Additionally, __setstate__ calls __init__ with the wrong arguments for the new constructor signature (it no longer accepts just write_options and point_settings). This makes pickling/unpickling unusable.

    def __getstate__(self):
        """Return a dict of attributes that you want to pickle."""
        state = self.__dict__.copy()
        # Remove rx
        del state['_subject']
        del state['_disposable']
        del state['_write_service']
        return state

    def __setstate__(self, state):
        """Set your object with the provided dict."""
        self.__dict__.update(state)
        # Init Rx
        self.__init__(self._write_options,
                      self._point_settings,
                      success_callback=self._success_callback,
                      error_callback=self._error_callback,
                      retry_callback=self._retry_callback)

Copilot

Pull request overview

Copilot reviewed 19 out of 19 changed files in this pull request and generated 2 comments.

Comments suppressed due to low confidence (3)

influxdb_client_3/write_client/client/write_api.py:1181

__getstate__ deletes state['_write_service'], but WriteApi no longer defines _write_service, so pickling will raise KeyError. Additionally, __setstate__ calls __init__ with the wrong signature (it now requires token, bucket, org, etc.), so unpickling will fail even if __getstate__ succeeds. Safer approach: pop() optional fields and recreate the Rx pipeline based on _write_options without re-calling __init__.

    def __getstate__(self):
        """Return a dict of attributes that you want to pickle."""
        state = self.__dict__.copy()
        # Remove rx
        del state['_subject']
        del state['_disposable']
        del state['_write_service']
        return state

    def __setstate__(self, state):
        """Set your object with the provided dict."""
        self.__dict__.update(state)
        # Init Rx
        self.__init__(self._write_options,
                      self._point_settings,
                      success_callback=self._success_callback,
                      error_callback=self._error_callback,
                      retry_callback=self._retry_callback)

tests/test_write_api.py:252

WriteApi.write() expects positional arguments as (bucket, org, record, ...), but this test passes them as (org, bucket, record). That swaps query params and makes the test exercise the wrong behavior. Use keyword args (or correct positional order) to match the write() signature.
tests/test_write_api.py:294
This write() call also swaps (org, bucket) positional order. Since the assertion is about timeout propagation rather than parameter order, using keyword args avoids accidentally testing the wrong thing.

Copilot

Pull request overview

Copilot reviewed 19 out of 19 changed files in this pull request and generated 3 comments.

Comments suppressed due to low confidence (2)

tests/test_write_api.py:252

In this test, WriteApi.write() expects positional args in the order (bucket, org, record). Passing org first swaps the routing parameters and can hide bugs in request construction/error handling.
tests/test_write_api.py:293
WriteApi.write() positional parameters are (bucket, org, record). This call passes org first, which makes the test exercise the wrong request routing.

Copilot

Pull request overview

Copilot reviewed 19 out of 19 changed files in this pull request and generated 3 comments.

Comments suppressed due to low confidence (3)

influxdb_client_3/write_client/client/write_api.py:1179

__getstate__() deletes _write_service, but WriteApi no longer sets this attribute, so pickling will raise KeyError. Additionally, __setstate__() calls __init__() with the wrong signature (it now requires token/bucket/org/...).

    def __getstate__(self):
        """Return a dict of attributes that you want to pickle."""
        state = self.__dict__.copy()
        # Remove rx
        del state['_subject']
        del state['_disposable']
        del state['_write_service']
        return state

    def __setstate__(self, state):
        """Set your object with the provided dict."""
        self.__dict__.update(state)
        # Init Rx
        self.__init__(self._write_options,
                      self._point_settings,
                      success_callback=self._success_callback,
                      error_callback=self._error_callback,
                      retry_callback=self._retry_callback)

tests/test_write_api.py:252

Argument order is swapped in this write() call: WriteApi.write(bucket, org, record, ...) expects bucket first, org second, but this passes org then bucket. This makes the test less meaningful and can hide real regressions.
tests/test_write_api.py:293
Argument order is swapped in this write() call: WriteApi.write(bucket, org, record, ...) expects bucket first, org second. As written, the test exercises the wrong parameter mapping (and still "works" only because values are strings).

karel-rehor · 2026-07-03T13:17:01Z

OK after researching differences between why a segmentation fault is occurring in Linux but apparently not in Darwin, I came across the +AI+ suggestion that Darwin often uses the "spawn" start method by default in multiprocessing, while Linux relies on the traditional "fork" approach. In multiprocessing_helper:__init__ I tried the following locally.

lines 121-132

   ...
    def __init__(self, **kwargs) -> None:
        """
        Initialize defaults.

        For more information how to initialize the writer see the examples above.

        :param kwargs: arguments are passed into ``__init__`` function of ``InfluxDBClient`` and ``write_api``.
        """
        multiprocessing.set_start_method("spawn", force=True)
        multiprocessing.Process.__init__(self)
        self.kwargs = kwargs
   ...

🎉 The integration test test_multuprocessing_helper now passes without encountering SIGSEGV. However I'm still not comfortable with this as an ultimate solution.

It still does not address all of the concerns mention by me earlier or in Aleses post. I still prefer this be removed and added as an advanced example.

We might now consider calling the set_start_method as shown here, along with marking the feature as deprecated, as we may want to remove it in a future refactoring with the objective of further simplifying the library.

N.B.

After adding the above change to my local fork. I see that the integration test now passes in CircleCI without issue.

Addendum

Further documentation https://docs.python.org/3.14/library/multiprocessing.html#contexts-and-start-methods. Note that according to this product documentation "set_start_method() should not be used more than once in the program.". So perhaps my suggestion here is not the best solution or work-around, but it is an indicator of how this might be fixed.

btw. it is mentioned in the multiprocessing documentation that "spawn... (is) the default on Windows and macOS".

bednar · 2026-07-07T06:18:19Z

A lot of “spam” here… 😉

Decision

The purpose of multiprocessing is to fully utilize the user’s CPU during writes. So if we already have tests for this, let’s fix the implementation so it does not create a QueryAPI instance for every forked process.

@NguyenHoangSon96:

Fix all requirements except the multiprocessing helper.
Fix the multiprocessing helper.
Update the PR description and CHANGELOG.md to reflect the actual changes in the PR.

Copilot

Pull request overview

Copilot reviewed 20 out of 20 changed files in this pull request and generated 1 comment.

Copilot

Pull request overview

Copilot reviewed 20 out of 20 changed files in this pull request and generated 2 comments.

Copilot

Pull request overview

Copilot reviewed 21 out of 21 changed files in this pull request and generated 1 comment.

Copilot

Pull request overview

Copilot reviewed 21 out of 21 changed files in this pull request and generated 4 comments.

Copilot

Pull request overview

Copilot reviewed 21 out of 21 changed files in this pull request and generated 3 comments.

NguyenHoangSon96 · 2026-07-08T07:58:21Z

Hi guys, based on Jakub's new comments, I have tried to work on the refactor of rest client first then working on Multiprocessing help class later.
I have fixed all "High" priorities issues from Copilot.
Please review this PR again. I don't want to continue trigger Copilot reviews because I don't want to drain all credits
like the last month.

bednar

I found two remaining issues around pickle support.

Copilot

Pull request overview

Copilot reviewed 21 out of 21 changed files in this pull request and generated 4 comments.

NguyenHoangSon96 · 2026-07-08T11:43:01Z

@bednar
I edited code based on your suggestions.
Additionally, I edited the code based on two new suggestions from Copilot.
#217 (comment)
#217 (comment)

bednar

One batching pickle regression remains.

bednar

Two additional backward-compatibility requirements remain.

bednar

The changelog needs to describe the public compatibility impact of this refactor.

Copilot

Pull request overview

Copilot reviewed 22 out of 22 changed files in this pull request and generated 6 comments.

NguyenHoangSon96 · 2026-07-09T07:21:20Z

+        org = 'my-org'
+        writer = MultiprocessingWriter(
+            host=self.host,
+            database=self.database,
+            token=self.token,
+            org=org,
+            write_options=WriteOptions(batch_size=1))
+        writer.start()
+        measurement = f'test{random_hex(6)}'.lower()
+        for x in range(1, 10):
+            time.sleep(0.2)
+            writer.write(
+                bucket=self.database,
+                record=f"{measurement},tag=a value=\"number{x}\" {time.time_ns()}"
+            )
+        writer.__del__()
+


Will fix this later.

NguyenHoangSon96 · 2026-07-09T07:20:53Z

            self.write_api.__del__()
            self.write_api = None


Will fix this later.

bednar

Two debug compatibility issues remain in the new RestClient.

Copilot

Pull request overview

Copilot reviewed 22 out of 22 changed files in this pull request and generated 2 comments.

NguyenHoangSon96 · 2026-07-09T09:15:28Z

@bednar

Logger has been setting properly when debug=True in RestClient class.
Added debug to __setstate__().

bednar

The breaking changes entry needs migration guidance.

Copilot

Pull request overview

Copilot reviewed 22 out of 22 changed files in this pull request and generated 4 comments.

Copilot

Pull request overview

Copilot reviewed 22 out of 22 changed files in this pull request and generated 5 comments.

Copilot

Pull request overview

Copilot reviewed 22 out of 22 changed files in this pull request and generated 9 comments.

+        r = RESTResponse(r)
+        if isinstance(r.data, (bytes, bytearray)):
+            r.data = r.data.decode('utf-8')
+        elif r.data is None:
+            r.data = ''
+


karel-rehor

Still in progress, but before I leave at COB, I already have one minor request, which is also related to some unit tests.

karel-rehor · 2026-07-14T15:43:19Z

+        if self._token is not None:
+            default_header['Authorization'] = f'{auth_schema} {self._token}'
+        self.base_url = f"{scheme}://{hostname}:{port}"
+        self.default_header = default_header


I see that the default value for host is None. The constructor up to this point fails to enforce a useful definition for a key attribute such as 'host'. It can lead to unstable or nonsensical base_url strings like "https://None:443".

I might recommend a guard for arguments that should always be defined at the start of the __init__() method like....

for key in ["host", "token", "database"]: if locals().get(key) is None: raise Exception(f"The '{key}' key is required")

This is of course unless I've overlooked another type enforcement mechanism in Python, of which I'm not yet aware.

Currently this is not an issue for unit tests using a mock, since the mock essentially just makes use of or ignores the 'None' part of the URL. However, outside of a testing framework, this should not be allowed.

Safeguard like your snipped suggest has been implemented.

Key parameters like "host", "token", "database" have been added to all test cases.

Tests added for missing key parameters.
Please click "Resolve conversations" when these fixed are good enough.

NguyenHoangSon96 self-assigned this Jun 12, 2026

Copilot AI review requested due to automatic review settings June 12, 2026 15:16

Copilot started reviewing on behalf of NguyenHoangSon96 June 12, 2026 15:17 View session

Copilot AI reviewed Jun 12, 2026

View reviewed changes

NguyenHoangSon96 force-pushed the refactor/rest-client branch from 2b26e5c to 7d74438 Compare June 12, 2026 16:42

NguyenHoangSon96 marked this pull request as draft June 16, 2026 11:15

NguyenHoangSon96 force-pushed the refactor/rest-client branch from 52cee20 to a7076f9 Compare June 17, 2026 09:38

NguyenHoangSon96 requested a review from Copilot June 18, 2026 05:45

Copilot started reviewing on behalf of NguyenHoangSon96 June 18, 2026 05:45 View session