MINIFICPP-2718 Windows based docker tests by martinzink · Pull Request #2133 · apache/nifi-minifi-cpp

martinzink · 2026-03-11T15:37:21Z

This PR allows windows based docker containers to run the behave test suite with some limitations:

I only enabled a handful of features because some of the tests are requiring thirdparty containers (linux only and windows doesnt allow mixed (windows and linux) containers)
Even the tests that dont require thirdparty containers might rely on extra linux commands that needs to be replicated in powershell
Its unfeasable to enable these in github CI, because this requires nested virtualization (which is quite rare for VMs).

So this is only the MVP for this feature, but it is still a huge help for future developments especially for using it to develop external extensions using the upcoming C api. It cuts down the local verification time.

I've also replaced the previous m2crypto based ssl_utils with the industry standard cryptography, this should help with the setup on macs and windows systems (m2crypto was quite difficult to install on these systems)

Thank you for submitting a contribution to Apache NiFi - MiNiFi C++.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

For all changes:

Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
Does your PR title start with MINIFICPP-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
Has your PR been rebased against the latest commit within the target branch (typically main)?
Is your initial contribution a single, squashed commit?

For code changes:

If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
If applicable, have you updated the LICENSE file?
If applicable, have you updated the NOTICE file?

For documentation related changes:

Have you ensured that format looks appropriate for the output in which it is rendered?

Note:

Please ensure that once the PR is submitted, you check GitHub Actions CI results for build issues and submit an update to your PR as soon as possible.

extensions/couchbase/tests/features/containers/couchbase_server_container.py

behave_framework/src/minifi_test_framework/containers/container_windows.py

martinzink · 2026-03-11T15:38:00Z

behave_framework/pyproject.toml

 [tool.setuptools]
 package-dir = {"" = "src"}
-packages = ["minifi_test_framework"]
+
+[tool.setuptools.packages.find]
+where = ["src"]
+include = ["minifi_test_framework*"]
+


this is required for wheel building

extensions/couchbase/tests/features/containers/couchbase_server_container.py

Copilot

Pull request overview

This PR enables running a subset of the Behave integration test suite with Windows-based Docker containers, and refactors the test framework container abstractions to support both Linux and Windows. It also replaces the previous M2Crypto/PyOpenSSL-based SSL helpers with cryptography, improving cross-platform dependency installation.

Changes:

Add Windows container support (new Windows container implementation, Windows MiNiFi container, and scenario tag gating via @SUPPORTS_WINDOWS).
Refactor test containers to explicitly target Linux (LinuxContainer) and extract controller/config helpers.
Replace SSL utilities implementation and dependencies with cryptography, updating affected test containers accordingly.

Reviewed changes

Copilot reviewed 42 out of 44 changed files in this pull request and generated 9 comments.

Show a summary per file

File	Description
extensions/standard-processors/tests/features/steps/steps.py	Switch SSL setup to shared helper function.
extensions/standard-processors/tests/features/steps/minifi_controller_steps.py	Route controller actions through a `controller` helper object; update retry decorator args.
extensions/standard-processors/tests/features/steps/minifi_c2_server_container.py	Switch to `cryptography`-based cert/key serialization; base class to `LinuxContainer`.
extensions/standard-processors/tests/features/replace_text.feature	Mark feature as supporting Windows.
extensions/standard-processors/tests/features/environment.py	Avoid Alpine image inspection on Windows host.
extensions/standard-processors/tests/features/core_functionality.feature	Mark feature as supporting Windows; increase timeouts.
extensions/standard-processors/tests/features/containers/tcp_client_container.py	Base class switched to `LinuxContainer`.
extensions/standard-processors/tests/features/containers/syslog_container.py	Base class switched to `LinuxContainer`.
extensions/standard-processors/tests/features/containers/diag_slave_container.py	Base class switched to `LinuxContainer`.
extensions/sql/tests/features/containers/postgress_server_container.py	Base class switched to `LinuxContainer`.
extensions/splunk/tests/features/containers/splunk_container.py	Base class switched to `LinuxContainer`; switch to `cryptography` dump helpers.
extensions/prometheus/tests/features/containers/prometheus_container.py	Base class switched to `LinuxContainer`; switch to `cryptography` dump helpers.
extensions/opc/tests/features/containers/opc_ua_server_container.py	Base class switched to `LinuxContainer`.
extensions/mqtt/tests/features/containers/mqtt_broker_container.py	Base class switched to `LinuxContainer`.
extensions/kafka/tests/features/containers/kafka_server_container.py	Base class switched to `LinuxContainer`; switch keystore/truststore material to `cryptography` dump helpers.
extensions/grafana-loki/tests/features/containers/reverse_proxy_container.py	Base class switched to `LinuxContainer`.
extensions/grafana-loki/tests/features/containers/grafana_loki_container.py	Base class switched to `LinuxContainer`; switch to `cryptography` dump helpers.
extensions/gcp/tests/features/containers/fake_gcs_server_container.py	Base class switched to `LinuxContainer`.
extensions/elasticsearch/tests/features/containers/opensearch_container.py	Switch to `cryptography` dump helpers.
extensions/elasticsearch/tests/features/containers/elasticsearch_container.py	Switch to `cryptography` dump helpers.
extensions/elasticsearch/tests/features/containers/elastic_base_container.py	Base class switched to `LinuxContainer`.
extensions/couchbase/tests/features/containers/couchbase_server_container.py	Base class switched to `LinuxContainer`; switch to `cryptography` dump helpers; update retry decorator arg names.
extensions/azure/tests/features/containers/azure_server_container.py	Base class switched to `LinuxContainer`.
extensions/aws/tests/features/containers/s3_server_container.py	Base class switched to `LinuxContainer`.
extensions/aws/tests/features/containers/kinesis_server_container.py	Base class switched to `LinuxContainer`.
docker/installed/win.Dockerfile	New Windows MiNiFi installed-image Dockerfile.
docker/RunBehaveTests.sh	Improve feature file collection portability (avoid `mapfile`).
behave_framework/src/minifi_test_framework/steps/configuration_steps.py	Switch from container methods to protocol helper functions.
behave_framework/src/minifi_test_framework/core/ssl_utils.py	Replace M2Crypto/PyOpenSSL SSL utilities with `cryptography`.
behave_framework/src/minifi_test_framework/core/minifi_test_context.py	Introduce OS-/image-based MiNiFi container selection (Windows vs Linux, FHS vs normal).
behave_framework/src/minifi_test_framework/core/hooks.py	Skip scenarios on Windows unless tagged `SUPPORTS_WINDOWS`; guard cleanup if setup didn’t run.
behave_framework/src/minifi_test_framework/containers/nifi_container.py	Base class switched to `LinuxContainer`; switch to `cryptography` dump helpers.
behave_framework/src/minifi_test_framework/containers/minifi_win_container.py	New Windows MiNiFi container implementation.
behave_framework/src/minifi_test_framework/containers/minifi_protocol.py	New protocol + shared property-setting helpers.
behave_framework/src/minifi_test_framework/containers/minifi_linux_container.py	New Linux MiNiFi container implementation (normal vs FHS deployment).
behave_framework/src/minifi_test_framework/containers/minifi_controller.py	New helper encapsulating `minifi-controller` interactions.
behave_framework/src/minifi_test_framework/containers/minifi_container.py	Removed monolithic MiNiFi container (replaced by linux/windows split).
behave_framework/src/minifi_test_framework/containers/http_proxy_container.py	Base class switched to `LinuxContainer`.
behave_framework/src/minifi_test_framework/containers/container_windows.py	New Windows container implementation for file ops + exec + log helpers.
behave_framework/src/minifi_test_framework/containers/container_protocol.py	New protocol defining container interface used by the framework.
behave_framework/src/minifi_test_framework/containers/container_linux.py	Rename/reshape base container into `LinuxContainer`; move shared helpers.
behave_framework/pyproject.toml	Bump framework version; swap SSL deps to `cryptography`; adjust package discovery.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

behave_framework/src/minifi_test_framework/core/minifi_test_context.py

behave_framework/src/minifi_test_framework/core/ssl_utils.py

behave_framework/src/minifi_test_framework/containers/minifi_protocol.py

docker/installed/win.Dockerfile

extensions/standard-processors/tests/features/steps/minifi_c2_server_container.py

behave_framework/src/minifi_test_framework/containers/minifi_win_container.py

behave_framework/src/minifi_test_framework/containers/container_protocol.py

behave_framework/src/minifi_test_framework/containers/minifi_controller.py

apache#2133

martinzink · 2026-04-13T13:28:47Z

extensions/standard-processors/tests/features/core_functionality.feature


-    Then at least one file with the content "first_custom_text" is placed in the "/tmp/output" directory in less than 20 seconds
-    And at least one file with the content "second_custom_text" is placed in the "/tmp/output" directory in less than 20 seconds
+    Then at least one file with the content "first_custom_text" is placed in the "/tmp/output" directory in less than 200 seconds


fgerlits · 2026-04-10T09:52:28Z

behave_framework/src/minifi_test_framework/containers/container_linux.py

+        exit_code, output = self.exec_run(["awk", "/VmRSS/ { printf \"%d\\n\", $2 }", "/proc/1/status"])
+        if exit_code != 0:
+            return None
+        memory_usage_in_bytes = int(output.strip()) * 1024


very minor, but why do we add the \n in line 489 only to strip it out in line 492? also, I would include "kB" in the regex, just in case the format of /proc/1/status changes in the future:

Suggested change

exit_code, output = self.exec_run(["awk", "/VmRSS/ { printf \"%d\\n\", $2 }", "/proc/1/status"])

if exit_code != 0:

return None

memory_usage_in_bytes = int(output.strip()) * 1024

exit_code, output = self.exec_run(["awk", "/VmRSS.*kB/ { printf \"%d\", $2 }", "/proc/1/status"])

if exit_code != 0:

return None

memory_usage_in_bytes = int(output) * 1024

fgerlits · 2026-04-10T10:18:35Z

behave_framework/src/minifi_test_framework/containers/container_protocol.py

+    def directory_contains_file_with_content(self, directory_path: str, expected_content: str) -> bool:
+        ...


It's a bit strange that we say these stubs return bool, str or int, but in fact we return None. Maybe raise an exception instead?

fgerlits · 2026-04-10T13:15:26Z

behave_framework/src/minifi_test_framework/containers/container_windows.py

+        self.client: docker.DockerClient = docker.from_env()
+        self.container: Optional[Container] = None
+        self.files: List[File] = []
+        self.dirs: List[Directory] = []


minor, but List is deprecated, we should use list instead: https://docs.python.org/3/library/typing.html#aliases-to-built-in-types

fgerlits · 2026-04-10T13:20:58Z

behave_framework/src/minifi_test_framework/containers/container_windows.py

+    def _normalize_path(self, path: str) -> str:
+        clean_path = path.strip().replace("/", "\\")
+        if clean_path.startswith("\\"):
+            clean_path = clean_path[1:]
+
+        # If it doesn't already have a drive letter, assume C:
+        if ":" not in clean_path:
+            return f"C:\\{clean_path}"
+        return clean_path


would os.path.normpath() work instead of this function?

fgerlits · 2026-04-14T12:38:41Z

behave_framework/src/minifi_test_framework/containers/container_windows.py

+        exit_code, _ = self._run_powershell(ps_script)
+
+        return exit_code == 0


we could log the error (if any) here, too, as in the other functions

fgerlits · 2026-04-14T13:01:36Z

behave_framework/src/minifi_test_framework/containers/container_windows.py

+            f"if ((Test-Path -LiteralPath '{win_path}' -PathType Container) "
+            f"-and (Get-ChildItem -LiteralPath '{win_path}' -Force | Select-Object -First 1)) "
+            f"{{ exit 0 }} else {{ exit 1 }}"


When I run this on a test directory, I get True on both empty and non-empty directories. A StackOverflow answer suggests adding Measure-Object:

Suggested change

f"if ((Test-Path -LiteralPath '{win_path}' -PathType Container) "

f"-and (Get-ChildItem -LiteralPath '{win_path}' -Force | Select-Object -First 1)) "

f"{{ exit 0 }} else {{ exit 1 }}"

f"if ((Test-Path -LiteralPath '{win_path}' -PathType Container) "

f"-and ((Get-ChildItem -LiteralPath '{win_path}' -Force | "

f" Select-Object -First 1 | Measure-Object).Count -gt 0)) "

f"{{ exit 0 }} else {{ exit 1 }}"

fgerlits · 2026-04-14T16:16:24Z

behave_framework/src/minifi_test_framework/core/ssl_utils.py

+    cert = x509.CertificateBuilder().subject_name(subject).issuer_name(issuer).public_key(key.public_key()).serial_number(
+        x509.random_serial_number()).not_valid_before(datetime.datetime.now(datetime.timezone.utc)).not_valid_after(
+        datetime.datetime.now(datetime.timezone.utc) + datetime.timedelta(days=3650)).add_extension(
+        x509.SubjectKeyIdentifier.from_public_key(key.public_key()), critical=False, ).add_extension(x509.BasicConstraints(ca=True, path_length=None),


nitpicking, but I think breaking into lines like this would be easier to read:

Suggested change

cert = x509.CertificateBuilder().subject_name(subject).issuer_name(issuer).public_key(key.public_key()).serial_number(

x509.random_serial_number()).not_valid_before(datetime.datetime.now(datetime.timezone.utc)).not_valid_after(

datetime.datetime.now(datetime.timezone.utc) + datetime.timedelta(days=3650)).add_extension(

x509.SubjectKeyIdentifier.from_public_key(key.public_key()), critical=False, ).add_extension(x509.BasicConstraints(ca=True, path_length=None),

cert = (x509.CertificateBuilder().subject_name(subject).issuer_name(issuer).public_key(key.public_key())

.serial_number(x509.random_serial_number())

.not_valid_before(datetime.datetime.now(datetime.timezone.utc))

.not_valid_after(datetime.datetime.now(datetime.timezone.utc) + datetime.timedelta(days=3650))

.add_extension(x509.SubjectKeyIdentifier.from_public_key(key.public_key()), critical=False)

.add_extension(x509.BasicConstraints(ca=True, path_length=None), critical=True)

.sign(key, hashes.SHA256()))

fgerlits · 2026-04-14T16:24:46Z

extensions/standard-processors/tests/features/core_functionality.feature

+    Then at least one file with the content "first_custom_text" is placed in the "/tmp/output" directory in less than 200 seconds
+    And at least one file with the content "second_custom_text" is placed in the "/tmp/output" directory in less than 200 seconds


Is this much slower on Windows, or was the timeout only changed for testing and it can be reverted?

martinzink force-pushed the MINIFICPP-2718 branch from 47057e5 to 721de91 Compare March 11, 2026 15:39

szaszm reviewed Mar 11, 2026

View reviewed changes

extensions/couchbase/tests/features/containers/couchbase_server_container.py Outdated Show resolved Hide resolved

behave_framework/src/minifi_test_framework/containers/container_windows.py Outdated Show resolved Hide resolved

martinzink commented Mar 11, 2026

View reviewed changes

martinzink marked this pull request as ready for review March 11, 2026 17:15

martinzink added the minifi-behave label Mar 12, 2026

martinzink modified the milestone: minifi-rust Mar 25, 2026

martinzink force-pushed the MINIFICPP-2718 branch from e3429cc to c04433a Compare March 25, 2026 13:28

martinzink mentioned this pull request Mar 25, 2026

MINIFICPP-2756 Rename minifi_test_framework to minifi_behave #2146

Open

8 tasks

martinzink requested a review from Copilot March 25, 2026 17:55

Copilot started reviewing on behalf of martinzink March 25, 2026 17:55 View session

Copilot AI reviewed Mar 25, 2026

View reviewed changes

martinzink mentioned this pull request Mar 27, 2026

MINIFICPP-2757 Build windows artifacts in create-release-artifacts CI #2145

Closed

8 tasks

martinzink added 6 commits March 30, 2026 17:46

MINIFICPP-2718 Windows based docker tests

6e8c658

review changes

77a2355

flake8 fix

a585b93

retry_check with named arguments

cf36bc6

review changes

d32f4aa

add flow_definition to MinifiProtocol

b6ba4f2

martinzink force-pushed the MINIFICPP-2718 branch from dd90af2 to b6ba4f2 Compare March 30, 2026 15:46

build fix

7b344d9

martinzink added a commit that referenced this pull request Mar 31, 2026

MINIFICPP-2718 Windows based docker tests #2133

05d243f

martinzink added a commit to martinzink/nifi-minifi-cpp that referenced this pull request Apr 7, 2026

MINIFICPP-2718 Windows based docker tests

bb9283c

apache#2133

martinzink added a commit to martinzink/nifi-minifi-cpp that referenced this pull request Apr 8, 2026

MINIFICPP-2718 Windows based docker tests

904f438

apache#2133

szaszm self-requested a review April 13, 2026 13:28

martinzink commented Apr 13, 2026

View reviewed changes

fgerlits self-requested a review April 14, 2026 10:15

fgerlits reviewed Apr 14, 2026

View reviewed changes

		def directory_contains_file_with_content(self, directory_path: str, expected_content: str) -> bool:
		...

		exit_code, _ = self._run_powershell(ps_script)

		return exit_code == 0

-    cert = x509.CertificateBuilder().subject_name(subject).issuer_name(issuer).public_key(key.public_key()).serial_number(
-        x509.random_serial_number()).not_valid_before(datetime.datetime.now(datetime.timezone.utc)).not_valid_after(
-        datetime.datetime.now(datetime.timezone.utc) + datetime.timedelta(days=3650)).add_extension(
-        x509.SubjectKeyIdentifier.from_public_key(key.public_key()), critical=False, ).add_extension(x509.BasicConstraints(ca=True, path_length=None),
+    cert = (x509.CertificateBuilder().subject_name(subject).issuer_name(issuer).public_key(key.public_key())
+        .serial_number(x509.random_serial_number())
+        .not_valid_before(datetime.datetime.now(datetime.timezone.utc))
+        .not_valid_after(datetime.datetime.now(datetime.timezone.utc) + datetime.timedelta(days=3650))
+        .add_extension(x509.SubjectKeyIdentifier.from_public_key(key.public_key()), critical=False)
+        .add_extension(x509.BasicConstraints(ca=True, path_length=None), critical=True)
+        .sign(key, hashes.SHA256()))

		Then at least one file with the content "first_custom_text" is placed in the "/tmp/output" directory in less than 200 seconds
		And at least one file with the content "second_custom_text" is placed in the "/tmp/output" directory in less than 200 seconds

Conversation

martinzink commented Mar 11, 2026

For all changes:

For code changes:

For documentation related changes:

Note:

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants