Skip to content

Support for Azure storage#571

Merged
tramora merged 3 commits intomainfrom
244-support-azure-storage
Apr 30, 2026
Merged

Support for Azure storage#571
tramora merged 3 commits intomainfrom
244-support-azure-storage

Conversation

@tramora
Copy link
Copy Markdown
Collaborator

@tramora tramora commented Apr 10, 2026

  • Among all the Azure storages, Khiops supports only (via its specific driver) "Files" and "Blobs" (Binary Large Objects)
  • The only supported authentication method is currently the AZURE_STORAGE_CONNECTION_STRING embedding the account name and account key

Completes #244 and fixes #576


TODO Before Asking for a Review

  • Rebase your branch to the latest version of main (or main-v10)
  • Make sure all CI workflows are green
  • When adding a public feature/fix: Update the Unreleased section of CHANGELOG.md (no date)
  • Self-Review: Review "Files Changed" tab and fix any problems you find
  • API Docs (only if there are changes in docstrings, rst files or samples):
    • Check the docs build without warning: see the log of the API Docs workflow
    • Check that your changes render well in HTML: download the API Docs artifact and open index.html
    • If there are any problems it is faster to iterate by building locally the API Docs

@tramora tramora force-pushed the 244-support-azure-storage branch 3 times, most recently from be97119 to 08f91e2 Compare April 10, 2026 14:34
@tramora tramora force-pushed the 244-support-azure-storage branch 4 times, most recently from 17fe956 to 80691c3 Compare April 12, 2026 12:59
@tramora
Copy link
Copy Markdown
Collaborator Author

tramora commented Apr 13, 2026

The automated tests on conda environments currently fail because of an outdate version of the azure-storage-file-share package in the conda-forge channel. This PR conda-forge/azure-storage-file-share-feedstock#9 should fix the issue but it is blocked for the moment.

@tramora tramora force-pushed the 244-support-azure-storage branch from 80691c3 to 701678b Compare April 13, 2026 16:32
Comment thread khiops/core/internals/filesystems.py Outdated
Comment thread khiops/core/internals/filesystems.py Outdated
Comment thread khiops/core/internals/filesystems.py
Comment thread khiops/core/internals/filesystems.py Outdated
Comment thread khiops/core/internals/filesystems.py Outdated
Comment thread khiops/core/internals/filesystems.py Outdated
Comment thread tests/test_remote_access.py Outdated
Comment thread tests/test_remote_access.py
Comment thread khiops/core/internals/filesystems.py
@tramora tramora force-pushed the 244-support-azure-storage branch from 701678b to 19aa231 Compare April 15, 2026 16:39
Copy link
Copy Markdown
Member

@folmos-at-orange folmos-at-orange left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is an opportunity to have better docs on file system.

Currently, very little is documented, and we now have 3 different remote systems of which only one (S3) is well documented.

I'd a section for each remote system in the module's docstring. Each section contains how to configure it in each case, and a URI example.

Comment thread khiops/core/internals/filesystems.py Outdated
Comment thread khiops/core/internals/filesystems.py Outdated
Comment thread khiops/core/internals/filesystems.py Outdated
Comment thread khiops/core/internals/filesystems.py
@tramora tramora force-pushed the 244-support-azure-storage branch from 19aa231 to 4dc0953 Compare April 16, 2026 14:39
Comment thread khiops/core/internals/filesystems.py Outdated
Comment thread khiops/core/internals/filesystems.py Outdated
Comment thread khiops/core/internals/filesystems.py Outdated
Comment thread khiops/core/internals/filesystems.py Outdated
Comment thread khiops/core/internals/filesystems.py Outdated
Comment thread khiops/core/internals/filesystems.py Outdated
Comment thread khiops/core/internals/filesystems.py Outdated
Comment thread khiops/core/internals/filesystems.py Outdated
Comment thread khiops/core/internals/filesystems.py Outdated
Comment thread khiops/core/internals/filesystems.py Outdated
Comment thread packaging/docker/khiopspydev/Dockerfile.debian Outdated
Comment thread packaging/docker/khiopspydev/Dockerfile.debian Outdated
Comment thread packaging/docker/khiopspydev/Dockerfile.ubuntu Outdated
Comment thread packaging/docker/khiopspydev/Dockerfile.ubuntu Outdated
@tramora
Copy link
Copy Markdown
Collaborator Author

tramora commented Apr 24, 2026

Check the remaining comments. Check failing remote tests in https://github.com/KhiopsML/khiops-python/actions/runs/24792699160, e.g. https://github.com/KhiopsML/khiops-python/actions/runs/24792699160/job/72554199345#step:12:279 and https://github.com/KhiopsML/khiops-python/actions/runs/24792699160/job/72554199345#step:12:294.

The error message was not very explicit; khiops failed only with a fatal error : Command file : Batch mode failure.

Naturally the latest ubuntu dev image didn't have the azure driver (system-wide or Conda) as the current PR is not yet merged.

The latest Conda packages for GCS (0.0.16) and Azure (0.0.6) do not exist (compared to their counterpart on github).
And the previous versions are not installable under Conda because a (new) dependency incompatibility (fmt).

Anyway the khiopspydev-ubuntu22.04:11.0.0.244-support-azure-storage image contains the system-wide drivers that should be always be the fallback drivers during the executions for both "pyXX" (simulated venv) and "pyXX_conda" (Conda environment). However despite of this, the execution in a Conda environment fails. I tried to launch locally the container as described in khiops-core#1009

@tramora tramora requested a review from popescu-v April 24, 2026 23:04
Comment thread packaging/docker/khiopspydev/Dockerfile.debian Outdated
Comment thread packaging/docker/khiopspydev/Dockerfile.ubuntu Outdated
@tramora tramora force-pushed the 244-support-azure-storage branch 2 times, most recently from 76d67b9 to 465ec37 Compare April 28, 2026 14:52
@tramora tramora requested a review from popescu-v April 28, 2026 15:00
@tramora tramora force-pushed the 244-support-azure-storage branch 3 times, most recently from 07b9694 to cc98005 Compare April 28, 2026 16:02
Comment thread .github/workflows/tests.yml Outdated
Comment thread .github/workflows/tests.yml Outdated
Comment thread .github/workflows/tests.yml Outdated
Comment thread packaging/docker/khiopspydev/Dockerfile.rocky Outdated
Comment thread packaging/docker/khiopspydev/Dockerfile.ubuntu Outdated
Comment thread packaging/docker/khiopspydev/Dockerfile.debian Outdated
@popescu-v
Copy link
Copy Markdown
Collaborator

Check the remaining comments. Check failing remote tests in https://github.com/KhiopsML/khiops-python/actions/runs/24792699160, e.g. https://github.com/KhiopsML/khiops-python/actions/runs/24792699160/job/72554199345#step:12:279 and https://github.com/KhiopsML/khiops-python/actions/runs/24792699160/job/72554199345#step:12:294.

The error message was not very explicit; khiops failed only with a fatal error : Command file : Batch mode failure.

Naturally the latest ubuntu dev image didn't have the azure driver (system-wide or Conda) as the current PR is not yet merged.

The latest Conda packages for GCS (0.0.16) and Azure (0.0.6) do not exist (compared to their counterpart on github). And the previous versions are not installable under Conda because a (new) dependency incompatibility (fmt).

Anyway the khiopspydev-ubuntu22.04:11.0.0.244-support-azure-storage image contains the system-wide drivers that should be always be the fallback drivers during the executions for both "pyXX" (simulated venv) and "pyXX_conda" (Conda environment). However despite of this, the execution in a Conda environment fails. I tried to launch locally the container as described in khiops-core#1009

Noted. However:

  • the native drivers are not the fallback drivers in Conda environments, because the drivers are selected according to the value of the KHIOPS_DRIVERS_PATH environment variable as defined in khiops_env. Its value is specific to the Conda environment khiops is launched in;
  • however, the error message in case the driver is missing could be more helpful IMHO, rather than just stating that the file is not found; I'd file an issue in the khiops repo to this end.

Copy link
Copy Markdown
Collaborator

@popescu-v popescu-v left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A few remaining comments.

@tramora tramora force-pushed the 244-support-azure-storage branch from cc98005 to b7be497 Compare April 30, 2026 09:07
@tramora
Copy link
Copy Markdown
Collaborator Author

tramora commented Apr 30, 2026

Check the remaining comments. Check failing remote tests in https://github.com/KhiopsML/khiops-python/actions/runs/24792699160, e.g. https://github.com/KhiopsML/khiops-python/actions/runs/24792699160/job/72554199345#step:12:279 and https://github.com/KhiopsML/khiops-python/actions/runs/24792699160/job/72554199345#step:12:294.

The error message was not very explicit; khiops failed only with a fatal error : Command file : Batch mode failure.
Naturally the latest ubuntu dev image didn't have the azure driver (system-wide or Conda) as the current PR is not yet merged.
The latest Conda packages for GCS (0.0.16) and Azure (0.0.6) do not exist (compared to their counterpart on github). And the previous versions are not installable under Conda because a (new) dependency incompatibility (fmt).
Anyway the khiopspydev-ubuntu22.04:11.0.0.244-support-azure-storage image contains the system-wide drivers that should be always be the fallback drivers during the executions for both "pyXX" (simulated venv) and "pyXX_conda" (Conda environment). However despite of this, the execution in a Conda environment fails. I tried to launch locally the container as described in khiops-core#1009

Noted. However:

* the native drivers are not the fallback drivers in Conda environments, because the drivers are selected according to the value of the `KHIOPS_DRIVERS_PATH` environment variable as defined in `khiops_env`. Its value is specific to the Conda environment `khiops` is launched in;

* however, the error message in case the driver is missing could be more helpful IMHO, rather than just stating that the file is not found; I'd file an issue in the `khiops` repo to this end.

When the conditions described in the issue khiops-core#1009 are not met (which make the process fail far before), khiops-core is explicit enough in telling it cannot open the remote file. Thus there is no need opening a new issue.

@tramora tramora requested a review from popescu-v April 30, 2026 09:10
@tramora tramora force-pushed the 244-support-azure-storage branch 2 times, most recently from f38a090 to 7c8090f Compare April 30, 2026 09:46
Thierry RAMORASOAVINA added 3 commits April 30, 2026 11:53
- Among all the Azure storages, Khiops supports only (via its specific driver) "Files" and "Blobs" (Binary Large Objects)
 - The only supported authentication method is currently the `AZURE_STORAGE_CONNECTION_STRING` embedding the account name and account key
- Remove the environments creation in the docker files
- Remove the installation of the Conda packages
- Remove the usage of these environments in the tests (unit and integration tests)
@tramora tramora force-pushed the 244-support-azure-storage branch from 7c8090f to fa62037 Compare April 30, 2026 09:54
Copy link
Copy Markdown
Collaborator

@popescu-v popescu-v left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.
Update Azure driver version and remove # XXX comments accordingly in the testing infrastructure when this becomes possible.

@tramora tramora merged commit ab5b9a3 into main Apr 30, 2026
59 of 63 checks passed
@tramora tramora deleted the 244-support-azure-storage branch April 30, 2026 14:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Comprehensive integration tests of FilesystemResource implementations

3 participants