Skip to content

feat: add global OpenSearch index prefix support (#8489)#8514

Open
mezzeddinee wants to merge 1 commit into
DIRACGrid:integrationfrom
mezzeddinee:feature/8489-opensearch-global-index-prefix
Open

feat: add global OpenSearch index prefix support (#8489)#8514
mezzeddinee wants to merge 1 commit into
DIRACGrid:integrationfrom
mezzeddinee:feature/8489-opensearch-global-index-prefix

Conversation

@mezzeddinee

@mezzeddinee mezzeddinee commented Apr 14, 2026

Copy link
Copy Markdown

Fixes #8489 by introducing index prefixing as a global OpenSearch setting and removing MonitoringDB-local prefix handling.

What changed

  • src/DIRAC/ConfigurationSystem/Client/Utilities.py

    • Read /Systems/NoSQLDatabases/IndexPrefix in getElasticDBParameters.
    • Normalize value (strip().lower()) and pass it as IndexPrefix.
  • src/DIRAC/Core/Base/ElasticDB.py

    • Read global IndexPrefix from DB parameters.
    • Forward it to ElasticSearchDB as globalIndexPrefix.
  • src/DIRAC/Core/Utilities/ElasticSearchDB.py

    • Added globalIndexPrefix constructor parameter and property setter.
    • Added _withGlobalPrefix() helper to prepend the global prefix to index names/patterns.
    • Applied prefixing across index/template/search/read/write/delete operations.
    • Refactored bulk_index flow to avoid apparent double-prepending.
  • src/DIRAC/MonitoringSystem/DB/MonitoringDB.py

    • Removed MonitoringDB-specific IndexPrefix lookup from CS.
    • Rely on inherited global prefix mechanism via ElasticDB.
  • docs/source/AdministratorGuide/ExternalsSupport/index.rst

    • Documented IndexPrefix under Systems/NoSQLDatabases and clarified it is lower-case.
  • docs/source/AdministratorGuide/Systems/MonitoringSystem/index.rst

    • Updated docs to reference global Systems/NoSQLDatabases/IndexPrefix.
  • src/DIRAC/Core/scripts/install_full.cfg

    • Added commented example for global IndexPrefix under NoSQLDatabases.
  • src/DIRAC/Core/Utilities/test/Test_ElasticSearchDB.py

    • Added and updated unit tests to verify expected index names across ElasticSearchDB interactions when a global index prefix is configured.

@mezzeddinee mezzeddinee marked this pull request as draft April 14, 2026 08:01
Comment thread docs/source/AdministratorGuide/ExternalsSupport/index.rst Outdated
Comment thread src/DIRAC/Core/Utilities/ElasticSearchDB.py Outdated
Comment thread src/DIRAC/Core/Utilities/ElasticSearchDB.py Outdated
Comment thread src/DIRAC/Core/Utilities/ElasticSearchDB.py Outdated
Comment thread src/DIRAC/Core/Utilities/ElasticSearchDB.py
@aldbr

aldbr commented Apr 15, 2026

Copy link
Copy Markdown
Contributor

Note: @mezzeddinee if you updated your PR and you want a reviewer having another look at it, please mark it as "Ready for review", cheers

@mezzeddinee

Copy link
Copy Markdown
Author

Note: @mezzeddinee if you updated your PR and you want a reviewer having another look at it, please mark it as "Ready for review", cheers
Addressed all requested changes in 8a86d4d (docs lower-case note, globalIndexPrefix property, removed list/tuple branch, bulk_index double-prefix fix). PR is now ready for re-review.

@mezzeddinee mezzeddinee marked this pull request as ready for review April 18, 2026 08:01
@mezzeddinee mezzeddinee force-pushed the feature/8489-opensearch-global-index-prefix branch 3 times, most recently from 93b2b0c to bd18638 Compare April 19, 2026 16:09

@fstagni fstagni left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would you try and write (or expand) a unit test to make sure that the index names are the expected ones, on every interaction?

This is quite delicate and have been bitten by this before.

@mezzeddinee mezzeddinee marked this pull request as draft April 25, 2026 12:18
@mezzeddinee mezzeddinee force-pushed the feature/8489-opensearch-global-index-prefix branch 6 times, most recently from 122e54d to 0b5fe4e Compare April 26, 2026 11:30
@aldbr

aldbr commented Apr 27, 2026

Copy link
Copy Markdown
Contributor

Is it ready to be reviewed @mezzeddinee?

@mezzeddinee mezzeddinee force-pushed the feature/8489-opensearch-global-index-prefix branch 7 times, most recently from cc4aad6 to fdb4d89 Compare May 9, 2026 09:14
@mezzeddinee

mezzeddinee commented May 9, 2026

Copy link
Copy Markdown
Author

I added/expanded ElasticSearchDB unit tests for global index prefix behavior, and they pass locally. Current remaining failure appear perhaps unrelated to this change: DiracX Unit Tests failing in CI setup (pixi add local file:// path resolution).

Any guidance on these CI issues is appreciated.

@mezzeddinee mezzeddinee marked this pull request as ready for review May 9, 2026 09:42
@mezzeddinee mezzeddinee force-pushed the feature/8489-opensearch-global-index-prefix branch 5 times, most recently from 0c84afd to 497c80d Compare May 12, 2026 08:42
@aldbr

aldbr commented May 13, 2026

Copy link
Copy Markdown
Contributor
  • Test in certification

@mezzeddinee mezzeddinee force-pushed the feature/8489-opensearch-global-index-prefix branch 2 times, most recently from e7f84a9 to 21fd60d Compare May 13, 2026 08:18
@mezzeddinee mezzeddinee force-pushed the feature/8489-opensearch-global-index-prefix branch 2 times, most recently from a9179ad to f254268 Compare May 21, 2026 06:52
@mezzeddinee mezzeddinee force-pushed the feature/8489-opensearch-global-index-prefix branch from f254268 to eb65f42 Compare June 8, 2026 06:51
:param dict orderBy: it is a dictionary in case we want to order the result {key:'desc'} or {key:'asc'}
:returns: a list of unique value for a certain key from the dictionary.
"""
indexName = self._withGlobalPrefix(indexName)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you're right. It was functionally safe because _withGlobalPrefix() is idempotent, but the call was redundant. I removed prefixing from getUniqueValue() so _Search() is solely responsible for applying the global prefix.

indexName = self._withGlobalPrefix(indexName)
sLog.debug(f"Getting indices alias of {indexName}")
# we only return indexes which belong to a specific prefix for example 'lhcb-production' or 'dirac-production etc.
return list(self.client.indices.get_alias(index=f"{indexName}*"))

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just wondering, what happens if indexName is None 😅

@mezzeddinee mezzeddinee Jun 8, 2026

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch. When indexName was omitted, _withGlobalPrefix("") returned an empty string and getIndexes() queried all indices with *. I changed it to use globalIndexPrefix when no index name is provided, resulting in <globalIndexPrefix>*. Without a configured global prefix, it preserves the existing * behavior.

@mezzeddinee mezzeddinee force-pushed the feature/8489-opensearch-global-index-prefix branch from eb65f42 to 2d77bfd Compare June 8, 2026 10:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature] Introduce the concept of a global prefix for OpenSearch indexes

3 participants