feat: add global OpenSearch index prefix support (#8489)#8514
feat: add global OpenSearch index prefix support (#8489)#8514mezzeddinee wants to merge 1 commit into
Conversation
|
Note: @mezzeddinee if you updated your PR and you want a reviewer having another look at it, please mark it as "Ready for review", cheers |
|
93b2b0c to
bd18638
Compare
fstagni
left a comment
There was a problem hiding this comment.
Would you try and write (or expand) a unit test to make sure that the index names are the expected ones, on every interaction?
This is quite delicate and have been bitten by this before.
122e54d to
0b5fe4e
Compare
|
Is it ready to be reviewed @mezzeddinee? |
cc4aad6 to
fdb4d89
Compare
|
I added/expanded ElasticSearchDB unit tests for global index prefix behavior, and they pass locally. Current remaining failure appear perhaps unrelated to this change: DiracX Unit Tests failing in CI setup (pixi add local file:// path resolution). Any guidance on these CI issues is appreciated. |
0c84afd to
497c80d
Compare
|
e7f84a9 to
21fd60d
Compare
a9179ad to
f254268
Compare
f254268 to
eb65f42
Compare
| :param dict orderBy: it is a dictionary in case we want to order the result {key:'desc'} or {key:'asc'} | ||
| :returns: a list of unique value for a certain key from the dictionary. | ||
| """ | ||
| indexName = self._withGlobalPrefix(indexName) |
There was a problem hiding this comment.
_withGlobalPrefix is also called as part of _Search, is that expected?
https://github.com/mezzeddinee/DIRAC/blob/eb65f42578f58f8bf2ed9b3cff0335dd70b75ea7/src/DIRAC/Core/Utilities/ElasticSearchDB.py#L391
There was a problem hiding this comment.
you're right. It was functionally safe because _withGlobalPrefix() is idempotent, but the call was redundant. I removed prefixing from getUniqueValue() so _Search() is solely responsible for applying the global prefix.
| indexName = self._withGlobalPrefix(indexName) | ||
| sLog.debug(f"Getting indices alias of {indexName}") | ||
| # we only return indexes which belong to a specific prefix for example 'lhcb-production' or 'dirac-production etc. | ||
| return list(self.client.indices.get_alias(index=f"{indexName}*")) |
There was a problem hiding this comment.
Just wondering, what happens if indexName is None 😅
There was a problem hiding this comment.
Good catch. When indexName was omitted, _withGlobalPrefix("") returned an empty string and getIndexes() queried all indices with *. I changed it to use globalIndexPrefix when no index name is provided, resulting in <globalIndexPrefix>*. Without a configured global prefix, it preserves the existing * behavior.
eb65f42 to
2d77bfd
Compare
Fixes #8489 by introducing index prefixing as a global OpenSearch setting and removing MonitoringDB-local prefix handling.
What changed
src/DIRAC/ConfigurationSystem/Client/Utilities.py
/Systems/NoSQLDatabases/IndexPrefixingetElasticDBParameters.strip().lower()) and pass it asIndexPrefix.src/DIRAC/Core/Base/ElasticDB.py
IndexPrefixfrom DB parameters.ElasticSearchDBasglobalIndexPrefix.src/DIRAC/Core/Utilities/ElasticSearchDB.py
globalIndexPrefixconstructor parameter and property setter._withGlobalPrefix()helper to prepend the global prefix to index names/patterns.bulk_indexflow to avoid apparent double-prepending.src/DIRAC/MonitoringSystem/DB/MonitoringDB.py
IndexPrefixlookup from CS.ElasticDB.docs/source/AdministratorGuide/ExternalsSupport/index.rst
IndexPrefixunderSystems/NoSQLDatabasesand clarified it is lower-case.docs/source/AdministratorGuide/Systems/MonitoringSystem/index.rst
Systems/NoSQLDatabases/IndexPrefix.src/DIRAC/Core/scripts/install_full.cfg
IndexPrefixunderNoSQLDatabases.src/DIRAC/Core/Utilities/test/Test_ElasticSearchDB.py