feat(db): bulk all-VO read queries for ResourceStatusDB#939
Open
chrisburr wants to merge 1 commit into
Open
Conversation
Replaces the per-resource lookups with bulk queries returning every site/resource status across all VOs, keyed by VO so that rows for the same resource in different VOs cannot overwrite each other, plus max(DateEffective)/count queries to serve as cache revisions. The logic layer maps the rows to the rss Pydantic models per VO, skipping storage elements with an incomplete set of access rows. Co-authored-by: Loxeris <30194187+Loxeris@users.noreply.github.com>
5 tasks
Loxeris
reviewed
Jun 11, 2026
| SiteStatus.status, | ||
| SiteStatus.reason, | ||
| SiteStatus.vo, | ||
| ).where(SiteStatus.status_type == "all") |
Member
There was a problem hiding this comment.
Status type is expected to always be "all" for sites I believe. Is this condition needed ?
Suggested change
| ).where(SiteStatus.status_type == "all") | |
| ) |
| stmt = select( | ||
| func.max(SiteStatus.date_effective), | ||
| func.count(), | ||
| ).where(SiteStatus.status_type == "all") |
Member
There was a problem hiding this comment.
same here
Suggested change
| ).where(SiteStatus.status_type == "all") | |
| ) |
Member
There was a problem hiding this comment.
AFAIK diracx-logic tests are not running when using pytest.
I think the base pyproject needs to be updated: pyproject.toml
| async def get_resource_statuses( | ||
| self, | ||
| name: str, | ||
| status_types: list[str] | None = None, |
Member
There was a problem hiding this comment.
Don't we also need to be able to filter the element type?
Comment on lines
+85
to
+93
| all_rows = await resource_status_db.get_resource_statuses(["all"]) | ||
|
|
||
| result: dict[str, dict[str, FTSStatus]] = {} | ||
| for vo, names in all_rows.items(): | ||
| result[vo] = { | ||
| name: FTSStatus(all=map_status(rows["all"].Status, rows["all"].Reason)) | ||
| for name, rows in names.items() | ||
| } | ||
|
|
Member
There was a problem hiding this comment.
Here we would get every resource status casted as a FTS status, right? i.e {TestSE: FTSStatus}
I think we need to filter element_type to fts in get_resource_statuses()
Same for get_compute_statuses, and get_storage_statuses.
Comment on lines
+177
to
+185
| stmt = insert(SiteStatus).values( | ||
| Name=name, | ||
| Status=status, | ||
| StatusType="all", | ||
| VO=vo, | ||
| Reason=reason, | ||
| DateEffective=date_effective or now, | ||
| LastCheckTime=last_check_time or now, | ||
| ) |
Member
There was a problem hiding this comment.
Suggested change
| stmt = insert(SiteStatus).values( | |
| Name=name, | |
| Status=status, | |
| StatusType="all", | |
| VO=vo, | |
| Reason=reason, | |
| DateEffective=date_effective or now, | |
| LastCheckTime=last_check_time or now, | |
| ) | |
| stmt = insert(SiteStatus).values( | |
| Name=name, | |
| Status=status, | |
| StatusType="all", | |
| VO=vo, | |
| Reason=reason, | |
| ElementType="Site", | |
| DateEffective=date_effective or now, | |
| LastCheckTime=last_check_time or now, | |
| ) |
| name: str, | ||
| status: str, | ||
| status_type: str, | ||
| vo: str, |
Member
There was a problem hiding this comment.
Suggested change
| vo: str, | |
| vo: str, | |
| element_type: str, |
| name: Resource name. | ||
| status: Status value. | ||
| status_type: One of "all", "ReadAccess", "WriteAccess", etc. | ||
| vo: Virtual organisation (e.g. "lhcb", "all"). |
Member
There was a problem hiding this comment.
Suggested change
| vo: Virtual organisation (e.g. "lhcb", "all"). | |
| vo: Virtual organisation (e.g. "lhcb", "all"). | |
| element_type: One of "ComputeElement", "FTS", "StorageElement". |
| Name=name, | ||
| Status=status, | ||
| StatusType=status_type, | ||
| VO=vo, |
Member
There was a problem hiding this comment.
Suggested change
| VO=vo, | |
| VO=vo, | |
| ElementType=element_type, |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Replaces the per-resource lookups with bulk queries returning every site/resource status across all VOs, keyed by VO so that rows for the same resource in different VOs cannot overwrite each other, plus max(DateEffective)/count queries to serve as cache revisions. The logic layer maps the rows to the rss Pydantic models per VO, skipping storage elements with an incomplete set of access rows.
Replaces #910
Part of #839