HDDS-15440. Combine pipelineMap and pipeline2container to a single map in PipelineStateMap

[omkenge]

What changes were proposed in this pull request?

HDDS-15440. Combine pipeline metadata and container mappings into a single map

This patch refactors PipelineStateMap by combining the existing pipelineMap and pipeline2container data structures into a single map that stores both the Pipeline object and its associated container information.

Previously, PipelineStateMap maintained two separate maps keyed by PipelineID:

Map<PipelineID, Pipeline> pipelineMap
Map<PipelineID, NavigableSet<ContainerID>> pipeline2container

Maintaining related information in separate maps increased code complexity and introduced the possibility of the two maps becoming inconsistent. This patch introduces a unified data structure that encapsulates both the pipeline metadata and its container set, ensuring that all pipeline-related state is managed together.

The change simplifies state management, improves maintainability, and reduces the risk of synchronization issues between multiple maps while preserving the existing behavior of PipelineStateMap.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-15440

How was this patch tested?

The patch was tested by:

• Running the existing PipelineStateMap unit tests.
• Verifying that pipelines can be added, retrieved, and removed successfully after the refactoring.
• Verifying that container mappings associated with a pipeline are correctly maintained through add/remove operations.
• Building and running the SCM module tests to ensure there are no regressions caused by the internal data structure change.

No functional behavior changes were introduced; the patch is a refactoring focused on improving the internal implementation of PipelineStateMap.

[omkenge]

Cc @adoroszlai

[adoroszlai]

Thanks @omkenge for the patch. Please enable the build-branch workflow in your fork.

[omkenge]

@adoroszlai Done

[omkenge]

CC @szetszwo

[szetszwo]

+1 the change looks good.

[omkenge]

CC @sumitagrawl

[ivandika3]

@omkenge Please check the failing tests.

[omkenge]

CC @ivandika3 @sumitagrawl

[ivandika3]

@omkenge The test failures do not seem transient (https://github.com/omkenge/ozone/actions/runs/27664987145). PTAL.

[adoroszlai]

retrigger after transient failures

The failures are not transient. Please fix them instead of simply retriggering CI.

To find the problem, you can download test logs from https://github.com/omkenge/ozone/actions/runs/27664987145 (see Artifacts at the bottom).

There is at least one big problem that fails all tests that write any data:

java.lang.UnsupportedOperationException
	at java.base/java.util.Collections$UnmodifiableCollection.removeAll(Collections.java:1105)
	at org.apache.hadoop.hdds.scm.container.ContainerManagerImpl.getMatchingContainer(ContainerManagerImpl.java:381)
	at org.apache.hadoop.hdds.scm.pipeline.WritableRatisContainerProvider.selectContainer(WritableRatisContainerProvider.java:184)
	at org.apache.hadoop.hdds.scm.pipeline.WritableRatisContainerProvider.getContainer(WritableRatisContainerProvider.java:154)
	at org.apache.hadoop.hdds.scm.pipeline.WritableRatisContainerProvider.getContainer(WritableRatisContainerProvider.java:83)
	at org.apache.hadoop.hdds.scm.pipeline.WritableContainerFactory.getContainer(WritableContainerFactory.java:72)
	at org.apache.hadoop.hdds.scm.block.BlockManagerImpl.allocateBlock(BlockManagerImpl.java:163)

[szetszwo]

@omkenge , thanks for the update!

• Please see the comment inlined.
• There are still some test failures in https://github.com/omkenge/ozone/actions/runs/27710532153 . Please take a look.

[omkenge]

CC @ivandika3 @szetszwo
Thanks for Your patience :)

[omkenge]

Cc @sumitagrawl

[szetszwo]

@omkenge , Now we are copying in all cases.

Could you take a look at this #10454 (comment) ?

[omkenge]

cc @szetszwo
Open for guidance and suggestions:)

[omkenge]

@szetszwo It would be great if you can guide me on failing CI