HDDS-15024. Track pending containers in SCM to prevent Datanode over-allocation

[ashishkumar50]

What changes were proposed in this pull request?

Introduce PendingContainerTracker in SCM to track container allocations that are issued but not yet fully realized on DataNodes.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-15024

How was this patch tested?

Unit test

[szetszwo]

@ashishkumar50 , thanks a lot for splitting the PR

• For simplicity, let's don't remove buckets. The number of datanodes is small (~10k). We just keep all of them in the map.
• Then, the map becomes very simple. Let's create a class.
• Always roll before return.

  class DatanodeBuckets {
    private final ConcurrentHashMap<DatanodeID, TwoWindowBucket> map = new ConcurrentHashMap<>();

    TwoWindowBucket get(DatanodeID id) {
      final TwoWindowBucket bucket = map.compute(id, (k, b) -> b != null ? b : new TwoWindowBucket(rollIntervalMs));
      bucket.rollIfNeeded();
      return bucket;
    }

    TwoWindowBucket get(DatanodeDetails dn) {
      return map.get(dn.getID());
    }
  }

[szetszwo]

Just use:

      if (usableSpace - pendingBytes >= containerSize) {

[ashishkumar50]

@szetszwo thanks for the review, handled the comments.

[szetszwo]

@ashishkumar50 , thanks for the update! The change looks mostly good.

• All the synchronized (bucket) should be removed.
• Are there legitimate cases to pass null to the methods and ignore it? If yes, please add a comment describing the cases. Otherwise, please replace the null check with Objects.requireNonNull(..).
  We usually use Objects.requireNonNull(..) to detect bugs. If we ignore and return, it hides the bug and may lead to more serious problem such as data loss.

[szetszwo]

@ashishkumar50 , thanks for the quick update!

+1 the change looks good.

[rakeshadr]

Thanks @ashishkumar50 for the continuous efforts. Added two log improvements, please take care.

+1 LGTM