Skip to content

HBASE-30242 Add Region server level HFile Compression Metrics #8389

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Umeshkumar9414 wants to merge 2 commits into
base: master
Choose a base branch
from
Open

HBASE-30242 Add Region server level HFile Compression Metrics #8389

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
e8d6c6e
HBASE-30242 Add Region server level HFile Compression Metrics
Umeshkumar9414 Jun 21, 2026
c1633d7
HBASE-30242 reimplement ratio as uncompressed / compressed
Umeshkumar9414 Jul 1, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions ...-compat/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsRegionServerSource.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -233,6 +233,8 @@ public interface MetricsRegionServerSource extends BaseSource, JvmPauseMonitorSo
String MEMSTORE_OFFHEAP_SIZE_DESC = "Off-heap Size of the memstore";
String STOREFILE_SIZE = "storeFileSize";
String STOREFILE_SIZE_GROWTH_RATE = "storeFileSizeGrowthRate";
String STOREFILE_UNCOMPRESSED_SIZE = "storeFileUncompressedSize";
String STOREFILE_COMPRESSION_RATIO = "storeFileCompressionRatio";
String MAX_STORE_FILE_AGE = "maxStoreFileAge";
String MIN_STORE_FILE_AGE = "minStoreFileAge";
String AVG_STORE_FILE_AGE = "avgStoreFileAge";
Expand All @@ -244,6 +246,10 @@ public interface MetricsRegionServerSource extends BaseSource, JvmPauseMonitorSo
String STOREFILE_SIZE_DESC = "Size of storefiles being served.";
String STOREFILE_SIZE_GROWTH_RATE_DESC =
"Bytes per second by which the size of storefiles being served grows.";
String STOREFILE_UNCOMPRESSED_SIZE_DESC = "Total uncompressed size of storefiles being served.";
String STOREFILE_COMPRESSION_RATIO_DESC =
"Compression ratio of storefiles (uncompressed/compressed). Higher values indicate better"
+ " compression. Returns 0.0 when there is no data.";
String TOTAL_REQUEST_COUNT = "totalRequestCount";
String TOTAL_REQUEST_COUNT_DESC =
"Total number of requests this RegionServer has answered; increments the count once for "
Expand Down
4 changes: 4 additions & 0 deletions ...pat/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsRegionServerSourceImpl.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -519,6 +519,10 @@ private MetricsRecordBuilder addGaugesToMetricsRecordBuilder(MetricsRecordBuilde
.addGauge(Interns.info(MEMSTORE_OFFHEAP_SIZE, MEMSTORE_OFFHEAP_SIZE_DESC),
rsWrap.getOffHeapMemStoreSize())
.addGauge(Interns.info(STOREFILE_SIZE, STOREFILE_SIZE_DESC), rsWrap.getStoreFileSize())
.addGauge(Interns.info(STOREFILE_UNCOMPRESSED_SIZE, STOREFILE_UNCOMPRESSED_SIZE_DESC),
rsWrap.getStoreFileUncompressedSize())
.addGauge(Interns.info(STOREFILE_COMPRESSION_RATIO, STOREFILE_COMPRESSION_RATIO_DESC),
rsWrap.getStoreFileCompressionRatio())
.addGauge(Interns.info(STOREFILE_SIZE_GROWTH_RATE, STOREFILE_SIZE_GROWTH_RATE_DESC),
rsWrap.getStoreFileSizeGrowthRate())
.addGauge(Interns.info(MAX_STORE_FILE_AGE, MAX_STORE_FILE_AGE_DESC),
Expand Down
11 changes: 11 additions & 0 deletions ...compat/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsRegionServerWrapper.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -116,6 +116,17 @@ public interface MetricsRegionServerWrapper {
*/
long getStoreFileSize();

/**
* Get the total uncompressed size of the store files this region server is serving from.
*/
long getStoreFileUncompressedSize();

/**
* Get the compression ratio of store files on this region server. This is the ratio of
* uncompressed data size to compressed on-disk size. Returns 0.0 when there is no data.
*/
double getStoreFileCompressionRatio();

/**
* Get the growth rate of the store files this region server is serving from.
*/
Expand Down
16 changes: 16 additions & 0 deletions ...er/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsRegionServerWrapperImpl.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -533,6 +533,20 @@ public long getStoreFileSize() {
return aggregate.storeFileSize;
}

@Override
public long getStoreFileUncompressedSize() {
return aggregate.storeFileUncompressedSize;
}

@Override
public double getStoreFileCompressionRatio() {
Comment thread
apurtell marked this conversation as resolved.
long uncompressed = aggregate.storeFileUncompressedSize;

@mnpoonia mnpoonia Aug 14, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

aggregate is a volatile snapshot as andrew mentioned and can be replaced between these two reads. That can calculate the ratio from different refreshes, including nonzero / 0 when regions are removed. Could we capture aggregate once, read both values from that snapshot?

RegionMetricAggregate current = aggregate;
if (current.storeFileSize <= 0) {
  return 0.0;
}
return (double) current.storeFileUncompressedSize / current.storeFileSize;

if (uncompressed == 0) {
Comment thread
apurtell marked this conversation as resolved.
return 0.0;
}
return (double) uncompressed / aggregate.storeFileSize;
}

@Override
public double getStoreFileSizeGrowthRate() {
return aggregate.storeFileSizeGrowthRate;
Expand Down Expand Up @@ -783,6 +797,7 @@ private static final class RegionMetricAggregate {
private long onHeapMemstoreSize = 0;
private long offHeapMemstoreSize = 0;
private long storeFileSize = 0;
private long storeFileUncompressedSize = 0;
private double storeFileSizeGrowthRate = 0;
private long maxStoreFileCount = 0;
private long maxStoreFileAge = 0;
Expand Down Expand Up @@ -979,6 +994,7 @@ private StoreFileStats aggregateStores(List<HStore> stores) {
onHeapMemstoreSize += store.getMemStoreSize().getHeapSize();
offHeapMemstoreSize += store.getMemStoreSize().getOffHeapSize();
storeFileSize += store.getStorefilesSize();
storeFileUncompressedSize += store.getStoreSizeUncompressed();
maxStoreFileCount = Math.max(maxStoreFileCount, store.getStorefilesCount());

maxStoreFileAge =
Expand Down
10 changes: 10 additions & 0 deletions ...er/src/test/java/org/apache/hadoop/hbase/regionserver/MetricsRegionServerWrapperStub.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -87,6 +87,16 @@ public long getStoreFileSize() {
return 1900;
}

@Override

@mnpoonia mnpoonia Aug 14, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This verifies source export from a fixed wrapper stub, but it does not exercise MetricsRegionServerWrapperImpl aggregation or ratio calculation. Could we extend TestMetricsRegionServerAggregate to assert the summed compressed and uncompressed sizes, along with the resulting uncompressed / compressed ratio, including the empty-aggregate case?

Using stores with different individual ratios would also verify that this remains a weighted aggregate rather than an average of per-store ratios.

Basically +1 to Andrew's comment.

public long getStoreFileUncompressedSize() {
return 3800;
}

@Override
public double getStoreFileCompressionRatio() {
return 2;
}

@Override
public double getStoreFileSizeGrowthRate() {
return 50.0;
Expand Down
2 changes: 2 additions & 0 deletions hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestMetricsRegionServer.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -90,6 +90,8 @@ public void testWrapperSource() {
HELPER.assertGauge("memstoreHeapSize", 500, serverSource);
HELPER.assertGauge("memstoreOffHeapSize", 600, serverSource);
HELPER.assertGauge("storeFileSize", 1900, serverSource);
HELPER.assertGauge("storeFileUncompressedSize", 3800, serverSource);
HELPER.assertGauge("storeFileCompressionRatio", 2.0, serverSource);
HELPER.assertGauge("storeFileSizeGrowthRate", 50.0, serverSource);
HELPER.assertCounter("totalRequestCount", 899, serverSource);
HELPER.assertCounter("totalRowActionRequestCount",
Expand Down