[OSDOCS-16926]: CQA: etcd practices

etcd/etcd-practices.adoc

@@ -6,7 +6,8 @@ include::_attributes/common-attributes.adoc[]
 
 toc::[]
 
-The following documentation provides information about recommended performance and scalability practices for etcd.
+[role="_abstract"]
+Follow storage, latency, and hardware validation guidance for etcd to reduce leader elections, API timeouts, and control plane instability on {product-title}.
 
 // Storage practices for etcd
 include::modules/recommended-etcd-practices.adoc[leveloffset=+1]
@@ -16,12 +17,14 @@ include::modules/recommended-cluster-latency-etcd.adoc[leveloffset=+1]
 
 [role="_additional-resources"]
 .Additional resources
+
 * xref:../etcd/etcd-performance.adoc#etcd-tuning-parameters_etcd-performance[Setting tuning parameters for etcd]
 
 // Validating the hardware for etcd
 include::modules/etcd-verify-hardware.adoc[leveloffset=+1]
 
 [role="_additional-resources"]
 .Additional resources
+
 * link:https://access.redhat.com/solutions/4885641[How to use `fio` to check etcd disk performance in {product-title}]
 * link:https://access.redhat.com/articles/6271341[etcd performance troubleshooting guide for {product-title}]

modules/etcd-verify-hardware.adoc

@@ -6,38 +6,48 @@
 [id="etcd-verify-hardware_{context}"]
 = Validating the hardware for etcd
 
-To validate the hardware for etcd before or after you create the {product-title} cluster, you can use fio.
+[role="_abstract"]
+Validate control plane disk performance with `fio` before or after you create the {product-title} cluster so that you can confirm that storage meets etcd latency requirements.
+
+The output of the following procedure reports whether the disk is fast enough to host etcd by comparing the 99th percentile of the fsync metric captured from the run to see if it is less than 10 ms. 
+
+A few of the most important etcd metrics that might affected by I/O performance are as follows:
+
+* `etcd_disk_wal_fsync_duration_seconds_bucket` reports the etcd WAL fsync duration.
+* `etcd_disk_backend_commit_duration_seconds_bucket` reports the etcd backend commit latency duration.
+* `etcd_server_leader_changes_seen_total` reports the leader changes.
+
+Because etcd replicates the requests among all the members, its performance strongly depends on network input/output (I/O) latency. High network latencies result in etcd heartbeats taking longer than the election timeout, which results in leader elections that are disruptive to the cluster. A key metric to monitor on a deployed {product-title} cluster is the 99th percentile of etcd network peer latency on each etcd cluster member. Use Prometheus to track the metric.
+
+The `histogram_quantile(0.99, rate(etcd_network_peer_round_trip_time_seconds_bucket[2m]))` metric reports the round trip time for etcd to finish replicating the client requests between the members. Ensure that it is less than 50 ms.
 
 .Prerequisites
 
-* Container runtimes such as Podman or Docker are installed on the machine that you are testing.
-* Data is written to the `/var/lib/etcd` path.
+* A container runtime, such as Podman or Docker, is installed on the machine that you are testing.
+* The `/var/lib/etcd` path is available for writing test data.
 
 .Procedure
-* Run fio and analyze the results:
+
+* Run `fio` and analyze the results:
 +
 --
-** If you use Podman, run this command:
-[source,terminal]
+** If you use Podman, run the following command:
 +
+[source,terminal]
 ----
 $ sudo podman run --volume /var/lib/etcd:/var/lib/etcd:Z quay.io/cloud-bulldozer/etcd-perf
 ----
 
-** If you use Docker, run this command:
-[source,terminal]
+** If you use Docker, run the following command:
 +
+[source,terminal]
 ----
 $ sudo docker run --volume /var/lib/etcd:/var/lib/etcd:Z quay.io/cloud-bulldozer/etcd-perf
 ----
 --
 
-The output reports whether the disk is fast enough to host etcd by comparing the 99th percentile of the fsync metric captured from the run to see if it is less than 10 ms. A few of the most important etcd metrics that might affected by I/O performance are as follows:
+.Verification
 
-* `etcd_disk_wal_fsync_duration_seconds_bucket` metric reports the etcd's WAL fsync duration
-* `etcd_disk_backend_commit_duration_seconds_bucket`  metric reports the etcd backend commit latency duration
-* `etcd_server_leader_changes_seen_total` metric reports the leader changes
+* Confirm that the 99th percentile of the fsync metric from the run is less than 10 ms.
 
-Because etcd replicates the requests among all the members, its performance strongly depends on network input/output (I/O) latency. High network latencies result in etcd heartbeats taking longer than the election timeout, which results in leader elections that are disruptive to the cluster. A key metric to monitor on a deployed {product-title} cluster is the 99th percentile of etcd network peer latency on each etcd cluster member. Use Prometheus to track the metric.
-
-The `histogram_quantile(0.99, rate(etcd_network_peer_round_trip_time_seconds_bucket[2m]))` metric reports the round trip time for etcd to finish replicating the client requests between the members. Ensure that it is less than 50 ms.
+* Confirm that the report indicates that the disk is fast enough to host etcd.

modules/recommended-cluster-latency-etcd.adoc

@@ -7,21 +7,20 @@
 = Cluster latency requirements for etcd
 
 [role="_abstract"]
-Two important constraints should be addressed to provide a low-latency, high-availability network for etcd: 
+etcd requires low network and disk I/O latency so that Raft consensus can commit changes quickly and recover from leader failures without destabilizing the cluster.
 
-* network I/O latency 
-* disk I/O latency
+Address two constraints to provide a low-latency, high-availability network for etcd: network I/O latency and disk I/O latency.
 
-etcd uses the Raft consensus algorithm, and every change should replicate to a majority of the cluster members before it commits. This process is highly sensitive to network and disk performance. The minimum time for an etcd request is the Round-Trip Time (RTT) between members, plus the time required for data to write to permanent storage.
+etcd uses the Raft consensus algorithm. Every change must replicate to a majority of cluster members before it commits. This process is highly sensitive to network and disk performance. The minimum time for an etcd request is the round-trip time (RTT) between members, plus the time required to write data to permanent storage.
 
-To achieve high availability, etcd should detect and recover from a leader failure quickly. This depends on two key tuning parameters:
+To achieve high availability, etcd must detect and recover from a leader failure quickly. This depends on two key tuning parameters:
 
 Heartbeat Interval:: The frequency that the leader sends a heartbeat to followers. This value should be close to the average RTT between members.
 Election Timeout:: The time a follower waits without hearing a heartbeat before it attempts to become the new leader. This should be at least 10 times the RTT value to account for network variance.
 
-In a healthy cluster, the round-trip time between members should be less than 50 ms to ensure stability and avoid frequent leader elections. This is why etcd clusters are often deployed within a single data center or availability zone to minimize physical distance and network latency.
+In a healthy cluster, the round-trip time between members should be less than 50 ms. This limit helps ensure stability and avoid frequent leader elections. For this reason, etcd clusters are often deployed within a single data center or availability zone to minimize physical distance and network latency.
 
-To support a low-latency, high-availability network, especially during the leader election process, an arbiter site should be located where it provides an RTT latency of less than 10 ms. The arbiter component of a network maintains consistency and availability in a distributed system. 
+To support a low-latency, high-availability network during leader election, place an arbiter site where RTT latency is less than 10 ms. The arbiter maintains consistency and availability in a distributed system.
 
 // Need to clarify so the impression is that the arbiter is not counted in the number of nodes
-// In the case of leader election and similar processes, the arbiter is used when clusters have an odd number of nodes, so a majority vote determines the system state.
+// In the case of leader election and similar processes, the arbiter is used when clusters have an odd number of nodes, so a majority vote determines the system state.

modules/recommended-etcd-practices.adoc

@@ -6,17 +6,20 @@
 [id="recommended-etcd-practices_{context}"]
 = Storage practices for etcd
 
-Because etcd writes data to disk and persists proposals on disk, its performance depends on disk performance. Although etcd is not particularly I/O intensive, it requires a low latency block device for optimal performance and stability. Because the consensus protocol for etcd depends on persistently storing metadata to a log (WAL), etcd is sensitive to disk-write latency. Slow disks and disk activity from other processes can cause long fsync latencies.
+[role="_abstract"]
+etcd performance depends on disk latency and throughput. Follow these storage practices to keep etcd stable and avoid API timeouts on {product-title}.
 
-Those latencies can cause etcd to miss heartbeats, not commit new proposals to the disk on time, and ultimately experience request timeouts and temporary leader loss. High write latencies also lead to an OpenShift API slowness, which affects cluster performance. Because of these reasons, avoid colocating other workloads on the control-plane nodes that are I/O sensitive or intensive and share the same underlying I/O infrastructure.
+Because etcd writes data to disk and persists proposals on disk, its performance depends on disk performance. Although etcd is not particularly I/O intensive, it requires a low latency block device for optimal performance and stability. Because the consensus protocol for etcd depends on persistently storing metadata to a write-ahead log (WAL), etcd is sensitive to disk-write latency. Slow disks and disk activity from other processes can cause long fsync latencies.
 
-Run etcd on a block device that can write at least 50 IOPS of 8KB sequentially, including fdatasync, in under 10ms. For heavy loaded clusters, sequential 500 IOPS of 8000 bytes (2 ms) are recommended. To measure those numbers, you can use a benchmarking tool, such as the `fio` command.
+Those latencies can cause etcd to miss heartbeats, not commit new proposals to the disk on time, and ultimately experience request timeouts and temporary leader loss. High write latencies also lead to {product-title} API slowness, which affects cluster performance. Because of these reasons, avoid colocating other workloads on the control-plane nodes that are I/O sensitive or intensive and share the same underlying I/O infrastructure.
 
-To achieve such performance, run etcd on machines that are backed by SSD or NVMe disks with low latency and high throughput. Consider single-level cell (SLC) solid-state drives (SSDs), which provide 1 bit per memory cell, are durable and reliable, and are ideal for write-intensive workloads.
+Run etcd on a block device that can write at least 50 input/output operations per second (IOPS) of 8KB sequentially, including fdatasync, in under 10ms. For heavily loaded clusters, sequential 500 IOPS of 8000 bytes (2 ms) are recommended. To measure those numbers, you can use a benchmarking tool, such as the `fio` command.
+
+To achieve such performance, run etcd on machines that are backed by solid-state drives (SSDs) or Non-Volatile Memory Express (NVMe) disks with low latency and high throughput. Consider single-level cell (SLC) SSDs, which provide 1 bit per memory cell, are durable and reliable, and are ideal for write-intensive workloads.
 
 [NOTE]
 ====
-The load on etcd arises from static factors, such as the number of nodes and pods, and dynamic factors, including changes in endpoints due to pod autoscaling, pod restarts, job executions, and other workload-related events. To accurately size your etcd setup, you must analyze the specific requirements of your workload. Consider the number of nodes, pods, and other relevant factors that impact the load on etcd.
+The load on etcd arises from static factors, such as the number of nodes and pods, and from dynamic factors. Dynamic factors include endpoint changes from pod autoscaling, pod restarts, job executions, and other workload-related events. To accurately size your etcd setup, analyze the specific requirements of your workload. Consider the number of nodes, pods, and other relevant factors that impact the load on etcd.
 ====
 
 The following hard drive practices provide optimal etcd performance:
@@ -27,12 +30,22 @@ The following hard drive practices provide optimal etcd performance:
 * Prefer high-bandwidth reads for faster recovery from failures.
 * Use solid state drives as a minimum selection. Prefer NVMe drives for production environments.
 * Use server-grade hardware for increased reliability.
-* Avoid NAS or SAN setups and spinning drives. Ceph Rados Block Device (RBD) and other types of network-attached storage can result in unpredictable network latency. To provide fast storage to etcd nodes at scale, use PCI passthrough to pass NVM devices directly to the nodes.
+* Avoid network-attached storage (NAS) or storage area network (SAN) setups and spinning drives. Ceph Rados Block Device (RBD) and other types of network-attached storage can result in unpredictable network latency. To provide fast storage to etcd nodes at scale, use Peripheral Component Interconnect (PCI) passthrough to pass NVM devices directly to the nodes.
 * Always benchmark by using utilities such as `fio`. You can use such utilities to continuously monitor the cluster performance as it increases.
 * Avoid using the Network File System (NFS) protocol or other network based file systems.
 
 Some key metrics to monitor on a deployed {product-title} cluster are p99 of etcd disk write ahead log duration and the number of etcd leader changes. Use Prometheus to track these metrics.
 
+Important etcd metrics that I/O performance might affect include the following:
+
+* `etcd_disk_wal_fsync_duration_seconds_bucket` reports the etcd WAL fsync duration.
+* `etcd_disk_backend_commit_duration_seconds_bucket` reports the etcd backend commit latency duration.
+* `etcd_server_leader_changes_seen_total` reports the leader changes.
+
+Because etcd replicates the requests among all the members, its performance strongly depends on network input/output (I/O) latency. High network latencies result in etcd heartbeats taking longer than the election timeout, which results in leader elections that are disruptive to the cluster. A key metric to monitor on a deployed {product-title} cluster is the 99th percentile of etcd network peer latency on each etcd cluster member. Use Prometheus to track the metric.
+
+The `histogram_quantile(0.99, rate(etcd_network_peer_round_trip_time_seconds_bucket[2m]))` metric reports the round trip time for etcd to finish replicating the client requests between the members. Ensure that it is less than 50 ms.
+
 [NOTE]
 ====
 The etcd member database sizes can vary in a cluster during normal operations. This difference does not affect cluster upgrades, even if the leader size is different from the other members.