Skip to content
Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
68 changes: 34 additions & 34 deletions modules/install/pages/sizing-general.adoc
Original file line number Diff line number Diff line change
@@ -1,11 +1,11 @@
= Sizing Guidelines
:description: Evaluate the overall performance and capacity goals that you have for Couchbase, and use that information to determine the necessary resources that you'll need in your deployment.
:description: Evaluate the overall performance and capacity goals that you have for Couchbase and use that information to determine the necessary resources that you'll need in your deployment.
:stem: latexmath

[abstract]
{description}

The most common and important questions you need to ask when deploying a new Couchbase Server cluster are how many nodes you need, and what size they need to be.
The most common and important questions you need to ask when deploying a new Couchbase Server cluster are how many nodes you need and what size they need to be.

With the increasing number of Couchbase services and the flexibility of the Couchbase Data Platform, the answer to this question can be challenging.
This guide aims to help you better size your deployment.
Expand All @@ -17,9 +17,9 @@ NOTE: The sizing recommendations and calculations discussed in this guide are ba
== General Considerations

The sizing of your Couchbase Server cluster is critical to its overall stability and performance.
While there are some xref:pre-install.adoc[basic system requirements] to run Couchbase Server, you still need to evaluate the overall performance and capacity requirements for your workload and dataset, and then divide that into the hardware and resources you have available.
While there are some xref:pre-install.adoc[basic system requirements] to run Couchbase Server, you still need to evaluate the overall performance and capacity requirements for your workload and dataset and then divide that into the hardware and resources you have available.

Your application wants the majority of reads to come out of the cache, and to have the I/O capacity to handle the writes.
Your application wants the majority of reads to come out of the cache and to have the I/O capacity to handle the writes.
There needs to be enough capacity in all areas to support everything the system is doing while maintaining the required level of performance.

=== Multi-Dimensional Scaling
Expand All @@ -31,9 +31,9 @@ This independent service model allows you to take advantage of Multi-Dimensional
Multi-Dimensional Scaling lets you fine-tune your cluster for optimal handling of changing workload-requirements, for each individual Couchbase Service.

Every Service has different demands on hardware resources.
Multi-Dimensional Scaling plays an important role when sizing your Couchbase cluster, both pre and post-deployment.
Multi-Dimensional Scaling plays an important role when sizing your Couchbase cluster, both pre- and post-deployment.
For example, core Data Service operations can often benefit from scaling out smaller commodity nodes.
Low latency operations with the Query Service might see a greater benefit from scaling up hardware resources on a given node.
Low-latency operations with the Query Service might see a greater benefit from scaling up hardware resources on a given node.

For more information about the nature and resource demands of each Couchbase Service, see xref:learn:services-and-indexes/services/services.adoc[Services].

Expand Down Expand Up @@ -61,7 +61,7 @@ Determine your allocation based on the following factors:
Some components that require RAM are:
+
--
** All index storage types which need sufficient memory quota allocation for proper functioning.
** All index storage types that need sufficient memory quota allocation for proper functioning.
** The Search Service.
--
+
Expand Down Expand Up @@ -96,24 +96,24 @@ Requirements for your disk subsystem are:
+
--
* *Disk size* — Specifies the disk storage space needed to hold your entire dataset.
* *Disk I/O* — Combines your sustained read/write rate, database file compaction, and any other operations that requires disk access.
* *Disk I/O* — Combines your sustained read/write rate, database file compaction, and any other operations that require disk access.
--
+
To better support Couchbase Server, keep in mind the following:
+
--
* Disk space continues to grow if fragmentation ratio keeps climbing.
To mitigate this, add enough buffer in your disk space to store all of the data.
* Disk space continues to grow if the fragmentation ratio keeps climbing.
To mitigate this, add enough buffer in your disk space to store all the data.
Monitor your cluster's fragmentation ratio in the Couchbase Server Web Console and trigger compaction processes as needed.
* Couchbase recommends using Solid State Drives (SSD) when possible.
An SSD gives much better performance than a Hard Disk Drive (HDD) when it comes to disk throughput and latency.
--

Network::
Enough network bandwidth is vital to the performance of Couchbase Server.
A reliable high-speed network for intra-cluster and inter-cluster communications has a huge effect on overall performance and scalability of Couchbase Server.
A reliable high-speed network for intra-cluster and inter-cluster communications has a huge effect on the overall performance and scalability of Couchbase Server.
+
Most deployments can achieve optimal performance with 1 Gbps interconnects, but some may need 10 Gbps.
NOTE: For optimal performance, Couchbase Server must be deployed with a minimum bandwidth of _at least_ 1 Gbps, but some deployments may need more.

== Sizing Data Service Nodes

Expand All @@ -125,15 +125,15 @@ For information, see xref:learn:buckets-memory-and-storage/storage-engines.adoc[

It's important to keep use-cases and application workloads in mind since different application workloads have different resource requirements.
For example, if your working set needs to be fully in-memory, your cluster might need more RAM.
If your application requires only 10% of data in-memory, you need disks with enough space to store all of the data, and that are fast enough for your read/write operations.
If your application requires only 10% of data in-memory, you need disks with enough space to store all the data, and that are fast enough for your read/write operations.

=== RAM Sizing for Data Service Nodes

You can start sizing the Data Service nodes by answering the following questions:

* Is the application primarily using individual document access?
* Do you plan to use XDCR?
* What's your working set size and what are your data operation throughput and latency requirements?
* What's your working set size, and what are your data operation throughput and latency requirements?

Answers to the above questions can help you better understand the capacity requirement of your cluster and provide a better estimation for sizing.

Expand Down Expand Up @@ -246,7 +246,7 @@ If this value constantly grows, consider the growth rate into the future when si
Couchbase Server always compresses documents when storing them on disk.
See xref:learn:buckets-memory-and-storage/compression.adoc[] for more information about compression in Couchbase Server.
Documents containing JSON data or binaries can be compressed.
Binary data that's already compressed (such as compressed images or videos) cannot be compressed further.
Binary data that's already compressed, (such as compressed images or videos,) cannot be compressed further.

+
Couchbase Server uses the https://en.wikipedia.org/wiki/Snappy_(compression)[Snappy^] compression algorithm, which prioritizes speed while still providing reasonable compression.
Expand All @@ -266,7 +266,7 @@ Also, Couchbase Server creates a tombstone record for each deleted document.
This record consumes a small amount of additional disk space.

* The metadata purge interval you'll use.
This purge process removes tombstones that records the deletion of documents.
This purge process removes tombstones that record the deletion of documents.
The default purge interval is 3 days.
For more information about the purge interval, see xref:manage:manage-settings/configure-compact-settings.adoc#tombstone-purge-interval[Metadata Purge Interval].

Expand Down Expand Up @@ -354,7 +354,7 @@ Using the formulas above, you can calculate the total disk space required as fol
+
[stem]
++++
S_{\mathrm{dataset}} = 1,000,000 \times 10,000 \times 0.7 = 7,000,000,000 \text{bytes}
S_{\mathrm{dataset}} = 1,000,000 \times 10,000 \times 0.7 = 7,000,000,000 \text{bytes}
++++

. Calculate the total metadata size:
Expand All @@ -378,7 +378,7 @@ S_{\mathrm{keys}} = 1,000,000 \times 32 = 32,000,000 \text{bytes}
+
[stem]
++++
S_{\mathrm{tombstones}} = (32 + 60) \times 3 \times (1 + 1) \times 5,000 = 2,760,000 \text{bytes}
S_{\mathrm{tombstones}} = (32 + 60) \times 3 \times (1 + 1) \times 5,000 = 2,760,000 \text{bytes}
++++

. Calculate the total disk space:
Expand All @@ -388,8 +388,8 @@ S_{\mathrm{tombstones}} = (32 + 60) \times 3 \times (1 + 1) \times 5,000 = 2,76
++++
\begin{equation}
\begin{split}
\text{total disk space} = & ( 7,000,000,000 \times (1 + 1) \\
& + 56,000,000 + 32,000,000 ) \\
\text{total disk space} = & (7,000,000,000 \times (1 + 1) \\
& + 56,000,000 + 32,000,000) \\
& \times 2.2 \\
& + 2,760,000 \\
& = 30,996,360,000 \text{bytes}
Expand Down Expand Up @@ -423,10 +423,10 @@ Similar to the nodes that run the Data Service, answer the following questions t
* What is the minimum, maximum, or average value size of the indexed fields?
* How many indexes do you need?
* How many documents need to be indexed?
* What is the working set percentage of index required memory?
* What is the working set percentage of index-required memory?
--

Answers to these questions can help you better understand the capacity requirement of your cluster, and provide a better estimation for sizing.
Answers to these questions can help you better understand the capacity requirement of your cluster and provide a better estimation for sizing.

*The following is an example use-case for sizing RAM for the Index service:*

Expand Down Expand Up @@ -515,10 +515,10 @@ Based on the above formula, these are the suggested sizing guidelines:
| (3200000000) * (1 + 0.25) * 0.2 = 800000000 bytes
|===

The previous example shows the memory requirement of a secondary index with 10M index entries, each with a 50 bytes secondary key and a 30 bytes DocumentID.
The previous example shows the memory requirement of a secondary index with 10M index entries, each with a 50-byte secondary key and a 30-byte DocumentID.
The memory usage requirements are 2.5{nbsp}GB (Nitro, 100% resident), 1{nbsp}GB (plasma, 20% resident), 800{nbsp}MB (ForestDB, 20% resident).

NOTE: The storage engine used in the sizing calculation corresponds to the storage mode chosen for Index Service as explained in the table below.
NOTE: The storage engine used in the sizing calculation corresponds to the storage mode chosen for Index Service, as explained in the table below.

.Storage engine and storage mode
|===
Expand Down Expand Up @@ -557,7 +557,7 @@ To size the Search Service nodes in your cluster, you need the following informa
* A sample document or documents that show the structure of your data.
* The specific queries per second (QPS) target you need from the Search Service.

You should also consider your replication, recovery, and high availability needs.
You should also consider your replication, recovery, and high-availability needs.

With all this information, you can work with Couchbase Support to get the most accurate sizing for your Search workload.

Expand All @@ -578,7 +578,7 @@ For example, if your target QPS is 30,000 and your queries are less complex, div
30,0000_{\mathrm{QPS}} \div 200_{\mathrm{Mid}} = 150_{\mathrm{vCPUs}}
++++

The formula gives a target of 150 vCPUs for a mid range workload with a less complex query.
The formula gives a target of 150 vCPUs for a mid-range workload with a less complex query.

If your queries were more complex, but the QPS target was the same, the calculation changes to use a value of 150 and a result of 200 vCPUs:

Expand Down Expand Up @@ -622,7 +622,7 @@ Use the following formula first to calculate the number of bytes per document in
++++
\begin{equation}
\begin{split}
\text{Per Doc Index Bytes} = ( ( W \cdot 1024 \cdot \text{f_text} \cdot \text{m_text} ) + ( W \cdot 1024 \cdot \text{f_kw} \cdot \text{m_kw} ) + B ) \times (1 + D)
\text{Per Doc Index Bytes} = (( W \cdot 1024 \cdot \text{f_text} \cdot \text{m_text} ) + (W \cdot 1024 \cdot \text{f_kw} \cdot \text{m_kw} ) + B) \times (1 + D)
\end{split}
\end{equation}
++++
Expand All @@ -639,7 +639,7 @@ You need to know the following variables for the formula:
| stem:[{\text{f_text}}]
a| A measure of the analyzed text from your JSON documents.

You can omit this value if you're using primarily keyword searches and do not have longer-form text fields that require an xref:search:customize-index.adoc#analyzers[analyzer].
You can omit this value if you're primarily using keyword searches and do not have longer-form text fields that require an xref:search:customize-index.adoc#analyzers[analyzer].

You can use the following value ranges based on the kind of analyzed text you have in your index:

Expand Down Expand Up @@ -673,7 +673,7 @@ For a good planning range, try a value between `0.10-0.18`.
To get the most accurate values for stem:[{\text{m_kw}}] and your RAM sizing calculations, contact Couchbase Support.

| stem:[B]
a| The number of bytes needed for storing field values for your documents, if xref:search:child-field-options-reference.adoc#store[store] is enabled for a child field mapping.
a| The number of bytes needed for storing field values for your documents if xref:search:child-field-options-reference.adoc#store[store] is enabled for a child field mapping.

If you're not storing any field values in your Search index, set this value to `0`.

Expand All @@ -688,11 +688,11 @@ If you want to add numeric and geospatial fields to your sizing estimate, change

[asciimath]
++++
\text{Per Doc Index Bytes} = ( ( W \cdot 1024 \cdot \text{f_text} \cdot \text{m_text} ) + ( W \cdot 1024 \cdot \text{f_kw} \cdot \text{m_kw} )
\text{Per Doc Index Bytes} = (( W \cdot 1024 \cdot \text{f_text} \cdot \text{m_text} ) + (W \cdot 1024 \cdot \text{f_kw} \cdot \text{m_kw})

+ ( W \cdot 1024 \cdot 0.02_\text{f_numeric} \cdot 2.0_\text{m_numeric} )
+ (W \cdot 1024 \cdot 0.02_\text{f_numeric} \cdot 2.0_\text{m_numeric})

+ ( W \cdot 1024 \cdot 0.002_\text{f_geo} \cdot 2.0_\text{m_geo} )+ B ) \times (1 + D)
+ (W \cdot 1024 \cdot 0.002_\text{f_geo} \cdot 2.0_\text{m_geo} )+ B) \times (1 + D)
++++

The values provided in the preceding formula for stem:[{\text{f_numeric}}], stem:[{\text{m_numeric}}], stem:[{\text{f_geo}}] and stem:[{\text{m_geo}}] are reasonable defaults for most numeric and geospatial search workloads.
Expand Down Expand Up @@ -857,7 +857,7 @@ Answer the following questions to help size the Query Service nodes on your clus
Different queries have different resource requirements.
A simple query might return results within milliseconds while a complex query may require several seconds.

The formula used to calculate the number of queries that's processed simultaneously is `CPU_cores * 4`.
The formula used to calculate the number of queries that are processed simultaneously is `CPU_cores * 4`.
The formula used to calculate the maximum queue-length for queries is `CPU_cores * 256`.
If you reach either limit, the system rejects additional queries with a 503 error.

Expand All @@ -879,7 +879,7 @@ The type of query you want to run determines the required amount of temporary di
For example, queries with heavy JOINs, aggregates, windowing, or additional predicates require more temporary disk space.
Typically, the temporary disk space can be 2x the data space.

* The percent of data shadowed, which is dependent on your use case.
* The percentage of data shadowed, which is dependent on your use case.

* When you load data from the Data Service into the Analytics Service, you can apply a filter to reduce both the loaded data size and the Analytics Service storage requirements proportionally.

Expand Down