What are the hardware recommendations for my cluster?

Summary

You want to allocate resources for a QuasarDB cluster and need to have an estimate on the hardware requirements in terms of storage, cache and speed.

Getting started

When determining the minimal cluster size, the dataset and how you interact with it is the driving force behind the decision making. We will be looking at the following variables:

  • Ingestion speed
  • Data retention period
  • Querying patterns

Throughout this article we use a trading firm as an example, but the same strategy can be applied to any use case.

Row size

The average size (in bytes) of a row is the basis of most of the calculations in this article. To calculate the average row size, please determine:

  • The number of columns per row
  • The data types for each of these columns
  • 8 additional bytes for storing row's associated timestamp

Look up the size of each of the data types in the following table:

Integer 8 bytes
Double 8 bytes
Timestamp 16 bytes
Blob Average blob size

For example, a dataset for storing stock ticks might look like this:

exchange id open close volume
NSDQ AAPL 17.50 17.83 1727

Here we're counting 5 columns, and can calculate the row size as follows: 

bytes = 8 + 4 + 4 + 8 + 8 + 8

As such, the row above would occupy at most 40 bytes.

Note: QuasarDB employs on-the-fly compression of data, and as such your storage requirements will most likely be a lot less. Since the compression factor depends upon your exact dataset, we will ignore it for the purposes of this article.

We encourage you to do some benchmarks of yourself to get an idea what the compression factor for your dataset is.

Ingestion speed

We measure the ingestion speed in terms of peak required performance per second and total volume per day per node. To calculate your ingestion speed requirements, please determine:

  • The number of nodes in your cluster
  • Your replication factor
  • Your peak ingestion speed in terms of rows per second
  • Your average daily ingestion volume in terms of rows per day

You can now calculate your peak I/O requirements and daily dataset size as follows:

peak_io [B/s]     = row_size [B] * peak_rows [1/s] * (replication_factor [1] / number_of_nodes [1])
bytes_per_day [B] = row_size [B] * rows_per_day [1] * (replication_factor [1] / number_of_nodes [1])

In our stock tick examples, assuming a 5-node cluster with a replication factor of 3, peak volume of 50,000 rows per second and a daily volume of 2 billion rows, we would get the following results:

peak_io [B/s]     = 40 * 50000 * (3 / 5) = 1200000 Bytes per sec
bytes_per_day [B] = 40 * 2000000000 * (3 / 5) = 48000000000 Bytes per day

 As such, our ingestion IO would peak at ~1.2MB/sec and generate a total of ~48GB per day. Note that these values are based on uncompressed data, and in a real world scenario it would most likely be less.

Total disk storage

To calculate the total required disk storage per peer, please determine:

  • Your data retention period in days

Using this data retention period, we can calculate storage requirements per node as follows:

peak_io = row_size * peak_rows_per_second * (replication_factor / number_of_nodes)

total_bytes [B] = bytes_per_day [B] * retention_days [1]

In our example, assuming we're targeting a data retention period of 90 days, this would result in the following results:

total_bytes [B] = 48GB * 90 = 4320GB

Total memory

The memory capacity calculation brings with it a clear trade-off between costs and performance. Where you can ingest data into QuasarDB as fast as you can without affecting memory/cache, we base the memory requirements on your query interactions with QuasarDB. 

Specifically, to calculate your memory requirements, please determine:

  • The recency of data you wish to query in realtime

Typically this means the time range for queries where you wish a response in under 1ms. Examples of appropriate use cases will be real-time monitoring and the default time range of customer-facing dashboards.

We will let this recency requirement guide our calculation to determine the amount of memory you should provision per node:

memory [B] = bytes_per_day [B] * recency_days [1]

In our example, if we have customer facing dashboards which typically stay within the 1-week range, we would have the following results:

memory_bytes [B] = 48GB * 7 = 336GB

Provisioning your node with this amount of RAM will ensure that the part of your dataset that has the highest performance requirements always resides in memory.

Note: compared to other database engines, QuasarDB is extremely fast in processing data on disk as well, and can easily process tens of terrabytes in a matter of seconds on moderately sized clusters. It can be worthwhile to invest in faster disk storage instead. 

Disk storage type

There are several choices to make between what type of disk storage to use. We wil base the decision on the peak throughput your disks need.

Please determine:

  • How often (in queries / sec) do you query "cold" data that is not in memory?
  • What is the timerange (in seconds) used in these queries that is not part of the cached range?
  • What are your latency requirements for each of these queries ?

Once we know these variables, we can employ two different ways to establish a desired performance in GB/s of the database.

Peak load

When you're planning to use QuasarDB for realtime analytical purposes (e.g. customer-facing dashboards), we must work with strict latency requirements and determine our peak desired performance in terms of GB/s. We can do this using the following formula:

# Calculate the amount of bytes we store per day
bytes_per_day [B] = row_size [B] * rows_per_day [1] * (replication_factor [1] / number_of_nodes [1])

# Calculate how much data (in seconds) we must process every second, minus the cached data
days_per_sec [1/s] = (query_days [1] - cached_days [1]) / time_available [s]

# Now that we know how much data we must process per second, and how many bytes there
# is stored for every second, we can calculate the throughput.
throughput_bytes [B/s] = days_per_sec [1/s] * bytes_per_day [B]

Using our example where we keep two weeks of data in cache, and have 90 days of data retention, we could have a situation where we issue two types of queries:

  • 1 query that touches the entire dataset every hour and should be returned within 1 minute,
  • 1 query that touches the last 4 weeks every minute that should be returned within 3 seconds.

Our calculation for peak load would then look like this:

bytes_per_day [B]         = 40 * 2000000000 * (3 / 5) = 48GB/day

days_per_sec_query1 [1/s] = (90 - 14) [1] / 60 [s] = 1.25 Days/s
throughput_query1 [B/s] = 1.25 [1/s] * 48GB [B] = 60GB/s

days_per_sec_query2 [1/s] = (28 - 14) [1] / 15 [s] = 1 Days/s
throughput_query2 [B/s] = 1 [1/s] * 48GB [B] = 48GB/s

throughput [B/s] = 60GB/s + 48GB/s = 108GB/sec
Average load

When you're planning to use QuasarDB for more batch-oriented purposes, and have less strict latency requirements on queries, we can make a slight adjustment to the formula to establish the average desired performance in terms of GB/s:

# Calculate the amount of bytes we store per day
bytes_per_day [B] = row_size [B] * rows_per_day [1] * (replication_factor [1] / number_of_nodes [1])

# Calculate how much data (in seconds) we must process every second, minus the cached data
days_per_sec [1/s] = (query_days [1] - cached_days [1]) / query_frequency [s]

# Now that we know how much data we must process per second, and how many bytes there
# is stored for every second, we can calculate the throughput.
throughput_bytes [B/s] = days_per_sec [1/s] * bytes_per_day [B]

The difference here is that rather than determining days_per_sec using the available time for the query (i.e. latency requirement), we use the frequency of the query (in terms of seconds per query) to establish how much time we have to process each of these queries. For our earlier example, our calculation might then look like this:

bytes_per_day [B]         = 40 * 2000000000 * (3 / 5) = 48GB/day

days_per_sec_query1 [1/s] = (90 - 14) [1] / 3600 [s] = 0.02 Days/s
throughput_query1 [B/s] = 0.02 [1/s] * 48GB [B] = 1GB/s

days_per_sec_query2 [1/s] = (28 - 14) [1] / 60 [s] = 0.25 Days/s
throughput_query2 [B/s] = 0.25 [1/s] * 48GB [B] = 12GB/s

throughput [B/s] = 1GB/s + 12GB/s = 13GB/sec
Choosing disk storage

Based on your results, we can then recommend the following:

  • If your numbers require less than 100MB/sec of throughput per node, magnetic disks will be enough;
  • If your numbers stay below 250MB/sec, SATA SSD is good enough
  • Once you go above these numbers, SATA quickly becomes the bottleneck and we recommend getting NVMe SSD due to PCIe lanes providing much more bandwidth: one M-key NVMe SSD can provide 20Git/sec of performance, and scale horizontally provided your environment has enough PCIe lanes available
  • For the best read/write performance, we recommend Intel Optane disks

CPU

The choice of CPU depends a lot on your exact dataset, but as a rule of thumb we recommend:

  • Single-socket CPUs, because NUMA employed in multi-socket setups makes memory latency unpredictable
  • Account for 0.5 core of CPU per gbit of I/O

The gbit of I/O should include ingestion, exporting and querying (including cached). For example, if we end up consuming 750MB/sec, we would recommend:

core_count [1] = 750 [MB/s] / 1Gbit * 0.5 = ~ 3 cores

Network interface

For most workloads, the network interface is the biggest bottleneck and limits QuasarDB from ingesting data faster. As such, we recommend basing your network interface on the peak ingestion speed determined earlier. 

As such, for many customers we recommend 10gbit network interfaces.

Was this article helpful?
0 out of 0 found this helpful