C++ Low-latency Logging

Table of contents

Objective

To focus on the performance of the latest binlog, Quill, and fmtlog libraries on common data (string, double, complex structures, etc.).

Methods of Comparison

There are two methods:

• Method 1: Warm up a certain number of times compare to say 10,000 times and then for loop 100,000 times, and then divide the total time by 100,000. However, this method fully utilizes the cache, and it is only a lower limit of the ideal delay situation, which does not match the actual situation.
• Method 2: Record the time difference each time, and then add up the 100,000 times. Note that you need to insert some meaningless operations between each time to simulate a real caching scenario. (poison L1 cache)

All of the following results are from tests run on a kernel-isolated machine with kernel tethering.

Single-thread results

log one Foo struct (char data[200] + int64_t + double):

result of Method 1:

result of Method 2:

Note that in order to optimize Quill’s speed, I went ahead and changed the defaut_queue_capacity of the config to 67108864 (64MB), to prevent the need to create a new SPSC queue while keeping it running (since Quill internally has a thread_local SPSC queue for each thread, and the logger’s background queue pops out of each thread’s SPSC queue and writes files to it). (Because Quill has a thread_local SPSC queue for each thread, and then the logger’s backend queue pops out of each thread’s SPSC queue and writes to the file), as does fmtlog.

In method two, I create a volatile char array and rewrite the size of the L1 cache over and over again, which is equivalent to inserting a poison L1 cache operation in between each time.

Multi-thread results

Multi-thread results are all in Method 2.

4-thread results:

8-thread results:

Conclusions

The theoretical limit of single-threaded Quill is about the same as binlog, but the actual result is still faster than binlog. But for multithreading, 4-threaded Quill is slightly worse than the new version of binlog, and 8-threaded Quill is about the same as the new version of binlog.

In terms of single-threaded performance, on simple objects, single-threaded performance fmtlog≈Quill>binlog. On complex struct, single-thread performance binlog>fmtlog>Quill, but binlog performs worse at 99.999 and 100th percentile.

Based on the multithread and single-thread results, basically the latency of each of the multithreading results is higher. The performance of multithreading to log the Foo struct is binlog>fmtlog>Quill, but multithreading to log a simple variable is fmtlog>binlog>Quill.

This should be because the Foo struct has an advantage over the binlog, but if you consider more complex scenarios, such as when a user may need to log a struct, a string, or a double, then it is important to look at the performance of a single simple object.

Realizations

Quill:

binlog:

References

• Quill: Asynchronous Low Latency C++ Logging Library
• Binlog: A high performance C++ log library, producing structured binary logs
• fmtlog: A performant fmtlib-style logging library with latency in nanoseconds.