path: root/doc/design-thoughts/ring-v2.txt

2024-02-20 - Ring buffer v2
===========================

Goals:
  - improve the multi-thread performance of rings so that traces can be written
    from all threads in parallel without the huge bottleneck of the lock that
    is currently necessary to protect the buffer. This is important for mmapped
    areas that are left as a file when the process crashes.

  - keep traces synchronous within a given thread, i.e. when the TRACE() call
    returns, the trace is either written into the ring or lost due to slow
    readers.

  - try hard to limit the cache line bounces between threads due to the use of
    a shared work area.

  - make waiting threads not disturb working ones

  - continue to work on all supported platforms, with a particular focus on
    performance for modern platforms (memory ordering, DWCAS etc can be used if
    they provide any benefit), with a fallback for inferior platforms.

  - do not reorder traces within a given thread.

  - do not break existing features

  - do not significantly increase memory usage


Analysis of the current situation
=================================

Currently, there is a read lock around the call to __sink_write() in order to
make sure that an attempt to write the number of lost messages is delivered
with highest priority and is consistent with the lost counter. This doesn't
seem to pose any problem at this point though if it were, it could possibly
be revisited.

__sink_write() calls ring_write() which first measures the input string length
from the multiple segments, and locks the ring:
  - while trying to free space
  - while copying the message, due to the buffer's API

Because of this, there is a huge serialization and threads wait in queue. Tests
involving a split of the lock and a release around the message copy have shown
a +60% performance increase, which is still not acceptable.


First proposed approach
=======================

The first approach would have consisted in writing messages in small parts:
  1) write 0xFF in the tag to mean "size not filled yet"
  2) write the message's length and write a zero tag after the message's
     location
  3) replace the first tag to 0xFE to indicate the size is known, but the
     message is not filled yet.
  4) memcpy() of the message to the area
  5) replace the first tag to 0 to mark the entry as valid.

It's worth noting that doing that without any lock will allow a second thread
looping on the first tag to jump to the second tag after step 3. But the cost
is high: in a 64-thread scenario where each of them wants to send one message,
the work would look like this:
  - 64 threads try to CAS the tag. One gets it, 63 fail. They loop on the byte
    in question in read-only mode, waiting for the byte to change. This loop
    constantly forces the cache line to switch from MODIFIED to SHARED in the
    writer thread, and makes it a pain for it to write the message's length
    just after it.

  - once the first writer thread finally manages to write the length (step 2),
    it writes 0xFE on the tag to release the waiting threads, and starts with
    step 4. At this point, 63 threads try a CAS on the same entry, and this
    hammering further complicates the memcpy() of step 4 for the first 63 bytes
    of the message (well, 32 on avg since the tag is not necessarily aligned).
    One thread wins, 62 fail. All read the size field and jump to the next tag,
    waiting in read loops there. The second thread starts to write its size and
    faces the same difficulty as described above, facing 62 competitors when
    writing its size and the beginning of its message.

  - when the first writer thread writes the end of its message, it gets close
    to the final tag where the 62 waiting threads are still reading, causing
    a slow down again with the loss of exclusivity on the cache line. This is
    the same for the second thread etc.

Thus, on average, a writing thread is hindered by N-1 threads at the beginning
of its message area (in the first 32 bytes on avg) and by N-2 threads at the
end of its area (in the last 32 bytes on avg). Given that messages are roughly
218 bytes on avg for HTTP/1, this means that roughly 1/3 of the message is
written under severe cache contention.

In addition to this, the buffer's tail needs to be updated once all threads are
ready, something that adds the need for synchronization so that the last writing
threads (the most likely to complete fast due to less perturbations) needs to
wait for all previous ones. This also means N atomic writes to the tail.


New proposal
============

In order to address the contention scenarios above, let's try to factor the
work as much as possible. The principle is that threads that want to write will
either do it themselves or declare their intent and wait for a writing thread
to do it for them. This aims at ensuring a maximum usage of read-only data
between threads, and to leave the work area read-write between very few
threads, and exclusive for multiple messages at once, avoiding the bounces.

First, the buffer will have 2 indexes:
  - head: where the valid data start
  - tail: where new data need to be appended

When a thread starts to work, it will keep a copy of $tail and push it forward
by as many bytes as needed to write all the messages it has to. In order to
guarantee that neither the previous nor the new $tail point to an outdated or
overwritten location but that there is always a tag there, $tail contains a
lock bit in its highest bit that will guarantee that only one at a time will
update it. The goal here is to perform as few atomic ops as possible in the
contended path so as to later amortize the costs and make sure to limit the
number of atomic ops on the wait path to the strict minimum so that waiting
threads do not hinder the workers:

  Fast path:
    1 load($tail) to check the topmost bit
    1 CAS($tail,$tail|BIT63) to set the bit (atomic_fetch_or / atomic_bts also work)
    1 store(1 byte tag=0xFF) at the beginning to mark the area busy
    1 store($tail) to update the new value
    1 copy of the whole message
    1 store(1 byte tag=0) at the beginning to release the message

  Contented path:
    N load($tail) while waiting for the bit to be zero
    M CAS($tail,$tail|BIT63) to try to set the bit on tail, competing with others
    1 store(1 byte tag=0xFF) at the beginning to mark the area busy
    1 store($tail) to update the new value
    1 copy of the whole message
    1 store(1 byte tag=0) at the beginning to release the message

Queue
-----

In order to limit the contention, writers will not start to write but will wait
in a queue, announcing their message pointers/lengths and total lengths. The
queue is made of a (ptr, len) pair that points to one such descriptor, located
in the waiter thread's stack, that itself points to the next pair. In fact
messages are ordered in a LIFO fashion but that isn't important since intra-
thread ordering is preserved (and in the worst case it will also be possible
to write them from end to beginning).

The approach is the following: a writer loasd $tail and sees it's busy, there's
no point continuing, it will add itself to the queue, announcing (ptr, len +
next->len) so that by just reading the first entry, one knows the total size
of the queue. And it will wait there as long as $tail has its topmost bit set
and the queue points to itself (meaning it's the queue's leader), so that only
one thread in the queue watches $tail, limiting the number of cache line
bounces. If the queue doesn't point anymore to the current thread, it means
another thread has taken it over so there's no point continuing, this thread
just becomes passive. If the lock bit is dropped from $tail, the watching
thread needs to re-check that it's still the queue's leader before trying to
grab the lock, so that only the leading thread will attempt it. Indeed, a few
of the last leading threads might still be looping, unaware that they're no
longer leaders. A CAS(&queue, self, self) will do it. Upon failure, the thread
just becomes a passive thread. Upon success, the thread is a confirmed leader,
it must then try to grab the tail lock. Only this thread and a few potential
newcomers will compete on this one. If the leading thread wins, it brings all
the queue with it and the newcomers will queue again. If the leading thread
loses, it needs to loop back to the point above, watching $tail and the
queue. In this case a newcomer might have grabbed the lock. It will notice
the non-empty queue and will take it with it. Thus in both cases the winner
thread does a CAS(queue, queue, NULL) to reset the queue, keeping the previous
pointer.

At this point the winner thread considers its own message size plus the
retrieved queue's size as the total required size and advances $tail by as
much, and will iterate over all messages to copy them in turn. The passive
threads are released by doing XCHG(&ptr->next, ptr) for each message, that
is normally impossible otherwise. As such, a passive thread just has to
loop over its own value, stored in its own stack, reading from its L1 cache
in loops without any risk of disturbing others, hence no need for EBO.

During the time it took to update $tail, more messages will have been
accumulating in the queue from various other threads, and once $tail is
written, one thread can pick them up again.

The benefit here is that the longer it takes one thread to free some space,
the more messages add up in the queue and the larger the next batch, so that
there are always very few contenders on the ring area and on the tail index.
At worst, the queue pointer is hammered but it's not on the fast path, since
wasting time here means all waiters will be queued.

Also, if we keep the first tag unchanged after it's set to 0xFF, it allows to
avoid atomic ops inside all the message. Indeed there's no reader in the area
as long as the tag is 0xFF, so we can just write all contents at once including
the varints and subsequent message tags without ever using atomic ops, hence
not forcing ordered writes. So maybe in the end there is some value in writing
the messages backwards from end to beginning, and just writing the first tag
atomically but not the rest.

The scenario would look like this:

  (without queue)

    - before starting to work:
        do {
            while (ret=(load(&tail) & BIT63))
                ;
        } while (!cas(&tail, &ret, ret | BIT63));

    - at this point, alone on it and guaranteed not to change
    - after new size is calculated, write it and drop the lock:

        store(&tail, new_tail & ~BIT63);

    - that's sufficient to unlock other waiters.

  (with queue)

      in_queue = 0;
      do {
         ret = load(&tail);
         if (ret & BIT63) {
             if (!in_queue) {
                 queue_this_node();
                 in_queue = 1;
             }
             while (ret & BIT63)
                 ;
         }
      } while (!cas(&tail, &ret, ret | BIT63));

      dequeue(in_queue) etc.

  Fast path:
    1 load($tail) to check the topmost bit
    1 CAS($tail,$tail|BIT63) to set the bit (atomic_fetch_or / atomic_bts also work)
    1 load of the queue to see that it's empty
    1 store(1 byte tag=0xFF) at the beginning to mark the area busy
    1 store($tail) to update the new value
    1 copy of the whole message
    1 store(1 byte tag=0) at the beginning to release the message

  Contented path:
    1 load($tail) to see the tail is changing
    M CAS(queue,queue,self) to try to add the thread to the queue (avgmax nbthr/2)
    N load($tail) while waiting for the lock bit to become zero
    1 CAS(queue,self,self) to check the leader still is
    M CAS($tail,$tail|BIT63) to try to set the bit on tail, competing with others
    1 CAS(queue,queue,NULL) to reset the queue
    1 store(1 byte tag=0xFF) at the beginning to mark the area busy
    1 store($tail) to update the new value
    1 copy of the whole message
    P copies of individual messages
    P stores of individual pointers to release writers
    1 store(1 byte tag=0) at the beginning to release the message

Optimal approach (later if needed?): multiple queues. Each thread has one queue
assigned, either from a thread group, or using a modulo from the thread ID.
Same as above then.


Steps
-----

It looks that the queue is what allows the process to scale by amortizing a
single lock for every N messages, but that it's not a prerequisite to start,
without a queue threads can just wait on $tail.


Options
-------

It is possible to avoid the extra check on CAS(queue,self,self) by forcing
writers into the queue all the time. It would slow down the fast path but
may improve the slow path, both of which would become the same:

  Contented path:
    1 XCHG(queue,self) to try to add the thread to the queue
    N load($tail) while waiting for the lock bit to become zero
    M CAS($tail,$tail|BIT63) to try to set the bit on tail, competing with others
    1 CAS(queue,self,NULL) to reset the queue
    1 store(1 byte tag=0xFF) at the beginning to mark the area busy
    1 store($tail) to update the new value
    1 copy of the whole message
    P copies of individual messages
    P stores of individual pointers to release writers
    1 store(1 byte tag=0) at the beginning to release the message

There seems to remain a race when resetting the queue, where a newcomer thread
would queue itself while not being the leader. It seems it can be addressed by
deciding that whoever gets the bit is not important, what matters is the thread
that manages to reset the queue. This can then be done using another XCHG:

    1 XCHG(queue,self) to try to add the thread to the queue
    N load($tail) while waiting for the lock bit to become zero
    M CAS($tail,$tail|BIT63) to try to set the bit on tail, competing with others
    1 XCHG(queue,NULL) to reset the queue
    1 store(1 byte tag=0xFF) at the beginning to mark the area busy
    1 store($tail) to update the new value
    1 copy of the whole message
    P copies of individual messages
    P stores of individual pointers to release writers
    1 store(1 byte tag=0) at the beginning to release the message

However this time this can cause fragmentation of multiple sub-queues that will
need to be reassembled. So finally the CAS is better, the leader thread should
recognize itself.

It seems tricky to reliably store the next pointer in each element, and a DWCAS
wouldn't help here either. Maybe uninitialized elements should just have a
special value (eg 0x1) for their next pointer, meaning "not initialized yet",
and that the thread will then replace with the previous queue pointer. A reader
would have to wait on this value when meeting it, knowing the pointer is not
filled yet but is coming.