Adding upstream version 2.9.5.upstream/2.9.5

Signed-off-by: Daniel Baumann <daniel.baumann@progress-linux.org>

1 files changed, 224 insertions, 0 deletions

diff --git a/doc/design-thoughts/connection-reuse.txt b/doc/design-thoughts/connection-reuse.txt
new file mode 100644
index 0000000..4eb22f6
--- /dev/null
+++ b/doc/design-thoughts/connection-reuse.txt
@@ -0,0 +1,224 @@
+2015/08/06 - server connection sharing
+
+Improvements on the connection sharing strategies
+-------------------------------------------------
+
+4 strategies are currently supported :
+  - never
+  - safe
+  - aggressive
+  - always
+
+The "aggressive" and "always" strategies take into account the fact that the
+connection has already been reused at least once or not. The principle is that
+second requests can be used to safely "validate" connection reuse on newly
+added connections, and that such validated connections may be used even by
+first requests from other sessions. A validated connection is a connection
+which has already been reused, hence proving that it definitely supports
+multiple requests. Such connections are easy to verify : after processing the
+response, if the txn already had the TX_NOT_FIRST flag, then it was not the
+first request over that connection, and it is validated as safe for reuse.
+Validated connections are put into a distinct list : server->safe_conns.
+
+Incoming requests with TX_NOT_FIRST first pick from the regular idle_conns
+list so that any new idle connection is validated as soon as possible.
+
+Incoming requests without TX_NOT_FIRST only pick from the safe_conns list for
+strategy "aggressive", guaranteeing that the server properly supports connection
+reuse, or first from the safe_conns list, then from the idle_conns list for
+strategy "always".
+
+Connections are always stacked into the list (LIFO) so that there are higher
+changes to convert recent connections and to use them. This will first optimize
+the likeliness that the connection works, and will avoid TCP metrics from being
+lost due to an idle state, and/or the congestion window to drop and the
+connection going to slow start mode.
+
+
+Handling connections in pools
+-----------------------------
+
+A per-server "pool-max" setting should be added to permit disposing unused idle
+connections not attached anymore to a session for use by future requests. The
+principle will be that attached connections are queued from the front of the
+list while the detached connections will be queued from the tail of the list.
+
+This way, most reused connections will be fairly recent and detached connections
+will most often be ignored. The number of detached idle connections in the lists
+should be accounted for (pool_used) and limited (pool_max).
+
+After some time, a part of these detached idle connections should be killed.
+For this, the list is walked from tail to head and connections without an owner
+may be evicted. It may be useful to have a per-server pool_min setting
+indicating how many idle connections should remain in the pool, ready for use
+by new requests. Conversely, a pool_low metric should be kept between eviction
+runs, to indicate the lowest amount of detached connections that were found in
+the pool.
+
+For eviction, the principle of a half-life is appealing. The principle is
+simple : over a period of time, half of the connections between pool_min and
+pool_low should be gone. Since pool_low indicates how many connections were
+remaining unused over a period, it makes sense to kill some of them.
+
+In order to avoid killing thousands of connections in one run, the purge
+interval should be split into smaller batches. Let's call N the ratio of the
+half-life interval and the effective interval.
+
+The algorithm consists in walking over them from the end every interval and
+killing ((pool_low - pool_min) + 2 * N - 1) / (2 * N). It ensures that half
+of the unused connections are killed over the half-life period, in N batches
+of population/2N entries at most.
+
+Unsafe connections should be evicted first. There should be quite few of them
+since most of them are probed and become safe. Since detached connections are
+quickly recycled and attached to a new session, there should not be too many
+detached connections in the pool, and those present there may be killed really
+quickly.
+
+Another interesting point of pools is that when a pool-max is not null, then it
+makes sense to automatically enable pretend-keep-alive on non-private connections
+going to the server in order to be able to feed them back into the pool. With
+the "aggressive" or "always" strategies, it can allow clients making a single
+request over their connection to share persistent connections to the servers.
+
+
+
+2013/10/17 - server connection management and reuse
+
+Current state
+-------------
+
+At the moment, a connection entity is needed to carry any address
+information. This means in the following situations, we need a server
+connection :
+
+- server is elected and the server's destination address is set
+
+- transparent mode is elected and the destination address is set from
+  the incoming connection
+
+- proxy mode is enabled, and the destination's address is set during
+  the parsing of the HTTP request
+
+- connection to the server fails and must be retried on the same
+  server using the same parameters, especially the destination
+  address (SN_ADDR_SET not removed)
+
+
+On the accepting side, we have further requirements :
+
+- allocate a clean connection without a stream interface
+
+- incrementally set the accepted connection's parameters without
+  clearing it, and keep track of what is set (eg: getsockname).
+
+- initialize a stream interface in established mode
+
+- attach the accepted connection to a stream interface
+
+
+This means several things :
+
+- the connection has to be allocated on the fly the first time it is
+  needed to store the source or destination address ;
+
+- the connection has to be attached to the stream interface at this
+  moment ;
+
+- it must be possible to incrementally set some settings on the
+  connection's addresses regardless of the connection's current state
+
+- the connection must not be released across connection retries ;
+
+- it must be possible to clear a connection's parameters for a
+  redispatch without having to detach/attach the connection ;
+
+- we need to allocate a connection without an existing stream interface
+
+So on the accept() side, it looks like this :
+
+  fd = accept();
+  conn = new_conn();
+  get_some_addr_info(&conn->addr);
+  ...
+  si = new_si();
+  si_attach_conn(si, conn);
+  si_set_state(si, SI_ST_EST);
+  ...
+  get_more_addr_info(&conn->addr);
+
+On the connect() side, it looks like this :
+
+  si = new_si();
+  while (!properly_connected) {
+    if (!(conn = si->end)) {
+      conn = new_conn();
+      conn_clear(conn);
+      si_attach_conn(si, conn);
+    }
+    else {
+      if (connected) {
+        f = conn->flags & CO_FL_XPRT_TRACKED;
+        conn->flags &= ~CO_FL_XPRT_TRACKED;
+        conn_close(conn);
+        conn->flags |= f;
+      }
+      if (!correct_dest)
+        conn_clear(conn);
+    }
+    set_some_addr_info(&conn->addr);
+    si_set_state(si, SI_ST_CON);
+    ...
+    set_more_addr_info(&conn->addr);
+    conn->connect();
+    if (must_retry) {
+      close_conn(conn);
+    }
+  }
+
+Note: we need to be able to set the control and transport protocols.
+On outgoing connections, this is set once we know the destination address.
+On incoming connections, this is set the earliest possible (once we know
+the source address).
+
+The problem analysed below was solved on 2013/10/22
+
+| ==> the real requirement is to know whether a connection is still valid or not
+|     before deciding to close it. CO_FL_CONNECTED could be enough, though it
+|     will not indicate connections that are still waiting for a connect to occur.
+|     This combined with CO_FL_WAIT_L4_CONN and CO_FL_WAIT_L6_CONN should be OK.
+| 
+|     Alternatively, conn->xprt could be used for this, but needs some careful checks
+|     (it's used by conn_full_close at least).
+| 
+| Right now, conn_xprt_close() checks conn->xprt and sets it to NULL.
+| conn_full_close() also checks conn->xprt and sets it to NULL, except
+| that the check on ctrl is performed within xprt. So conn_xprt_close()
+| followed by conn_full_close() will not close the file descriptor.
+| Note that conn_xprt_close() is never called, maybe we should kill it ?
+| 
+| Note: at the moment, it's problematic to leave conn->xprt to NULL before doing
+|       xprt_init() because we might end up with a pending file descriptor. Or at
+|       least with some transport not de-initialized. We might thus need
+|       conn_xprt_close() when conn_xprt_init() fails.
+| 
+| The fd should be conditioned by ->ctrl only, and the transport layer by ->xprt.
+| 
+| - conn_prepare_ctrl(conn, ctrl) 
+| - conn_prepare_xprt(conn, xprt) 
+| - conn_prepare_data(conn, data) 
+| 
+| Note: conn_xprt_init() needs conn->xprt so it's not a problem to set it early.
+| 
+| One problem might be with conn_xprt_close() not being able to know if xprt_init()
+| was called or not. That's where it might make sense to only set ->xprt during init.
+| Except that it does not fly with outgoing connections (xprt_init is called after
+| connect()).
+| 
+| => currently conn_xprt_close() is only used by ssl_sock.c and decides whether
+|    to do something based on ->xprt_ctx which is set by ->init() from xprt_init().
+|    So there is nothing to worry about. We just need to restore conn_xprt_close()
+|    and rely on ->ctrl to close the fd instead of ->xprt.
+| 
+| => we have the same issue with conn_ctrl_close() : when is the fd supposed to be
+|    valid ? On outgoing connections, the control is set much before the fd...