summaryrefslogtreecommitdiffstats
path: root/addons/ot/README
diff options
context:
space:
mode:
authorDaniel Baumann <daniel.baumann@progress-linux.org>2024-04-13 12:18:05 +0000
committerDaniel Baumann <daniel.baumann@progress-linux.org>2024-04-13 12:18:05 +0000
commitb46aad6df449445a9fc4aa7b32bd40005438e3f7 (patch)
tree751aa858ca01f35de800164516b298887382919d /addons/ot/README
parentInitial commit. (diff)
downloadhaproxy-b46aad6df449445a9fc4aa7b32bd40005438e3f7.tar.xz
haproxy-b46aad6df449445a9fc4aa7b32bd40005438e3f7.zip
Adding upstream version 2.9.5.upstream/2.9.5
Signed-off-by: Daniel Baumann <daniel.baumann@progress-linux.org>
Diffstat (limited to 'addons/ot/README')
-rw-r--r--addons/ot/README794
1 files changed, 794 insertions, 0 deletions
diff --git a/addons/ot/README b/addons/ot/README
new file mode 100644
index 0000000..a08f471
--- /dev/null
+++ b/addons/ot/README
@@ -0,0 +1,794 @@
+ -----------------------------------------
+ The HAProxy OpenTracing filter (OT)
+ Version 1.0
+ ( Last update: 2020-12-10 )
+ -----------------------------------------
+ Author : Miroslav Zagorac
+ Contact : mzagorac at haproxy dot com
+
+
+SUMMARY
+--------
+
+ 0. Terms
+ 1. Introduction
+ 2. Build instructions
+ 3. Basic concepts in OpenTracing
+ 4. OT configuration
+ 4.1. OT scope
+ 4.2. "ot-tracer" section
+ 4.3. "ot-scope" section
+ 4.4. "ot-group" section
+ 5. Examples
+ 5.1 Benchmarking results
+ 6. OT CLI
+ 7. Known bugs and limitations
+
+
+0. Terms
+---------
+
+* OT: The HAProxy OpenTracing filter
+
+ OT is the HAProxy filter that allows you to send data to distributed
+ tracing systems via the OpenTracing API.
+
+
+1. Introduction
+----------------
+
+Nowadays there is a growing need to divide a process into microservices and
+there is a problem of monitoring the work of the same process. One way to
+solve this problem is to use distributed tracing service in a central location,
+the so-called tracer.
+
+OT is a feature introduced in HAProxy 2.4. This filter enables communication
+via the OpenTracing API with OpenTracing compatible servers (tracers).
+Currently, tracers that support this API include Datadog, Jaeger, LightStep
+and Zipkin.
+
+The OT filter was primarily tested with the Jaeger tracer, while configurations
+for both Datadog and Zipkin tracers were also set in the test directory.
+
+The OT filter is a standard HAProxy filter, so what applies to others also
+applies to this one (of course, by that I mean what is described in the
+documentation, more precisely in the doc/internals/filters.txt file).
+
+The OT filter activation is done explicitly by specifying it in the HAProxy
+configuration. If this is not done, the OT filter in no way participates
+in the work of HAProxy.
+
+As for the impact on HAProxy speed, this is documented with several tests
+located in the test directory, and the result is found in the README-speed-*
+files. In short, the speed of operation depends on the way it is used and
+the complexity of the configuration, from an almost immeasurable impact to
+a significant deceleration (5x and more). I think that in some normal use
+the speed of HAProxy with the filter on will be quite satisfactory with a
+slowdown of less than 4% (provided that no more than 10% of requests are
+sent to the tracer, which is determined by the keyword 'rate-limit').
+
+The OT filter allows intensive use of ACLs, which can be defined anywhere in
+the configuration. Thus, it is possible to use the filter only for those
+connections that are of interest to us.
+
+
+2. Build instructions
+----------------------
+
+OT is the HAProxy filter and as such is compiled together with HAProxy.
+
+To communicate with some OpenTracing compatible tracer, the OT filter uses the
+OpenTracing C Wrapper library (which again uses the OpenTracing CPP library).
+This means that we must have both libraries installed on the system on which
+we want to compile or use HAProxy.
+
+Instructions for compiling and installing both required libraries can be
+found at https://github.com/haproxytech/opentracing-c-wrapper .
+
+Also, to use the OT filter when running HAProxy we need to have an OpenTracing
+plugin for the tracer we want to use. We will return to this later, in
+section 5.
+
+The OT filter can be more easily compiled using the pkg-config tool, if we
+have the OpenTracing C Wrapper library installed so that it contains pkg-config
+files (which have the .pc extension). If the pkg-config tool cannot be used,
+then the path to the directory where the include files and libraries are
+located can be explicitly specified.
+
+Below are examples of the two ways to compile HAProxy with the OT filter, the
+first using the pkg-congfig tool and the second explicitly specifying the path
+to the OpenTracing C Wrapper include and library.
+
+Note: prompt '%' indicates that the command is executed under a unprivileged
+ user, while prompt '#' indicates that the command is executed under the
+ root user.
+
+Example of compiling HAProxy using the pkg-congfig tool (assuming the
+OpenTracing C Wrapper library is installed in the /opt directory):
+
+ % PKG_CONFIG_PATH=/opt/lib/pkgconfig make USE_OT=1 TARGET=linux-glibc
+
+The OT filter can also be compiled in debug mode as follows:
+
+ % PKG_CONFIG_PATH=/opt/lib/pkgconfig make USE_OT=1 OT_DEBUG=1 TARGET=linux-glibc
+
+HAProxy compilation example explicitly specifying path to the OpenTracing C
+Wrapper include and library:
+
+ % make USE_OT=1 OT_INC=/opt/include OT_LIB=/opt/lib TARGET=linux-glibc
+
+In case we want to use debug mode, then it looks like this:
+
+ % make USE_OT=1 OT_DEBUG=1 OT_INC=/opt/include OT_LIB=/opt/lib TARGET=linux-glibc
+
+If the library we want to use is not installed on a unix system, then a locally
+installed library can be used (say, which is compiled and installed in the user
+home directory). In this case instead of /opt/include and /opt/lib the
+equivalent paths to the local installation should be specified. Of course,
+in that case the pkg-config tool can also be used if we have a complete
+installation (with .pc files).
+
+last but not least, if the pkg-config tool is not used when compiling, then
+HAProxy executable may not be able to find the OpenTracing C Wrapper library
+at startup. This can be solved in several ways, for example using the
+LD_LIBRARY_PATH environment variable which should be set to the path where the
+library is located before starting the HAProxy.
+
+ % LD_LIBRARY_PATH=/opt/lib /path-to/haproxy ...
+
+Another way is to add RUNPATH to HAProxy executable that contains the path to
+the library in question.
+
+ % make USE_OT=1 OT_RUNPATH=1 OT_INC=/opt/include OT_LIB=/opt/lib TARGET=linux-glibc
+
+After HAProxy is compiled, we can check if the OT filter is enabled:
+
+ % ./haproxy -vv | grep opentracing
+ --- command output ----------
+ [ OT] opentracing
+ --- command output ----------
+
+
+3. Basic concepts in OpenTracing
+---------------------------------
+
+Basic concepts of OpenTracing can be read on the OpenTracing documentation
+website https://opentracing.io/docs/overview/.
+
+Here we will list only the most important elements of distributed tracing and
+these are 'trace', 'span' and 'span context'. Trace is a description of the
+complete transaction we want to record in the tracing system. A span is an
+operation that represents a unit of work that is recorded in a tracing system.
+Span context is a group of information related to a particular span that is
+passed on to the system (from service to service). Using this context, we can
+add new spans to already open trace (or supplement data in already open spans).
+
+An individual span may contain one or more tags, logs and baggage items.
+The tag is a key-value element that is valid for the entire span. Log is a
+key-value element that allows you to write some data at a certain time, it
+can be used for debugging. A baggage item is a key-value data pair that can
+be used for the duration of an entire trace, from the moment it is added to
+the span.
+
+
+4. OT configuration
+--------------------
+
+In order for the OT filter to be used, it must be included in the HAProxy
+configuration, in the proxy section (frontend / listen / backend):
+
+ frontend ot-test
+ ...
+ filter opentracing [id <id>] config <file>
+ ...
+
+If no filter id is specified, 'ot-filter' is used as default. The 'config'
+parameter must be specified and it contains the path of the file used to
+configure the OT filter.
+
+
+4.1 OT scope
+-------------
+
+If the filter id is defined for the OT filter, then the OT scope with
+the same name should be defined in the configuration file. In the same
+configuration file we can have several defined OT scopes.
+
+Each OT scope must have a defined (only one) "ot-tracer" section that is
+used to configure the operation of the OT filter and define the used groups
+and scopes.
+
+OT scope starts with the id of the filter specified in square brackets and
+ends with the end of the file or when a new OT scope is defined.
+
+For example, this defines two OT scopes in the same configuration file:
+ [my-first-ot-filter]
+ ot-tracer tracer1
+ ...
+ ot-group group1
+ ...
+ ot-scope scope1
+ ...
+
+ [my-second-ot-filter]
+ ...
+
+
+4.2. "ot-tracer" section
+-------------------------
+
+Only one "ot-tracer" section must be defined for each OT scope.
+
+There are several keywords that must be defined for the OT filter to work.
+These are 'config' which defines the configuration file for the OpenTracing
+API, and 'plugin' which defines the OpenTracing plugin used.
+
+Through optional keywords can be defined ACLs, logging, rate limit, and groups
+and scopes that define the tracing model.
+
+
+ot-tracer <name>
+ A new OT with the name <name> is created.
+
+ Arguments :
+ name - the name of the tracer section
+
+
+ The following keywords are supported in this section:
+ - mandatory keywords:
+ - config
+ - plugin
+
+ - optional keywords:
+ - acl
+ - debug-level
+ - groups
+ - [no] log
+ - [no] option disabled
+ - [no] option dontlog-normal
+ - [no] option hard-errors
+ - rate-limit
+ - scopes
+
+
+acl <aclname> <criterion> [flags] [operator] <value> ...
+ Declare or complete an access list.
+
+ To configure and use the ACL, see section 7 of the HAProxy Configuration
+ Manual.
+
+
+config <file>
+ 'config' is one of the two mandatory keywords associated with the OT tracer
+ configuration. This keyword sets the path of the configuration file for the
+ OpenTracing tracer plugin. To set the contents of this configuration file,
+ it is best to look at the documentation related to the OpenTracing tracer we
+ want to use.
+
+ Arguments :
+ file - the path of the configuration file
+
+
+debug-level <value>
+ This keyword sets the value of the debug level related to the display of
+ debug messages in the OT filter. The 'debug-level' value is binary, ie
+ a single value bit enables or disables the display of the corresponding
+ debug message that uses that bit. The default value is set via the
+ FLT_OT_DEBUG_LEVEL macro in the include/config.h file. Debug level value
+ is used only if the OT filter is compiled with the debug mode enabled,
+ otherwise it is ignored.
+
+ Arguments :
+ value - binary value ranging from 0 to 255 (8 bits)
+
+
+groups <name> ...
+ A list of "ot-group" groups used for the currently defined tracer is declared.
+ Several groups can be specified in one line.
+
+ Arguments :
+ name - the name of the OT group
+
+
+log global
+log <addr> [len <len>] [format <fmt>] <facility> [<level> [<minlevel>]]
+no log
+ Enable per-instance logging of events and traffic.
+
+ To configure and use the logging system, see section 4.2 of the HAProxy
+ Configuration Manual.
+
+
+option disabled
+no option disabled
+ Keyword which turns the operation of the OT filter on or off. By default
+ the filter is on.
+
+
+option dontlog-normal
+no option dontlog-normal
+ Enable or disable logging of normal, successful processing. By default,
+ this option is disabled. For this option to be considered, logging must
+ be turned on.
+
+ See also: 'log' keyword description.
+
+
+option hard-errors
+no option hard-errors
+ During the operation of the filter, some errors may occur, caused by
+ incorrect configuration of the tracer or some error related to the operation
+ of HAProxy. By default, such an error will not interrupt the filter
+ operation for the stream in which the error occurred. If the 'hard-error'
+ option is enabled, the operation error prohibits all further processing of
+ events and groups in the stream in which the error occurred.
+
+
+plugin <file>
+ 'plugin' is one of the two mandatory keywords associated with the OT tracer
+ configuration. This keyword sets the path of the OpenTracing tracer plugin.
+
+ Arguments :
+ file - the name of the plugin used
+
+
+rate-limit <value>
+ This option allows limiting the use of the OT filter, ie it can be influenced
+ whether the OT filter is activated for a stream or not. Determining whether
+ or not a filter is activated depends on the value of this option that is
+ compared to a randomly selected value when attaching the filter to the stream.
+ By default, the value of this option is set to 100.0, ie the OT filter is
+ activated for each stream.
+
+ Arguments :
+ value - floating point value ranging from 0.0 to 100.0
+
+
+scopes <name> ...
+ This keyword declares a list of "ot-scope" definitions used for the currently
+ defined tracer. Multiple scopes can be specified in the same line.
+
+ Arguments :
+ name - the name of the OT scope
+
+
+4.3. "ot-scope" section
+------------------------
+
+Stream processing begins with filter attachment, then continues with the
+processing of a number of defined events and groups, and ends with filter
+detachment. The "ot-scope" section is used to define actions related to
+individual events. However, this section may be part of a group, so the
+event does not have to be part of the definition.
+
+
+ot-scope <name>
+ Creates a new OT scope definition named <name>.
+
+ Arguments :
+ name - the name of the OT scope
+
+
+ The following keywords are supported in this section:
+ - acl
+ - baggage
+ - event
+ - extract
+ - finish
+ - inject
+ - log
+ - span
+ - tag
+
+
+acl <aclname> <criterion> [flags] [operator] <value> ...
+ Declare or complete an access list.
+
+ To configure and use the ACL, see section 7 of the HAProxy Configuration
+ Manual.
+
+
+baggage <name> <sample> ...
+ Baggage items allow the propagation of data between spans, ie allow the
+ assignment of metadata that is propagated to future children spans.
+ This data is formatted in the style of key-value pairs and is part of
+ the context that can be transferred between processes that are part of
+ a server architecture.
+
+ This kewyord allows setting the baggage for the currently active span. The
+ data type is always a string, ie any sample type is converted to a string.
+ The exception is a binary value that is not supported by the OT filter.
+
+ See the 'tag' keyword description for the data type conversion table.
+
+ Arguments :
+ name - key part of a data pair
+ sample - sample expression (value part of a data pair), at least one
+ sample must be present
+
+
+event <name> [{ if | unless } <condition>]
+ Set the event that triggers the 'ot-scope' to which it is assigned.
+ Optionally, it can be followed by an ACL-based condition, in which case it
+ will only be evaluated if the condition is true.
+
+ ACL-based conditions are executed in the context of a stream that processes
+ the client and server connections. To configure and use the ACL, see
+ section 7 of the HAProxy Configuration Manual.
+
+ Arguments :
+ name - the event name
+ condition - a standard ACL-based condition
+
+ Supported events are (the table gives the names of the events in the OT
+ filter and the corresponding equivalent in the SPOE filter):
+
+ -------------------------------------|------------------------------
+ the OT filter | the SPOE filter
+ -------------------------------------|------------------------------
+ on-client-session-start | on-client-session
+ on-frontend-tcp-request | on-frontend-tcp-request
+ on-http-wait-request | -
+ on-http-body-request | -
+ on-frontend-http-request | on-frontend-http-request
+ on-switching-rules-request | -
+ on-backend-tcp-request | on-backend-tcp-request
+ on-backend-http-request | on-backend-http-request
+ on-process-server-rules-request | -
+ on-http-process-request | -
+ on-tcp-rdp-cookie-request | -
+ on-process-sticking-rules-request | -
+ on-client-session-end | -
+ on-server-unavailable | -
+ -------------------------------------|------------------------------
+ on-server-session-start | on-server-session
+ on-tcp-response | on-tcp-response
+ on-http-wait-response | -
+ on-process-store-rules-response | -
+ on-http-response | on-http-response
+ on-server-session-end | -
+ -------------------------------------|------------------------------
+
+
+extract <name-prefix> [use-vars | use-headers]
+ For a more detailed description of the propagation process of the span
+ context, see the description of the keyword 'inject'. Only the process
+ of extracting data from the carrier is described here.
+
+ Arguments :
+ name-prefix - data name prefix (ie key element prefix)
+ use-vars - data is extracted from HAProxy variables
+ use-headers - data is extracted from the HTTP header
+
+
+ Below is an example of using HAProxy variables to transfer span context data:
+
+ --- test/ctx/ot.cfg --------------------------------------------------------
+ ...
+ ot-scope client_session_start_2
+ extract "ot_ctx_1" use-vars
+ span "Client session" child-of "ot_ctx_1"
+ ...
+ ----------------------------------------------------------------------------
+
+
+finish <name> ...
+ Closing a particular span or span context. Instead of the name of the span,
+ there are several specially predefined names with which we can finish certain
+ groups of spans. So it can be used as the name '*req*' for all open spans
+ related to the request channel, '*res*' for all open spans related to the
+ response channel and '*' for all open spans regardless of which channel they
+ are related to. Several spans and/or span contexts can be specified in one
+ line.
+
+ Arguments :
+ name - the name of the span or context context
+
+
+inject <name-prefix> [use-vars] [use-headers]
+ In OpenTracing, the transfer of data related to the tracing process between
+ microservices that are part of a larger service is done through the
+ propagation of the span context. The basic operations that allow us to
+ access and transfer this data are 'inject' and 'extract'.
+
+ 'inject' allows us to extract span context so that the obtained data can
+ be forwarded to another process (microservice) via the selected carrier.
+ 'inject' in the name actually means inject data into carrier. Carrier is
+ an interface here (ie a data structure) that allows us to transfer tracing
+ state from one process to another.
+
+ Data transfer can take place via one of two selected storage methods, the
+ first is by adding data to the HTTP header and the second is by using HAProxy
+ variables. Only data transfer via HTTP header can be used to transfer data
+ to another process (ie microservice). All data is organized in the form of
+ key-value data pairs.
+
+ No matter which data transfer method you use, we need to specify a prefix
+ for the key element. All alphanumerics (lowercase only) and underline
+ character can be used to construct the data name prefix. Uppercase letters
+ can actually be used, but they will be converted to lowercase when creating
+ the prefix.
+
+ Arguments :
+ name-prefix - data name prefix (ie key element prefix)
+ use-vars - HAProxy variables are used to store and transfer data
+ use-headers - HTTP headers are used to store and transfer data
+
+
+ Below is an example of using HTTP headers and variables, and how this is
+ reflected in the internal data of the HAProxy process.
+
+ --- test/ctx/ot.cfg --------------------------------------------------------
+ ...
+ ot-scope client_session_start_1
+ span "HAProxy session" root
+ inject "ot_ctx_1" use-headers use-vars
+ ...
+ ----------------------------------------------------------------------------
+
+ - generated HAProxy variable (key -> value):
+ txn.ot_ctx_1.uberDtraceDid -> 8f1a05a3518d2283:8f1a05a3518d2283:0:1
+
+ - generated HTTP header (key: value):
+ ot_ctx_1-uber-trace-id: 8f1a05a3518d2283:8f1a05a3518d2283:0:1
+
+ Because HAProxy does not allow the '-' character in the variable name (which
+ is automatically generated by the OpenTracing API and on which we have no
+ influence), it is converted to the letter 'D'. We can see that there is no
+ such conversion in the name of the HTTP header because the '-' sign is allowed
+ there. Due to this conversion, initially all uppercase letters are converted
+ to lowercase because otherwise we would not be able to distinguish whether
+ the disputed sign '-' is used or not.
+
+ Thus created HTTP headers and variables are deleted when executing the
+ 'finish' keyword or when detaching the stream from the filter.
+
+
+log <name> <sample> ...
+ This kewyord allows setting the log for the currently active span. The
+ data type is always a string, ie any sample type is converted to a string.
+ The exception is a binary value that is not supported by the OT filter.
+
+ See the 'tag' keyword description for the data type conversion table.
+
+ Arguments :
+ name - key part of a data pair
+ sample - sample expression (value part of a data pair), at least one
+ sample must be present
+
+
+span <name> [<reference>]
+ Creating a new span (or referencing an already opened one). If a new span
+ is created, it can be a child of the referenced span, follow from the
+ referenced span, or be root 'span'. In case we did not specify a reference
+ to the previously created span, the new span will become the root span.
+ We need to pay attention to the fact that in one trace there can be only
+ one root span. In case we have specified a non-existent span as a reference,
+ a new span will not be created.
+
+ Arguments :
+ name - the name of the span being created or referenced (operation
+ name)
+ reference - span or span context to which the created span is referenced
+
+
+tag <name> <sample> ...
+ This kewyord allows setting a tag for the currently active span. The first
+ argument is the name of the tag (tag ID) and the second its value. A value
+ can consist of one or more data. If the value is only one data, then the
+ type of that data depends on the type of the HAProxy sample. If the value
+ contains more data, then the data type is string. The data conversion table
+ is below:
+
+ HAProxy sample data type | the OpenTracing data type
+ --------------------------+---------------------------
+ NULL | NULL
+ BOOL | BOOL
+ INT32 | INT64
+ UINT32 | UINT64
+ INT64 | INT64
+ UINT64 | UINT64
+ IPV4 | STRING
+ IPV6 | STRING
+ STRING | STRING
+ BINARY | UNSUPPORTED
+ --------------------------+---------------------------
+
+ Arguments :
+ name - key part of a data pair
+ sample - sample expression (value part of a data pair), at least one
+ sample must be present
+
+
+4.4. "ot-group" section
+------------------------
+
+This section allows us to define a group of OT scopes, that is not activated
+via an event but is triggered from TCP or HTTP rules. More precisely, these
+are the following rules: 'tcp-request', 'tcp-response', 'http-request',
+'http-response' and 'http-after-response'. These rules can be defined in the
+HAProxy configuration file.
+
+
+ot-group <name>
+ Creates a new OT group definition named <name>.
+
+ Arguments :
+ name - the name of the OT group
+
+
+ The following keywords are supported in this section:
+ - scopes
+
+
+scopes <name> ...
+ 'ot-scope' sections that are part of the specified group are defined. If
+ the mentioned 'ot-scope' sections are used only in some OT group, they do
+ not have to have defined events. Several 'ot-scope' sections can be
+ specified in one line.
+
+ Arguments :
+ name - the name of the 'ot-scope' section
+
+
+5. Examples
+------------
+
+Several examples of the OT filter configuration can be found in the test
+directory. A brief description of the prepared configurations follows:
+
+cmp - the configuration very similar to that of the spoa-opentracing project.
+ It was made to compare the speed of the OT filter with the
+ implementation of distributed tracing via spoa-opentracing application.
+
+sa - the configuration in which all possible events are used.
+
+ctx - the configuration is very similar to the previous one, with the only
+ difference that the spans are opened using the span context as a span
+ reference.
+
+fe be - a slightly more complicated example of the OT filter configuration
+ that uses two cascaded HAProxy services. The span context between
+ HAProxy processes is transmitted via the HTTP header.
+
+empty - the empty configuration in which the OT filter is initialized but
+ no event is triggered. It is not very usable, except to check the
+ behavior of the OT filter in the case of a similar configuration.
+
+
+In order to be able to collect data (and view results via the web interface)
+we need to install some of the supported tracers. We will use the Jaeger
+tracer as an example. Installation instructions can be found on the website
+https://www.jaegertracing.io/download/. For the impatient, here we will list
+how the image to test the operation of the tracer system can be installed
+without much reading of the documentation.
+
+ # docker pull jaegertracing/all-in-one:latest
+ # docker run -d --name jaeger -e COLLECTOR_ZIPKIN_HTTP_PORT=9411 \
+ -p 5775:5775/udp -p 6831:6831/udp -p 6832:6832/udp -p 5778:5778 \
+ -p 16686:16686 -p 14268:14268 -p 9411:9411 jaegertracing/all-in-one:latest
+
+The last command will also initialize and run the Jaeger container. If we
+want to use that container later, it can be started and stopped in the classic
+way, using the 'docker container start/stop' commands.
+
+
+In order to be able to use any of the configurations from the test directory,
+we must also have a tracer plugin in that directory (all examples use the
+Jaeger tracer plugin). The simplest way is to download the tracer plugin
+using the already prepared shell script get-opentracing-plugins.sh.
+The script accepts one argument, the directory in which the download is made.
+If run without an argument, the script downloads all plugins to the current
+directory.
+
+ % ./get-opentracing-plugins.sh
+
+After that, we can run one of the pre-configured configurations using the
+provided script run-xxx.sh (where xxx is the name of the configuration being
+tested). For example:
+
+ % ./run-sa.sh
+
+The script will create a new log file each time it is run (because part of the
+log file name is the start time of the script).
+
+Eh, someone will surely notice that all test configurations use the Jaeger
+tracing plugin that cannot be downloaded using the get-opentracing-plugins.sh
+script. Unfortunately, the latest precompiled version that can be downloaded
+is 0.4.2, for newer ones only the source code can be found. Version 0.4.2 has
+a bug that can cause the operation of the OT filter to get stuck, so it is
+better not to use this version. Here is the procedure by which we can compile
+a newer version of the plugin (in our example it is 0.5.0).
+
+Important note: the GCC version must be at least 4.9 or later.
+
+ % wget https://github.com/jaegertracing/jaeger-client-cpp/archive/v0.5.0.tar.gz
+ % tar xf v0.5.0.tar.gz
+ % cd jaeger-client-cpp-0.5.0
+ % mkdir build
+ % cd build
+ % cmake -DCMAKE_INSTALL_PREFIX=/opt -DJAEGERTRACING_PLUGIN=ON -DHUNTER_CONFIGURATION_TYPES=Release -DHUNTER_BUILD_SHARED_LIBS=OFF ..
+ % make
+
+After the plugin is compiled, it will be in the current directory. The name
+of the plugin is libjaegertracing_plugin.so.
+
+
+5.1. Benchmarking results
+--------------------------
+
+To check the operation of the OT filter, several different test configurations
+have been made which are located in the test directory. The test results of
+the same configurations (with the names README-speed-xxx, where xxx is the name
+of the configuration being tested) are also in the directory of the same name.
+
+All tests were performed on the same debian 9.13 system, CPU i7-4770, 32 GB RAM.
+For the purpose of testing, the thttpd web server on port 8000 was used.
+Testing was done with the wrk utility running via run-xxx.sh scripts; that is,
+via the test-speed.sh script that is run as follows:
+
+ % ./test-speed.sh all
+
+The above mentioned thttpd web server is run from that script and it should be
+noted that we need to have the same installed on the system (or change the path
+to the thttpd server in that script if it is installed elsewhere).
+
+Each test is performed several times over a period of 5 minutes per individual
+test. The only difference when running the tests for the same configuration
+was in changing the 'rate-limit' parameter (and the 'option disabled' option),
+which is set to the following values: 100.0, 50.0, 10.0, 2.5 and 0.0 percent.
+Then a test is performed with the OT filter active but disabled for request
+processing ('option disabled' is included in the ot.cfg configuration). In
+the last test, the OT filter is not used at all, ie it is not active and does
+not affect the operation of HAProxy in any way.
+
+
+6. OT CLI
+----------
+
+Via the HAProxy CLI interface we can find out the current status of the OT
+filter and change several of its settings.
+
+All supported CLI commands can be found in the following way, using the
+socat utility with the assumption that the HAProxy CLI socket path is set
+to /tmp/haproxy.sock (of course, instead of socat, nc or other utility can
+be used with a change in arguments when running the same):
+
+ % echo "help" | socat - UNIX-CONNECT:/tmp/haproxy.sock | grep flt-ot
+ --- command output ----------
+ flt-ot debug [level] : set the OT filter debug level (default: get current debug level)
+ flt-ot disable : disable the OT filter
+ flt-ot enable : enable the OT filter
+ flt-ot soft-errors : turning off hard-errors mode
+ flt-ot hard-errors : enabling hard-errors mode
+ flt-ot logging [state] : set logging state (default: get current logging state)
+ flt-ot rate [value] : set the rate limit (default: get current rate value)
+ flt-ot status : show the OT filter status
+ --- command output ----------
+
+'flt-ot debug' can only be used in case the OT filter is compiled with the
+debug mode enabled.
+
+
+7. Known bugs and limitations
+------------------------------
+
+The name of the span context definition can contain only letters, numbers and
+characters '_' and '-'. Also, all uppercase letters in the name are converted
+to lowercase. The character '-' is converted internally to the 'D' character,
+and since a HAProxy variable is generated from that name, this should be taken
+into account if we want to use it somewhere in the HAProxy configuration.
+The above mentioned span context is used in the 'inject' and 'extract' keywords.
+
+Let's look a little at the example test/fe-be (configurations are in the
+test/fe and test/be directories, 'fe' is here the abbreviation for frontend
+and 'be' for backend). In case we have the 'rate-limit' set to a value less
+than 100.0, then distributed tracing will not be started with each new HTTP
+request. It also means that the span context will not be delivered (via the
+HTTP header) to the backend HAProxy process. The 'rate-limit' on the backend
+HAProxy must be set to 100.0, but because the frontend HAProxy does not send
+a span context every time, all such cases will cause an error to be reported
+on the backend server. Therefore, the 'hard-errors' option must be set on the
+backend server, so that processing on that stream is stopped as soon as the
+first error occurs. Such cases will slow down the backend server's response
+a bit (in the example in question it is about 3%).