diff options
Diffstat (limited to '')
| -rw-r--r-- | addons/ot/README | 794 | ||||
| -rw-r--r-- | addons/ot/README-func | 298 | ||||
| -rw-r--r-- | addons/ot/README-pool | 25 |
3 files changed, 1117 insertions, 0 deletions
diff --git a/addons/ot/README b/addons/ot/README new file mode 100644 index 0000000..a08f471 --- /dev/null +++ b/addons/ot/README @@ -0,0 +1,794 @@ + ----------------------------------------- + The HAProxy OpenTracing filter (OT) + Version 1.0 + ( Last update: 2020-12-10 ) + ----------------------------------------- + Author : Miroslav Zagorac + Contact : mzagorac at haproxy dot com + + +SUMMARY +-------- + + 0. Terms + 1. Introduction + 2. Build instructions + 3. Basic concepts in OpenTracing + 4. OT configuration + 4.1. OT scope + 4.2. "ot-tracer" section + 4.3. "ot-scope" section + 4.4. "ot-group" section + 5. Examples + 5.1 Benchmarking results + 6. OT CLI + 7. Known bugs and limitations + + +0. Terms +--------- + +* OT: The HAProxy OpenTracing filter + + OT is the HAProxy filter that allows you to send data to distributed + tracing systems via the OpenTracing API. + + +1. Introduction +---------------- + +Nowadays there is a growing need to divide a process into microservices and +there is a problem of monitoring the work of the same process. One way to +solve this problem is to use distributed tracing service in a central location, +the so-called tracer. + +OT is a feature introduced in HAProxy 2.4. This filter enables communication +via the OpenTracing API with OpenTracing compatible servers (tracers). +Currently, tracers that support this API include Datadog, Jaeger, LightStep +and Zipkin. + +The OT filter was primarily tested with the Jaeger tracer, while configurations +for both Datadog and Zipkin tracers were also set in the test directory. + +The OT filter is a standard HAProxy filter, so what applies to others also +applies to this one (of course, by that I mean what is described in the +documentation, more precisely in the doc/internals/filters.txt file). + +The OT filter activation is done explicitly by specifying it in the HAProxy +configuration. If this is not done, the OT filter in no way participates +in the work of HAProxy. + +As for the impact on HAProxy speed, this is documented with several tests +located in the test directory, and the result is found in the README-speed-* +files. In short, the speed of operation depends on the way it is used and +the complexity of the configuration, from an almost immeasurable impact to +a significant deceleration (5x and more). I think that in some normal use +the speed of HAProxy with the filter on will be quite satisfactory with a +slowdown of less than 4% (provided that no more than 10% of requests are +sent to the tracer, which is determined by the keyword 'rate-limit'). + +The OT filter allows intensive use of ACLs, which can be defined anywhere in +the configuration. Thus, it is possible to use the filter only for those +connections that are of interest to us. + + +2. Build instructions +---------------------- + +OT is the HAProxy filter and as such is compiled together with HAProxy. + +To communicate with some OpenTracing compatible tracer, the OT filter uses the +OpenTracing C Wrapper library (which again uses the OpenTracing CPP library). +This means that we must have both libraries installed on the system on which +we want to compile or use HAProxy. + +Instructions for compiling and installing both required libraries can be +found at https://github.com/haproxytech/opentracing-c-wrapper . + +Also, to use the OT filter when running HAProxy we need to have an OpenTracing +plugin for the tracer we want to use. We will return to this later, in +section 5. + +The OT filter can be more easily compiled using the pkg-config tool, if we +have the OpenTracing C Wrapper library installed so that it contains pkg-config +files (which have the .pc extension). If the pkg-config tool cannot be used, +then the path to the directory where the include files and libraries are +located can be explicitly specified. + +Below are examples of the two ways to compile HAProxy with the OT filter, the +first using the pkg-congfig tool and the second explicitly specifying the path +to the OpenTracing C Wrapper include and library. + +Note: prompt '%' indicates that the command is executed under a unprivileged + user, while prompt '#' indicates that the command is executed under the + root user. + +Example of compiling HAProxy using the pkg-congfig tool (assuming the +OpenTracing C Wrapper library is installed in the /opt directory): + + % PKG_CONFIG_PATH=/opt/lib/pkgconfig make USE_OT=1 TARGET=linux-glibc + +The OT filter can also be compiled in debug mode as follows: + + % PKG_CONFIG_PATH=/opt/lib/pkgconfig make USE_OT=1 OT_DEBUG=1 TARGET=linux-glibc + +HAProxy compilation example explicitly specifying path to the OpenTracing C +Wrapper include and library: + + % make USE_OT=1 OT_INC=/opt/include OT_LIB=/opt/lib TARGET=linux-glibc + +In case we want to use debug mode, then it looks like this: + + % make USE_OT=1 OT_DEBUG=1 OT_INC=/opt/include OT_LIB=/opt/lib TARGET=linux-glibc + +If the library we want to use is not installed on a unix system, then a locally +installed library can be used (say, which is compiled and installed in the user +home directory). In this case instead of /opt/include and /opt/lib the +equivalent paths to the local installation should be specified. Of course, +in that case the pkg-config tool can also be used if we have a complete +installation (with .pc files). + +last but not least, if the pkg-config tool is not used when compiling, then +HAProxy executable may not be able to find the OpenTracing C Wrapper library +at startup. This can be solved in several ways, for example using the +LD_LIBRARY_PATH environment variable which should be set to the path where the +library is located before starting the HAProxy. + + % LD_LIBRARY_PATH=/opt/lib /path-to/haproxy ... + +Another way is to add RUNPATH to HAProxy executable that contains the path to +the library in question. + + % make USE_OT=1 OT_RUNPATH=1 OT_INC=/opt/include OT_LIB=/opt/lib TARGET=linux-glibc + +After HAProxy is compiled, we can check if the OT filter is enabled: + + % ./haproxy -vv | grep opentracing + --- command output ---------- + [ OT] opentracing + --- command output ---------- + + +3. Basic concepts in OpenTracing +--------------------------------- + +Basic concepts of OpenTracing can be read on the OpenTracing documentation +website https://opentracing.io/docs/overview/. + +Here we will list only the most important elements of distributed tracing and +these are 'trace', 'span' and 'span context'. Trace is a description of the +complete transaction we want to record in the tracing system. A span is an +operation that represents a unit of work that is recorded in a tracing system. +Span context is a group of information related to a particular span that is +passed on to the system (from service to service). Using this context, we can +add new spans to already open trace (or supplement data in already open spans). + +An individual span may contain one or more tags, logs and baggage items. +The tag is a key-value element that is valid for the entire span. Log is a +key-value element that allows you to write some data at a certain time, it +can be used for debugging. A baggage item is a key-value data pair that can +be used for the duration of an entire trace, from the moment it is added to +the span. + + +4. OT configuration +-------------------- + +In order for the OT filter to be used, it must be included in the HAProxy +configuration, in the proxy section (frontend / listen / backend): + + frontend ot-test + ... + filter opentracing [id <id>] config <file> + ... + +If no filter id is specified, 'ot-filter' is used as default. The 'config' +parameter must be specified and it contains the path of the file used to +configure the OT filter. + + +4.1 OT scope +------------- + +If the filter id is defined for the OT filter, then the OT scope with +the same name should be defined in the configuration file. In the same +configuration file we can have several defined OT scopes. + +Each OT scope must have a defined (only one) "ot-tracer" section that is +used to configure the operation of the OT filter and define the used groups +and scopes. + +OT scope starts with the id of the filter specified in square brackets and +ends with the end of the file or when a new OT scope is defined. + +For example, this defines two OT scopes in the same configuration file: + [my-first-ot-filter] + ot-tracer tracer1 + ... + ot-group group1 + ... + ot-scope scope1 + ... + + [my-second-ot-filter] + ... + + +4.2. "ot-tracer" section +------------------------- + +Only one "ot-tracer" section must be defined for each OT scope. + +There are several keywords that must be defined for the OT filter to work. +These are 'config' which defines the configuration file for the OpenTracing +API, and 'plugin' which defines the OpenTracing plugin used. + +Through optional keywords can be defined ACLs, logging, rate limit, and groups +and scopes that define the tracing model. + + +ot-tracer <name> + A new OT with the name <name> is created. + + Arguments : + name - the name of the tracer section + + + The following keywords are supported in this section: + - mandatory keywords: + - config + - plugin + + - optional keywords: + - acl + - debug-level + - groups + - [no] log + - [no] option disabled + - [no] option dontlog-normal + - [no] option hard-errors + - rate-limit + - scopes + + +acl <aclname> <criterion> [flags] [operator] <value> ... + Declare or complete an access list. + + To configure and use the ACL, see section 7 of the HAProxy Configuration + Manual. + + +config <file> + 'config' is one of the two mandatory keywords associated with the OT tracer + configuration. This keyword sets the path of the configuration file for the + OpenTracing tracer plugin. To set the contents of this configuration file, + it is best to look at the documentation related to the OpenTracing tracer we + want to use. + + Arguments : + file - the path of the configuration file + + +debug-level <value> + This keyword sets the value of the debug level related to the display of + debug messages in the OT filter. The 'debug-level' value is binary, ie + a single value bit enables or disables the display of the corresponding + debug message that uses that bit. The default value is set via the + FLT_OT_DEBUG_LEVEL macro in the include/config.h file. Debug level value + is used only if the OT filter is compiled with the debug mode enabled, + otherwise it is ignored. + + Arguments : + value - binary value ranging from 0 to 255 (8 bits) + + +groups <name> ... + A list of "ot-group" groups used for the currently defined tracer is declared. + Several groups can be specified in one line. + + Arguments : + name - the name of the OT group + + +log global +log <addr> [len <len>] [format <fmt>] <facility> [<level> [<minlevel>]] +no log + Enable per-instance logging of events and traffic. + + To configure and use the logging system, see section 4.2 of the HAProxy + Configuration Manual. + + +option disabled +no option disabled + Keyword which turns the operation of the OT filter on or off. By default + the filter is on. + + +option dontlog-normal +no option dontlog-normal + Enable or disable logging of normal, successful processing. By default, + this option is disabled. For this option to be considered, logging must + be turned on. + + See also: 'log' keyword description. + + +option hard-errors +no option hard-errors + During the operation of the filter, some errors may occur, caused by + incorrect configuration of the tracer or some error related to the operation + of HAProxy. By default, such an error will not interrupt the filter + operation for the stream in which the error occurred. If the 'hard-error' + option is enabled, the operation error prohibits all further processing of + events and groups in the stream in which the error occurred. + + +plugin <file> + 'plugin' is one of the two mandatory keywords associated with the OT tracer + configuration. This keyword sets the path of the OpenTracing tracer plugin. + + Arguments : + file - the name of the plugin used + + +rate-limit <value> + This option allows limiting the use of the OT filter, ie it can be influenced + whether the OT filter is activated for a stream or not. Determining whether + or not a filter is activated depends on the value of this option that is + compared to a randomly selected value when attaching the filter to the stream. + By default, the value of this option is set to 100.0, ie the OT filter is + activated for each stream. + + Arguments : + value - floating point value ranging from 0.0 to 100.0 + + +scopes <name> ... + This keyword declares a list of "ot-scope" definitions used for the currently + defined tracer. Multiple scopes can be specified in the same line. + + Arguments : + name - the name of the OT scope + + +4.3. "ot-scope" section +------------------------ + +Stream processing begins with filter attachment, then continues with the +processing of a number of defined events and groups, and ends with filter +detachment. The "ot-scope" section is used to define actions related to +individual events. However, this section may be part of a group, so the +event does not have to be part of the definition. + + +ot-scope <name> + Creates a new OT scope definition named <name>. + + Arguments : + name - the name of the OT scope + + + The following keywords are supported in this section: + - acl + - baggage + - event + - extract + - finish + - inject + - log + - span + - tag + + +acl <aclname> <criterion> [flags] [operator] <value> ... + Declare or complete an access list. + + To configure and use the ACL, see section 7 of the HAProxy Configuration + Manual. + + +baggage <name> <sample> ... + Baggage items allow the propagation of data between spans, ie allow the + assignment of metadata that is propagated to future children spans. + This data is formatted in the style of key-value pairs and is part of + the context that can be transferred between processes that are part of + a server architecture. + + This kewyord allows setting the baggage for the currently active span. The + data type is always a string, ie any sample type is converted to a string. + The exception is a binary value that is not supported by the OT filter. + + See the 'tag' keyword description for the data type conversion table. + + Arguments : + name - key part of a data pair + sample - sample expression (value part of a data pair), at least one + sample must be present + + +event <name> [{ if | unless } <condition>] + Set the event that triggers the 'ot-scope' to which it is assigned. + Optionally, it can be followed by an ACL-based condition, in which case it + will only be evaluated if the condition is true. + + ACL-based conditions are executed in the context of a stream that processes + the client and server connections. To configure and use the ACL, see + section 7 of the HAProxy Configuration Manual. + + Arguments : + name - the event name + condition - a standard ACL-based condition + + Supported events are (the table gives the names of the events in the OT + filter and the corresponding equivalent in the SPOE filter): + + -------------------------------------|------------------------------ + the OT filter | the SPOE filter + -------------------------------------|------------------------------ + on-client-session-start | on-client-session + on-frontend-tcp-request | on-frontend-tcp-request + on-http-wait-request | - + on-http-body-request | - + on-frontend-http-request | on-frontend-http-request + on-switching-rules-request | - + on-backend-tcp-request | on-backend-tcp-request + on-backend-http-request | on-backend-http-request + on-process-server-rules-request | - + on-http-process-request | - + on-tcp-rdp-cookie-request | - + on-process-sticking-rules-request | - + on-client-session-end | - + on-server-unavailable | - + -------------------------------------|------------------------------ + on-server-session-start | on-server-session + on-tcp-response | on-tcp-response + on-http-wait-response | - + on-process-store-rules-response | - + on-http-response | on-http-response + on-server-session-end | - + -------------------------------------|------------------------------ + + +extract <name-prefix> [use-vars | use-headers] + For a more detailed description of the propagation process of the span + context, see the description of the keyword 'inject'. Only the process + of extracting data from the carrier is described here. + + Arguments : + name-prefix - data name prefix (ie key element prefix) + use-vars - data is extracted from HAProxy variables + use-headers - data is extracted from the HTTP header + + + Below is an example of using HAProxy variables to transfer span context data: + + --- test/ctx/ot.cfg -------------------------------------------------------- + ... + ot-scope client_session_start_2 + extract "ot_ctx_1" use-vars + span "Client session" child-of "ot_ctx_1" + ... + ---------------------------------------------------------------------------- + + +finish <name> ... + Closing a particular span or span context. Instead of the name of the span, + there are several specially predefined names with which we can finish certain + groups of spans. So it can be used as the name '*req*' for all open spans + related to the request channel, '*res*' for all open spans related to the + response channel and '*' for all open spans regardless of which channel they + are related to. Several spans and/or span contexts can be specified in one + line. + + Arguments : + name - the name of the span or context context + + +inject <name-prefix> [use-vars] [use-headers] + In OpenTracing, the transfer of data related to the tracing process between + microservices that are part of a larger service is done through the + propagation of the span context. The basic operations that allow us to + access and transfer this data are 'inject' and 'extract'. + + 'inject' allows us to extract span context so that the obtained data can + be forwarded to another process (microservice) via the selected carrier. + 'inject' in the name actually means inject data into carrier. Carrier is + an interface here (ie a data structure) that allows us to transfer tracing + state from one process to another. + + Data transfer can take place via one of two selected storage methods, the + first is by adding data to the HTTP header and the second is by using HAProxy + variables. Only data transfer via HTTP header can be used to transfer data + to another process (ie microservice). All data is organized in the form of + key-value data pairs. + + No matter which data transfer method you use, we need to specify a prefix + for the key element. All alphanumerics (lowercase only) and underline + character can be used to construct the data name prefix. Uppercase letters + can actually be used, but they will be converted to lowercase when creating + the prefix. + + Arguments : + name-prefix - data name prefix (ie key element prefix) + use-vars - HAProxy variables are used to store and transfer data + use-headers - HTTP headers are used to store and transfer data + + + Below is an example of using HTTP headers and variables, and how this is + reflected in the internal data of the HAProxy process. + + --- test/ctx/ot.cfg -------------------------------------------------------- + ... + ot-scope client_session_start_1 + span "HAProxy session" root + inject "ot_ctx_1" use-headers use-vars + ... + ---------------------------------------------------------------------------- + + - generated HAProxy variable (key -> value): + txn.ot_ctx_1.uberDtraceDid -> 8f1a05a3518d2283:8f1a05a3518d2283:0:1 + + - generated HTTP header (key: value): + ot_ctx_1-uber-trace-id: 8f1a05a3518d2283:8f1a05a3518d2283:0:1 + + Because HAProxy does not allow the '-' character in the variable name (which + is automatically generated by the OpenTracing API and on which we have no + influence), it is converted to the letter 'D'. We can see that there is no + such conversion in the name of the HTTP header because the '-' sign is allowed + there. Due to this conversion, initially all uppercase letters are converted + to lowercase because otherwise we would not be able to distinguish whether + the disputed sign '-' is used or not. + + Thus created HTTP headers and variables are deleted when executing the + 'finish' keyword or when detaching the stream from the filter. + + +log <name> <sample> ... + This kewyord allows setting the log for the currently active span. The + data type is always a string, ie any sample type is converted to a string. + The exception is a binary value that is not supported by the OT filter. + + See the 'tag' keyword description for the data type conversion table. + + Arguments : + name - key part of a data pair + sample - sample expression (value part of a data pair), at least one + sample must be present + + +span <name> [<reference>] + Creating a new span (or referencing an already opened one). If a new span + is created, it can be a child of the referenced span, follow from the + referenced span, or be root 'span'. In case we did not specify a reference + to the previously created span, the new span will become the root span. + We need to pay attention to the fact that in one trace there can be only + one root span. In case we have specified a non-existent span as a reference, + a new span will not be created. + + Arguments : + name - the name of the span being created or referenced (operation + name) + reference - span or span context to which the created span is referenced + + +tag <name> <sample> ... + This kewyord allows setting a tag for the currently active span. The first + argument is the name of the tag (tag ID) and the second its value. A value + can consist of one or more data. If the value is only one data, then the + type of that data depends on the type of the HAProxy sample. If the value + contains more data, then the data type is string. The data conversion table + is below: + + HAProxy sample data type | the OpenTracing data type + --------------------------+--------------------------- + NULL | NULL + BOOL | BOOL + INT32 | INT64 + UINT32 | UINT64 + INT64 | INT64 + UINT64 | UINT64 + IPV4 | STRING + IPV6 | STRING + STRING | STRING + BINARY | UNSUPPORTED + --------------------------+--------------------------- + + Arguments : + name - key part of a data pair + sample - sample expression (value part of a data pair), at least one + sample must be present + + +4.4. "ot-group" section +------------------------ + +This section allows us to define a group of OT scopes, that is not activated +via an event but is triggered from TCP or HTTP rules. More precisely, these +are the following rules: 'tcp-request', 'tcp-response', 'http-request', +'http-response' and 'http-after-response'. These rules can be defined in the +HAProxy configuration file. + + +ot-group <name> + Creates a new OT group definition named <name>. + + Arguments : + name - the name of the OT group + + + The following keywords are supported in this section: + - scopes + + +scopes <name> ... + 'ot-scope' sections that are part of the specified group are defined. If + the mentioned 'ot-scope' sections are used only in some OT group, they do + not have to have defined events. Several 'ot-scope' sections can be + specified in one line. + + Arguments : + name - the name of the 'ot-scope' section + + +5. Examples +------------ + +Several examples of the OT filter configuration can be found in the test +directory. A brief description of the prepared configurations follows: + +cmp - the configuration very similar to that of the spoa-opentracing project. + It was made to compare the speed of the OT filter with the + implementation of distributed tracing via spoa-opentracing application. + +sa - the configuration in which all possible events are used. + +ctx - the configuration is very similar to the previous one, with the only + difference that the spans are opened using the span context as a span + reference. + +fe be - a slightly more complicated example of the OT filter configuration + that uses two cascaded HAProxy services. The span context between + HAProxy processes is transmitted via the HTTP header. + +empty - the empty configuration in which the OT filter is initialized but + no event is triggered. It is not very usable, except to check the + behavior of the OT filter in the case of a similar configuration. + + +In order to be able to collect data (and view results via the web interface) +we need to install some of the supported tracers. We will use the Jaeger +tracer as an example. Installation instructions can be found on the website +https://www.jaegertracing.io/download/. For the impatient, here we will list +how the image to test the operation of the tracer system can be installed +without much reading of the documentation. + + # docker pull jaegertracing/all-in-one:latest + # docker run -d --name jaeger -e COLLECTOR_ZIPKIN_HTTP_PORT=9411 \ + -p 5775:5775/udp -p 6831:6831/udp -p 6832:6832/udp -p 5778:5778 \ + -p 16686:16686 -p 14268:14268 -p 9411:9411 jaegertracing/all-in-one:latest + +The last command will also initialize and run the Jaeger container. If we +want to use that container later, it can be started and stopped in the classic +way, using the 'docker container start/stop' commands. + + +In order to be able to use any of the configurations from the test directory, +we must also have a tracer plugin in that directory (all examples use the +Jaeger tracer plugin). The simplest way is to download the tracer plugin +using the already prepared shell script get-opentracing-plugins.sh. +The script accepts one argument, the directory in which the download is made. +If run without an argument, the script downloads all plugins to the current +directory. + + % ./get-opentracing-plugins.sh + +After that, we can run one of the pre-configured configurations using the +provided script run-xxx.sh (where xxx is the name of the configuration being +tested). For example: + + % ./run-sa.sh + +The script will create a new log file each time it is run (because part of the +log file name is the start time of the script). + +Eh, someone will surely notice that all test configurations use the Jaeger +tracing plugin that cannot be downloaded using the get-opentracing-plugins.sh +script. Unfortunately, the latest precompiled version that can be downloaded +is 0.4.2, for newer ones only the source code can be found. Version 0.4.2 has +a bug that can cause the operation of the OT filter to get stuck, so it is +better not to use this version. Here is the procedure by which we can compile +a newer version of the plugin (in our example it is 0.5.0). + +Important note: the GCC version must be at least 4.9 or later. + + % wget https://github.com/jaegertracing/jaeger-client-cpp/archive/v0.5.0.tar.gz + % tar xf v0.5.0.tar.gz + % cd jaeger-client-cpp-0.5.0 + % mkdir build + % cd build + % cmake -DCMAKE_INSTALL_PREFIX=/opt -DJAEGERTRACING_PLUGIN=ON -DHUNTER_CONFIGURATION_TYPES=Release -DHUNTER_BUILD_SHARED_LIBS=OFF .. + % make + +After the plugin is compiled, it will be in the current directory. The name +of the plugin is libjaegertracing_plugin.so. + + +5.1. Benchmarking results +-------------------------- + +To check the operation of the OT filter, several different test configurations +have been made which are located in the test directory. The test results of +the same configurations (with the names README-speed-xxx, where xxx is the name +of the configuration being tested) are also in the directory of the same name. + +All tests were performed on the same debian 9.13 system, CPU i7-4770, 32 GB RAM. +For the purpose of testing, the thttpd web server on port 8000 was used. +Testing was done with the wrk utility running via run-xxx.sh scripts; that is, +via the test-speed.sh script that is run as follows: + + % ./test-speed.sh all + +The above mentioned thttpd web server is run from that script and it should be +noted that we need to have the same installed on the system (or change the path +to the thttpd server in that script if it is installed elsewhere). + +Each test is performed several times over a period of 5 minutes per individual +test. The only difference when running the tests for the same configuration +was in changing the 'rate-limit' parameter (and the 'option disabled' option), +which is set to the following values: 100.0, 50.0, 10.0, 2.5 and 0.0 percent. +Then a test is performed with the OT filter active but disabled for request +processing ('option disabled' is included in the ot.cfg configuration). In +the last test, the OT filter is not used at all, ie it is not active and does +not affect the operation of HAProxy in any way. + + +6. OT CLI +---------- + +Via the HAProxy CLI interface we can find out the current status of the OT +filter and change several of its settings. + +All supported CLI commands can be found in the following way, using the +socat utility with the assumption that the HAProxy CLI socket path is set +to /tmp/haproxy.sock (of course, instead of socat, nc or other utility can +be used with a change in arguments when running the same): + + % echo "help" | socat - UNIX-CONNECT:/tmp/haproxy.sock | grep flt-ot + --- command output ---------- + flt-ot debug [level] : set the OT filter debug level (default: get current debug level) + flt-ot disable : disable the OT filter + flt-ot enable : enable the OT filter + flt-ot soft-errors : turning off hard-errors mode + flt-ot hard-errors : enabling hard-errors mode + flt-ot logging [state] : set logging state (default: get current logging state) + flt-ot rate [value] : set the rate limit (default: get current rate value) + flt-ot status : show the OT filter status + --- command output ---------- + +'flt-ot debug' can only be used in case the OT filter is compiled with the +debug mode enabled. + + +7. Known bugs and limitations +------------------------------ + +The name of the span context definition can contain only letters, numbers and +characters '_' and '-'. Also, all uppercase letters in the name are converted +to lowercase. The character '-' is converted internally to the 'D' character, +and since a HAProxy variable is generated from that name, this should be taken +into account if we want to use it somewhere in the HAProxy configuration. +The above mentioned span context is used in the 'inject' and 'extract' keywords. + +Let's look a little at the example test/fe-be (configurations are in the +test/fe and test/be directories, 'fe' is here the abbreviation for frontend +and 'be' for backend). In case we have the 'rate-limit' set to a value less +than 100.0, then distributed tracing will not be started with each new HTTP +request. It also means that the span context will not be delivered (via the +HTTP header) to the backend HAProxy process. The 'rate-limit' on the backend +HAProxy must be set to 100.0, but because the frontend HAProxy does not send +a span context every time, all such cases will cause an error to be reported +on the backend server. Therefore, the 'hard-errors' option must be set on the +backend server, so that processing on that stream is stopped as soon as the +first error occurs. Such cases will slow down the backend server's response +a bit (in the example in question it is about 3%). diff --git a/addons/ot/README-func b/addons/ot/README-func new file mode 100644 index 0000000..273c7f9 --- /dev/null +++ b/addons/ot/README-func @@ -0,0 +1,298 @@ +Here I will write down some specifics of certain parts of the source, these are +just some of my thoughts and clues and they are probably not too important for +a wider audience. + +src/parser.c +------------------------------------------------------------------------------ +The first thing to run when starting the HAProxy is the flt_ot_parse() function +which actually parses the filter configuration. + +In case of correct configuration, the function returns ERR_NONE (or 0), while +in case of incorrect configuration it returns the combination of ERR_* flags +(ERR_NONE here does not belong to that bit combination because its value is 0). + +One of the parameters of the function is <char **err> in which an error message +can be returned, if it exists. In that case the return value of the function +should have some of the ERR_* flags set. + +Let's look at an example of the following filter configuration what the function +call sequence looks like. + +Filter configuration line: + filter opentracing [id <id>] config <file> + +Function call sequence: + flt_ot_parse(<err>) { + /* Initialization of the filter configuration data. */ + flt_ot_conf_init() { + } + + /* Setting the filter name. */ + flt_ot_parse_keyword(<err>) { + flt_ot_parse_strdup(<err>) { + } + } + + /* Setting the filter configuration file name. */ + flt_ot_parse_keyword(<err>) { + flt_ot_parse_strdup(<err>) { + } + } + + /* Checking the configuration of the filter. */ + flt_ot_parse_cfg(<err>) { + flt_ot_parse_cfg_tracer() { + } + ... + flt_ot_post_parse_cfg_tracer() { + } + flt_ot_parse_cfg_group() { + } + ... + flt_ot_post_parse_cfg_group() { + } + flt_ot_parse_cfg_scope() { + } + ... + flt_ot_post_parse_cfg_scope() { + } + } + } + +Checking the filter configuration is actually much more complicated, only the +name of the main function flt_ot_parse_cfg() that does it is listed here. + +All functions that use the <err> parameter should set the error status using +that pointer. All other functions (actually these are all functions called +by the flt_ot_parse_cfg() function) should set the error message using the +ha_warning()/ha_alert() HAProxy functions. Of course, the return value (the +mentioned combination of ERR_* bits) is set in all these functions and it +indicates whether the filter configuration is correct or not. + + +src/group.c +------------------------------------------------------------------------------ +The OT filter allows the use of groups within which one or more 'ot-scope' +declarations can be found. These groups can be used using several HAProxy +rules, more precisely 'http-request', 'http-response', 'tcp-request', +'tcp-response' and 'http-after-response' rules. + +Configuration example for the specified rules: + <rule> ot-group <filter-id> <group-name> [ { if | unless } <condition> ] + +Parsing each of these rules is performed by the flt_ot_group_parse() function. +After parsing the configuration, its verification is performed via the +flt_ot_group_check() function. One parsing function and one configuration +check function are called for each defined rule. + + flt_ot_group_parse(<err>) { + } + ... + flt_ot_group_check() { + } + ... + + +When deinitializing the module, the function flt_ot_group_release() is called +(which is actually an release_ptr callback function from one of the above +rules). One callback function is called for each defined rule. + + flt_ot_group_release() { + } + ... + + +src/filter.c +------------------------------------------------------------------------------ +After parsing and checking the configuration, the flt_ot_check() function is +called which associates the 'ot-group' and 'ot-scope' definitions with their +declarations. This procedure concludes the configuration of the OT filter and +after that its initialization is possible. + + flt_ops.check = flt_ot_check; + flt_ot_check() { + } + + +The initialization of the OT filter is done via the flt_ot_init() callback +function. In this function the OpenTracing API library is also initialized. +It is also possible to initialize for each thread individually, but nothing +is being done here for now. + + flt_ops.init = flt_ot_init; + flt_ot_init() { + flt_ot_cli_init() { + } + /* Initialization of the OpenTracing API. */ + ot_init(<err>) { + } + } + + flt_ops.init_per_thread = flt_ot_init_per_thread; + flt_ot_init_per_thread() { + } + ... + + +After the filter instance is created and attached to the stream, the +flt_ot_attach() function is called. In this function a new OT runtime +context is created, and flags are set that define which analyzers are used. + + flt_ops.attach = flt_ot_attach; + flt_ot_attach() { + /* In case OT is disabled, nothing is done on this stream further. */ + flt_ot_runtime_context_init(<err>) { + flt_ot_pool_alloc() { + } + /* Initializing and setting the variable 'sess.ot.uuid'. */ + if (flt_ot_var_register(<err>) != -1) { + flt_ot_var_set(<err>) { + } + } + } + } + + +When a stream is started, this function is called. At the moment, nothing +is being done in it. + + flt_ops.stream_start = flt_ot_stream_start; + flt_ot_stream_start() { + } + + +Channel analyzers are called when executing individual filter events. +For each of the four analyzer functions, the events associated with them +are listed. + + Events: + - 1 'on-client-session-start' + - 15 'on-server-session-start' +------------------------------------------------------------------------ + flt_ops.channel_start_analyze = flt_ot_channel_start_analyze; + flt_ot_channel_start_analyze() { + flt_ot_event_run() { + /* Run event. */ + flt_ot_scope_run() { + /* Processing of all ot-scopes defined for the current event. */ + } + } + } + + + Events: + - 2 'on-frontend-tcp-request' + - 4 'on-http-body-request' + - 5 'on-frontend-http-request' + - 6 'on-switching-rules-request' + - 7 'on-backend-tcp-request' + - 8 'on-backend-http-request' + - 9 'on-process-server-rules-request' + - 10 'on-http-process-request' + - 11 'on-tcp-rdp-cookie-request' + - 12 'on-process-sticking-rules-request + - 16 'on-tcp-response' + - 18 'on-process-store-rules-response' + - 19 'on-http-response' +------------------------------------------------------------------------ + flt_ops.channel_pre_analyze = flt_ot_channel_pre_analyze; + flt_ot_channel_pre_analyze() { + flt_ot_event_run() { + /* Run event. */ + flt_ot_scope_run() { + /* Processing of all ot-scopes defined for the current event. */ + } + } + } + + + Events: + - 3 'on-http-wait-request' + - 17 'on-http-wait-response' +------------------------------------------------------------------------ + flt_ops.channel_post_analyze = flt_ot_channel_post_analyze; + flt_ot_channel_post_analyze() { + flt_ot_event_run() { + /* Run event. */ + flt_ot_scope_run() { + /* Processing of all ot-scopes defined for the current event. */ + } + } + } + + + Events: + - 13 'on-client-session-end' + - 14 'on-server-unavailable' + - 20 'on-server-session-end' +------------------------------------------------------------------------ + flt_ops.channel_end_analyze = flt_ot_channel_end_analyze; + flt_ot_channel_end_analyze() { + flt_ot_event_run() { + /* Run event. */ + flt_ot_scope_run() { + /* Processing of all ot-scopes defined for the current event. */ + } + } + + /* In case the backend server does not work, event 'on-server-unavailable' + is called here before event 'on-client-session-end'. */ + if ('on-server-unavailable') { + flt_ot_event_run() { + /* Run event. */ + flt_ot_scope_run() { + /* Processing of all ot-scopes defined for the current event. */ + } + } + } + } + + +After the stream has stopped, this function is called. At the moment, nothing +is being done in it. + + flt_ops.stream_stop = flt_ot_stream_stop; + flt_ot_stream_stop() { + } + + +Then, before the instance filter is detached from the stream, the following +function is called. It deallocates the runtime context of the OT filter. + + flt_ops.detach = flt_ot_detach; + flt_ot_detach() { + flt_ot_runtime_context_free() { + flt_ot_pool_free() { + } + } + } + + +Module deinitialization begins with deinitialization of individual threads +(as many threads as configured for the HAProxy process). Because nothing +special is connected to the process threads, nothing is done in this function. + + flt_ops.deinit_per_thread = flt_ot_deinit_per_thread; + flt_ot_deinit_per_thread() { + } + ... + + +For this function see the above description related to the src/group.c file. + + flt_ot_group_release() { + } + ... + + +Module deinitialization ends with the flt_ot_deinit() function, in which all +memory occupied by module operation (and OpenTracing API operation, of course) +is freed. + + flt_ops.deinit = flt_ot_deinit; + flt_ot_deinit() { + ot_close() { + } + flt_ot_conf_free() { + } + } diff --git a/addons/ot/README-pool b/addons/ot/README-pool new file mode 100644 index 0000000..8164b04 --- /dev/null +++ b/addons/ot/README-pool @@ -0,0 +1,25 @@ +Used pools: + +-------------------------------+-----------------------------+----------------------------- + head / name | size | define +-------------------------------+-----------------------------+----------------------------- + pool_head_ buffer | global.tune.bufsize = 16384 | USE_POOL_BUFFER + pool_head_ trash | 32 + 16384 | USE_TRASH_CHUNK +-------------------------------+-----------------------------+----------------------------- + pool_head_ ot_scope_span | 96 | USE_POOL_OT_SCOPE_SPAN + pool_head_ ot_scope_context | 64 | USE_POOL_OT_SCOPE_CONTEXT + pool_head_ ot_runtime_context | 128 | USE_POOL_OT_RUNTIME_CONTEXT + pool_head_ ot_span_context | 96 | USE_POOL_OT_SPAN_CONTEXT +-------------------------------+-----------------------------+----------------------------- + +By defining individual definitions in file include/config.h, it is possible to +switch individual pools on / off. If a particular pool is not used, memory is +used in a 'normal' way instead, using malloc()/free() functions. + +This is made only from the aspect of debugging the program, i.e. comparing the +speed of operation using different methods of working with memory. + +In general, it would be better to use memory pools, due to less fragmentation +of memory space after long operation of the program. The speed of operation +is similar to when using standard allocation functions (when testing it was +shown that pool use was fast by about 1%). |