Adding upstream version 1:7.0.3.upstream/1%7.0.3

Signed-off-by: Daniel Baumann <daniel.baumann@progress-linux.org>

5 files changed, 705 insertions, 0 deletions

diff --git a/doc/userguide/partials/commands-pcap-sc.rst b/doc/userguide/partials/commands-pcap-sc.rst
new file mode 100644
index 0000000..4e5e496
--- /dev/null
+++ b/doc/userguide/partials/commands-pcap-sc.rst
@@ -0,0 +1,38 @@
+.. Consider converting `.. description` to `.. option` when the
+   minimum version of Sphinx on the primary distributions are all
+   updated to generate duplicate reference links. For example, we
+   can't use `.. option` on CentOS 7 which has Sphinx 1.1.3, but
+   Fedora 30 with Sphinx 1.8.4 is fine.
+
+.. describe:: pcap-file <file> <dir> [tenant] [continuous] [delete-when-done]
+
+   Add pcap files to Suricata for sequential processing. The generated
+   log/alert files will be put into the directory specified as second argument.
+   Make sure to provide absolute path to the files and directory. It is
+   acceptable to add multiple files without waiting the result.
+
+.. describe:: pcap-file-continuous <file> <dir> [tenant] [delete-when-done]
+
+   Add pcap files to Suricata for sequential processing. Directory will be
+   monitored for new files being added until there is a use of
+   **pcap-interrupt** or directory is moved or deleted.
+
+.. describe:: pcap-file-number
+
+   Number of pcap files waiting to get processed.
+
+.. describe:: pcap-file-list
+
+   List of queued pcap files.
+
+.. describe:: pcap-last-processed
+
+   Processed time of last file in milliseconds since epoch.
+
+.. describe:: pcap-interrupt
+
+   Terminate the current state by interrupting directory processing.
+
+.. describe:: pcap-current
+
+   Currently processed file.
diff --git a/doc/userguide/partials/commands-sc.rst b/doc/userguide/partials/commands-sc.rst
new file mode 100644
index 0000000..d34707f
--- /dev/null
+++ b/doc/userguide/partials/commands-sc.rst
@@ -0,0 +1,125 @@
+.. Consider converting `.. description` to `.. option` when the
+   minimum version of Sphinx on the primary distributions are all
+   updated to generate duplicate reference links. For example, we
+   can't use `.. option` on CentOS 7 which has Sphinx 1.1.3, but
+   Fedora 30 with Sphinx 1.8.4 is fine.
+
+.. Start with the most common basic commands.
+
+.. describe:: shutdown
+
+   Shut Suricata instance down.
+
+.. describe:: command-list
+
+   List available commands.
+
+.. describe:: help
+
+   Get help about the available commands.
+
+.. describe:: version
+
+   Print the version of Suricata instance.
+
+.. describe:: uptime
+
+   Display the uptime of Suricata.
+
+.. describe:: running-mode
+
+   Display running mode. This can either be *workers*, *autofp* or *single*.
+
+.. describe:: capture-mode
+
+   Display the capture mode. This can be either of *PCAP_DEV*,
+   *PCAP_FILE*, *PFRING(DISABLED)*, *NFQ*, *NFLOG*, *IPFW*, *ERF_FILE*,
+   *ERF_DAG*, *AF_PACKET_DEV*, *NETMAP(DISABLED)*, *UNIX_SOCKET* or
+   *WINDIVERT(DISABLED)*.
+
+.. describe:: conf-get <variable>
+
+   Get configuration value for a given variable. Variable to be provided can be
+   either of the configuration parameters that are written in suricata.yaml.
+
+.. describe:: dump-counters
+
+   Dump Suricata's performance counters.
+
+.. describe:: ruleset-reload-rules
+
+   Reload the ruleset and wait for completion.
+
+.. describe:: reload-rules
+
+   Alias .. describe *ruleset-reload-rules*.
+
+.. describe:: ruleset-reload-nonblocking
+
+   Reload ruleset and proceed without waiting.
+
+.. describe:: ruleset-reload-time
+
+   Return time of last reload.
+
+.. describe:: ruleset-stats
+
+   Display the number of rules loaded and failed.
+
+.. describe:: ruleset-failed-rules
+
+   Display the list of failed rules.
+
+.. describe:: register-tenant-handler <id> <htype> [hargs]
+
+   Register a tenant handler with the specified mapping.
+
+.. describe:: unregister-tenant-handler <id> <htype> [hargs]
+
+   Unregister a tenant handler with the specified mapping.
+
+.. describe:: register-tenant <id> <filename>
+
+   Register tenant with a particular ID and filename.
+
+.. describe:: reload-tenant <id> [filename]
+
+   Reload a tenant with specified ID. A filename to a tenant yaml can be
+   specified. If it is omitted, the original yaml that was used to load
+   / last reload the tenant is used.
+
+.. describe:: reload-tenants
+
+   Reload all registered tenants by reloading their yaml.
+
+.. describe:: unregister-tenant <id>
+
+   Unregister tenant with a particular ID.
+
+.. describe:: add-hostbit <ipaddress> <hostbit> <expire>
+
+   Add hostbit on a host IP with a particular bit name and time of expiry.
+
+.. describe:: remove-hostbit <ipaddress> <hostbit>
+
+   Remove hostbit on a host IP with specified IP address and bit name.
+
+.. describe:: list-hostbit <ipaddress>
+
+   List hostbit for a particular host IP.
+
+.. describe:: reopen-log-files
+
+   Reopen log files to be run after external log rotation.
+
+.. describe:: memcap-set <config> <memcap>
+
+   Update memcap value of a specified item.
+
+.. describe:: memcap-show <config>
+
+   Show memcap value of a specified item.
+
+.. describe:: memcap-list
+
+   List all memcap values available.
diff --git a/doc/userguide/partials/eve-log.yaml b/doc/userguide/partials/eve-log.yaml
new file mode 100644
index 0000000..9652257
--- /dev/null
+++ b/doc/userguide/partials/eve-log.yaml
@@ -0,0 +1,187 @@
+outputs:
+  # Extensible Event Format (nicknamed EVE) event log in JSON format
+  - eve-log:
+      enabled: yes
+      filetype: regular #regular|syslog|unix_dgram|unix_stream|redis
+      filename: eve.json
+      # Enable for multi-threaded eve.json output; output files are amended
+      # with an identifier, e.g., eve.9.json
+      #threaded: false
+      #prefix: "@cee: " # prefix to prepend to each log entry
+      # the following are valid when type: syslog above
+      #identity: "suricata"
+      #facility: local5
+      #level: Info ## possible levels: Emergency, Alert, Critical,
+                   ## Error, Warning, Notice, Info, Debug
+      #redis:
+      #  server: 127.0.0.1
+      #  port: 6379
+      #  async: true ## if redis replies are read asynchronously
+      #  mode: list ## possible values: list|lpush (default), rpush, channel|publish
+      #             ## lpush and rpush are using a Redis list. "list" is an alias for lpush
+      #             ## publish is using a Redis channel. "channel" is an alias for publish
+      #  key: suricata ## key or channel to use (default to suricata)
+      # Redis pipelining set up. This will enable to only do a query every
+      # 'batch-size' events. This should lower the latency induced by network
+      # connection at the cost of some memory. There is no flushing implemented
+      # so this setting as to be reserved to high traffic suricata.
+      #  pipelining:
+      #    enabled: yes ## set enable to yes to enable query pipelining
+      #    batch-size: 10 ## number of entry to keep in buffer
+
+      # Include top level metadata. Default yes.
+      #metadata: no
+
+      types:
+        - alert:
+            # payload: yes             # enable dumping payload in Base64
+            # payload-buffer-size: 4kb # max size of payload buffer to output in eve-log
+            # payload-printable: yes   # enable dumping payload in printable (lossy) format
+            # packet: yes              # enable dumping of packet (without stream segments)
+            # http-body: yes           # Requires metadata; enable dumping of http body in Base64
+            # http-body-printable: yes # Requires metadata; enable dumping of http body in printable format
+
+            # Enable the logging of tagged packets for rules using the
+            # "tag" keyword.
+            tagged-packets: yes
+
+            # Configure the metadata to be logged along with an
+            # alert. The following shows the default configuration
+            # which is used if this field is not provided or simply
+            # set to a truthful value. Setting of this section is only
+            # required if you wish to enable/disable specific fields.
+            #metadata:
+
+              # Include the decoded application layer (ie. http, dns)
+              app-layer: true
+
+              # Log the current state of the flow record.
+              flow: true
+
+              rule:
+                # Log the metadata field from the rule in a structured
+                # format.
+                metadata: true
+
+                # Log the raw rule text.
+                raw: false
+
+            # HTTP X-Forwarded-For support by adding an extra field or overwriting
+            # the source or destination IP address (depending on flow direction)
+            # with the one reported in the X-Forwarded-For HTTP header. This is
+            # helpful when reviewing alerts for traffic that is being reverse
+            # or forward proxied.
+            xff:
+              enabled: no
+              # Two operation modes are available, "extra-data" and "overwrite".
+              mode: extra-data
+              # Two proxy deployments are supported, "reverse" and "forward". In
+              # a "reverse" deployment the IP address used is the last one, in a
+              # "forward" deployment the first IP address is used.
+              deployment: reverse
+              # Header name where the actual IP address will be reported, if more
+              # than one IP address is present, the last IP address will be the
+              # one taken into consideration.
+              header: X-Forwarded-For
+        - http:
+            extended: yes     # enable this for extended logging information
+            # custom allows additional http fields to be included in eve-log
+            # the example below adds three additional fields when uncommented
+            #custom: [Accept-Encoding, Accept-Language, Authorization]
+        - dns:
+            # Use version 2 logging with the new format:
+            # dns answers will be logged in one single event
+            # rather than an event for each of the answers.
+            # Without setting a version the version
+            # will fallback to 1 for backwards compatibility.
+            version: 2
+
+            # Enable/disable this logger. Default: enabled.
+            #enabled: no
+
+            # Control logging of requests and responses:
+            # - requests: enable logging of DNS queries
+            # - responses: enable logging of DNS answers
+            # By default both requests and responses are logged.
+            #requests: no
+            #responses: no
+
+            # Format of answer logging:
+            # - detailed: array item per answer
+            # - grouped: answers aggregated by type
+            # Default: all
+            #answer-format: [detailed, grouped]
+
+            # Answer types to log.
+            # Default: all
+            #answer-types: [a, aaaa, cname, mx, ns, ptr, txt]
+        - dns:
+            # Version 1 DNS logger.
+            # Deprecated: Will be removed by May 2022.
+            version: 1
+
+            enabled: no
+            # control logging of queries and answers
+            # default yes, no to disable
+            query: yes     # enable logging of DNS queries
+            answer: yes    # enable logging of DNS answers
+            # control which RR types are logged
+            # all enabled if custom not specified
+            #custom: [a, aaaa, cname, mx, ns, ptr, txt]
+        - tls:
+            extended: yes     # enable this for extended logging information
+            # output TLS transaction where the session is resumed using a
+            # session id
+            #session-resumption: no
+            # custom allows to control which tls fields that are included
+            # in eve-log
+            #custom: [subject, issuer, session_resumed, serial, fingerprint, sni, version, not_before, not_after, certificate, chain]
+        - files:
+            force-magic: no   # force logging magic on all logged files
+            # force logging of checksums, available hash functions are md5,
+            # sha1 and sha256
+            #force-hash: [md5]
+        #- drop:
+        #    alerts: yes      # log alerts that caused drops
+        #    flows: all       # start or all: 'start' logs only a single drop
+        #                     # per flow direction. All logs each dropped pkt.
+        - smtp:
+            #extended: yes # enable this for extended logging information
+            # this includes: bcc, message-id, subject, x_mailer, user-agent
+            # custom fields logging from the list:
+            #  reply-to, bcc, message-id, subject, x-mailer, user-agent, received,
+            #  x-originating-ip, in-reply-to, references, importance, priority,
+            #  sensitivity, organization, content-md5, date
+            #custom: [received, x-mailer, x-originating-ip, relays, reply-to, bcc]
+            # output md5 of fields: body, subject
+            # for the body you need to set app-layer.protocols.smtp.mime.body-md5
+            # to yes
+            #md5: [body, subject]
+
+        # NFS logging.
+        - nfs
+        # IKE logging.
+        - ike
+        # BitTorrent DHT logging.
+        - bittorrent-dht
+        - ssh
+        - stats:
+            totals: yes       # stats for all threads merged together
+            threads: no       # per thread stats
+            deltas: no        # include delta values
+        - dhcp:
+            # DHCP logging.
+            enabled: yes
+            # When extended mode is on, all DHCP messages are logged
+            # with full detail. When extended mode is off (the
+            # default), just enough information to map a MAC address
+            # to an IP address is logged.
+            extended: no
+        # bi-directional flows
+        - flow
+        # uni-directional flows
+        #- netflow
+
+        # An event for logging metadata, specifically pktvars when
+        # they are set, but will also include the full metadata object.
+        #- metadata
diff --git a/doc/userguide/partials/options-unittests.rst b/doc/userguide/partials/options-unittests.rst
new file mode 100644
index 0000000..971e8fd
--- /dev/null
+++ b/doc/userguide/partials/options-unittests.rst
@@ -0,0 +1,25 @@
+.. Options for developers - unittests.
+
+.. option:: -u
+
+   Run the unit tests and exit. Requires that Suricata be configured
+   with *--enable-unittests*.
+
+.. option:: -U, --unittest-filter=REGEX
+
+   With the -U option you can select which of the unit tests you want
+   to run. This option uses REGEX. Example of use: suricata -u -U
+   http
+
+.. option:: --list-unittests
+
+   Lists available unit tests.
+
+.. option:: --fatal-unittests
+
+   Enables fatal failure on a unit test error. Suricata will exit
+   instead of continuing more tests.
+
+.. option:: --unittests-coverage
+
+   Display unit test coverage report.
diff --git a/doc/userguide/partials/options.rst b/doc/userguide/partials/options.rst
new file mode 100644
index 0000000..34f8f6a
--- /dev/null
+++ b/doc/userguide/partials/options.rst
@@ -0,0 +1,330 @@
+.. Start with the most common basic options.
+
+.. option:: -h
+
+   Display a brief usage overview.
+
+.. option:: -V
+
+   Displays the version of Suricata.
+
+.. option:: -c <path>
+
+   Path to configuration file.
+
+.. option:: --include <path>
+
+   Additional configuration files to include. Multiple additional
+   configuration files can be provided and will be included in the
+   order specified on the command line.  These additional configuration
+   files are loaded as if they existed at the end of the main
+   configuration file.
+
+   Example including one additional file::
+
+     --include /etc/suricata/other.yaml
+
+   Example including more than one additional file::
+
+     --include /etc/suricata/other.yaml --include /etc/suricata/extra.yaml
+
+.. option:: -T
+
+   Test configuration.
+
+.. _cmdline-option-v:
+
+.. option:: -v
+
+   Increase the verbosity of the Suricata application logging by
+   increasing the log level from the default. This option can be
+   passed multiple times to further increase the verbosity.
+
+   - -v: INFO
+   - -vv: PERF
+   - -vvv: CONFIG
+   - -vvvv: DEBUG
+
+   This option will not decrease the log level set in the
+   configuration file if it is already more verbose than the level
+   requested with this option.
+
+.. Basic input options.
+
+.. option:: -r <path>
+
+   Run in pcap offline mode (replay mode) reading files from pcap file. If
+   <path> specifies a directory, all files in that directory will be processed
+   in order of modified time maintaining flow state between files.
+
+.. option:: --pcap-file-continuous
+
+   Used with the -r option to indicate that the mode should stay alive until
+   interrupted. This is useful with directories to add new files and not reset
+   flow state between files.
+
+.. option:: --pcap-file-recursive
+
+   Used with the -r option when the path provided is a directory.  This option
+   enables recursive traversal into subdirectories to a maximum depth of 255.
+   This option cannot be combined with --pcap-file-continuous.  Symlinks are
+   ignored.
+
+.. option:: --pcap-file-delete
+
+   Used with the -r option to indicate that the mode should delete pcap files
+   after they have been processed. This is useful with pcap-file-continuous to
+   continuously feed files to a directory and have them cleaned up when done. If
+   this option is not set, pcap files will not be deleted after processing.
+
+.. option::  -i <interface>
+
+   After the -i option you can enter the interface card you would like
+   to use to sniff packets from.  This option will try to use the best
+   capture method available. Can be used several times to sniff packets from
+   several interfaces.
+
+.. option:: --pcap[=<device>]
+
+   Run in PCAP mode. If no device is provided the interfaces
+   provided in the *pcap* section of the configuration file will be
+   used.
+   
+.. option:: --af-packet[=<device>]
+
+   Enable capture of packet using AF_PACKET on Linux. If no device is
+   supplied, the list of devices from the af-packet section in the
+   yaml is used.
+
+.. option:: --af-xdp[=<device>]
+
+   Enable capture of packet using AF_XDP on Linux. If no device is
+   supplied, the list of devices from the af-xdp section in the
+   yaml is used.
+
+.. option:: -q <queue id>
+
+   Run inline of the NFQUEUE queue ID provided. May be provided
+   multiple times.
+
+.. Back to other basic options.
+
+.. option:: -s <filename.rules>
+
+   With the -s option you can set a file with signatures, which will
+   be loaded together with the rules set in the yaml.
+
+   It is possible to use globbing when specifying rules files.
+   For example, ``-s '/path/to/rules/*.rules'``
+
+.. option:: -S <filename.rules>
+
+   With the -S option you can set a file with signatures, which will
+   be loaded exclusively, regardless of the rules set in the yaml.
+
+   It is possible to use globbing when specifying rules files.
+   For example, ``-S '/path/to/rules/*.rules'``
+
+.. option:: -l <directory>
+
+   With the -l option you can set the default log directory. If you
+   already have the default-log-dir set in yaml, it will not be used
+   by Suricata if you use the -l option. It will use the log dir that
+   is set with the -l option. If you do not set a directory with
+   the -l option, Suricata will use the directory that is set in yaml.
+
+.. option:: -D
+
+   Normally if you run Suricata on your console, it keeps your console
+   occupied. You can not use it for other purposes, and when you close
+   the window, Suricata stops running.  If you run Suricata as daemon
+   (using the -D option), it runs at the background and you will be
+   able to use the console for other tasks without disturbing the
+   engine running.
+
+.. option:: --runmode <runmode>
+
+   With the *--runmode* option you can set the runmode that you would
+   like to use. This command line option can override the yaml runmode
+   option.
+
+   Runmodes are: *workers*, *autofp* and *single*.
+
+   For more information about runmodes see :doc:`Runmodes
+   </performance/runmodes>` in the user guide.
+
+.. option:: -F <bpf filter file>
+
+   Use BPF filter from file.
+
+.. option:: -k [all|none]
+
+   Force (all) the checksum check or disable (none) all checksum
+   checks.
+
+.. option:: --user=<user>
+
+   Set the process user after initialization. Overrides the user
+   provided in the *run-as* section of the configuration file.
+
+.. option:: --group=<group>
+
+   Set the process group to group after initialization. Overrides the
+   group provided in the *run-as* section of the configuration file.
+
+.. option:: --pidfile <file>
+
+   Write the process ID to file. Overrides the *pid-file* option in
+   the configuration file and forces the file to be written when not
+   running as a daemon.
+
+.. option:: --init-errors-fatal
+
+   Exit with a failure when errors are encountered loading signatures.
+
+.. option:: --strict-rule-keywords[=all|<keyword>|<keywords(csv)]
+
+   Applies to: classtype, reference and app-layer-event.
+
+   By default missing reference or classtype values are warnings and
+   not errors. Additionally, loading outdated app-layer-event events are
+   also not treated as errors, but as warnings instead.
+
+   If this option is enabled these warnings are considered errors.
+
+   If no value, or the value 'all', is specified, the option applies to
+   all of the keywords above. Alternatively, a comma separated list can
+   be supplied with the keyword names it should apply to.
+
+.. option:: --disable-detection
+
+   Disable the detection engine.
+
+.. option:: --disable-hashing
+
+   Disable support for hash algorithms such as md5, sha1 and sha256.
+
+   By default hashing is enabled. Disabling hashing will also disable some
+   Suricata features such as the filestore, ja3, and rule keywords that use hash
+   algorithms.
+
+.. Information options.
+   
+.. option:: --dump-config
+
+   Dump the configuration loaded from the configuration file to the
+   terminal and exit.
+
+.. option:: --dump-features
+
+   Dump the features provided by Suricata modules and exit. Features
+   list (a subset of) the configuration values and are intended to
+   assist with comparing provided features with those required by
+   one or more rules.
+
+.. option:: --build-info
+
+   Display the build information the Suricata was built with.
+
+.. option:: --list-app-layer-protos
+
+   List all supported application layer protocols.
+
+.. option:: --list-keywords=[all|csv|<kword>]
+
+   List all supported rule keywords.
+
+.. option:: --list-runmodes
+
+   List all supported run modes.
+
+.. Advanced options.
+
+.. option:: --set <key>=<value>
+
+   Set a configuration value. Useful for overriding basic
+   configuration parameters. For example, to change the default log
+   directory::
+
+     --set default-log-dir=/var/tmp
+
+   This option cannot be used to add new entries to a list in the
+   configuration file, such as a new output. It can only be used to
+   modify a value in a list that already exists.
+
+   For example, to disable the ``eve-log`` in the default
+   configuration file::
+
+     --set outputs.1.eve-log.enabled=no
+
+   Also note that the index values may change as the ``suricata.yaml``
+   is updated.
+
+   See the output of ``--dump-config`` for existing values that could
+   be modified with their index.
+
+.. option:: --engine-analysis
+
+   Print reports on analysis of different sections in the engine and
+   exit. Please have a look at the conf parameter engine-analysis on
+   what reports can be printed
+
+.. option:: --unix-socket=<file>
+
+   Use file as the Suricata unix control socket. Overrides the
+   *filename* provided in the *unix-command* section of the
+   configuration file.
+
+.. option:: --reject-dev=<device>
+
+   Use *device* to send out RST / ICMP error packets with
+   the *reject* keyword.
+
+.. Advanced input options.
+
+.. option:: --pcap-buffer-size=<size>
+
+   Set the size of the PCAP buffer (0 - 2147483647).
+
+.. option:: --netmap[=<device>]
+
+   Enable capture of packet using NETMAP on FreeBSD or Linux. If no
+   device is supplied, the list of devices from the netmap section
+   in the yaml is used.
+
+.. option:: --pfring[=<device>]
+
+   Enable PF_RING packet capture. If no device provided, the devices in
+   the Suricata configuration will be used.
+  
+.. option:: --pfring-cluster-id <id>
+
+   Set the PF_RING cluster ID.
+   
+.. option:: --pfring-cluster-type <type>
+
+   Set the PF_RING cluster type (cluster_round_robin, cluster_flow).
+
+.. option:: -d <divert-port>
+
+   Run inline using IPFW divert mode.
+
+.. option:: --dag <device>
+
+   Enable packet capture off a DAG card. If capturing off a specific
+   stream the stream can be select using a device name like
+   "dag0:4". This option may be provided multiple times read off
+   multiple devices and/or streams.
+	    
+.. option:: --napatech
+
+   Enable packet capture using the Napatech Streams API.
+
+.. option:: --erf-in=<file>
+
+   Run in offline mode reading the specific ERF file (Endace
+   extensible record format).
+
+.. option:: --simulate-ips
+
+   Simulate IPS mode when running in a non-IPS mode.