Adding upstream version 8.2402.0+dfsg.upstream/8.2402.0+dfsg

Signed-off-by: Daniel Baumann <daniel.baumann@progress-linux.org>

1 files changed, 218 insertions, 0 deletions

diff --git a/source/whitepapers/syslog_protocol.rst b/source/whitepapers/syslog_protocol.rst
new file mode 100644
index 0000000..8373f71
--- /dev/null
+++ b/source/whitepapers/syslog_protocol.rst
@@ -0,0 +1,218 @@
+syslog-protocol support in rsyslog
+==================================
+
+`rsyslog <http://www.rsyslog.com/>`_ **provides a trial implementation
+of the proposed**
+`syslog-protocol <http://www.monitorware.com/Common/en/glossary/syslog-protocol.php>`_
+**standard.** The intention of this implementation is to find out what
+inside syslog-protocol is causing problems during implementation. As
+syslog-protocol is a standard under development, its support in rsyslog
+is highly volatile. It may change from release to release. So while it
+provides some advantages in the real world, users are cautioned against
+using it right now. If you do, be prepared that you will probably need
+to update all of your rsyslogds with each new release. If you try it
+anyhow, please provide feedback as that would be most beneficial for us.
+
+Currently supported message format
+----------------------------------
+
+Due to recent discussion on syslog-protocol, we do not follow any
+specific revision of the draft but rather the candidate ideas. The
+format supported currently is:
+
+**``<PRI>VERSION SP TIMESTAMP SP HOSTNAME SP APP-NAME SP PROCID SP MSGID SP [SD-ID]s  SP MSG``**
+
+Field syntax and semantics are as defined in IETF I-D
+syslog-protocol-15.
+
+Capabilities Implemented
+------------------------
+
+-  receiving message in the supported format (see above)
+-  sending messages in the supported format
+-  relaying messages
+-  receiving messages in either legacy or -protocol format and
+   transforming them into the other one
+-  virtual availability of TAG, PROCID, APP-NAME, MSGID, SD-ID no matter
+   if the message was received via legacy format, API or syslog-protocol
+   format (non-present fields are being emulated with great success)
+-  maximum message size is set via preprocessor #define
+-  syslog-protocol messages can be transmitted both over UDP and plain
+   TCP with some restrictions on compliance in the case of TCP
+
+Findings
+--------
+
+This lists what has been found during implementation:
+
+-  The same receiver must be able to support both legacy and
+   syslog-protocol syslog messages. Anything else would be a big
+   inconvenience to users and would make deployment much harder. The
+   detection must be done automatically (see below on how easy that is).
+-  **NUL characters inside MSG** cause the message to be truncated at
+   that point. This is probably a major point for many C-based
+   implementations. No measures have yet been taken against this.
+   Modifying the code to "cleanly" support NUL characters is
+   non-trivial, even though rsyslogd already has some byte-counted
+   string library (but this is new and not yet available everywhere).
+-  **character encoding in MSG**: is is problematic to do the right
+   UTF-8 encoding. The reason is that we pick up the MSG from the local
+   domain socket (which got it from the syslog(3) API). The text
+   obtained does not include any encoding information, but it does
+   include non US-ASCII characters. It may also include any other
+   encoding. Other than by guessing based on the provided text, I have
+   no way to find out what it is. In order to make the syslogd do
+   anything useful, I have now simply taken the message as is and
+   stuffed it into the MSG part. Please note that I think this will be a
+   route that other implementors would take, too.
+-  A minimal parser is easy to implement. It took me roughly 2 hours to
+   add it to rsyslogd. This includes the time for restructuring the code
+   to be able to parse both legacy syslog as well as syslog-protocol.
+   The parser has some restrictions, though
+
+   -  STRUCTURED-DATA field is extracted, but not validated. Structured
+      data "[test ]]" is not caught as an error. Nor are any other
+      errors caught. For my needs with this syslogd, that level of
+      structured data processing is probably sufficient. I do not want
+      to parse/validate it in all cases. This is also a performance
+      issue. I think other implementors could have the same view. As
+      such, we should not make validation a requirement.
+   -  MSG is not further processed (e.g. Unicode not being validated)
+   -  the other header fields are also extracted, but no validation is
+      performed right now. At least some validation should be easy to
+      add (not done this because it is a proof-of-concept and scheduled
+      to change).
+
+-  Universal access to all syslog fields (missing ones being emulated)
+   was also quite easy. It took me around another 2 hours to integrate
+   emulation of non-present fields into the code base.
+-  The version at the start of the message makes it easy to detect if we
+   have legacy syslog or syslog-protocol. Do NOT move it to somewhere
+   inside the middle of the message, that would complicate things. It
+   might not be totally fail-safe to just rely on "1 " as the "cookie"
+   for a syslog-protocol. Eventually, it would be good to add some more
+   uniqueness, e.g. "@#1 ".
+-  I have no (easy) way to detect truncation if that happens on the UDP
+   stack. All I see is that I receive e.g. a 4K message. If the message
+   was e.g. 6K, I received two chunks. The first chunk (4K) is correctly
+   detected as a syslog-protocol message, the second (2K) as legacy
+   syslog. I do not see what we could do against this. This questions
+   the usefulness of the TRUNCATE bit. Eventually, I could look at the
+   UDP headers and see that it is a fragment. I have looked at a network
+   sniffer log of the conversation. This looks like two
+   totally-independent messages were sent by the sender stack.
+-  The maximum message size is currently being configured via a
+   preprocessor #define. It can easily be set to 2K or 4K, but more than
+   4K is not possible because of UDP stack limitations. Eventually, this
+   can be worked around, but I have not done this yet.
+-  rsyslogd can accept syslog-protocol formatted messages but is able to
+   relay them in legacy format. I find this a must in real-life
+   deployments. For this, I needed to do some field mapping so that
+   APP-NAME/PROCID are mapped into a TAG.
+-  rsyslogd can also accept legacy syslog message and relay them in
+   syslog-protocol format. For this, I needed to apply some sub-parsing
+   of the TAG, which on most occasions provides correct results. There
+   might be some misinterpretations but I consider these to be mostly
+   non-intrusive.
+-  Messages received from the syslog API (the normal case under \*nix)
+   also do not have APP-NAME and PROCID and I must parse them out of TAG
+   as described directly above. As such, this algorithm is absolutely
+   vital to make things work on \*nix.
+-  I have an issue with messages received via the syslog(3) API (or, to
+   be more precise, via the local domain socket this API writes to):
+   These messages contain a timestamp, but that timestamp does neither
+   have the year nor the high-resolution time. The year is no real
+   issue, I just take the year of the reception of that message. There
+   is a very small window of exposure for messages read from the log
+   immediately after midnight Jan 1st. The message in the domain socket
+   might have been written immediately before midnight in the old year.
+   I think this is acceptable. However, I can not assign a
+   high-precision timestamp, at least it is somewhat off if I take the
+   timestamp from message reception on the local socket. An alternative
+   might be to ignore the timestamp present and instead use that one
+   when the message is pulled from the local socket (I am talking about
+   IPC, not the network - just a reminder...). This is doable, but
+   eventually not advisable. It looks like this needs to be resolved via
+   a configuration option.
+-  rsyslogd already advertised its origin information on application
+   startup (in a syslog-protocol-14 compatible format). It is fairly
+   easy to include that with any message if desired (not currently
+   done).
+-  A big problem I noticed are malformed messages. In -syslog-protocol,
+   we recommend/require to discard malformed messages. However, in
+   practice users would like to see everything that the syslogd
+   receives, even if it is in error. For the first version, I have not
+   included any error handling at all. However, I think I would
+   deliberately ignore any "discard" requirement. My current point of
+   view is that in my code I would eventually flag a message as being
+   invalid and allow the user to filter on this invalidness. So these
+   invalid messages could be redirected into special bins.
+-  The error logging recommendations (those I insisted on;)) are not
+   really practical. My application has its own error logging philosophy
+   and I will not change this to follow a draft.
+-  Relevance of support for leap seconds and senders without knowledge
+   of time is questionable. I have not made any specific provisions in
+   the code nor would I know how to handle that differently. I could,
+   however, pull the local reception timestamp in this case, so it might
+   be useful to have this feature. I do not think any more about this
+   for the initial proof-of-concept. Note it as a potential problem
+   area, especially when logging to databases.
+-  The HOSTNAME field for internally generated messages currently
+   contains the hostname part only, not the FQDN. This can be changed
+   inside the code base, but it requires some thinking so that thinks
+   are kept compatible with legacy syslog. I have not done this for the
+   proof-of-concept, but I think it is not really bad. Maybe an hour or
+   half a day of thinking.
+-  It is possible that I did not receive a TAG with legacy syslog or via
+   the syslog API. In this case, I can not generate the APP-NAME. For
+   consistency, I have used "-" in such cases (just like in PROCID,
+   MSGID and STRUCTURED-DATA).
+-  As an architectural side-effect, syslog-protocol formatted messages
+   can also be transmitted over non-standard syslog/raw tcp. This
+   implementation uses the industry-standard LF termination of tcp
+   syslog records. As such, syslog-protocol messages containing a LF
+   will be broken invalidly. There is nothing that can be done against
+   this without specifying a TCP transport. This issue might be more
+   important than one thinks on first thought. The reason is the wide
+   deployment of syslog/tcp via industry standard.
+
+**Some notes on syslog-transport-udp-06**
+
+-  I did not make any low-level modifications to the UDP code and think
+   I am still basically covered with this I-D.
+-  I deliberately violate section 3.3 insofar as that I do not
+   necessarily accept messages destined to port 514. This feature is
+   user-required and a must. The same applies to the destination port. I
+   am not sure if the "MUST" in section 3.3 was meant that this MUST be
+   an option, but not necessarily be active. The wording should be
+   clarified.
+-  section 3.6: I do not check checksums. See the issue with discarding
+   messages above. The same solution will probably be applied in my
+   code.
+
+ 
+
+Conlusions/Suggestions
+----------------------
+
+These are my personal conclusions and suggestions. Obviously, they must
+be discussed ;)
+
+-  NUL should be disallowed in MSG
+-  As it is not possible to definitely know the character encoding of
+   the application-provided message, MSG should **not** be specified to
+   use UTF-8 exclusively. Instead, it is suggested that any encoding may
+   be used but UTF-8 is preferred. To detect UTF-8, the MSG should start
+   with the UTF-8 byte order mask of "EF BB BF" if it is UTF-8 encoded
+   (see section 155.9 of
+   `http://www.unicode.org/versions/Unicode4.0.0/ch15.pdf <http://www.unicode.org/versions/Unicode4.0.0/ch15.pdf>`_)
+-  Requirements to drop messages should be reconsidered. I guess I would
+   not be the only implementor ignoring them.
+-  Logging requirements should be reconsidered and probably be removed.
+-  It would be advisable to specify "-" for APP-NAME is the name is not
+   known to the sender.
+-  The implications of the current syslog/tcp industry standard on
+   syslog-protocol should be further evaluated and be fully understood
+
+ 
+