diff options
Diffstat (limited to 'source/troubleshooting')
| -rw-r--r-- | source/troubleshooting/debug.rst | 198 | ||||
| -rw-r--r-- | source/troubleshooting/file_not_written.rst | 53 | ||||
| -rw-r--r-- | source/troubleshooting/howtodebug.rst | 57 | ||||
| -rw-r--r-- | source/troubleshooting/index.rst | 19 | ||||
| -rw-r--r-- | source/troubleshooting/selinux.rst | 27 | ||||
| -rw-r--r-- | source/troubleshooting/troubleshoot.rst | 407 |
6 files changed, 761 insertions, 0 deletions
diff --git a/source/troubleshooting/debug.rst b/source/troubleshooting/debug.rst new file mode 100644 index 0000000..f4de297 --- /dev/null +++ b/source/troubleshooting/debug.rst @@ -0,0 +1,198 @@ +Rsyslog Debug Support +===================== + +For harder to find issues, rsyslog has integrated debug support. Usually, +this is not required for finding configuration issues but rather +to hunt for program or plugin bugs. However, there are several +occasions where debug log has proven to be quite helpful in finding +out configuration issues. + +A :doc:`quick guide can be found here<../troubleshooting/howtodebug>`. + +Signals supported +----------------- + +**SIGUSR1** - turns debug messages on and off. Note that for this signal +to work, rsyslogd must be running with debugging enabled, either via the +-d command line switch or the environment options specified below. It is +**not** required that rsyslog was compiled with debugging enabled (but +depending on the settings this may lead to better debug info). + +**Note:** this signal **may go away** in later releases and may be +replaced by something else. + +Environment Variables +--------------------- + +There are two environment variables that set several debug settings: + +- The "RSYSLOG\_DEBUGLOG" (sample: + RSYSLOG\_DEBUGLOG="/path/to/debuglog/debug.log") writes (almost) all debug + message to the specified log file in addition to stdout. Some system + messages (e.g. segfault or abort message) are not written to the file + as we can not capture them. +- Runtime debug support is controlled by "RSYSLOG\_DEBUG". + + The "RSYSLOG\_DEBUG" environment variable contains an option string + with the following options possible (all are case insensitive): + + - **LogFuncFlow** - print out the logical flow of functions + (entering and exiting them) + - **FileTrace** - specifies which files to trace LogFuncFlow. If + **not** set (the default), a LogFuncFlow trace is provided for all + files. Set to limit it to the files specified. FileTrace may be + specified multiple times, one file each (e.g. export + RSYSLOG\_DEBUG="LogFuncFlow FileTrace=vm.c FileTrace=expr.c" + - **PrintFuncDB** - print the content of the debug function database + whenever debug information is printed (e.g. abort case)! + - **PrintAllDebugInfoOnExit** - print all debug information + immediately before rsyslogd exits (currently not implemented!) + - **PrintMutexAction** - print mutex action as it happens. Useful + for finding deadlocks and such. + - **NoLogTimeStamp** - do not prefix log lines with a timestamp + (default is to do that). + - **NoStdOut** - do not emit debug messages to stdout. If + RSYSLOG\_DEBUGLOG is not set, this means no messages will be + displayed at all. + - **Debug** - if present, turns on the debug system and enables + debug output + - **DebugOnDemand** - if present, turns on the debug system but does + not enable debug output itself. You need to send SIGUSR1 to turn + it on when desired. + - **OutputTidToStderr** - if present, makes rsyslog output + information about the thread id (tid) of newly created processes to + stderr. Note that not necessarily all new threads are reported + (depends on the code, e.g. of plugins). This is only available + under Linux. This usually does NOT work when privileges have been + dropped (that's not a bug, but the way it is). + - **help** - display a very short list of commands - hopefully a + life saver if you can't access the documentation... + + Individual options are separated by spaces. + +Why Environment Variables? +~~~~~~~~~~~~~~~~~~~~~~~~~~ + +You may ask why we use environment variables for debug-system parameters +and not the usual rsyslog.conf configuration commands. After all, +environment variables force one to change distro-specific configuration +files, whereas regular configuration directives would fit nicely into +the one central rsyslog.conf. + +Historically environment variables were necessary to initialize so-called +"rtinst" mode. This mode no longer exists, as the OS tools have improved. +Using environment variables still has the benefit that the work right from +initialization of rsyslogd. Most importantly, this is before the rsyslog.conf +is read. + +If that is no issue, rsyslog.conf global statements can be used to enable +debug mode and provide some settings. + +HOWEVER, if you have a too hard time to set debug instructions using the +environment variables, there is a cure, described in the next paragraph. + +Enabling Debug via rsyslog.conf +------------------------------- + +As described in the previous paragraph, enabling debug via rsyslog.conf +may not be perfect for some debugging needs, but basic debug output will +work - and that is what most often is required. There are limited +options available, but these cover the most important use cases. + +Debug processing is done via legacy config statements. There currently +is no plan to move these over to the v6+ config system. Available +settings are + +- $DebugFile <filename> - sets the debug file name +- $DebugLevel <0\|1\|2> - sets the respective debug level, where 0 + means debug off, 1 is debug on demand activated (but debug mode off) + and 2 is full debug mode. + +Note that in theory it is forbidden to specify these parameters more +than once. However, we do not enforce that and if it happens results are +undefined. + +Getting debug information from a running Instance +------------------------------------------------- + +It is possible to obtain debugging information from a running instance, +but this requires some setup. We assume that the instance runs in the +background, so debug output to stdout is not desired. As such, all debug +information needs to go into a log file. + +To create this setup, you need to + +- point the RSYSLOG\_DEBUGLOG environment variable to a file that is + accessible during the while runtime (we strongly suggest a file in + the local file system!) +- set RSYSLOG\_DEBUG at least to "DebugOnDeman NoStdOut" +- make sure these environment variables are set in the correct + (distro-specific) startup script if you do not run rsyslogd + interactively + +These settings enable the capability to react to SIGUSR1. The signal +will toggle debug status when received. So send it once to turn debug +logging on, and send it again to turn debug logging off again. The third +time it will be turned on again ... and so on. + +On a typical system, you can signal rsyslogd as follows: + +:: + + kill -USR1 $(cat /var/run/rsyslogd.pid) + +The debug log will show whether debug +logging has been turned on or off. There is no other indication of the +status. + +Note: running with DebugOnDemand by itself does in practice not have any +performance toll. However, switching debug logging on has a severe +performance toll. Also, debug logging synchronizes much of the code, +removing a lot of concurrency and thus potential race conditions. As +such, the very same running instance may behave very differently with +debug logging turned on vs. off. The on-demand debug log functionality +is considered to be very valuable to analyze hard-to-find bugs that only +manifest after a long runtime. Turning debug logging on a failing +instance may reveal the cause of the failure. However, depending on the +failure, debug logging may not even be successfully turned on. Also +note that with this rsyslog version we cannot obtain any debug +information on events that happened *before* debug logging was turned +on. + + +Interpreting the Logs +--------------------- + +Debug logs are primarily meant for rsyslog developers. But they may +still provide valuable information to users. Just be warned that logs +sometimes contains information the looks like an error, but actually is +none. We put a lot of extra information into the logs, and there are +some cases where it is OK for an error to happen, we just wanted to +record it inside the log. The code handles many cases automatically. So, +in short, the log may not make sense to you, but it (hopefully) makes +sense to a developer. Note that we developers often need many lines of +the log file, it is relatively rare that a problem can be diagnosed by +looking at just a couple of (hundred) log records. + +Security Risks +-------------- + +The debug log will reveal potentially sensible information, including +user accounts and passwords, to anyone able to read the log file. As +such, it is recommended to properly guard access to the log file. Also, +an instance running with debug log enabled runs much slower than one +without. An attacker may use this to place carry out a denial-of-service +attack or try to hide some information from the log file. As such, it is +suggested to enable DebugOnDemand mode only for a reason. Note that when +no debug mode is enabled, SIGUSR1 is completely ignored. + +When running in any of the debug modes (including on demand mode), an +interactive instance of rsyslogd can be aborted by pressing ctl-c. + +See Also +-------- + +- `How to use debug on + demand <http://www.rsyslog.com/how-to-use-debug-on-demand/>`_ +- :doc:`Quick debug guide<../troubleshooting/howtodebug>` + diff --git a/source/troubleshooting/file_not_written.rst b/source/troubleshooting/file_not_written.rst new file mode 100644 index 0000000..4ed915d --- /dev/null +++ b/source/troubleshooting/file_not_written.rst @@ -0,0 +1,53 @@ +Output File is not Being Written +================================ + +Note: current rsyslog versions have somewhat limited error reporting +inside omfile. If a problem persists, you may want to generate a +rsyslog debug log, which often can help you pinpoint the actual root +cause of the problem more quickly. + +To learn more about the current state of error reporting, follow +our `bug tracker <https://github.com/rsyslog/rsyslog/issues/548>`_ +for this issue. + +The following subsections list frequent causes for file writing +problems. You can quickly check this without the need to create a +debug log. + +SELinux +------- +This often stems back to **selinux** permission errors, especially +if files outside of the ``/var/log`` directory shall be written +to. + +Follow the :doc:`SELinux troubleshooting guide <selinux>` +to check for this condition. + +Max Number of Open Files +------------------------ +This can also be caused by a too low limit on number of open +file handles, especially when dynafiles are being written. + +Note that some versions of systemd limit the process +to 1024 files by default. The current +set limit can be validated by doing:: + + cat /proc/<pid>/limits + +and the currently open number of files can be obtained by doing:: + + ls /proc/<pid>/fd | wc -l + +Also make sure the system-wide max open files is appropriate using:: + + sysctl fs.file-max + +Some versions of systemd completely ignore +``/etc/security/limits*``. To change limits for a service in systemd, edit +``/usr/lib/systemd/system/rsyslog.service`` and under ``[Service]`` add: +``LimitNOFILE=<new value>``. + +Then run:: + + systemctl daemon-reload + systemctl restart rsyslog diff --git a/source/troubleshooting/howtodebug.rst b/source/troubleshooting/howtodebug.rst new file mode 100644 index 0000000..5d770ff --- /dev/null +++ b/source/troubleshooting/howtodebug.rst @@ -0,0 +1,57 @@ +How to create a debug log +========================= + +Creating a debug log is actually quite simple. You just have to add a +few lines to the configuration file. + +Regular debug output +-------------------- + +Add the following right at the beginning of the rsyslog.conf file. This +will ensure that debug support is the first thing to enable when the +rsyslog service is started: + +:: + + $DebugFile /var/log/rsyslog.debug + $DebugLevel 2 + +The actual file path and name may be changed if required. + +Having set the above, when rsyslog is restarted it will produce a continuous +debug file. + +Debug on Demand +--------------- + +For having rsyslog be ready to create a debug log (aka Debug on Demand), the +settings are a little different. + +:: + + $DebugFile /var/log/rsyslog.debug + $DebugLevel 1 + +Now, rsyslog will not create a debug log on restart, but wait for a USR signal +to the pid. When sent, the debug output will be triggered. When sent again, +debug output will be stopped. + +:: + + kill -USR1 `cat /var/run/rsyslogd.pid` + +Notes +----- + +- Having debug output enabled, the debug file will grow very quickly. Make sure + to not have it enabled permanently. The file will eventually fill up the disk. +- Debug mode is not to be used in a productive environment as a permanent setting. + It will affect the processing and performance. + +See Also +-------- + +- :doc:`Troubleshooting <../troubleshooting/troubleshoot>` doc page. +- :doc:`Debug Support<../troubleshooting/debug>` doc page with more detailed + information. + diff --git a/source/troubleshooting/index.rst b/source/troubleshooting/index.rst new file mode 100644 index 0000000..4fc3c95 --- /dev/null +++ b/source/troubleshooting/index.rst @@ -0,0 +1,19 @@ +Troubleshooting +=============== + +Typical Problems +---------------- +.. toctree:: + :maxdepth: 1 + + file_not_written + selinux + +General Procedure +----------------- +.. toctree:: + :maxdepth: 2 + + debug + troubleshoot + howtodebug diff --git a/source/troubleshooting/selinux.rst b/source/troubleshooting/selinux.rst new file mode 100644 index 0000000..5646067 --- /dev/null +++ b/source/troubleshooting/selinux.rst @@ -0,0 +1,27 @@ +Troubleshooting SELinux-Related Issues +====================================== + +SELinux by its very nature can block many features of rsyslog (or any +other process, of course), even when run under root. Actually, this is +what it is supposed to do, so there is nothing bad about it. + +If you suspect that some issues stems back to SELinux configuration, +do the following: + +* *temporarily* disable SELinux +* restart rsyslog +* retry the operation + +If it now succeeds, you know that you have a SELinux policy issue. +The solution here is **not** to keep SELinux disabled. Instead do: + +* reenable SELinux (set back to previous state, whatever that was) +* add proper SELinux policies for what you want to do with rsyslog + +With SELinux running, restart rsyslog +$ sudo audit2allow -a +audit2allow will read the audit.log and list any SELinux infractions, namely the rsyslog infractions +$ sudo audit2allow -a -M <FRIENDLY_NAME_OF_MODULE>.pp +audit2allow will create a module allowing all previous infractions to have access +$ sudo semodule -i <FRIENDLY_NAME_OF_MODULE>.pp +Your module is loaded! Restart rsyslog and continue to audit until no more infractions are detected and rsyslog has proper access. Additionally, you can save these modules and install them on future machines where rsyslog will need the same access. diff --git a/source/troubleshooting/troubleshoot.rst b/source/troubleshooting/troubleshoot.rst new file mode 100644 index 0000000..9881864 --- /dev/null +++ b/source/troubleshooting/troubleshoot.rst @@ -0,0 +1,407 @@ +troubleshooting problems +======================== + +**Having trouble with** `rsyslog <https://www.rsyslog.com>`_? This page +provides some tips on where to look for help and what to do if you need +to ask for assistance. This page is continuously being expanded. + +Useful troubleshooting resources are: + +- The `rsyslog documentation <https://www.rsyslog.com/doc>`_ - note that + the online version always covers the most recent development version. + However, there is a version-specific doc set in each tarball. If you + installed rsyslog from a package, there usually is a rsyslog-doc + package, that often needs to be installed separately. + +- Check the `rsyslog github issue tracker <https://github.com/rsyslog/rsyslog/issues>`_ and + `the bugzilla <http://bugzilla.adiscon.com>`_ to see if your + problem is a known (and even fixed ;)) bug. + **Note:** the preferred way to create new bugs is via github. + The bugzilla does no longer accept new bugs. It is just kept + to work on old ones and as a reference source for ChangeLog entries. + +Malformed Messages and Message Properties +----------------------------------------- + +A common trouble source are `ill-formed syslog +messages <syslog_parsing.html>`_, which lead to to all sorts of +interesting problems, including malformed hostnames and dates. Read the +quoted guide to find relief. A common symptom is that the %HOSTNAME% +property is used for generating dynafile names, but some gibberish +shows up. This is caused by the malformed syslog messages, so be sure to +read the :doc:`guide on syslog parsing <../whitepapers/syslog_parsing>` +if you face that problem. Just +let me add that the common work-around is to use %FROMHOST% or +%FROMHOST-IP% instead. These do not take the hostname from the message, +but rather use the host that sent the message (taken from the socket +layer). Of course, this does not work over NAT or relay chains, where +the only cure is to make sure senders emit well-formed messages. + +Configuration Problems +---------------------- + +Rsyslog has support for +configuration checking. It offers a special command line switch (-N<*value*>) +that puts it into "config verification mode". In that mode, it interprets +and checks the configuration file, but does not startup. This mode can be +used in parallel to a running instance of rsyslogd. + +Please note that many distros have (unfortunately) begun to split the rsyslog +config into many small config snippets (a.k.a input files), usually in `/etc/rsyslog.d`. From +rsyslog's point of view, only a single config file exists and these snippets +are logically combined into that single config. For that reason, config checking +does usually **not** work on snippets. Because the interdependencies are missing. +For example, binding a ruleset to a module that is not possible if the ruleset is +loaded outside of the snippet. + +**As a general guideline, never do config checks on config snippets. Always use +the main configuration file as a starting point** (usually /etc/rsyslog.conf`). + +The *value* given after -N is a set of binary values. Currently, there only is + +======= ====================================== +value meaning +1 turn on config checking +2 permit checking of include files +======= ====================================== + +Where 2 automatically turns on config checking mode, if not given. In that +sense ``-N2`` and ``-N3`` are equivalent. + +Values other than given in the table above are **not** supported and may lead +to unpredictable results. + +When set to check include files, some conditions are relaxed. For example, +rsyslog usually requires that at least one action is defined somewhere in +the configuration. For obvious reasons, it would not make much sense to run +an instance without any action. However, when an include file is checked, +it may happen that it contains no actions as all. As such, the requirement +to include one action has been lifted in include file checking. + +To check a full rsyslog configuration, run rsyslog interactively as follows: + +:: + + $ /path/to/rsyslogd -f/path/to/config-file -N1 + +You should also specify other options you usually give. +Any problems experienced are reported to stderr [aka +"your screen" (if not redirected)]. + +If you would like to check just an include file, instead use: + +:: + + $ /path/to/rsyslogd -f/path/to/config-file -N3 + +Sometimes problems are rooted in config include files, and especially the +order in which they are processed. To troubleshoot these kinds of issues, you +can use the rsyslogd `-o` option: it permits to specify a file that shall +receive a full copy of rsyslog's current configuration **as rsyslog sees it**. +This means all include file content is directly inside that file at +exactly the spot where rsyslog sees it. The output file is almost a +verbatim copy of the original full rsyslog config. For troubleshooting +purposes it additionally contains comment lines that indicate where +content from specific include files begins and ends. The include file +is correctly named in these comments. + +This option can be used together with `-N`. Again, it is best to run +rsyslog interactively. Do as such:: + + $ /path/to/rsyslogd -f/path/to/config-file -N3 -o /path/to/full-conf-file + + +Checking Connection Problems +---------------------------- + +If a client cannot connect via the network to the rsyslog server, you +can do a connection check via netcat. That will verify if the sender +is able to deliver to an application running on the receiver. Netcat +is a very simple receiver, so we can be sure that no netcat problem +will interfere with this test. + +With netcat, you can test UDP and TCP syslog connections, but not TLS. + +To do this test, you need to + +* on the client + + - stop the syslog sender process, if possible. If the sender is + rsyslog, you can use the same procedure described below for the + server. + +* on the rsyslog server + + - stop and/or disable rsyslog + On systemd systems (newer distro versions), systemd might + automatically restart rsyslog when data is written to the system + log socket. To be sure, we recommend to disable the service on + those systems. This sequence should work: + $ systemctl disable rsyslog.service + $ systemctl stop rsyslog.service + + - open a terminal session, and start a netcat listener **on the same + listening port** that you have configured inside rsyslog. Note that + if you use a privileged port, you need to execute nc as root. + We assume port 13515 is used for rsyslog, so do this: + $ nc -k -l <ip-of-server> 13515 # [FOR TCP] OR sudo nc ... + $ nc -u -l <ip-of-server> 13515 # [FOR UDP] OR sudo nc ... + +* on the syslog client + + - send a test message via netcat: + $ echo "test message 1" | nc <ip-of-server> 13515 # [FOR TCP] + $ echo "test message 1" | nc <ip-of-server> 13515 # [FOR UDP] + +* on the server + + - check if you received the test message. Note that you might also + have received additional messages if the original sender process + was not stopped. If you see garbage, most probably some sender + tries to send via TLS. + - you can stop nc by <ctl>-c + +If you did not see the test message arrive at the central server, +the problem is most probably rooted in the network configuration +or other parts of the system configuration. Things to check are +- firewall settings + +- for UDP: does the sender have a route back to the original sender? + This is often required by modern systems to prevent spoofing; if the + sender cannot be reached, UDP messages are discarded AFTER they have + been received by the OS (an app like netcat or rsyslog will never + see them) + +- if that doesn't help, use a network monitor (or tcpdump, Wireshark, ...) + to verify that the network packet at least reaches the system. + +If you saw the test message arrive at the central server, the problem +most probably is related to the rsyslog configuration or the system +configuration that affects rsyslog (SELinux, AppArmor, ...). + +A good next test is to run rsyslog interactively, just like you did +with netcat: + +* on the server + - make sure the rsyslog service is still stopped + + - run + $ sudo /usr/sbin/rsyslogd -n + +* on the client + + - send a test message + +* on the server + - check if the message arrived + + - terminate rsyslog by pressing <ctl>-c + +If the test message arrived, you definitely have a problem with the +system configuration, most probably in SELinux, AppArmor or a similar +subsystem. Note that your interactive security context is quite different +from the rsyslog system service context. + +If the test message did not arrive, it is time to generate a debug +log to see exactly what rsyslog does. A full description is in this file +a bit down below, but in essence you need to do + +* on the server + - make sure the rsyslog service is still stopped + - run + + $ sudo /usr/sbin/rsyslogd -nd 2> rsyslog-debug.log + +* on the client + - send a test message + +* on the server + - stop rsyslog by pressing <ctl>- + - review debug log + + +Asking for Help +--------------- + +If you can't find the answer yourself, you should look at these places +for community help. + +- The `rsyslog mailing + list <http://lists.adiscon.net/mailman/listinfo/rsyslog>`_. This is a + low-volume list which occasional gets traffic spikes. The mailing + list is probably a good place for complex questions. + This is the preferred method of obtaining support. +- The `rsyslog forum <http://kb.monitorware.com/rsyslog-f40.html>`_. + +Debug Log +--------- + +If you ask for help, there are chances that we need to ask for an +rsyslog debug log. The debug log is a detailed report of what rsyslog +does during processing. As such, it may even be useful for your very own +troubleshooting. People have seen things inside their debug log that +enabled them to find problems they did not see before. So having a look +at the debug log, even before asking for help, may be useful. + +Note that the debug log contains most of those things we consider +useful. This is a lot of information, but may still be too few. So it +sometimes may happen that you will be asked to run a specific version +which has additional debug output. Also, we revise from time to time +what is worth putting into the standard debug log. As such, log content +may change from version to version. We do not guarantee any specific +debug log contents, so do not rely on that. The amount of debug logging +can also be controlled via some environment options. Please see +`debugging support <debug.html>`_ for further details. + +In general, it is advisable to run rsyslogd in the foreground to obtain +the log. To do so, make sure you know which options are usually used +when you start rsyslogd as a background daemon. Let's assume "-c5" is +the only option used. Then, do the following: + +- make sure rsyslogd as a daemon is stopped (verify with ps -ef\|grep + rsyslogd) +- make sure you have a console session with root permissions +- run rsyslogd interactively: ```/sbin/rsyslogd ..your options.. -dn > + logfile``` + where "your options" is what you usually use. /sbin/rsyslogd is the + full path to the rsyslogd binary (location different depending on + distro). In our case, the command would be + ```/sbin/rsyslogd -c5 -dn > logfile``` +- press ctrl-C when you have sufficient data (e.g. a device logged a + record) + **NOTE: rsyslogd will NOT stop automatically - you need to ctrl-c out + of it!** +- Once you have done all that, you can review logfile. It contains the + debug output. +- When you are done, make sure you re-enable (and start) the background + daemon! + +If you need to submit the logfile, you may want to check if it contains +any passwords or other sensitive data. If it does, you can change it to +some **consistent** meaningless value. **Do not delete the lines**, as +this renders the debug log unusable (and makes Rainer quite angry for +wasted time, aka significantly reduces the chance he will remain +motivated to look at your problem ;)). For the same reason, make sure +whatever you change is change consistently. Really! + +Debug log file can get quite large. Before submitting them, it is a good +idea to zip them. Rainer has handled files of around 1 to 2 GB. If +your's is larger ask before submitting. Often, it is sufficient to +submit the first 2,000 lines of the log file and around another 1,000 +around the area where you see a problem. Also, ask you can submit a file +via private mail. Private mail is usually a good way to go for large +files or files with sensitive content. However, do NOT send anything +sensitive that you do not want the outside to be known. While Rainer so +far made effort no to leak any sensitive information, there is no +guarantee that doesn't happen. If you need a guarantee, you are probably +a candidate for a `commercial support +contract <http://www.rsyslog.com/professional-services/>`_. Free support comes without any +guarantees, include no guarantee on confidentiality [aka "we don't want +to be sued for work were are not even paid for ;)]. **So if you submit +debug logs, do so at your sole risk**. By submitting them, you accept +this policy. + +Segmentation Faults +------------------- + +Rsyslog has a very rapid development process, complex capabilities and +now gradually gets more and more exposure. While we are happy about +this, it also has some bad effects: some deployment scenarios have +probably never been tested and it may be impossible to test them for the +development team because of resources needed. So while we try to avoid +this, you may see a serious problem during deployments in demanding, +non-standard, environments (hopefully not with a stable version, but +chances are good you'll run into troubles with the development +versions). + +In order to aid the debugging process, it is useful to have debug symbols +on the system. If you build rsyslog yourself, make sure that the ``-g`` +option is included in CFLAGS. If you use packages, the debug symbols come +in their own package. **It is highly recommended to install that package +as it provides tremendous extra benefit.** To do so, do: + +:: + + yum install rsyslog-debuginfo + +Obviously, this is for RPM-based systems, but it's essentially the same +with other packaging systems, just use the native commands. Note that +the package may be named slightly different, but it should always be +fairly easy to locate. + + +Active support from the user base is very important to help us track +down those things. Most often, serious problems are the result of some +memory misaddressing. During development, we routinely use valgrind, a +very well and capable memory debugger. This helps us to create pretty +clean code. But valgrind can not detect everything, most importantly not +code paths that are never executed. So of most use for us is +information about aborts and abort locations. + +Unfortunately, faults rooted in addressing errors typically show up only +later, so the actual abort location is in an unrelated spot. To help +track down the original spot, `libc later than 5.4.23 offers +support <http://www.gnu.org/software/hello/manual/libc/Heap-Consistency-Checking.html>`_ +for finding, and possible temporary relief from it, by means of the +MALLOC\_CHECK\_ environment variable. Setting it to 2 is a useful +troubleshooting aid for us. It will make the program abort as soon as +the check routines detect anything suspicious (unfortunately, this may +still not be the root cause, but hopefully closer to it). Setting it to +0 may even make some problems disappear (but it will NOT fix them!). +With functionality comes cost, and so exporting MALLOC\_CHECK\_ without +need comes at a performance penalty. However, we strongly recommend +adding this instrumentation to your test environment should you see any +serious problems. Chances are good it will help us interpret a dump +better, and thus be able to quicker craft a fix. + +In order to get useful information, we need some backtrace of the abort. +First, you need to make sure that a core file is created. Under Fedora, +for example, that means you need to have an "ulimit -c unlimited" in +place. + +Now let's assume you got a core file (e.g. in /core.1234). So what to do +next? Sending a core file to us is most often pointless - we need to +have the exact same system configuration in order to interpret it +correctly. Obviously, chances are extremely slim for this to be. So we +would appreciate if you could extract the most important information. +This is done as follows: + +:: + + $ gdb /path/to/rsyslogd + $ core /core.1234 + $ info thread + $ thread apply all bt full + $ q # quits gdb + +The same method can be applied to a running rsyslog process that suffers +from a lock condition. E.g. if you experience that rsyslog is no longer +forwarding log messages, but this cannot be reproduced in our lab. Using +gdb to review the state of the active threads may be an option to see +which thread is causing the problem (e.g. by locking itself or being in a +wait state). + +Again, basically the same steps can be applied. But, instead of using a +core file, we will require the currently used PID. So make sure to acquire +the PID before executing gdb. + +:: + + $ gdb /path/to/rsyslogd + $ attach PID # numerical value + $ info thread + $ thread apply all bt full + $ q # quits gdb + +Then please send all information that gdb spit out to the development +team. It is best to first ask on the forum or mailing list on how to do +that. The developers will keep in contact with you and, I fear, will +probably ask for other things as well ;) + + +Note that we strive for highest reliability of the engine even in +unusual deployment scenarios. Unfortunately, this is hard to achieve, +especially with limited resources. So we are depending on cooperation +from users. This is your chance to make a big contribution to the +project without the need to program or do anything else except get a +problem solved. |