summaryrefslogtreecommitdiffstats
path: root/README_FILES/DEBUG_README
diff options
context:
space:
mode:
Diffstat (limited to '')
-rw-r--r--README_FILES/DEBUG_README402
1 files changed, 402 insertions, 0 deletions
diff --git a/README_FILES/DEBUG_README b/README_FILES/DEBUG_README
new file mode 100644
index 0000000..a277d96
--- /dev/null
+++ b/README_FILES/DEBUG_README
@@ -0,0 +1,402 @@
+PPoossttffiixx DDeebbuuggggiinngg HHoowwttoo
+
+-------------------------------------------------------------------------------
+
+PPuurrppoossee ooff tthhiiss ddooccuummeenntt
+
+This document describes how to debug parts of the Postfix mail system when
+things do not work according to expectation. The methods vary from making
+Postfix log a lot of detail, to running some daemon processes under control of
+a call tracer or debugger.
+
+The text assumes that the Postfix main.cf and master.cf configuration files are
+stored in directory /etc/postfix. You can use the command "ppoossttccoonnff
+ccoonnffiigg__ddiirreeccttoorryy" to find out the actual location of this directory on your
+machine.
+
+Listed in order of increasing invasiveness, the debugging techniques are as
+follows:
+
+ * Look for obvious signs of trouble
+ * Debugging Postfix from inside
+ * Try turning off chroot operation in master.cf
+ * Verbose logging for specific SMTP connections
+ * Record the SMTP session with a network sniffer
+ * Making Postfix daemon programs more verbose
+ * Manually tracing a Postfix daemon process
+ * Automatically tracing a Postfix daemon process
+ * Running daemon programs with the interactive ddd debugger
+ * Running daemon programs with the interactive gdb debugger
+ * Running daemon programs under a non-interactive debugger
+ * Unreasonable behavior
+ * Reporting problems to postfix-users@postfix.org
+
+LLooookk ffoorr oobbvviioouuss ssiiggnnss ooff ttrroouubbllee
+
+Postfix logs all failed and successful deliveries to a logfile.
+
+ * When Postfix uses syslog logging (the default), the file is usually called
+ /var/log/maillog, /var/log/mail, or something similar; the exact pathname
+ is configured in a file called /etc/syslog.conf, /etc/rsyslog.conf, or
+ something similar.
+
+ * When Postfix uses its own logging system (see MAILLOG_README), the location
+ of the logfile is configured with the Postfix maillog_file parameter.
+
+When Postfix does not receive or deliver mail, the first order of business is
+to look for errors that prevent Postfix from working properly:
+
+ % eeggrreepp ''((wwaarrnniinngg||eerrrroorr||ffaattaall||ppaanniicc))::'' //ssoommee//lloogg//ffiillee || mmoorree
+
+Note: the most important message is near the BEGINNING of the output. Error
+messages that come later are less useful.
+
+The nature of each problem is indicated as follows:
+
+ * "ppaanniicc" indicates a problem in the software itself that only a programmer
+ can fix. Postfix cannot proceed until this is fixed.
+
+ * "ffaattaall" is the result of missing files, incorrect permissions, incorrect
+ configuration file settings that you can fix. Postfix cannot proceed until
+ this is fixed.
+
+ * "eerrrroorr" reports an error condition. For safety reasons, a Postfix process
+ will terminate when more than 13 of these happen.
+
+ * "wwaarrnniinngg" indicates a non-fatal error. These are problems that you may not
+ be able to fix (such as a broken DNS server elsewhere on the network) but
+ may also indicate local configuration errors that could become a problem
+ later.
+
+DDeebbuuggggiinngg PPoossttffiixx ffrroomm iinnssiiddee
+
+Postfix version 2.1 and later can produce mail delivery reports for debugging
+purposes. These reports not only show sender/recipient addresses after address
+rewriting and alias expansion or forwarding, they also show information about
+delivery to mailbox, delivery to non-Postfix command, responses from remote
+SMTP servers, and so on.
+
+Postfix can produce two types of mail delivery reports for debugging:
+
+ * What-if: report what would happen, but do not actually deliver mail. This
+ mode of operation is requested with:
+
+ % //uussrr//ssbbiinn//sseennddmmaaiill --bbvv aaddddrreessss......
+ Mail Delivery Status Report will be mailed to <your login name>.
+
+ * What happened: deliver mail and report successes and/or failures, including
+ replies from remote SMTP servers. This mode of operation is requested with:
+
+ % //uussrr//ssbbiinn//sseennddmmaaiill --vv aaddddrreessss......
+ Mail Delivery Status Report will be mailed to <your login name>.
+
+These reports contain information that is generated by Postfix delivery agents.
+Since these run as daemon processes that cannot interact with users directly,
+the result is sent as mail to the sender of the test message. The format of
+these reports is practically identical to that of ordinary non-delivery
+notifications.
+
+For a detailed example of a mail delivery status report, see the debugging
+section at the end of the ADDRESS_REWRITING_README document.
+
+TTrryy ttuurrnniinngg ooffff cchhrroooott ooppeerraattiioonn iinn mmaasstteerr..ccff
+
+A common mistake is to turn on chroot operation in the master.cf file without
+going through all the necessary steps to set up a chroot environment. This
+causes Postfix daemon processes to fail due to all kinds of missing files.
+
+The example below shows an SMTP server that is configured with chroot turned
+off:
+
+ /etc/postfix/master.cf:
+ # =============================================================
+ # service type private unpriv cchhrroooott wakeup maxproc command
+ # (yes) (yes) ((yyeess)) (never) (100)
+ # =============================================================
+ smtp inet n - nn - - smtpd
+
+Inspect master.cf for any processes that have chroot operation not turned off.
+If you find any, save a copy of the master.cf file, and edit the entries in
+question. After executing the command "ppoossttffiixx rreellooaadd", see if the problem has
+gone away.
+
+If turning off chrooted operation made the problem go away, then
+congratulations. Leaving Postfix running in this way is adequate for most
+sites. If you prefer chrooted operation, see the Postfix
+BASIC_CONFIGURATION_README file for information about how to prepare Postfix
+for chrooted operation.
+
+VVeerrbboossee llooggggiinngg ffoorr ssppeecciiffiicc SSMMTTPP ccoonnnneeccttiioonnss
+
+In /etc/postfix/main.cf, list the remote site name or address in the
+debug_peer_list parameter. For example, in order to make the software log a lot
+of information to the syslog daemon for connections from or to the loopback
+interface:
+
+ /etc/postfix/main.cf:
+ debug_peer_list = 127.0.0.1
+
+You can specify one or more hosts, domains, addresses or net/masks. To make the
+change effective immediately, execute the command "ppoossttffiixx rreellooaadd".
+
+RReeccoorrdd tthhee SSMMTTPP sseessssiioonn wwiitthh aa nneettwwoorrkk ssnniiffffeerr
+
+This example uses ttccppdduummpp. In order to record a conversation you need to
+specify a large enough buffer with the "--ss" option or else you will miss some
+or all of the packet payload.
+
+ # ttccppdduummpp --ww //ffiillee//nnaammee --ss 00 hhoosstt eexxaammppllee..ccoomm aanndd ppoorrtt 2255
+
+Older tcpdump versions don't support "--ss 00"; in that case, use "--ss 22000000"
+instead.
+
+Run this for a while, stop with Ctrl-C when done. To view the data use a binary
+viewer, eetthheerreeaall, or good old lleessss.
+
+MMaakkiinngg PPoossttffiixx ddaaeemmoonn pprrooggrraammss mmoorree vveerrbboossee
+
+Append one or more "--vv" options to selected daemon definitions in /etc/postfix/
+master.cf and type "ppoossttffiixx rreellooaadd". This will cause a lot of activity to be
+logged to the syslog daemon. For example, to make the Postfix SMTP server
+process more verbose:
+
+ /etc/postfix/master.cf:
+ smtp inet n - n - - smtpd -v
+
+To diagnose problems with address rewriting specify a "--vv" option for the
+cleanup(8) and/or trivial-rewrite(8) daemon, and to diagnose problems with mail
+delivery specify a "--vv" option for the qmgr(8) or oqmgr(8) queue manager, or
+for the lmtp(8), local(8), pipe(8), smtp(8), or virtual(8) delivery agent.
+
+MMaannuuaallllyy ttrraacciinngg aa PPoossttffiixx ddaaeemmoonn pprroocceessss
+
+Many systems allow you to inspect a running process with a system call tracer.
+For example:
+
+ # ttrraaccee --pp pprroocceessss--iidd (SunOS 4)
+ # ssttrraaccee --pp pprroocceessss--iidd (Linux and many others)
+ # ttrruussss --pp pprroocceessss--iidd (Solaris, FreeBSD)
+ # kkttrraaccee --pp pprroocceessss--iidd (generic 4.4BSD)
+
+Even more informative are traces of system library calls. Examples:
+
+ # llttrraaccee --pp pprroocceessss--iidd (Linux, also ported to FreeBSD and BSD/OS)
+ # ssoottrruussss --pp pprroocceessss--iidd (Solaris)
+
+See your system documentation for details.
+
+Tracing a running process can give valuable information about what a process is
+attempting to do. This is as much information as you can get without running an
+interactive debugger program, as described in a later section.
+
+AAuuttoommaattiiccaallllyy ttrraacciinngg aa PPoossttffiixx ddaaeemmoonn pprroocceessss
+
+Postfix can attach a call tracer whenever a daemon process starts. Call tracers
+come in several kinds.
+
+ 1. System call tracers such as ttrraaccee, ttrruussss, ssttrraaccee, or kkttrraaccee. These show the
+ communication between the process and the kernel.
+
+ 2. Library call tracers such as ssoottrruussss and llttrraaccee. These show calls of
+ library routines, and give a better idea of what is going on within the
+ process.
+
+Append a --DD option to the suspect command in /etc/postfix/master.cf, for
+example:
+
+ /etc/postfix/master.cf:
+ smtp inet n - n - - smtpd -D
+
+Edit the debugger_command definition in /etc/postfix/main.cf so that it invokes
+the call tracer of your choice, for example:
+
+ /etc/postfix/main.cf:
+ debugger_command =
+ PATH=/bin:/usr/bin:/usr/local/bin;
+ (truss -p $process_id 2>&1 | logger -p mail.info) & sleep 5
+
+Type "ppoossttffiixx rreellooaadd" and watch the logfile.
+
+RRuunnnniinngg ddaaeemmoonn pprrooggrraammss wwiitthh tthhee iinntteerraaccttiivvee dddddd ddeebbuuggggeerr
+
+If you have X Windows installed on the Postfix machine, then an interactive
+debugger such as dddddd can be convenient.
+
+Edit the debugger_command definition in /etc/postfix/main.cf so that it invokes
+dddddd:
+
+ /etc/postfix/main.cf:
+ debugger_command =
+ PATH=/bin:/usr/bin:/usr/local/bin:/usr/X11R6/bin
+ ddd $daemon_directory/$process_name $process_id & sleep 5
+
+Be sure that ggddbb is in the command search path, and export XXAAUUTTHHOORRIITTYY so that X
+access control works, for example:
+
+ % sseetteennvv XXAAUUTTHHOORRIITTYY ~~//..XXaauutthhoorriittyy (csh syntax)
+ $ eexxppoorrtt XXAAUUTTHHOORRIITTYY==$$HHOOMMEE//..XXaauutthhoorriittyy (sh syntax)
+
+Append a --DD option to the suspect daemon definition in /etc/postfix/master.cf,
+for example:
+
+ /etc/postfix/master.cf:
+ smtp inet n - n - - smtpd -D
+
+Stop and start the Postfix system. This is necessary so that Postfix runs with
+the proper XXAAUUTTHHOORRIITTYY and DDIISSPPLLAAYY settings.
+
+Whenever the suspect daemon process is started, a debugger window pops up and
+you can watch in detail what happens.
+
+RRuunnnniinngg ddaaeemmoonn pprrooggrraammss wwiitthh tthhee iinntteerraaccttiivvee ggddbb ddeebbuuggggeerr
+
+If you have the screen command installed on the Postfix machine, then you can
+run an interactive debugger such as ggddbb as follows.
+
+Edit the debugger_command definition in /etc/postfix/main.cf so that it runs
+ggddbb inside a detached ssccrreeeenn session:
+
+ /etc/postfix/main.cf:
+ debugger_command =
+ PATH=/bin:/usr/bin:/sbin:/usr/sbin; export PATH; HOME=/root;
+ export HOME; screen -e^tt -dmS $process_name gdb
+ $daemon_directory/$process_name $process_id & sleep 2
+
+Be sure that ggddbb is in the command search path.
+
+Append a --DD option to the suspect daemon definition in /etc/postfix/master.cf,
+for example:
+
+ /etc/postfix/master.cf:
+ smtp inet n - n - - smtpd -D
+
+Execute the command "ppoossttffiixx rreellooaadd" and wait until a daemon process is started
+(you can see this in the maillog file).
+
+Then attach to the screen, and debug away:
+
+ # HOME=/root screen -r
+ gdb) continue
+ gdb) where
+
+RRuunnnniinngg ddaaeemmoonn pprrooggrraammss uunnddeerr aa nnoonn--iinntteerraaccttiivvee ddeebbuuggggeerr
+
+If you do not have X Windows installed on the Postfix machine, or if you are
+not familiar with interactive debuggers, then you can try to run ggddbb in non-
+interactive mode, and have it print a stack trace when the process crashes.
+
+Edit the debugger_command definition in /etc/postfix/main.cf so that it invokes
+the ggddbb debugger:
+
+ /etc/postfix/main.cf:
+ debugger_command =
+ PATH=/bin:/usr/bin:/usr/local/bin; export PATH; (echo cont; echo
+ where; sleep 8640000) | gdb $daemon_directory/$process_name
+ $process_id 2>&1
+ >$config_directory/$process_name.$process_id.log & sleep 5
+
+Append a --DD option to the suspect daemon in /etc/postfix/master.cf, for
+example:
+
+ /etc/postfix/master.cf:
+ smtp inet n - n - - smtpd -D
+
+Type "ppoossttffiixx rreellooaadd" to make the configuration changes effective.
+
+Whenever a suspect daemon process is started, an output file is created, named
+after the daemon and process ID (for example, smtpd.12345.log). When the
+process crashes, a stack trace (with output from the "wwhheerree" command) is
+written to its logfile.
+
+UUnnrreeaassoonnaabbllee bbeehhaavviioorr
+
+Sometimes the behavior exhibited by Postfix just does not match the source
+code. Why can a program deviate from the instructions given by its author?
+There are two possibilities.
+
+ * The compiler has erred. This rarely happens.
+
+ * The hardware has erred. Does the machine have ECC memory?
+
+In both cases, the program being executed is not the program that was supposed
+to be executed, so anything could happen.
+
+There is a third possibility:
+
+ * Bugs in system software (kernel or libraries).
+
+Hardware-related failures usually do not reproduce in exactly the same way
+after power cycling and rebooting the system. There's little Postfix can do
+about bad hardware. Be sure to use hardware that at the very least can detect
+memory errors. Otherwise, Postfix will just be waiting to be hit by a bit
+error. Critical systems deserve real hardware.
+
+When a compiler makes an error, the problem can be reproduced whenever the
+resulting program is run. Compiler errors are most likely to happen in the code
+optimizer. If a problem is reproducible across power cycles and system reboots,
+it can be worthwhile to rebuild Postfix with optimization disabled, and to see
+if optimization makes a difference.
+
+In order to compile Postfix with optimizations turned off:
+
+ % mmaakkee ttiiddyy
+ % mmaakkee mmaakkeeffiilleess OOPPTT==
+
+This produces a set of Makefiles that do not request compiler optimization.
+
+Once the makefiles are set up, build the software:
+
+ % mmaakkee
+ % ssuu
+ Password:
+ # mmaakkee iinnssttaallll
+
+If the problem goes away, then it is time to ask your vendor for help.
+
+RReeppoorrttiinngg pprroobblleemmss ttoo ppoossttffiixx--uusseerrss@@ppoossttffiixx..oorrgg
+
+The people who participate on postfix-users@postfix.org are very helpful,
+especially if YOU provide them with sufficient information. Remember, these
+volunteers are willing to help, but their time is limited.
+
+When reporting a problem, be sure to include the following information.
+
+ * A summary of the problem. Please do not just send some logging without
+ explanation of what YOU believe is wrong.
+
+ * Complete error messages. Please use cut-and-paste, or use attachments,
+ instead of reciting information from memory.
+
+ * Postfix logging. See the text at the top of the DEBUG_README document to
+ find out where logging is stored. Please do not frustrate the helpers by
+ word wrapping the logging. If the logging is more than a few kbytes of
+ text, consider posting an URL on a web or ftp site.
+
+ * Consider using a test email address so that you don't have to reveal email
+ addresses or passwords of innocent people.
+
+ * If you can't use a test email address, please anonymize email addresses and
+ host names consistently. Replace each letter by "A", each digit by "D" so
+ that the helpers can still recognize syntactical errors.
+
+ * Command output from:
+
+ o "ppoossttccoonnff --nn". Please do not send your main.cf file, or 1000+ lines of
+ ppoossttccoonnff command output.
+
+ o "ppoossttccoonnff --MMff" (Postfix 2.9 or later).
+
+ * Better, provide output from the ppoossttffiinnggeerr tool. This can be found at http:
+ //ftp.wl0.org/SOURCES/postfinger.
+
+ * If the problem is SASL related, consider including the output from the
+ ssaassllffiinnggeerr tool. This can be found at http://postfix.state-of-mind.de/
+ patrick.koetter/saslfinger/.
+
+ * If the problem is about too much mail in the queue, consider including
+ output from the qqsshhaappee tool, as described in the QSHAPE_README file.
+
+ * If the problem is protocol related (connections time out, or an SMTP server
+ complains about syntax errors etc.) consider recording a session with
+ ttccppdduummpp, as described in the DEBUG_README document.
+