Monitoring Postfix

The goal here is to monitor servers running Postfix to determine the number of email messages delivered locally and abroad and to graph that data. The data will be made available via Net-SNMP for collection using your NMS of choice.

Basis for this Work

The methods outlined below are based on the work of Craig Sanders <http://taz.net.au/postfix/mrtg/>. Some things that are done different here:

  • Only one database file is used for storing the data instead of two.
  • Instead of recording two sets of numbers (the current reading and the previous reading), subtracting them, and then returning the difference, the method below lets MRTG/Cacti/whatever you’re using perform that operation itself (as they’ve been designed to). This simplifies the process a fair bit. [On a side note, the reason Craig’s method still works is because he’s using the gauge option in his mrtg.cfg file]
  • The script that pulls the data out of the statistics database has been rewritten in C in order to eek out a bit more performance and reduce the overhead of feeding the data to Net-SNMP.

Code

update-mailstats.pl
Watches the Postfix maillog file and stores a tally of sent and received email in a small database. This script requires the File::Tail module from CPAN.
postfix-stats-get.c
Retrieves data out of the database file.

Compile postfix-stats-get.c and move it to /var/net-snmp.

# gcc -o postfix-stats-get postfix-stats-get.c
# mv postfix-stats-get /var/net-snmp

Linux users: Make sure you have a copy of Berkeley DB installed (tested up to and including v5.3) and then compile using this command:

# gcc -DLINUX -o postfix-stats-get postfix-stats-get.c -ldb

Next, add update-mailstats.pl to the system start-up scripts or to a cronjob so that it runs when the system boots. The script will automatically daemonize itself and begin tailing the /var/log/maillog file.

Getting Stats from Postfix

Unfortunately, Postfix doesn’t store internal statistics due do the way it’s designed. Because of this, statistics will have to be culled from the log file.

The update-mailstats.pl script runs in the background watching the /var/log/maillog file. It records the number of sent and recieved email messages based on whether the message is to or from a local address or a remote one. It also records the number of 4XX and 5XX error codes Postfix returns for incoming mail and the number seen when trying to deliver mail. The data is recorded in a hash table, /tmp/postfix-mailstats.db.

The update-mailstats.pl script should be added to the system start-up scripts or to a cronjob that fires when the system boots.

Serving Stats via Net-SNMP

Since the goal is to use SNMP to monitor the mail server, the data in the statistics file must be made available via SNMP. The Net-SNMP daemon allows for data to be retrieved using local shell scripts or programs. The data retrieved from these scripts is made available under special table .1.3.6.1.4.1.2021.8.1. More information on how this works is available in the snmpd.conf manpage (look for the exec keyword).

The postfix-stats-get program will retrieve data from the database and pass it back to Net-SNMP. The program takes one command line argument which indicates the datapoint to retrieve:

  • sent:smtp – Number of messages sent via SMTP
  • sent:local – Number of messages delivered locally
  • recv:smtp – Number of messages received via SMTP
  • recv:local – Number of messages received locally
  • smtp:4xx – Number of 4XX responses seen when trying to deliver mail
  • smtp:5xx – Number of 5XX responses seen when trying to deliver mail
  • smtpd:4xx – Number of 4XX responses sent to remote servers when they tried to deliver mail to us
  • smtpd:5xx – Number of 5XX responses sent to remote servers when they tried to deliver mail to us

If you have defined custom delivery methods in Postfix’s master.cf file (for example, I’ve defined a method called “legal” which inserts a legal disclaimer at the bottom of outgoing emails) then these tools should automatically create statistics for them too. In my case I can query “sent:legal” and it just works.

Edit snmpd.conf and add exec statements for each data point you want to query. Note that the argument after “exec” is arbitrary; it’s just displayed in the extNames oid.

exec postfix-sent-smtp /var/net-snmp/postfix-stats-get sent:smtp exec postfix-recv-smtp /var/net-snmp/postfix-stats-get recv:smtp exec postfix-sent-local /var/net-snmp/postfix-stats-get sent:local exec postfix-recv-local /var/net-snmp/postfix-stats-get recv:local

Once snmpd is restarted, a walk of the .1.3.6.1.4.1.2021.8.1 MIB will show the data from the hash table.

enterprises.ucdavis.extTable.extEntry.extNames.1 = postfix-sent-smtp
enterprises.ucdavis.extTable.extEntry.extNames.2 = postfix-recv-smtp
enterprises.ucdavis.extTable.extEntry.extNames.3 = postfix-sent-local
enterprises.ucdavis.extTable.extEntry.extNames.4 = postfix-recv-local
enterprises.ucdavis.extTable.extEntry.extCommand.1 = /var/net-snmp/postfix-stats-get sent:smtp
enterprises.ucdavis.extTable.extEntry.extCommand.2 = /var/net-snmp/postfix-stats-get recv:smtp
enterprises.ucdavis.extTable.extEntry.extCommand.3 = /var/net-snmp/postfix-stats-get sent:local
enterprises.ucdavis.extTable.extEntry.extCommand.4 = /var/net-snmp/postfix-stats-get recv:local
enterprises.ucdavis.extTable.extEntry.extResult.1 = 0
enterprises.ucdavis.extTable.extEntry.extResult.2 = 0
enterprises.ucdavis.extTable.extEntry.extResult.3 = 0
enterprises.ucdavis.extTable.extEntry.extResult.4 = 0
enterprises.ucdavis.extTable.extEntry.extOutput.1 = 0
enterprises.ucdavis.extTable.extEntry.extOutput.2 = 215
enterprises.ucdavis.extTable.extEntry.extOutput.3 = 219
enterprises.ucdavis.extTable.extEntry.extOutput.4 = 4

Of interest are the extOutput lines which correspond to messages sent via SMTP, recieved via SMTP, sent locally, and recieved locally, respectively. Don’t get confused by the exResult lines; these are actually the exit status of the postfix-stats-get command and not the email message counts.

Graphing Postfix Stats

Now that Postfix’s statistics are available via SNMP, they can be graphed the same as any other.

Postfix SMTP Messages

Point your NMS at the appropriate extOutput oid(s) and begin graphing. Note that if you don’t have UCD-SNMP-MIB loaded then you won’t be able to refer to any of the extOutput oids by name. Instead use .1.3.6.1.4.1.2021.8.1.101.X where X is the specific output you want.

Notes

The hash file /tmp/postfix-stats.db has a fixed size; it won’t increase in size over time. If the file is deleted for some reason (e.g., if the system reloads and clears out /tmp on start-up), update-mailstats.pl will recreate it.

As explained in the snmpd.conf manpage, when snmpd runs external commands such as postfix-stats-get, it caches the results in the file /var/net-snmp/.snmp-exec-cache. This file must be writeable by the user that snmpd is running as or else it will not return the output from the external script being ran.

The File::Tail perl module does not read the maillog in real time therefore the database is not updated in real time. There may be up to 60 seconds between database updates.

Networking. Unix. Cyber Security. Code. Protocols. System Design. My Blog.