Tag Archives: snmp

Why I Enthusiastically Switched from Cacti to Zabbix for System Monitoring

Cacti is a “complete network graphing solution” according to their website. It has also been a thorn in my side for a long time.

See what I did there? Thorn… because it’s a cactus… never mind.

When Cacti is in a steady state–when I could get it to a steady state–it was good. Not great, because there was a lot of effort to get it into what I consider “steady state”, but good. The rest of the time… thorny.

There are five major things that have driven me up the wall. In no particular order: Continue reading Why I Enthusiastically Switched from Cacti to Zabbix for System Monitoring

OpenBSD 5.1 SNMP MIBs

It’s May and that means a new version of OpenBSD is out. My SNMP MIBs have been updated for 5.1 and are available for download on the OpenBSD SNMP MIBs page.

THIS WILL BE ONE OF THE LAST RELEASES OF THE MIBS FOR NET-SNMP

During the OpenBSD 5.1 development cycle, I committed the CARP MIB to the base OpenBSD snmpd. The kernel sensor MIB has been in the base snmpd for a few releases now. That leaves the pf MIB which was committed to 5.1-current some weeks ago and will be present in the 5.2 release.

So, you’ve got a few options.

  1. Want to still run Net-SNMP with the extra MIBS? Go to the SNMP MIBs page and follow the directions. No change from previous versions. However, make plans to migrate away from Net-SNMP for OpenBSD 5.2.
  2. Only use the CARP or kernel sensors MIB? Use the base snmpd(8). There’s no configuration necessary, just run the daemon. The MIB files are in /usr/share/snmp/mibs/ (The pf MIB file is present there, but the implementation is not part of snmpd(8) in 5.1). You should also read my guide on what’s changed between the Net-SNMP and snmpd(8) implementations of the MIBs.
  3. Want to use the base snmpd(8) but still have a requirement for Net-SNMP? See my blog post on using both together.

Net-SNMP and snmpd Coexistence on OpenBSD

Although it would be awesome to ditch Net-SNMP altogether now that the base OpenBSD SNMP daemon has support for all of the OpenBSD-related MIBS (CARP, PF, kernel sensors), reality is that Net-SNMP still offers some features that are needed. OpenBSD doesn’t have any SNMP tools (snmpwalk, snmpset, etc) so these are still required from Net-SNMP. There’s also some unique features in the Net-SNMP daemon that are still useful if you want to do things like monitor BIND9 or Postfix statistics.

Here’s how to run both at the same time and leverage snmpd for the OpenBSD-related MIBs and the Net-SNMP daemon for its ability to retrieve data from scripts and extend itself using loadable modules and smux sub-agents. Continue reading Net-SNMP and snmpd Coexistence on OpenBSD

Switching from Net-SNMP to snmpd for CARP, PF and Sensor Monitoring

Update: For help running both snmpds at the same time, see Net-SNMP and snmpd Coexistence on OpenBSD.

Now that OPENBSD-CARP-MIB and OPENBSD-PF-MIB have been added to the base snmpd in OpenBSD (CARP-MIB will be in 5.1-release, PF-MIB in 5.2, and the SENSOR MIB has been there since 4.5), I wanted to document the differences between these MIBs and the corresponding implementation of the MIBs that I wrote for Net-SNMP.

Both implementations provide the same set of OIDs and allow the same data to be retrieved. Whatever you were querying via Net-SNMP is available via snmpd.

What has changed is the base OID where the CARP and PF MIBs are rooted at as well as the name of certain OIDs. Continue reading Switching from Net-SNMP to snmpd for CARP, PF and Sensor Monitoring

Net-SNMP 5.6.1 Missing hrSystemProcesses OID

I just upgraded a couple of machines to OpenBSD 4.9 and noticed the hrSystemProcesses OID was not being returned by Net-SNMP 5.6.1 (from the 4.9 ports/packages collection) .

jwk@theta:~% snmpwalk -v2c -c public theta .1.3.6.1.2.1.25.1.6.0
SNMPv2-SMI::mib-2.25.1.6.0 = No Such Instance currently exists 
    at this OID

I know for sure this worked on OpenBSD 4.8/Net-SNMP 5.4.2.1.

Turns out there is a bug in Net-SNMP 5.6.1 (bug 3166568) that’s causing this. It’s been fixed in their SVN tree. If you download this patch, place it into your ports/net/net-snmp/patches/ directory and recompile the port, you’ll be good to go.

OpenBSD SNMP MIBs

The following SNMP MIBs and the accompanying code that extend the Net-SNMP daemon allow administrators to query information from various OpenBSD subsystems. Currently, stats can be queried from:

  • Packet Filter
  • The kernel sensors framework
  • Common Address Redundancy Protocol (CARP)

These MIBs are being integrated into OpenBSD’s own snmpd. OpenBSD 5.1 has the kernel sensor and CARP MIBs. OpenBSD 5.1-current has and the future 5.2 release will have the pf MIB. See this post for a bit more detail.

Table of Contents

SNMP Introduction

SNMP is the Simple Network Management Protocol. It’s used to manage nodes (routers, switches, servers, etc) on an IP network. SNMP allows an administrator to query the current status of a node, collect data for analysis and historical reasons, and to make configuration changes to the node. SNMP also has provisions for a node to send an alarm or a “trap” to a central monitoring station to alert the administrator to possible trouble.

When the node being monitored is a router or firewall, some of the more commonly monitored objects include:

  • Interface bytes per second
  • Interface packets per second
  • Bytes passed/blocked
  • Packets passed/blocked
  • CPU usage
  • Memory usage
  • State table size

The list of objects that can be monitored by SNMP is stored in a Management Information Base, or MIB. The MIB defines the name of the object, its data type (integer, string, etc) and the object’s location within the MIB heirarchy.

Packet Filter MIB

The PF MIB allows for the querying of objects relating to OpenBSD’s Packet Filter firewall software. The following information can be queried from the MIB:

  • General info (is PF enabled, the runtime, the hostid)
  • Packet counters
  • State table counters
  • Log interface counters
  • Source track counters
  • Memory limits
  • Protocol timeouts
  • Interface stats (number of rules, number of states, bytes passed/blocked, packets passed/blocked, etc)
  • Table stats (number of addresses, evaluations, bytes passed/blocked, packets passed/blocked, etc)
  • Table content stats (bytes passed/blocked and packets passed/blocked for each address/network within a table)
  • Filter rule label counters
    • This feature was contributed by Sven Ingebright Ulland <email hidden; JavaScript is required>
  • pfsync(4) protocol counters

The limitations of the MIB:

The MIB tree can be viewed here: PF MIB tree
A sample walk of the MIB can be seen here: PF MIB walk

Kernel Sensors MIB

The Kernel Sensors MIB allows for querying information from the OpenBSD kernel sensors. The kernel sensors provide data from various environmental sensors, disk drives, etc.

The MIB reports data in much the same way as the “sysctl hw.sensors” command does.

The MIB tree can be viewed here: Sensors MIB tree
A sample walk of the MIB can be seen here: Sensors MIB walk

CARP MIB

The CARP MIB allows for querying the status of carp(4) interfaces. It can be used to determine how many CARP groups are configured on a node, what their state is, and the various operational settings of the CARP group, including advskew, advbase and the carpdev.

The limitations of the MIB:

  • The value of the CARP demote counter associated with interface groups is not handled.
  • The IP address(es) assigned to the CARP interface is not returned. This can be queried in the ifTable.
  • When doing load balancing via CARP, the “carpnodes” are not shown via the MIB.

The MIB tree can be viewed here: CARP MIB tree
A sample walk of the MIB can be seen here: CARP MIB walk

Download

The MIBs are available as a patch to the OpenBSD net-snmp port.

Earlier releases only included the PF MIB. These are listed below.

The pf-mib39 release is backwards compatible with OpenBSD 3.8, however the diff will not apply cleanly to the OpenBSD 3.8 net-snmp port. You will have to resolve any failures by hand.

The OLD, ucd-snmp patch is still available here: pf-mib-ucd.diff. This diff works with ucd-snmp 4.2.6 on OpenBSD 3.5 and 3.6.

Installation

Perform these steps to install the MIB:

  1. Retrieve the net-snmp port from the OpenBSD ports system
  2. Extract the patch tarball
  3. Apply the included patch
  4. Build and install the port

Example:

# cd /usr/ports/net/net-snmp
# ftp http://www.packetmischief.ca/files/openbsd/snmp/obsd-mibsXX.tar
# tar xf obsd-mibsXX.tar
# patch < obsd-mibs.diff
# make install

The MIBs are now installed as part of the net-snmp port.

Usage

The path to the MIBs is .1.3.6.1.4.1.64512 which translates to iso.org.dod.internet.private.enterprises.openBSD. 64512 is a somewhat arbitrarily chosen number that is not currently assigned.

To see what objects are available in the MIB you can either walk the MIB using an SNMP client…

# snmpwalk -v2c -c community host OPENBSD-PF-MIB::pfMIBObjects
# snmpwalk -v2c -c community host OPENBSD-SENSORS-MIB::sensorsMIBObjects
# snmwwalk -v2c -c community host OPENBSD-CARP-MIB::carpMIBObjects

… or you can view the tree:

# snmptranslate -Tp -OS OPENBSD-PF-MIB::pfMIBObjects
# snmptranslate -Tp -OS OPENBSD-SENSORS-MIB::sensorsMIBObjects
# snmptranslate -Tp -OS OPENBSD-CARP-MIB::carpMIBObjects

To query an object in the MIB, find the path to the object you want and append it to .enterprises.openBSD. For example, if you wanted to query the running object, you would determine its path to be pfMIBObjects.info.running.0. Note the 0 (zero) on the end; it is important, so don’t omit it. Next you need to tell your snmp client to load the MIB so that it’s able to translate the above path to the numeric path. The sequence looks like this:

# export MIBS="+/usr/local/share/snmp/mibs/OPENBSD-PF-MIB.txt"
# snmpget -v2c -c community host \
.enterprises.openBSD.pfMIBObjects.info.running.0
enterprises.openBSD.pfMIBObjects.info.running.0 = true(1)

The output from the snmpget is shown in bold.

Example Graphs

Below are a couple of graphs from a Cacti server that is monitoring some OpenBSD firewalls that have these MIBs installed.

 

Third-Party Tools

Below is a list of tools that others have written to take advantage of the MIBs available here. If you want something added here, please contact me.

  • Nagios check_pf_carp. Generates alarms for CARP hosts that are not in MASTER state. By Brian A. Seklecki

Credits

The following people have contributed by writing code, reporting bugs, offering suggestions, or sending alchohol :-)

  • Sven Ingebrigt Ulland (PF-MIB: labels support)
  • Brian A. Seklecki
  • Bryan Kaplan
  • Bill Cameron
  • Clint Byrum

Thank you for your contributions.

Related Blog Posts

References

Monitoring Postfix

The goal here is to monitor servers running Postfix to determine the number of email messages delivered locally and abroad and to graph that data. The data will be made available via Net-SNMP for collection using your NMS of choice.

Basis for this Work

The methods outlined below are based on the work of Craig Sanders <http://taz.net.au/postfix/mrtg/>. Some things that are done different here:

  • Only one database file is used for storing the data instead of two.
  • Instead of recording two sets of numbers (the current reading and the previous reading), subtracting them, and then returning the difference, the method below lets MRTG/Cacti/whatever you’re using perform that operation itself (as they’ve been designed to). This simplifies the process a fair bit. [On a side note, the reason Craig’s method still works is because he’s using the gauge option in his mrtg.cfg file]
  • The script that pulls the data out of the statistics database has been rewritten in C in order to eek out a bit more performance and reduce the overhead of feeding the data to Net-SNMP.

Code

update-mailstats.pl
Watches the Postfix maillog file and stores a tally of sent and received email in a small database. This script requires the File::Tail module from CPAN.
postfix-stats-get.c
Retrieves data out of the database file.

Compile postfix-stats-get.c and move it to /var/net-snmp.

# gcc -o postfix-stats-get postfix-stats-get.c
# mv postfix-stats-get /var/net-snmp

Linux users: Make sure you have a copy of Berkeley DB installed (tested up to and including v5.3) and then compile using this command:

# gcc -DLINUX -o postfix-stats-get postfix-stats-get.c -ldb

Next, add update-mailstats.pl to the system start-up scripts or to a cronjob so that it runs when the system boots. The script will automatically daemonize itself and begin tailing the /var/log/maillog file.

Getting Stats from Postfix

Unfortunately, Postfix doesn’t store internal statistics due do the way it’s designed. Because of this, statistics will have to be culled from the log file.

The update-mailstats.pl script runs in the background watching the /var/log/maillog file. It records the number of sent and recieved email messages based on whether the message is to or from a local address or a remote one. It also records the number of 4XX and 5XX error codes Postfix returns for incoming mail and the number seen when trying to deliver mail. The data is recorded in a hash table, /tmp/postfix-mailstats.db.

The update-mailstats.pl script should be added to the system start-up scripts or to a cronjob that fires when the system boots.

Serving Stats via Net-SNMP

Since the goal is to use SNMP to monitor the mail server, the data in the statistics file must be made available via SNMP. The Net-SNMP daemon allows for data to be retrieved using local shell scripts or programs. The data retrieved from these scripts is made available under special table .1.3.6.1.4.1.2021.8.1. More information on how this works is available in the snmpd.conf manpage (look for the exec keyword).

The postfix-stats-get program will retrieve data from the database and pass it back to Net-SNMP. The program takes one command line argument which indicates the datapoint to retrieve:

  • sent:smtp – Number of messages sent via SMTP
  • sent:local – Number of messages delivered locally
  • recv:smtp – Number of messages received via SMTP
  • recv:local – Number of messages received locally
  • smtp:4xx – Number of 4XX responses seen when trying to deliver mail
  • smtp:5xx – Number of 5XX responses seen when trying to deliver mail
  • smtpd:4xx – Number of 4XX responses sent to remote servers when they tried to deliver mail to us
  • smtpd:5xx – Number of 5XX responses sent to remote servers when they tried to deliver mail to us

If you have defined custom delivery methods in Postfix’s master.cf file (for example, I’ve defined a method called “legal” which inserts a legal disclaimer at the bottom of outgoing emails) then these tools should automatically create statistics for them too. In my case I can query “sent:legal” and it just works.

Edit snmpd.conf and add exec statements for each data point you want to query. Note that the argument after “exec” is arbitrary; it’s just displayed in the extNames oid.

exec postfix-sent-smtp /var/net-snmp/postfix-stats-get sent:smtp exec postfix-recv-smtp /var/net-snmp/postfix-stats-get recv:smtp exec postfix-sent-local /var/net-snmp/postfix-stats-get sent:local exec postfix-recv-local /var/net-snmp/postfix-stats-get recv:local

Once snmpd is restarted, a walk of the .1.3.6.1.4.1.2021.8.1 MIB will show the data from the hash table.

enterprises.ucdavis.extTable.extEntry.extNames.1 = postfix-sent-smtp
enterprises.ucdavis.extTable.extEntry.extNames.2 = postfix-recv-smtp
enterprises.ucdavis.extTable.extEntry.extNames.3 = postfix-sent-local
enterprises.ucdavis.extTable.extEntry.extNames.4 = postfix-recv-local
enterprises.ucdavis.extTable.extEntry.extCommand.1 = /var/net-snmp/postfix-stats-get sent:smtp
enterprises.ucdavis.extTable.extEntry.extCommand.2 = /var/net-snmp/postfix-stats-get recv:smtp
enterprises.ucdavis.extTable.extEntry.extCommand.3 = /var/net-snmp/postfix-stats-get sent:local
enterprises.ucdavis.extTable.extEntry.extCommand.4 = /var/net-snmp/postfix-stats-get recv:local
enterprises.ucdavis.extTable.extEntry.extResult.1 = 0
enterprises.ucdavis.extTable.extEntry.extResult.2 = 0
enterprises.ucdavis.extTable.extEntry.extResult.3 = 0
enterprises.ucdavis.extTable.extEntry.extResult.4 = 0
enterprises.ucdavis.extTable.extEntry.extOutput.1 = 0
enterprises.ucdavis.extTable.extEntry.extOutput.2 = 215
enterprises.ucdavis.extTable.extEntry.extOutput.3 = 219
enterprises.ucdavis.extTable.extEntry.extOutput.4 = 4

Of interest are the extOutput lines which correspond to messages sent via SMTP, recieved via SMTP, sent locally, and recieved locally, respectively. Don’t get confused by the exResult lines; these are actually the exit status of the postfix-stats-get command and not the email message counts.

Graphing Postfix Stats

Now that Postfix’s statistics are available via SNMP, they can be graphed the same as any other.

Postfix SMTP Messages

Point your NMS at the appropriate extOutput oid(s) and begin graphing. Note that if you don’t have UCD-SNMP-MIB loaded then you won’t be able to refer to any of the extOutput oids by name. Instead use .1.3.6.1.4.1.2021.8.1.101.X where X is the specific output you want.

Notes

The hash file /tmp/postfix-stats.db has a fixed size; it won’t increase in size over time. If the file is deleted for some reason (e.g., if the system reloads and clears out /tmp on start-up), update-mailstats.pl will recreate it.

As explained in the snmpd.conf manpage, when snmpd runs external commands such as postfix-stats-get, it caches the results in the file /var/net-snmp/.snmp-exec-cache. This file must be writeable by the user that snmpd is running as or else it will not return the output from the external script being ran.

The File::Tail perl module does not read the maillog in real time therefore the database is not updated in real time. There may be up to 60 seconds between database updates.