Thursday 8 December 2011

Remote firefox over ssh

To start firefox remotely on linux machines by forwarding "X" through ssh.

try running firefox as firefox -no-remote (setting MOZ_NO_REMOTE=1 as environment variable should work as well)

we can also use xvfb [ X virtual framebuffer that performs all graphical operations in memory, not showing any screen output to start X applications in background. This will be extremely useful in case of running selenium tests on a headless server.

 To install xvfb::

yum install xorg-x11-server-Xvfb.x86_64


Xvfb can be started as:

Xvfb :1 -screen 0 1024x768x24 &



References:

1) http://unix.stackexchange.com/questions/2794/starting-firefox-on-a-remote-host-over-ssh-opens-a-new-window-locally-what-is

2) http://www.theopensourcerer.com/2007/11/15/remote-firefox-over-xssh/

Wednesday 7 December 2011

Monitoring Linux servers using snmp and mrtg

Introduction

You can monitor your system’s Web performance quite easily with graphical Linux tools. You’ll learn how to use several in this chapter, including MRTG, which is based on SNMP and monitors raw network traffic, and Webalizer, which tracks Web site hits.

SNMP

Most servers, routers and firewalls keep their operational statistics in object identifiers (OIDs) that you can remotely retrieve via the Simple Network Management Protocol (SNMP). For ease of use, equipment vendors provide Management Information Base (MIB) files for their devices that define the functions of the OIDs they contain. That’s a lot of new terms to digest in two sentences, so take a moment to look more closely.

OIDs And MIBs

OIDs are arranged in a structure of management information (SMI) tree defined by the SNMP standard. The tree starts from a root node, which then descends through branches and leaves that each add their own reference value to the path separated by a period.. Figure 22-1 shows an OID structure in which the path to the enterprises OID branch passes through the org, dod, internet, and private branches first. The OID path for enterprises is, therefore, 1.3.6.1.4.1.
Management Information Bases (MIBs) are text definitions of each of the OID branches. Table 22-1 shows how some commonly used OIDs map to their MIB definitions. For example, the SMI org MIB defines all the topmost OIDs found at the next layer, which is named dod; the internet MIB under dod defines the function of the topmost OIDs in the directory, mgmt, experimental, and private branches. This MIB information is very useful for SNMP management programs, enabling you to click on an OID and see its value, type, and description.
Table 1 OIDs And Their Equivalent MIBs
OIDMIB
1.3org
1.3.6dod
1.3.6.1internet
1.3.6.1.1directory
1.3.6.1.2mgmt
1.3.6.1.3experimental
1.3.6.1.4private
1.3.6.1.4.1enterprises
You can refer to an OID by substituting the values in a branch with one of these more readable MIB aliases. For example, you can reference the OID 1.3.6.1.4.1.9.9.109.1.1.1.1.5 as enterprises.9.9.109.1.1.1.1.5.1 by substituting the branch name (enterprises) for its OID numbers (1.3.6.1.4.1).
Remember, only the OID value at the very tip of a branch, the leaf, actually has a readable value. Think of OIDs like the directory structure on a hard disk. Each branch is equivalent to a subdirectory, and the very last value at the tip (the leaf) correlates to a file containing data.
The Linux snmpget command outputs the value of a single leaf, and the snmpwalk command provides the values of all leaves under a branch. I’ll discuss these commands later; for now, all you need to know is that the command output frequently doesn’t list the entire OID, just the MIB file in which it was found and the alias within the MIB. For example
    SNMPv2-MIB::sysUpTime.0
Here the OID value was found in the SNMPv2-MIB file and occupies position zero in the sysUpTime alias.
Equipment manufacturers are usually assigned their own dedicated OID branch under the enterprises MIB, and they must also provide information in universally accepted OIDs for ease of manageability. For example, NIC interface data throughput values must always be placed in a predefined location in the general tree, but a memory use value on a customized processor card may be defined in a MIB under the manufacturer’s own OID branch.

SNMP Community Strings

As a security measure, you need to know the SNMP password, or community string, to query OIDs. There are a number of types of community strings, the most commonly used ones are the Read Only or “get” community string that only provides access for viewing statistics and system parameters. In many cases the Read Only community string or password is set to the word “public;” you should change it from this easy-to-guess value whenever possible. The Read/Write or “set” community string is for not only viewing statistics and system parameters but also for updating the parameters.

SNMP Versions

There are currently three versions of SNMP.
  1. SNMP Version 1: The first version of SNMP to be implemented, version 1 was designed to be a protocol to provide device statistics and error reporting without consuming a lot of system resources. Security was limited to community strings and access controls based on the IP address of the querying server. Data communication wasn’t encrypted.
  2. SNMP Version 2: The second version of SNMP, often referred to as v2c, expanded the number of supported error codes, increased the size of counters used to track data, and had the ability to do bulk queries that more efficiently loaded response packets with data. SNMP v2c is backward compatible with version 1.
  3. SNMP Version 3: This version provides greater security and remote configuration capabilities than its predecessors. Access isn’t limited to a single community string for read-only and read/write access, as usernames and passwords have been introduced. Views of OIDs in a MIB can also be limited on a per-user basis. Support for encrypted SNMP data transfer and transfer error detection is also provided.
Remember their differences, because you will need to specify the version number when doing SNMP queries.

Doing SNMP Queries

Configuring SNMP on a server isn’t hard, but it does require a number of detailed steps.

Installing SNMP Utilities on a Linux Server

If you intend to use your Linux box to query your network devices, other servers or even itself using MRTG or any other tool, you need to have the SNMP utility tools package net-snmp-utils installed. This package may also require pre-requisite packages, so it is best to use an automated package updater such yum or apt to do this.
When searching for these packages the filenames will start with the package name followed by a version number, as in net-snmp-utils-5.1.1-2.i386.rpm.

SNMP Utilities Command Syntax

The SNMP utility tools package installs a number of new commands on your system for doing SNMP queries, most notably snmpget for individual OIDs and snmpwalk for obtaining the contents of an entire MIB. Both commands require you to specify the community string with a -c operator. They also require you to specify the version of the SNMP query to be used with a -v 1, -v 2c, or -v 3 operator for versions 1, 2, and 3, respectively. The first argument is the name or IP address of the target device and all other arguments list the MIBs to be queried.
This example gets all the values in the interface MIB of the local server using SNMP version 1 and the community string of S3c999rity.
[root@SNMPSER tmp]# snmpwalk -v 1 -c S3c999rity localhost interface
...
...
IF-MIB::ifDescr.1 = STRING: lo
IF-MIB::ifDescr.2 = STRING: eth0
IF-MIB::ifDescr.3 = STRING: eth1
...
...
IF-MIB::ifPhysAddress.1 = STRING:
IF-MIB::ifPhysAddress.2 = STRING: 0:9:5b:2f:9e:d5
IF-MIB::ifPhysAddress.3 = STRING: 0:b0:d0:46:32:71
...
...
[root@SNMPSER tmp]#
Upon inspecting the output of the snmpwalk command, you can see that the second interface seems to have the name eth0 and the MAC address 0:9:5b:2f:9e:d5. You can now retrieve the individual MAC address using the snmpget command.
    [root@SNMPSER tmp]# snmpget -v 1 -c const1payted localhost ifPhysAddress.2
    IF-MIB::ifPhysAddress.2 = STRING: 0:9:5b:2f:9e:d5
    [root@SNMPSER tmp]#
You can confirm this information using the ifconfig command for interface eth0; the very first line shows a matching MAC address.
    [root@SNMPSER tmp]# ifconfig -a eth0
    eth0      Link encap:Ethernet  HWaddr 00:09:5B:2F:9E:D5
              inet addr:216.10.119.244  Bcast:216.10.119.255   Mask:255.255.255.240
    ...
    ...
    [root@SNMPSER tmp]#
You’ll now see how you can configure SNMP on your Linux server to achieve these results.

Configuring Simple SNMP on a Linux Server

By default Fedora, installs the net-snmp package as its SNMP server product. This package uses a configuration file named /etc/snmp/snmpd.conf in which the community strings and other parameters may be set. The version of the configuration file that comes with net-snmp is quite complicated. I suggest archiving it and using a much simpler version with only a single line containing the keyword rocommunity followed by the community string. Here is an example.
  • Save the old configuration file
    [root@SNMPSER tmp]# cd /etc/snmp/
    [root@SNMPSER snmp]# mv snmpd.conf snmpd.conf.old
    [root@SNMPSER snmp]# vi snmpd.conf
  • Enter the following line in the new configuration file to set the Read Only community string to S3c999rity.
    rocommunity S3c999rity
  • Configure Linux to start SNMP services on each reboot with the chkconfig command:
    [root@SNMPSER root]# chkconfig snmpd on
    [root@SNMPSER root]#
  • Start SNMP to load the current configuration file.
    [root@SNMPSER root]# service snmpd start
    Starting snmpd: [ OK ]
    [root@SNMPSER root]#
  • Test whether SNMP can read the system and interface MIBs using the snmpwalk command.
          [root@SNMPSER root]# snmpwalk -v 1 -c S3c999rity localhost system
          SNMPv2-MIB::sysDescr.0 = STRING: Linux bigboy 2.4.18-14 #1 Wed Sep 4 11:57:57 EDT 2002 i586
          SNMPv2-MIB::sysObjectID.0 = OID: NET-SNMP-MIB::netSnmpAgentOIDs.10
          SNMPv2-MIB::sysUpTime.0 = Timeticks: (425) 0:00:04.25
          SNMPv2-MIB::sysContact.0 = STRING: root@localhost
          SNMPv2-MIB::sysName.0 = STRING: bigboy
          ...
          ...
          ...
          [root@SNMPSER snmp]# snmpwalk -v 1 -c S3c999rity localhost interface
          IF-MIB::ifNumber.0 = INTEGER: 3
          IF-MIB::ifIndex.1 = INTEGER: 1
          IF-MIB::ifIndex.2 = INTEGER: 2
          IF-MIB::ifIndex.3 = INTEGER: 3
          IF-MIB::ifDescr.1 = STRING: lo
          IF-MIB::ifDescr.2 = STRING: wlan0
          IF-MIB::ifDescr.3 = STRING: eth0
          ...
          ...
          ...
          [root@SNMPSER root]#
Now that you know SNMP is working correctly on your Linux server, you can configure SNMP statistics gathering software, such as MRTG, to create online graphs of your traffic flows.

SNMP On Other Devices

In the example, you were polling localhost. You can poll any SNMP-aware network device that has SNMP enabled. All you need is the IP address and SNMP Read Only string and you’ll be able to get similar results. Here is an example of a query of a device with an IP address of 192.168.1.1.
    [root@SNMPSER root]# snmpwalk -v 1 -c chir1qui 192.168.1.1 interface
Note: When issuing snmpwalk and snmpget commands, remember to use the proper version switch (-v 1, -v 2c, or -v 3) for the version of SNMP you’re using.

Basic SNMP Security

The security precautions that need to be taken with SNMP vary depending on the version you are using. This section outlines the basic steps for protecting your MIB data.

SNMP Versions 1 and 2

The most commonly supported versions of SNMP don’t encrypt your community string password so you shouldn’t do queries over insecure networks, such as the Internet. You should also make sure that you use all reasonable security measures to allow queries only from trusted IP addresses either via a firewall or the SNMP security features available in the snmp.conf file. You can also configure your server to use the TCP wrappers feature outlined in Appendix I, “Miscellaneous Linux Topics,” to limit access to specific servers without the need of a firewall.
In case you need it, the snmpd.conf file can support limiting MIB access to trusted hosts and networks.
The snmpd.conf file has two security sections; a section with very restrictive access sits at the top of the file and is immediately followed by a less restrictive section. The example that follows is a modification of the less restrictive section. You will have to comment out the more restrictive statements at the top of the file for it to work correctly.
    ##       sec.name     source           community
    ##       ========     ======           =========
    com2sec  local       localhost        S3c999rity
    com2sec  network_1   172.16.1.0/24    S3c999rity
    com2sec  network_2   192.168.2.0/24   S3c999rity

    ##       Access.group.name   sec.model        sec.name
    ##       =================  =========         ========
    group    MyROGroup_1        v1                local
    group    MyROGroup_1        v1                network_1
    group    MyROGroup_2        v2c               network_2

    ##   MIB.view.name     incl/excl  MIB.subtree  mask
    ##   ==============   =========  ===========  ====
    view all-mibs         included   .1           80

    ##      MIB
    ##      group.name   context sec.model sec.level prefix read     write  notif
    ##      ==========   ======= ========= ========= ====== ====     =====  =====
    access  MyROGroup_1  ""       v1       noauth    exact  all-mibs none   none
    access  MyROGroup_2  ""       v2c      noauth    exact  all-mibs none   none
In our example:
  • Only three networks (localhost, 172.16.1.0/24, and 192.168.1.0/24) are allowed to access the server with the S3c999rity community string.
  • Each network is matched to a either a group called MyROGroup_1 using SNMP version 1, or group called MyROGroup_2 using SNMP version 2.
  • All the MIBs on the server are defined by the view named all-mibs.
  • An access statement ensures that only the defined networks have read only access to all the MIBs. MyROGroup_1 only has version 1 access with MyROGroup_2 only having version 2 access.
  • Modification of the MIBs via SNMP is denied because the word “none” is in the write section of the access statement.
These precautions are probably unnecessary in a home environment where access is generally limited to devices on the home network by a NAT firewall.

SNMP Version 3

SNMP Version 3 SNMPv3 is a much more secure alternative to earlier versions as it encrypts all its data and uses a username / password combination for clientauthentication. The username should be located in the /etc/snmp/snmpd.conf file with a corresponding automatically generated password located in the /var/net-snmp/snmpd.conf file. Here is how it’s done.
  • Install the net-snmp-devel package as it contains the utility that will allow you to generate the password
  • Stop the snmpd process.
    [root@SNMPSER root]# service snmpd stop
    Stopping snmpd: [  OK  ]
    [root@SNMPSER root]#
  • Automatically create the /etc/snmp/snmpd.conf and /var/net-snmp/snmpd.conf username and password entries using the net-snmp-config command. In this example the authentication password of “rootsrockreggae” for the read only (-ro) username “username4snmpv3″ is encrypted using the MD5 algorithm. The data received will not be encrypted.
    [root@SNMPSER root]# net-snmp-config --create-snmpv3-user -ro 
    -a MD5 -A rootsrockreggae username4snmpv3

    adding the following line to /var/net-snmp/snmpd.conf:
       createUser username4snmpv3 MD5 "rootsrockreggae" DES
    adding the following line to /etc/snmp/snmpd.conf:
       rouser username4snmpv3
    [root@SNMPSER root]# service snmpd start
    Starting snmpd: [  OK  ]
    [root@SNMPSER root]#
  • To encrypt the data received we use the net-snmp-config command again to create a username just for this purpose. A new read only username “securev3user” and authentication password “mandeville” are used, but this time the data will be encrypted using the DES algorithm with the privacy password of “savlamar”.
    [root@SNMPSER root]# net-snmp-config --create-snmpv3-user -ro 
    -a MD5 -A mandeville -x DES -X savlamar securev3user

    adding the following line to /var/net-snmp/snmpd.conf:
       createUser securev3user MD5 "mandeville" DES savlamar
    adding the following line to /etc/snmp/snmpd.conf:
       rouser securev3user
    [root@SNMPSER root]#
  • Start the snmpd process.
    [root@SNMPSER root]# service snmpd start
    Starting snmpd: [  OK  ]
    [root@SNMPSER root]#
  • First we’ll do a query from remote host smallfry. We specify the authentication password and authentication encryption method, and we also use the -l flag to indicate that authentication will be used, but that data privacy will be disabled (the authNoPriv option).
   [root@smallfry ~]# snmpget -v 3 -u username4snmpv3 -l authNoPriv 
   -a MD5 -A rootsrockreggae  192.168.1.100 SNMPv2-MIB::sysORDescr.8
   SNMPv2-MIB::sysORDescr.8 = STRING: The management information definitions for the SNMP User-based Security Model.
   [root@smallfry ~]#
The query returns an easy to read string, “The management information definitions for the SNMP User-based Security Model”. This unencrypted string can also be seen in the tshark packet capture of the server’s interface.
    [root@SNMPSER root]# tshark -n -i eth1 -x port 161
    Capturing on eth1
    ...
    ...
    ...

      0.005889 192.168.1.100 -> 192.168.1.50 SNMP get-response

    0000  00 c0 4f 46 0c 2e 00 b0 d0 46 32 71 08 00 45 00   ..OF.....F2q..E.
    0010  00 f0 00 00 40 00 40 11 b3 b2 c0 a8 02 c8 c0 a8   ....@.@.........
    0020  02 32 00 a1 80 0a 00 dc 87 38 30 81 d1 02 01 03   .2.......80.....
    0030  30 11 02 04 45 a2 23 54 02 03 00 ff e3 04 01 01   0...E.#T........
    0040  02 01 03 04 38 30 36 04 0d 80 00 1f 88 80 71 11   ....806.......q.
    0050  68 72 0e b1 e7 45 02 01 12 02 01 39 04 0f 75 73   hr...E.....9..us
    0060  65 72 6e 61 6d 65 34 73 6e 6d 70 76 33 04 0c 46   ername4snmpv3..F
    0070  6c 74 26 51 4d aa 65 61 59 06 1a 04 00 30 7f 04   lt&QM.eaY....0..
    0080  0d 80 00 1f 88 80 71 11 68 72 0e b1 e7 45 04 00   ......q.hr...E..
    0090  a2 6c 02 04 43 4e da d7 02 01 00 02 01 00 30 5e   .l..CN........0^
    00a0  30 5c 06 0a 2b 06 01 02 01 01 09 01 03 08 04 4e   0..+..........N
    00b0  54 68 65 20 6d 61 6e 61 67 65 6d 65 6e 74 20 69   The management i
    00c0  6e 66 6f 72 6d 61 74 69 6f 6e 20 64 65 66 69 6e   nformation defin
    00d0  69 74 69 6f 6e 73 20 66 6f 72 20 74 68 65 20 53   itions for the S
    00e0  4e 4d 50 20 55 73 65 72 2d 62 61 73 65 64 20 53   NMP User-based S
    00f0  65 63 75 72 69 74 79 20 4d 6f 64 65 6c 2e         ecurity Model.

    4 packets captured
    [root@SNMPSER root]#
  • Next we’ll do a query that will return a response over an encrypted data channel while crossing the network (the authPriv option).
    [root@smallfry ~]# snmpget -v 3 -u securev3user -l authPriv 
    -a MD5 -A mandeville  -x DES -X savlamar 192.168.1.100 SNMPv2-MIB::sysORDescr.8
    SNMPv2-MIB::sysORDescr.8 = STRING: The management information definitions for the SNMP User-based Security Model.
    [root@smallfry ~]# 

The query returns the same string, but the tshark packet capture only sees encrypted data, with only the username being visible. 

[root@SNMPSER root]# tshark -n -i eth1 -x port 161
Capturing on eth1
...
...
...
  0.003675 192.168.1.200 -> 192.168.1.50 SNMP Source port: 161  Destination port: 32778 [UDP CHECKSUM INCORRECT]

0000  00 c0 4f 46 0c 2e 00 b0 d0 46 32 71 08 00 45 00   ..OF.....F2q..E.
0010  01 00 00 00 40 00 40 11 b3 a2 c0 a8 02 c8 c0 a8   ....@.@.........
0020  02 32 00 a1 80 0a 00 ec 87 48 30 81 e1 02 01 03   .2.......H0.....
0030  30 11 02 04 17 52 82 96 02 03 00 ff e3 04 01 03   0....R..........
0040  02 01 03 04 3e 30 3c 04 0d 80 00 1f 88 80 71 11   ....>0<.......q.
0050  68 72 0e b1 e7 45 02 01 11 02 02 00 8e 04 0c 73   hr...E.........s
0060  65 63 75 72 65 76 33 75 73 65 72 04 0c 01 b2 00   ecurev3user.....
0070  6e 23 07 83 dc a2 b6 d6 3d 04 08 00 00 00 11 4e   n#......=......N
0080  df 19 a3 04 81 88 36 dd e0 ce e0 52 19 ff 58 7e   ......6....R..X~
0090  be fa d1 96 20 2b 28 65 59 30 e8 d4 cb 18 9f 8f   .... +(eY0......
00a0  1e 5b a3 d6 ae f7 4a 86 bd ed 2a 4b a8 df 52 fb   .[....J...*K..R.
00b0  00 b4 a8 37 3d 74 9e 6d 1d 56 9a ba f2 13 fa 72   ...7=t.m.V.....r
00c0  4d 47 fb 88 7b d3 54 e1 9d b3 66 f0 29 ab 8a 55   MG..{.T...f.)..U
00d0  6f 77 65 40 87 ab 0c 51 d9 0e bf 33 7f 9a cb ea   owe@...Q...3....
00e0  37 50 3c 8e 65 dd 8f 3c 49 71 96 59 f9 d3 a8 23   7P<.e..
  • Your password and privilege information are stored in /var/net-snmp/snmpd.conf using the format displayed when you used the net-snmp-config command. here is an example.
    # File: /var/net-snmp/snmpd.conf before SNMP starts

    createUser securev3user MD5 "mandeville" DES savlamar
    createUser username4snmpv3 MD5 "rootsrockreggae" DES
The snmpd daemon will completely encrypt the SNMP password data in this file when it restarts which helps to further increase security. We can see an example of this configuration here.
    # File: /var/net-snmp/snmpd.conf after SNMP starts

    usmUser 1 3 0x80001f88780711168720eb1e745
    0x73656375626576337573657200
    0x73656375726576337573657200
    NULL .1.3.6.1.6.3.10.1.1.2 0xd951
    ac1d95033f4afgf31243eb6907df .1.3.6.1.6.3.10.1.2.2
    0xf1f4bb00452211d27b50c273c09031ac 0x00
    usmUser 1 3 0x80001f8880711168720eb1e745
    0x757365726e61657534736e6d70763300
    0x757365726e616d6534736e6d70763300
    NULL .1.3.6.1.6.3.10.
    1.1.2 0x5e35c9f5352519aa4f53eded09bbdddd
    .1.3.6.1.6.3.10.1.2.2 0x5e35c9f5122519aa4f53eded09bbdddd ""
    setserialno 1464593474
Practice using the net-snmp-config command so that you can become familiar with the syntax it uses to edit the SNMP configuration files. When in doubt, you can get a full syntax listing of the command if you use it without any arguments like this:
    [root@SNMPSER root]# net-snmp-config
    ...
    ...
     SNMP Setup commands:

       --create-snmpv3-user [-ro] [-A authpass] [-X privpass]
                            [-a MD5|SHA] [-x DES|AES] [username]
    ...
    ...
    [root@SNMPSER root]#
With experience, you should become confident enough to edit the configuration files by yourself.
As you can see, SNMPv3 is more secure than previous versions and should be your first SNMP choice whenever possible.

Simple SNMP Troubleshooting

If your SNMP queries fail, then verify that:
  • You restarted your snmp.conf file so the configuration settings become active. Remember, the snmpd.conf file is only read by the snmpd daemon when it starts up.
  • You are using the correct community string.
  • Firewalls aren’t preventing SNMP queries from the SNMP client to the SNMP target.
  • Your SNMP security policy allows the query from your network.
  • Any TCP wrappers configuration on your SNMP target machine allows SNMP queries from your SNMP client. Generally speaking in a home environment protected by NAT your TCP wrappers files (/etc/hosts.allow) and (/etc/hosts.deny) should be blank.
  • Network routing between the client and target devices is correct. A simple ping or traceroute test should be sufficient.
  • The snmpd daemon is running on the SNMP client.
  • You are querying using the correct SNMP version.
  • Your /var/log/messages file does not contain errors that may have occurred while starting snmpd.
Troubleshooting to get functioning SNMP queries is important as many other supporting applications, such as MRTG which I’ll discuss next, rely on them in order to work correctly.

MRTG

MRTG (Multi-Router Traffic Grapher) is a public domain package for producing graphs of various router statistics via a Web page. You can easily create graphs of traffic flow statistics through your home network’s firewall/router or even your Linux box’s NIC cards using MRTG. The product is available from the MRTG Web site (www.mrtg.org) and also on your distribution CDs. Figure 2 shows a sample MRTG graph.

MRTG Download and Installation

You need to install MRTG before proceeding. Most RedHat and Fedora Linux software products are available in the RPM format. When searching for the file, remember that the MRTG RPM’s filename usually starts with mrtg and a version number, as in mrtg-2.10.5-3.i386.rpm.
In addition to MRTG, you need to install the SNMP utility tools as explained earlier and you need to have a Web server package installed for MRTG to work. RedHat Linux usually comes with the Apache Web server software preinstalled. The easiest way to tell if Apache is installed is to run the rpm -q httpd command. By default Apache expects the HTML files for your Web site to be located in /var/www/html. MRTG places its HTML files in /var/www/mrtg.

Configuring MRTG

By default, MRTG maps the inbound and outbound data throughput rates on the device it is polling. Methods for specifying other OIDs, such as CPU and memory usage, are discussed in “Advanced MRTG“. For now, I’ll stick with the default configuration.
When the MRTG RPM is installed, it creates a directory called /etc/mrtg in which all future configuration files are stored. To create a replacement default /etc/mrtg/mrtg.cfg configuration file for the server, follow these steps.
  • Use MRTG’s cfgmaker command to create a configuration file named mrtg.cfg for the server (bigboy) using a Read Only community string of S3c999rity. Place all data files in the directory /var/www/mrtg.
    [root@SNMPSER root]# cfgmaker --output=/etc/mrtg/mrtg.cfg 
    --global "workdir: /var/www/mrtg" -ifref=ip 
    --global 'options[_]: growright,bits' 
    S3c999rity@localhost

    --base: Get Device Info on S3c999rity@localhost:
    --base: Vendor Id:
    --base: Populating confcache
    --snpo: confcache S3c999rity@localhost: Descr lo --> 1
    --snpo: confcache S3c999rity@localhost: Descr wlan0 --> 2
    ...
    ...
    ...
     --base: Walking ifAdminStatus
    --base: Walking ifOperStatus
    --base: Writing /etc/mrtg/mrtg.cfg
    [root@SNMPSER root]#
SNMPv2: As explained in the SNMP section, there are different versions of SNMP. If your query doesn’t work, check to make sure you are using the required version and then check other SNMP configuration parameters on the target device. You can specify MRTG’s SNMP query version with the –snmp-options cfgmaker option. Here is an example of cfgmaker using an SNMP version 2 query of a router with an IP address of 192.168.1.3. The –snmp-options option’s five colons before the 2 are important.
    [root@SNMPSER root]# cfgmaker --output=/etc/mrtg/192.168.1.3.cfg 
    -ifref=ip --global "workdir: /var/www/mrtg" 
    --snmp-options=:::::2 S3c999rity@192.168.1.3
    [root@SNMPSER root]#
SNMPv3: The cfgmaker command can also be used to poll SNMPv3 enabled devices, but you have to first install the Net::SNMP PERL module.
    [root@SNMPSER root]# yum -y install perl-Net-SNMP

    or

    [root@SNMPSER root]# perl -MCPAN -e "install Net::SNMP"
If you fail to install the module, you will get an error looking like this:
    Undefined subroutine &main::snmpmapOID called at ./cfgmaker line 1480.
Next you need to know the SNMPv3 ID of the host you intend to poll. In fedora, this is value is located in the /var/net-snmp/snmpd.conf file. You can use the grep command to obtain it.
    [root@SNMPSER root]# grep oldEngineID /var/net-snmp/snmpd.conf
    oldEngineID 0x80001f8880711168720eb1e745
    [root@SNMPSER root]#
You can then repeat the cfgmaker command with options specific to the privacy and authentication schemes configured on the SNMP target device. Using the configurations of our previous SNMPv3 example, our configuration for encrypted authentication only would look like this:
    [root@SNMPSER root]# cfgmaker --global 'WorkDir: /var/www/mrtg' 
    --global 'Options[_]: growright, bits' 
    --output=/etc/mrtg/192.168.1.100.cfg 
    --enablesnmpv3 --username=username4snmpv3 
    --authpassword=rootsrockreggae --authproto=md5 
    --snmp-options=:::::3 
    --contextengineid=0x80001f8880711168720eb1e745 
    securev3user@192.168.1.100
Our configuration for encrypted authentication and data privacy only would look like this:
    [root@SNMPSER root]# cfgmaker --global 'WorkDir: /var/www/mrtg' 
    --global 'Options[_]: growright, bits' 
    --output=/etc/mrtg/192.168.1.100-secure.cfg 
    --enablesnmpv3 --username=securev3user --authpassword=mandeville 
    --authproto=md5 --privpassword=savlamar --privprotocol=des 
    --snmp-options=:::::3 
    --contextengineid=0x80001f8880711168720eb1e745 
    securev3user@192.168.1.100
Note: The MRTG cfgmaker command reliably supports SNMPv3 as of MRTG version 2.15. Prior to this version you would commonly see this error when attempting to do DNMPv3 queries.
    SNMP V3 requires a --username parameter as part of the User Security 
    Model for router securev3user@192.168.1.100:::::3 
    at ./cfgmaker line 121.
  • Edit /etc/mrtg/mrtg.cfg, and remove the sections related to interfaces you don’t need to monitor. A certain candidate would be the virtual loopback interface Lo: (with the IP address of 127.0.0.1), which doesn’t pass any network traffic at all.
  • Run MRTG using /etc/mrtg/mrtg.cfg as your argument three times. You’ll get an error the two times as MRTG tries to move old data files, and naturally, the first time it is run, MRTG has no data files to move.
    [root@SNMPSER root]# env LANG=C /usr/bin/mrtg /etc/mrtg/mrtg.cfg
    Rateup WARNING: /usr/bin/rateup could not read the primary log file for localhost_192.168.1.100
    Rateup WARNING: /usr/bin/rateup The backup log file for localhost_192.168.1.100 was invalid as well
    Rateup WARNING: /usr/bin/rateup Can't remove localhost_192.168.1.100.old updating log file
    Rateup WARNING: /usr/bin/rateup Can't rename localhost_192.168.1.100.log to localhost_192.168.1.100.old updating log file
    [root@SNMPSER root]# env LANG=C /usr/bin/mrtg /etc/mrtg/mrtg.cfg
    Rateup WARNING: /usr/bin/rateup Can't remove localhost_192.168.1.100.old updating log file
    [root@SNMPSER root]# env LANG=C /usr/bin/mrtg /etc/mrtg/mrtg.cfg
    [root@SNMPSER root]#
  • Use MRTG’s indexmaker command to create a Web index page using your new mrtg.cfg file as a guide. The MRTG Web GUI expects to find the index file in the default MRTG Web directory of /var/www/mrtg/, so the format of the command would be.
          [root@SNMPSER root]# indexmaker --output=/var/www/mrtg/index.html 
          /etc/mrtg/mrtg.cfg
  • MRTG is run every five minutes by default, and the file that governs this is /etc/cron.d/mrtg. For MRTG to work correctly, edit this file, replacing all occurrences of /usr/bin/mrtg with env LANG=C /usr/bin/mrtg. The explanation for changing the language character set for MRTG is given in the “Troubleshooting MRTG” section.
This isn’t all, you need to view the graphs too. This will be covered later, but first I’ll show you how to poll multiple devices.

Getting MRTG To Poll Multiple Devices

The Fedora Core MRTG installation process creates a cron file named /etc/cron.d/mrtg. This file tells the cron daemon to run MRTG using the /etc/mrtg/mrtg.cfg file every five minutes to poll your network devices. You can configure MRTG to poll multiple devices, each with a separate configuration file. Here’s how:
  • Create a new configuration file using the steps from the previous section; choose a filename that is not mrtg.cfg.
  • Add a new MRTG line in /etc/cron.d/mrtg for each new configuration file you create.
          0-59/5 * * * * root env LANG=C /usr/bin/mrtg /etc/mrtg/mrtg.cfg
          0-59/5 * * * * root env LANG=C /usr/bin/mrtg /etc/mrtg/device1.cfg
          0-59/5 * * * * root env LANG=C /usr/bin/mrtg /etc/mrtg/device2.cfg
  • Run the indexmaker command, and include all of your /etc/mrtg configuration files, to regenerate your Web index page.
          [root@SNMPSER root]# indexmaker --output=/var/www/mrtg/index.html 
          /etc/mrtg/mrtg.cfg /etc/mrtg/device1.cfg /etc/mrtg/device2.cfg
  • Other versions of Linux keep their MRTG cron entries inside the /etc/crontab file. Edit this file using the same syntax as the Fedora /etc/cron.d/mrtg file, and then restart the cron daemon to re-read the configuration:
          [root@SNMPSER root]# service crond restart
You could also create a script with the /usr/bin/mrtg /etc/mrtg/device.cfg entries in it and make cron run it every five minutes. This way you can just edit the script each time you add a device without having to restart cron.

Configuring Apache To Work With MRTG

MRTG is useful because it can provide a graphical representation of your server’s performance statistics via a Web browser.
With Fedora Core, MRTG creates an add-on configuration file named /etc/httpd/conf.d/mrtg.conf that includes all the necessary Apache commands for MRTG to work.
Some configuration may need to be done, because by default MRTG accepts Web requests from the Linux console only. You can add your home network to the file by inserting the network on the Allow from line, or you can allow universal access by commenting out that line along with the Deny from line. This example adds access from the 192.168.1.0 network.
    
        Order deny,allow
        Allow from all
    
Remember to restart Apache once you have made these modifications in order for these changes to take effect.
Note: With newer versions of Fedora, Apache automatically reads the add-on files in the /etc/httpd/conf.d/ directory. With Fedora Core 1, you have to specifically configure the Apache configuration file /etc/httpd/conf/httpd.conf to find it. You can do this yourself by inserting this line at the very bottom of the main Apache configuration file before restarting Apache for the change to take effect.
    include "/etc/httpd/conf.d/mrtg.conf"

Basic Security

If you are accessing MRTG graphs from the Internet, you may want to add password protection to the directory by using a .htaccess.

How To View The MRTG Graphs In Your Web Browser

You can now access your MRTG graphs by pointing your browser to the URL:
    http://server-ip-address/mrtg/

Using MRTG To Monitor Other Subsystems

MRTG will generate HTML pages with daily, weekly, monthly, and yearly statistics for your interfaces. By default, MRTG provides only network interface statistics.

Troubleshooting MRTG

There are many simple steps you can use to troubleshoot MRTG. Take a look at some of the most common ones.

Basic Steps

MRTG won’t work if SNMP queries don’t work. Make sure you follow the SNMP troubleshooting steps if you have any difficulties.

Setting The Correct Character Set

MRTG usually works only if your system uses an ASCII-based (Western European) character set. If it isn’t set, then you’ll get errors such as this every time you run MRTG from the command line or as part of a cron job:
    [root@SNMPSER root]# mrtg /etc/mrtg/mrtg.cfg
    -------------------------------------------------------------------
    ERROR: Mrtg will most likely not work propperly when the environment
           variable LANG is set to UTF-8. Please run mrtg in an envir..
           where this is not the case:

           env LANG=C /usr/bin/mrtg ...
    -------------------------------------------------------------------
    [root@SNMPSER root]#
Your system’s character set is defined in /etc/sysconfig/i18n, and the current Fedora default of en_US.UTF-8 won’t work, but en_US will after a system reboot. This is not necessarily a good idea, especially if the native language Linux uses on your system is not ASCII based, other things may fail to work.
A better solution is to always run MRTG using this command instead of using just plain /usr/bin/mrtg.
    env LANG=C /usr/bin/mrtg
This will modify the character set used by MRTG alone and shouldn’t affect anything else.

Fedora Core 1 MRTG Errors With Net-SNMP

A bug appears in the MRTG implementation for some Fedora Core 1 MRTG versions when polling another Fedora Core 1 server.
When using a -ifref=ip statement with the cfgmaker command, every line in the configuration file that is generated becomes commented out. When it works, this statement is very convenient, because it makes MRTG provide graphs sorted by the IP addresses of the interfaces instead of the default, which is the much harder to recognize interface MAC address. Upgrading to the latest Core 1 version of MRTG will fix the problem.
    ### Interface 6 >> Descr:  | Name:  | Ip: '192.168.1.100'
    ###
    ### The following interface is commented out because:
    ### * has a speed of which makes no sense
    ### * got 'Received SNMP response with error code
    ###       error status: noSuchName
    ###       index 1 (OID: 1.3.6.1.2.1.2.2.1.10.6)
    ###     SNMPv1_Session (remote host: "localhost" [127.0.0.1].161)
    ###                       community: "S3c999rity"
    ###                      request ID: 824482716
    ###                     PDU bufsize: 8000 bytes
    ###                         timeout: 2s
    ###                         retries: 5
    #
    # Target[localhost_192.168.1.100]: /192.168.1.100:S3c999rity@localhost:
    # SetEnv[localhost_192.168.1.100]: MRTG_INT_IP="192.168.1.100" MRTG_INT_DES
    # MaxBytes[localhost_192.168.1.100]: 0
    # Title[localhost_192.168.1.100]: Traffic Analysis for 192.168.1.100
    # PageTop[localhost_192.168.1.100]: Traffic Analysis for 192.168.1.100
As all the lines in the configuration file are commented out with a # character, indexmaker fails to create an index.html file and gives errors.
    [root@SNMPSER root]# indexmaker --output=/var/www/mrtg/stats/index.html /etc/mrtg/mrtg.cfg
    Use of uninitialized value in hash element at /usr/bin/indexmaker line 307.
    [root@SNMPSER root]#

Webalizer

Webalizer is a Web server log file analysis tool that comes installed by default on RedHat/Fedora Linux. Each night, Webalizer reads your Apache log files and creates a set of Web pages that enable you to view Web surfer statistics for your site. The information provided includes a list of your Web site’s most popular pages sorted by hits along with traffic graphs showing the times of day when your site is most popular.

How To View Your Webalizer Statistics

Fedora creates an add-on configuration file named /etc/httpd/conf.d/Webalizer.conf that includes all the necessary Apache commands for Webalizer to work. As in the case of the MRTG add-on file mentioned above, you have to edit it to allow access to the Webalizer pages from locations other than the Linux console. You also have to restart Apache to make the changes take effect.
By default, Webalizer places its index page in the directory /var/www/html/usage and allows you to view your data by visiting the URL http://server-ip-address/usage.

The Webalizer Configuration File

Webalizer stores its configuration in the file /etc/Webalizer.conf. The default settings should be sufficient for your Web server, but you may want to adjust the directory in which Webalizer places your graph statistics. This can be adjusted with the OutputDir directive in the file. After adjustments, Webalizer functions with few annoyances; however, be aware that running in quiet mode could hide deeper problems that could occur in future.

The top Command

You can monitor the amount of memory and CPU resources your system is using the top command.
    [root@SNMPSER tmp]# top

      3:04pm  up 25 days, 23:23,  2 users,  load average: 0.00, 0.02, 0.00
    78 processes: 76 sleeping, 2 running, 0 zombie, 0 stopped
    CPU states:  0.9% user,  0.5% system,  0.0% nice,  0.8% idle
    Mem:   384716K av,  327180K used,   57536K free,       0K shrd,  101544K buff
    Swap:  779112K av,       0K used,  779112K free                  130776K cached

      PID USER     PRI  NI  SIZE   RSS SHARE STAT %CPU %MEM   TIME COMMAND
    27191 root      15   0  1012 1012   780 R     5.6  0.2   0:00 top
     4545 root      16   0  5892 5888  4956 S     0.9  1.5 169:26 magicdev
        1 root      15   0   476   476   432 S     0.0  0.1   0:05 init
        2 root      15   0     0     0     0 SW    0.0  0.0   0:00 keventd
        5 root      15   0     0     0     0 SW    0.0  0.0   0:41 kswapd
        6 root      25   0     0     0     0 SW    0.0  0.0   0:00 bdflush

    [root@SNMPSER tmp]#
Here the CPU usage is under 1.0% and 14% of memory (57536K) is free. The amount of free memory may appear low, but in this case, the server doesn’t seem to be swapping idle processes from memory to the swap disk partition as it isn’t being used at all. Excessive swapping can cause your system to slow down dramatically, the simplest ways to avoid this is to add more RAM or reduce the number of processes or users that are active on your system.
If your system seems slow but the CPU and memory usage is low, then start looking at networking problems, such as poor duplex negotiation, bad cables, and network congestion due to excessive traffic.

The vmstat Command

You can also determine memory and swap usage with the vmstat command, which provides a summary of what top produces. In the example, memory is still 14% free (57,452MB used from a total of 130,780) and swap isn’t being used at all.
    [root@SNMPSER tmp]# vmstat
       procs                       memory    swap          io     system         cpu
      r  b  w   swpd   free   buff   cache  si  so    bi    bo   in    cs  us  sy  id
      0  0  0      0  57452 101584 130780   0   0     0     4   18     1   3   1   1
    [root@SNMPSER tmp]#
As your memory fills up, your system will temporarily store programs and data on your hard disk’s “swap” partition. Excess swapping of programs and data between disk and memory can cause your system to slow down significantly and memory usage should be monitored to allow you to plan ways to either increase RAM or tune the way your system operates. System tuning is beyond the scope of this book, but there are many reference guides which can show you how to do this.

The free Utility

The free utility can determine the amount of free RAM on your system. The output is easier to understand than vmstat’s. Here’s a sample.
    [root@SNMPSER tmp]# free
                 total       used       free     shared     buffers     cached
    Mem:        126060     119096       6964          0       58972      40028
    -/+ buffers/cache:      20096     105964
    Swap:       522072       15496     506576
    [root@SNMPSER tmp]#
You should generally try to make your system run with at least 20% free memory on average, which should allow it to handle moderate spikes in usage caused by running memory-intensive cron batch jobs or tape backups. If you cannot achieve this, consider running more efficient versions of programs, offloading some applications to servers with less load, and, of course, upgrading the capacity of your RAM.

Conclusion

Server monitoring is always a good practice, because it can help you predict when things are going to go wrong or long term trends in your Web traffic.
MRTG can be expanded not only to monitor traffic on your server’s NIC cards, but also to graph many of the statistics listed in top, free, and vmstat. 

 references: