OpenNMS - Interface Bandwidth Threshold

One of the tasks I have received is to set up email alert if a particular interface's bandwidth utilization is over certain threshold. If you are new to the application like me, this could be a difficult task. With some help and Google search, here are the steps and hope this will be helpful for someone out there:

1. Change directory to /opt/opennms/etc, and add following to "threshd-configuration.xml" - '.1.3.6.1.4.1.9.%' is for Cisco, and you might need to change it accordingly for other vendor's gear :

<package name="bandwidth-threshold">
        <filter>IPADDR != '0.0.0.0' &amp; (nodeSysOID LIKE '.1.3.6.1.4.1.9.%' )</filter>
        <include-range begin="1.1.1.1" end="254.254.254.254"/>
        <include-range begin="::1" end="ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff"/>
        <service name="SNMP" interval="300000" user-defined="false" status="on">
            <parameter key="thresholding-group" value="bandwidth-threshold"/>
        </service>
    </package>

2. Add following to "thresholds.xml" - this will create ingress and egress threshold and rearm setting. If 80% of the interface bandwidth is reached, will trigger the alert and rearm if it drops down to 60%.

 <group name="bandwidth-threshold" rrdRepository="/opt/opennms/share/rrd/snmp/">
        <expression type="high" ds-type="if" value="80.0" rearm="60.0"
            trigger="1" ds-label="snmpifdescr"
            triggeredUEI="uei.opennms.org/Interface/threshold/TrafficIN/highThresholdExceeded"
            rearmedUEI="uei.opennms.org/Interface/threshold/TrafficIN/highThresholdRearmed"
            filterOperator="or" expression="8 * ifHCInOctets / 1000000 / ifHighSpeed * 100">
            <resource-filter field="snmpifalias">.* #$</resource-filter>
        </expression>
        <expression type="high" ds-type="if" value="80.0" rearm="60.0"
            trigger="1" ds-label="snmpifdescr"
            triggeredUEI="uei.opennms.org/Interface/threshold/TrafficOUT/highThresholdExceeded"
            rearmedUEI="uei.opennms.org/Interface/threshold/TrafficOUT/highThresholdRearmed"
            filterOperator="or" expression="8 * ifHCOutOctets / 1000000 / ifHighSpeed * 100.0">
            <resource-filter field="snmpifalias">.* #$</resource-filter>
        </expression>
    </group>
3. You should have a "collectd-configuration.xml" file already. If not, here is a sample:
<?xml version="1.0" encoding="UTF-8"?>
<collectd-configuration threads="50">
    <package name="example1">
        <filter>IPADDR != '0.0.0.0'</filter>
        <include-range begin="1.1.1.1" end="254.254.254.254"/>
        <include-range begin="::1" end="ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff"/>
        <service name="SNMP" interval="300000" user-defined="false" status="on">
            <parameter key="collection" value="default"/>
            <parameter key="thresholding-enabled" value="true"/>
        </service>
        <service name="WMI" interval="300000" user-defined="false" status="off">
            <parameter key="collection" value="default"/>
            <parameter key="thresholding-enabled" value="true"/>
        </service>
        <service name="OpenNMS-JVM" interval="300000"
            user-defined="false" status="on">
            <parameter key="port" value="18980"/>
            <parameter key="retry" value="2"/>
            <parameter key="timeout" value="3000"/>
            <parameter key="protocol" value="rmi"/>
            <parameter key="urlPath" value="/jmxrmi"/>
            <parameter key="rrd-base-name" value="java"/>
            <parameter key="ds-name" value="opennms-jvm"/>
            <parameter key="friendly-name" value="opennms-jvm"/>
            <parameter key="collection" value="jsr160"/>
            <parameter key="thresholding-enabled" value="true"/>
        </service>
           </package>
  <collector service="SNMP" class-name="org.opennms.netmgt.collectd.SnmpCollector"/>
    <collector service="WMI" class-name="org.opennms.netmgt.collectd.WmiCollector"/>
    <collector service="OpenNMS-JVM" class-name="org.opennms.netmgt.collectd.Jsr160Collector"/>
</collectd-configuration>
4. Here is a sample of "poller-configuration.xml" that you might already have:

<?xml version="1.0" encoding="UTF-8"?>
<poller-configuration xmlns="http://xmlns.opennms.org/xsd/config/poller"
    threads="30" nextOutageId="SELECT nextval('outageNxtId')"
    serviceUnresponsiveEnabled="false" xmlrpc="false" pathOutageEnabled="true">
    <node-outage status="on" pollAllIfNoCriticalServiceDefined="true">
        <critical-service name="ICMP"/>
    </node-outage>
    <package name="example1">
        <filter>IPADDR != '0.0.0.0'</filter>
        <include-range begin="1.1.1.1" end="254.254.254.254"/>
        <include-range begin="::1" end="ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff"/>
        <rrd step="300">
            <rra>RRA:AVERAGE:0.5:1:2016</rra>
            <rra>RRA:AVERAGE:0.5:12:1488</rra>
            <rra>RRA:AVERAGE:0.5:288:366</rra>
            <rra>RRA:MAX:0.5:288:366</rra>
            <rra>RRA:MIN:0.5:288:366</rra>
        </rrd>
   <service name="ICMP" interval="300000" user-defined="false" status="on">
            <parameter key="retry" value="2"/>
            <parameter key="timeout" value="3000"/>
            <parameter key="rrd-repository" value="/opt/opennms/share/rrd/response"/>
            <parameter key="rrd-base-name" value="icmp"/>
            <parameter key="ds-name" value="icmp"/>
            <parameter key="thresholding-enabled" value="true"/>
        </service>
  <service name="SNMP" interval="300000" user-defined="false" status="on">
            <parameter key="port" value="161"/>
            <parameter key="oid" value=".1.3.6.1.2.1.1.2.0"/>
        </service>
  </package>

  <monitor service="ICMP" class-name="org.opennms.netmgt.poller.monitors.IcmpMonitor"/>
  <monitor service="SMTP" class-name="org.opennms.netmgt.poller.monitors.SmtpMonitor"/>
 
</poller-configuration>
5. Change directory to Events and add following to "programmatic.events.xml":

 <event>
        <uei>uei.opennms.org/Interface/threshold/TrafficIN/highThresholdExceeded</uei>
        <event-label>User-defined threshold event uei.opennms.org/Interface/threshold/TrafficIN/highThresholdExceeded</event-label>
        <descr>Threshold exceeded for %service% datasource %parm[ds]% on interface %interface%, parms: %parm[all]</descr>
        <logmsg dest="logndisplay">Threshold exceeded for %service% datasource %parm[ds]% on interface %interface%, parms: %parm[all]%</logmsg>
        <severity>Warning</severity>
    </event>
    <event>
        <uei>uei.opennms.org/Interface/threshold/TrafficIN/highThresholdRearmed</uei>
        <event-label>User-defined threshold event uei.opennms.org/Interface/threshold/TrafficIN/highThresholdRearmed</event-label>
        <descr>Threshold rearmed for %service% datasource %parm[ds]% on interface %interface%, parms: %parm[all]</descr>
        <logmsg dest="logndisplay">Threshold rearmed for %service% datasource %parm[ds]% on interface %interface%, parms: %parm[all]%</logmsg>
        <severity>Warning</severity>
    </event>
    <event>
        <uei>uei.opennms.org/Interface/threshold/TrafficOUT/highThresholdExceeded</uei>
        <event-label>User-defined threshold event uei.opennms.org/Interface/threshold/TrafficOUT/highThresholdExceeded</event-label>
        <descr>Threshold exceeded for %service% datasource %parm[ds]% on interface %interface%, parms: %parm[all]</descr>
        <logmsg dest="logndisplay">Threshold exceeded for %service% datasource %parm[ds]% on interface %interface%, parms: %parm[all]%</logmsg>
        <severity>Warning</severity>
    </event>
    <event>
        <uei>uei.opennms.org/Interface/threshold/TrafficOUT/highThresholdRearmed</uei>
        <event-label>User-defined threshold event uei.opennms.org/Interface/threshold/TrafficOUT/highThresholdRearmed</event-label>
        <descr>Threshold rearmed for %service% datasource %parm[ds]% on interface %interface%, parms: %parm[all]</descr>
        <logmsg dest="logndisplay">Threshold rearmed for %service% datasource %parm[ds]% on interface %interface%, parms: %parm[all]%</logmsg>
        <severity>Warning</severity>
    </event>
6. Restart Opennms:
service opennms stop
service opennms start

7. You can change some of the threshold setting from WebUI's Admin section:
     a. Under Operation, click Manage Thresholds/bandwidth-threshold, and select Edit
     b. Scroll to the right and click Edit

 

     c. The value of 80.0 and 60.0 are percentage of the interface bandwidth. Trigger value is the interval of scan.  The Reqular Expression controls which interface will be monitored and you will have to add a description to the interface and the description will need to be ended with a "#".

8. From Admin section, click on Configure Notifications, and then Add New Notification:



9. From the long list of Events, select the one you've added in step 5.

 

10. Assume you have reset of notification already configured, such as destination path. Here is a sample of notification message:

 

Switching Cisco Aironet AP between Autonomous Mode and Lightweight Mode

To use Wireless LAN Controllers manage older Cisco Aironet AP, such as Air-AP1231, you will need to convert it to lightweight mode. Occasionally, you will need to convert it back to autonomous mode and then reconvert it to lightweight mode if the AP constantly reboots itself and following message shows up in the console session:

*Mar  1 00:00:27.657: %LWAPP-3-CLIENTERRORLOG: LWAPP Crypto Init: could not start PKI session

*Mar  1 00:00:27.658: %LWAPP-3-CLIENTERRORLOG: Failed to initialize Crypto. Rebooting

*Mar  1 00:00:27.823: %SYS-5-RELOAD: Reload requested by CAPWAP CLIENT. Reload Reason: FAILED CRYPTO INIT.

*Mar  1 00:00:27.828: %LWAPP-5-CHANGED: CAPWAP changed state to DOWNXmodem file system is available.

Here is a list of software you will need to perform the tasks:

·         CiscoAironet-AP-to-LWAPP-Upgrade-Tool-v34.exe

·         TFTP server

·         LWAPP Recovery Image: c1200-rcvk9w8-tar.124-21a.JA2.tar

·         Supported IOS Image: c1200-k9w7-tar.123-7.JA1.tar

Here is the step to covert the AP to Lightweight mode:

1.       Install the Upgrade tool

2.       Connect the autonomous AP to network and get its IP. Make sure the AP is running the IOS image mentioned above.

3.       Create a text file with following content: “10.90.1.104,Cisco,Cisco,Cisco” without the quotation mark. Replace the IP address here with the AP’s IP. The sample file I used here is called “ap-file.txt”

4.       Launch the upgrading tool and fill in required field, where IP File is the text file contains AP’s IP; Select “Use Upgrade Tool TFTP Server” and points to the LWAPP image; The System IP Addr is your local machine’s IP address; Enter WLC’s IP, and credential that can add AP to controller; Select “Use Controller Time”; Click Start. The conversion process could take a few minutes. Click on Detailed Log for progress.

5.       Make sure your security policy on WLC allows the AP to join. Once it is done, you should be able to find the AP on your controller.

To convert the AP back to autonomous mode:

1.       Assign your machine with a static IP: 10.0.0.2/8, and connect it to a spare switch.

2.       Rename the IOS image to c1200-k9w7-tar.default , save it to the TFTP program’s root folder, and launch the TFTP program. Make sure “default” is the image extension – check your Folder Display Option.

3.       Plug the AP into the same VLAN port as your machine, hold the Mode button, and power on the AP.

4.       Wait the status LED turn to RED, not amber, and release the mode button. The AP will assign 10.0.0.1 to itself and download the IOS image.

Here is a reference doc from Cisco: 

http://www.cisco.com/en/US/docs/wireless/access_point/conversion/lwapp/upgrade/guide/lwapnote.html#wp160918

ASIC to switch port mapping / oversubscription

Interface/Module Connectivity Problems

Connectivity Problem or Packet Loss with WS-X6548-GE-TX and WS-X6148-GE-TX Modules used in a Server Farm

When you use either the WS-X6548-GE-TX or WS-X6148-GE-TX modules, there is a possibility that individual port utilization can lead to connectivity problems or packet loss on the surrounding interfaces. Especially when you use EtherChannel and Remote Switched Port Analyzer (RSPAN) in these line cards, you can potentially see the slow response due to packet loss. These line cards are oversubscription cards that are designed to extend gigabit to the desktop and might not be ideal for server farm connectivity. On these modules there is a single 1-Gigabit Ethernet uplink from the port ASIC that supports eight ports. These cards share a 1 Mb buffer between a group of ports (1-8, 9-16, 17-24, 25-32, 33-40, and 41-48) since each block of eight ports is 8:1 oversubscribed. The aggregate throughput of each block of eight ports cannot exceed 1 Gbps. Table 4 in the Cisco Catalyst 6500 Series 10/100- & 10/100/1000-Mbps Ethernet Interface Modules shows the different types of Ethernet interface modules and the supported buffer size per port.
Oversubscription happens due to multiple ports combined into a single Pinnacle ASIC. The Pinnacle ASIC is a direct memory access (DMA) engine that transfers packets between backplane switching bus and the network ports. If any port in this range receives or transmits traffic at a rate that exceeds its bandwidth or utilizes a large amount of buffers to handle bursts of traffic, the other ports in the same range can potentially experience packet loss. The buffer assignment on these modules is documented in Buffers, Queues & Thresholds on Catalyst 6500 Ethernet Modules.

For WS-X6748-GE-TX module, use following command to find out ASIC to switch port mapping:


Switch1#show interfaces capabilities module 3
GigabitEthernet1/3/1
  Model:                 WS-X6748-GE-TX
  Type:                  10/100/1000BaseT
  Speed:                 10,100,1000,auto
  Duplex:                half,full
.
.
.
  Ports-in-ASIC (Sub-port ASIC) : 1-24 (1-12) <------ First 24 ports on one fabric, of which, the first half of the ports maps to one ASIC and the other half maps to the second ASIC. Combined with the output below, 20G bandwidth for each of the two fabric on the module or 10G shared per 12 ports. 

.
.
.                          
  Remote switch uplink:  no
  Port-Security:         yes
  Dot1x:                 yes

Switch1#sh fabric status

 slot  channel speed module   fabric   hotStandby  Standby  Standby

                     status   status      support  module   fabric

    3        0   20G     OK       OK   Y(not-hot) 

    3        1   20G     OK       OK   Y(not-hot)

    4        0   20G     OK       OK   Y(not-hot)

    4        1   20G     OK       OK   Y(not-hot)

    5        0   20G     OK       OK          N/A

    5        1   20G     OK       OK          N/A

Use Cisco vWLC in VMWare Workstation

Cisco offers WLC virtual machine. Here are a few good online resources to get you started:
http://jeensern.blogspot.com/2012/09/cisco-virtual-wireless-lan-controller.html
http://www.labminutes.com/wlc0001_vwlc_7_3_vmware_installation
http://www.labminutes.com/blog0003_vwlc_7.3_installation_caveats
And here is the Cisco's deployment guide:
http://www.cisco.com/en/US/products/ps12723/products_tech_note09186a0080bd2d04.shtml

If you are using VMWare Workstation, you will need to download the latest VMWare OVF Tool and convert the ova file to virtual machine files.

Once you fire up the vWLC in VMware Workstation, you will see "Press any key to use this terminal as the default terminal.". If you have not configured other way to ssh or telnet to the VM, press any key here.

If you see screen "freeze" with "kernel direct mapping tables up to 100000000 @ 8000-d000", it is OK. The WLC is running fine and you can still access it though CLI or HTTPS.
Alternatively, you can

•  press any key at above stage, 
• press ESC key at following screen to get more booting options
• Or let it boot to primary image by default

You will see log in prompt soon:

Update:

Played with newer version of the vWLC and here is an update on the setup:

Device/Software List:

• PC with dual NICs
• WMware Workstation 11
• vWLC 8.0 and 8.3
• Cisco 3560 PoE Switch
• Cisco Thin AP

NIC configuration:

• The 01BottemTP is the first NIC, which has internet connection.
• The Local Area Connection 2 (Realtek PCIe GBE) is the second NIC and it supports VLANs.
• I created 3 VLANs using the second NIC's utility program and renamed them as VLAN11, 12 and 1 for easy identification purpose from original name Realtek Virtual Adapter 1, 2, & 3 in Windows 7. 

VMware Workstation Setup:

• In WMware Workstation Virtual Network, assign VMnet0 to Realtak Virtual Adapter 3.
• Assign VMnet1 to the second physical NIC.

Once the OVA file was imported to WMware, assign the first Network Adapter to VMnet0 and the second Network Adapter to VMNet1.

vWLC Setup:

The key point to initially configure the vWLC is highlight below:

• Select default for Service interface IP Address configuration
• Management Interface VLAN ID is 0

Switch Configuration:

Connect the PC's second NIC to a trunk port - Gi0/1

DHCP Relay on CheckPoint R75.20

In case of requesting IP address from a DHCP server sitting behind a different FW interface, DHCP relay needs to be configured. In following example, DHCP server is in internal network and corresponding firewall interface is eth0. The DHCP client is in a DMZ and the matching firewall interface is eth1.
Step 1: Enable DHCP Relay:

1. SSH to FW and run "sysconfig"
2. Select option 8: "DHCP Relay Configuration"
3. Define DHCP server IP address in "DHCP servers list"
4. Select both eth0 and eth1 interface in "Relay via interfaces"
5. Enable DHCP relay

Step 2: Configure Firewall Rules

1. Open SmartDashborad and create following node or network objects:
1. Server_DHCP - a node with DHCP server's IP address in the internal network
2. DHCP_255.255.255.255 - this is a node with 255.255.255.255 address
3. Network_X.X.X.X - this is the network of the DHCP scope
2. Create following three rules:
1. (source) Server_DHCP, (destination) Firewall object, (services) dhcp-relay && dhcp-rep-localmodule && dhcp-req-localmodule, Accept, Log
2. (source) Network_X.X.X.X, (destination) Server_DHCP, (services) dhcp-relay && dhcp-rep-localmodule && dhcp-req-localmodule, Accept, Log
3. (source) Any, (destination) DHCP_255.255.255.255, (services) dhcp-relay && dhcp-rep-localmodule && dhcp-req-localmodule, Accept, Log

dhcp-relay and dhcp-req-localmodule is UDP port 67 and dhcp-rep-localmodule is UDP 68. They should be predefined.

EEM Applet

Our network monitoring program detected  ICMP latency from a few WS-C3560CG-8PC-S switches recently. The "show process cpu sort" shows normal output. However, the "show process cpu history" indicates CPU utilization was always 60% and spike over 90% from time to time. To capture which process is the culprit, Cisco recommended to use following EEM (Embedded Event Manager) applet:

event manager applet high-cpu

!

event snmp oid 1.3.6.1.4.1.9.9.109.1.1.1.1.3 get-type next entry-op gt entry-val 80 poll-interval 5 exit-time 500 maxrun 600

!

action 0.1 cli command "enable"

action 0.2 syslog msg "TAC - Capturing high cpu information to flash:"

action 0.3 cli command "term length 0"

action 1.1 cli command "show process cpu sorted | redirect flash:eem-cpu1.txt"

action 1.2 cli command "show interface | redirect flash:eem-interface1.txt"

action 1.3 cli command "show interface stats | redirect flash:eem-stat1.txt"

action 1.4 cli command "show ip traffic | redirect flash:eem-traffic1.txt"

action 2.1 cli command "show process cpu sorted | redirect flash:eem-cpu2.txt"

action 2.2 cli command "show interface | redirect flash:eem-interface2.txt"

action 2.3 cli command "show interface stats | redirect flash:eem-stat2.txt"

action 2.4 cli command "show ip traffic | redirect flash:eem-traffic2.txt"

What this applet will do is when CPU process is over 80%, it will write those show output to text files in flash. This applet did capture the "virtual exec" was running over 90% of CPU process. 

Just felt this is a cool utility to share. You can find more EEM samples here: http://www.cisco.com/en/US/docs/switches/datacenter/sw/5_x/nx-os/system_management/configuration/guide/sm_eem_events_and_examples.html

BTW, the root cause of the high CPU utilization is a bug in the IOS. Upgraded to c3560c405ex-universalk9-mz.122-55.EX3.bin seems bring down the utilization to 20%.

Checkpoint Firewall lab with VMware

I am new to Checkpoint Firewall and just want to share my experience on setting up a Checkpoint Firewall lab with VMware workstation. This lab will have two management stations and three firewalls. you can configure the two management station and the first two firewall as HA later.

1. First, download Check_Point_R75.20.Splat.iso from Checkpoint site.
2. Second, create one VM in WMware Workstation and call it MGMT1. This is a Red Hat Linux 5 VM. You just need one virtual NIC for now.
3. Boot the VM with the iso file you downloaded and install the Security Platform. The detailed instruction can be found here: http://www.sysadmintutorials.com/installing-check-point-r75-secureplatform-tutorials/ and you will need to stop at step 11 of the second
4. Turn off the VM and use VMware to clone this VM as MGMT2, FW1, FW2, and FW3.
5. You will need to add two more virtual NICs (in different VMNet) to FW1 and FW2, and one more virtual NIC to FW2.
6. Here is the tricky or annonying part: all your VMs first NIC will have same MAC address. To change that, turn on each VM, login as admin with default password (admin). Enter "expert" at the prompt, and enter "admin" as initial password again. Using VI to open "/etc/sysconfig/netconf.C" and "/etc/sysconfig/netconf.C.keep" and modify the MAC address of the first NIC.
7. After the change, reboot the VM, log in as admin, and enter "ifconfig" to verify each NIC has unique MAC address. (I have spent some time to change the MAC address using VMware KB 507: http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=507, but that did not work. My colleague Larry helped me out on step 6 & 7 here.)
8. On MGMT1, enter "sysconfig" to install Security Management - continue the steps in the above tutorial. This will be the primary management station.
9. On MGMT2, follow the same step to install Security Management, except this will be the secondary management station.
10. One the rest of the VMs, use sysconfig to install Security Gateway - those will be the actual firewall.

Now you have a few raw machines to start learning Checkpoint FW. Have fun and good luck!