Wednesday, June 19, 2013

OpenNMS - Interface Bandwidth Threshold

One of the tasks I have received is to set up email alert if a particular interface's bandwidth utilization is over certain threshold. If you are new to the application like me, this could be a difficult task. With some help and Google search, here are the steps and hope this will be helpful for someone out there:

1. Change directory to /opt/opennms/etc, and add following to "threshd-configuration.xml" - '.1.3.6.1.4.1.9.%' is for Cisco, and you might need to change it accordingly for other vendor's gear :

<package name="bandwidth-threshold">
        <filter>IPADDR != '0.0.0.0' &amp; (nodeSysOID LIKE '.1.3.6.1.4.1.9.%' )</filter>
        <include-range begin="1.1.1.1" end="254.254.254.254"/>
        <include-range begin="::1" end="ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff"/>
        <service name="SNMP" interval="300000" user-defined="false" status="on">
            <parameter key="thresholding-group" value="bandwidth-threshold"/>
        </service>
    </package>


2. Add following to "thresholds.xml" - this will create ingress and egress threshold and rearm setting. If 80% of the interface bandwidth is reached, will trigger the alert and rearm if it drops down to 60%.

 <group name="bandwidth-threshold" rrdRepository="/opt/opennms/share/rrd/snmp/">
        <expression type="high" ds-type="if" value="80.0" rearm="60.0"
            trigger="1" ds-label="snmpifdescr"
            triggeredUEI="uei.opennms.org/Interface/threshold/TrafficIN/highThresholdExceeded"
            rearmedUEI="uei.opennms.org/Interface/threshold/TrafficIN/highThresholdRearmed"
            filterOperator="or" expression="8 * ifHCInOctets / 1000000 / ifHighSpeed * 100">
            <resource-filter field="snmpifalias">.* #$</resource-filter>
        </expression>
        <expression type="high" ds-type="if" value="80.0" rearm="60.0"
            trigger="1" ds-label="snmpifdescr"
            triggeredUEI="uei.opennms.org/Interface/threshold/TrafficOUT/highThresholdExceeded"
            rearmedUEI="uei.opennms.org/Interface/threshold/TrafficOUT/highThresholdRearmed"
            filterOperator="or" expression="8 * ifHCOutOctets / 1000000 / ifHighSpeed * 100.0">
            <resource-filter field="snmpifalias">.* #$</resource-filter>
        </expression>
    </group>
3. You should have a "collectd-configuration.xml" file already. If not, here is a sample:
<?xml version="1.0" encoding="UTF-8"?>
<collectd-configuration threads="50">
    <package name="example1">
        <filter>IPADDR != '0.0.0.0'</filter>
        <include-range begin="1.1.1.1" end="254.254.254.254"/>
        <include-range begin="::1" end="ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff"/>
        <service name="SNMP" interval="300000" user-defined="false" status="on">
            <parameter key="collection" value="default"/>
            <parameter key="thresholding-enabled" value="true"/>
        </service>
        <service name="WMI" interval="300000" user-defined="false" status="off">
            <parameter key="collection" value="default"/>
            <parameter key="thresholding-enabled" value="true"/>
        </service>
        <service name="OpenNMS-JVM" interval="300000"
            user-defined="false" status="on">
            <parameter key="port" value="18980"/>
            <parameter key="retry" value="2"/>
            <parameter key="timeout" value="3000"/>
            <parameter key="protocol" value="rmi"/>
            <parameter key="urlPath" value="/jmxrmi"/>
            <parameter key="rrd-base-name" value="java"/>
            <parameter key="ds-name" value="opennms-jvm"/>
            <parameter key="friendly-name" value="opennms-jvm"/>
            <parameter key="collection" value="jsr160"/>
            <parameter key="thresholding-enabled" value="true"/>
        </service>
           </package>
  <collector service="SNMP" class-name="org.opennms.netmgt.collectd.SnmpCollector"/>
    <collector service="WMI" class-name="org.opennms.netmgt.collectd.WmiCollector"/>
    <collector service="OpenNMS-JVM" class-name="org.opennms.netmgt.collectd.Jsr160Collector"/>
</collectd-configuration>
4. Here is a sample of "poller-configuration.xml" that you might already have:

<?xml version="1.0" encoding="UTF-8"?>
<poller-configuration xmlns="http://xmlns.opennms.org/xsd/config/poller"
    threads="30" nextOutageId="SELECT nextval('outageNxtId')"
    serviceUnresponsiveEnabled="false" xmlrpc="false" pathOutageEnabled="true">
    <node-outage status="on" pollAllIfNoCriticalServiceDefined="true">
        <critical-service name="ICMP"/>
    </node-outage>
    <package name="example1">
        <filter>IPADDR != '0.0.0.0'</filter>
        <include-range begin="1.1.1.1" end="254.254.254.254"/>
        <include-range begin="::1" end="ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff"/>
        <rrd step="300">
            <rra>RRA:AVERAGE:0.5:1:2016</rra>
            <rra>RRA:AVERAGE:0.5:12:1488</rra>
            <rra>RRA:AVERAGE:0.5:288:366</rra>
            <rra>RRA:MAX:0.5:288:366</rra>
            <rra>RRA:MIN:0.5:288:366</rra>
        </rrd>
   <service name="ICMP" interval="300000" user-defined="false" status="on">
            <parameter key="retry" value="2"/>
            <parameter key="timeout" value="3000"/>
            <parameter key="rrd-repository" value="/opt/opennms/share/rrd/response"/>
            <parameter key="rrd-base-name" value="icmp"/>
            <parameter key="ds-name" value="icmp"/>
            <parameter key="thresholding-enabled" value="true"/>
        </service>
  <service name="SNMP" interval="300000" user-defined="false" status="on">
            <parameter key="port" value="161"/>
            <parameter key="oid" value=".1.3.6.1.2.1.1.2.0"/>
        </service>
  </package>

  <monitor service="ICMP" class-name="org.opennms.netmgt.poller.monitors.IcmpMonitor"/>
  <monitor service="SMTP" class-name="org.opennms.netmgt.poller.monitors.SmtpMonitor"/>
 
</poller-configuration>
5. Change directory to Events and add following to "programmatic.events.xml":

 <event>
        <uei>uei.opennms.org/Interface/threshold/TrafficIN/highThresholdExceeded</uei>
        <event-label>User-defined threshold event uei.opennms.org/Interface/threshold/TrafficIN/highThresholdExceeded</event-label>
        <descr>Threshold exceeded for %service% datasource %parm[ds]% on interface %interface%, parms: %parm[all]</descr>
        <logmsg dest="logndisplay">Threshold exceeded for %service% datasource %parm[ds]% on interface %interface%, parms: %parm[all]%</logmsg>
        <severity>Warning</severity>
    </event>
    <event>
        <uei>uei.opennms.org/Interface/threshold/TrafficIN/highThresholdRearmed</uei>
        <event-label>User-defined threshold event uei.opennms.org/Interface/threshold/TrafficIN/highThresholdRearmed</event-label>
        <descr>Threshold rearmed for %service% datasource %parm[ds]% on interface %interface%, parms: %parm[all]</descr>
        <logmsg dest="logndisplay">Threshold rearmed for %service% datasource %parm[ds]% on interface %interface%, parms: %parm[all]%</logmsg>
        <severity>Warning</severity>
    </event>
    <event>
        <uei>uei.opennms.org/Interface/threshold/TrafficOUT/highThresholdExceeded</uei>
        <event-label>User-defined threshold event uei.opennms.org/Interface/threshold/TrafficOUT/highThresholdExceeded</event-label>
        <descr>Threshold exceeded for %service% datasource %parm[ds]% on interface %interface%, parms: %parm[all]</descr>
        <logmsg dest="logndisplay">Threshold exceeded for %service% datasource %parm[ds]% on interface %interface%, parms: %parm[all]%</logmsg>
        <severity>Warning</severity>
    </event>
    <event>
        <uei>uei.opennms.org/Interface/threshold/TrafficOUT/highThresholdRearmed</uei>
        <event-label>User-defined threshold event uei.opennms.org/Interface/threshold/TrafficOUT/highThresholdRearmed</event-label>
        <descr>Threshold rearmed for %service% datasource %parm[ds]% on interface %interface%, parms: %parm[all]</descr>
        <logmsg dest="logndisplay">Threshold rearmed for %service% datasource %parm[ds]% on interface %interface%, parms: %parm[all]%</logmsg>
        <severity>Warning</severity>
    </event>
6. Restart Opennms:
service opennms stop
service opennms start

7. You can change some of the threshold setting from WebUI's Admin section:
     a. Under Operation, click Manage Thresholds/bandwidth-threshold, and select Edit
     b. Scroll to the right and click Edit
 
     c. The value of 80.0 and 60.0 are percentage of the interface bandwidth. Trigger value is the interval of scan.  The Reqular Expression controls which interface will be monitored and you will have to add a description to the interface and the description will need to be ended with a "#".

8. From Admin section, click on Configure Notifications, and then Add New Notification:





9. From the long list of Events, select the one you've added in step 5.
 
10. Assume you have reset of notification already configured, such as destination path. Here is a sample of notification message:
 

No comments:

Post a Comment