NetenGU's Notes: June 2013

Another task on OpenNMS with help of Rob Coote and Alejandro Galue:

1. Add following xml file (Cisco.errdisable.events.xml) to /opennms/etc/events directory:

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<events>
<event>
<mask>
<maskelement>
<mename>id</mename>
<mevalue>.1.3.6.1.4.1.9.9.41.2</mevalue>
</maskelement>
<maskelement>
<mename>generic</mename>
<mevalue>6</mevalue>
</maskelement>
<maskelement>
<mename>specific</mename>
<mevalue>1</mevalue>
</maskelement>
<varbind>
<vbnumber>2</vbnumber>
<vbvalue>5</vbvalue>
</varbind>
<varbind>
<vbnumber>3</vbnumber>
<vbvalue>ERR_DISABLE</vbvalue>
</varbind>
</mask>
<uei>uei.opennms.org/vendor/Cisco/traps/syslog/errDisable</uei>
<event-label>CISCO-SYSLOG-MIB defined trap event: clogMessageGenerated-Warning (ERR_DISABLE)</event-label> <descr>When a syslog message is generated by the device a clogMessageGenerated notification is sent. The sending of these notifications can be enabled/disabled via the clogNotificationsEnabled object.<table> <tr><td> clogHistFacility</td><td>%parm[#1]%
</td><td><p;></td;></tr><tr><td>
clogHistSeverity</td><td>%parm[#2]%
</td><td><p;>
emergency(1) alert(2) critical(3) error(4) warning(5) notice(6) info(7) debug(8) </td;></tr><tr><td>
clogHistMsgName</td><td>%parm[#3]%
</td><td><p;></td;></tr><tr><td>
clogHistMsgText</td><td>%parm[#4]%
</td><td><p;></td;></tr><tr><td>
clogHistTimestamp</td><td>%parm[#5]%
</td><td><p;></td;></tr></table>
</descr>
<logmsg dest='logndisplay'>Cisco Event: SYSLOG Notification: %parm[#4]%.</logmsg>
<severity>Warning</severity>
</event>

</events>

2. Add "<event-file>events/Cisco.errdisable.events.xml</event-file>" to /opennms/etc/eventconf.xml as shown below:

<event-file>events/Cisco.CIDS.events.xml</event-file>

<event-file>events/Cisco.5300dchan.events.xml</event-file>

<event-file>events/Cisco.errdisable.events.xml</event-file>

<event-file>events/Cisco.mcast.events.xml</event-file>

<event-file>events/Cisco.SCE.events.xml</event-file>

<event-file>events/Cisco2.events.xml</event-file>

3. Issue following command to refresh the change: /opt/opennms/bin/send-event.pl uei.opennms.org/internal/eventsConfigChange

4. From WebUI, go go Admin/Configure Notifications/Add New Event Notification, select "Cisco-SYSLOG-MIB defined trap event:" and configure the alert

Note: Here is another version of Cisco.errdisable.events.xml, which will do the same job, except it does not provide interface information:

<?xml version="1.0" encoding="UTF-8" standalone="true"?>
-<events xmlns="http://xmlns.opennms.org/xsd/eventconf"> -<event> -<mask> -<maskelement> <mename>id</mename> <mevalue>.1.3.6.1.4.1.9.9.548.0.1</mevalue> </maskelement> -<maskelement> <mename>generic</mename> <mevalue>6</mevalue> </maskelement> -<maskelement> <mename>specific</mename> <mevalue>1</mevalue> </maskelement> </mask> <uei>uei.opennms.org/mib2opennms/cErrDisableInterfaceEvent</uei> <event-label>CISCO-ERR-DISABLE-MIB defined trap event: cErrDisableInterfaceEvent</event-label> <descr> The cErrDisableInterfaceEvent is generated when an interface or {interface, vlan} is error-disabled by the feature specified in cErrDisableIfStatusCause. cErrDisableInterfaceEvent is deprecated and replaced by cErrDisableInterfaceEventRev1.<table> <tr><td> cErrDisableIfStatusCause</td><td> %parm[#1]%;</td><td> udld(1) bpduGuard(2) channelMisconfig(3) pagpFlap(4) dtpFlap(5) linkFlap(6) l2ptGuard(7) dot1xSecurityViolation(8) portSecurityViolation(9) gbicInvalid(10) dhcpRateLimit(11) unicastFlood(12) vmps(13) stormControl(14) inlinePower(15) arpInspection(16) portLoopback(17) packetBuffer(18) macLimit(19) linkMonitorFailure(20) oamRemoteFailure(21) dot1adIncompEtype(22) dot1adIncompTunnel(23) sfpConfigMismatch(24) communityLimit(25) invalidPolicy(26) lsGroup(27) ekey(28) portModeFailure(29) pppoeIaRateLimit(30) oamRemoteCriticalEvent(31) oamRemoteDyingGasp(32) oamRemoteLinkFault(33) mvrp(34) tranceiverIncomp(35) </td></tr></table> </descr> <logmsg dest="logndisplay"> cErrDisableInterfaceEvent trap received cErrDisableIfStatusCause=%parm[#1]% </logmsg> <severity>Indeterminate</severity> -<varbindsdecode> <parmid>parm[#1]</parmid> <decode varbinddecodedstring="udld" varbindvalue="1"/> <decode varbinddecodedstring="bpduGuard" varbindvalue="2"/> <decode varbinddecodedstring="channelMisconfig" varbindvalue="3"/> <decode varbinddecodedstring="pagpFlap" varbindvalue="4"/> <decode varbinddecodedstring="dtpFlap" varbindvalue="5"/> <decode varbinddecodedstring="linkFlap" varbindvalue="6"/> <decode varbinddecodedstring="l2ptGuard" varbindvalue="7"/> <decode varbinddecodedstring="dot1xSecurityViolation" varbindvalue="8"/> <decode varbinddecodedstring="portSecurityViolation" varbindvalue="9"/> <decode varbinddecodedstring="gbicInvalid" varbindvalue="10"/> <decode varbinddecodedstring="dhcpRateLimit" varbindvalue="11"/> <decode varbinddecodedstring="unicastFlood" varbindvalue="12"/> <decode varbinddecodedstring="vmps" varbindvalue="13"/> <decode varbinddecodedstring="stormControl" varbindvalue="14"/> <decode varbinddecodedstring="inlinePower" varbindvalue="15"/> <decode varbinddecodedstring="arpInspection" varbindvalue="16"/> <decode varbinddecodedstring="portLoopback" varbindvalue="17"/> <decode varbinddecodedstring="packetBuffer" varbindvalue="18"/> <decode varbinddecodedstring="macLimit" varbindvalue="19"/> <decode varbinddecodedstring="linkMonitorFailure" varbindvalue="20"/> <decode varbinddecodedstring="oamRemoteFailure" varbindvalue="21"/> <decode varbinddecodedstring="dot1adIncompEtype" varbindvalue="22"/> <decode varbinddecodedstring="dot1adIncompTunnel" varbindvalue="23"/> <decode varbinddecodedstring="sfpConfigMismatch" varbindvalue="24"/> <decode varbinddecodedstring="communityLimit" varbindvalue="25"/> <decode varbinddecodedstring="invalidPolicy" varbindvalue="26"/> <decode varbinddecodedstring="lsGroup" varbindvalue="27"/> <decode varbinddecodedstring="ekey" varbindvalue="28"/> <decode varbinddecodedstring="portModeFailure" varbindvalue="29"/> <decode varbinddecodedstring="pppoeIaRateLimit" varbindvalue="30"/> <decode varbinddecodedstring="oamRemoteCriticalEvent" varbindvalue="31"/> <decode varbinddecodedstring="oamRemoteDyingGasp" varbindvalue="32"/> <decode varbinddecodedstring="oamRemoteLinkFault" varbindvalue="33"/> <decode varbinddecodedstring="mvrp" varbindvalue="34"/> <decode varbinddecodedstring="tranceiverIncomp" varbindvalue="35"/> </varbindsdecode> </event>
<events>

Note: Here is a link for more error disabled information on Cisco IOS devices:
http://www.cisco.com/en/US/tech/tk389/tk621/technologies_tech_note09186a00806cd87b.shtml

One of the tasks I have received is to set up email alert if a particular interface's bandwidth utilization is over certain threshold. If you are new to the application like me, this could be a difficult task. With some help and Google search, here are the steps and hope this will be helpful for someone out there:

1. Change directory to /opt/opennms/etc, and add following to "threshd-configuration.xml" - '.1.3.6.1.4.1.9.%' is for Cisco, and you might need to change it accordingly for other vendor's gear :

<package name="bandwidth-threshold">
 <filter>IPADDR != '0.0.0.0' & (nodeSysOID LIKE '.1.3.6.1.4.1.9.%' )</filter>
 <include-range begin="1.1.1.1" end="254.254.254.254"/>
 <include-range begin="::1" end="ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff"/>
 <service name="SNMP" interval="300000" user-defined="false" status="on">
 <parameter key="thresholding-group" value="bandwidth-threshold"/>
 </service>
 </package>

2. Add following to "thresholds.xml" - this will create ingress and egress threshold and rearm setting. If 80% of the interface bandwidth is reached, will trigger the alert and rearm if it drops down to 60%.

<group name="bandwidth-threshold" rrdRepository="/opt/opennms/share/rrd/snmp/">
 <expression type="high" ds-type="if" value="80.0" rearm="60.0"
 trigger="1" ds-label="snmpifdescr"
 triggeredUEI="uei.opennms.org/Interface/threshold/TrafficIN/highThresholdExceeded"
 rearmedUEI="uei.opennms.org/Interface/threshold/TrafficIN/highThresholdRearmed"
 filterOperator="or" expression="8 * ifHCInOctets / 1000000 / ifHighSpeed * 100">
 <resource-filter field="snmpifalias">.* #$</resource-filter>
 </expression>
 <expression type="high" ds-type="if" value="80.0" rearm="60.0"
 trigger="1" ds-label="snmpifdescr"
 triggeredUEI="uei.opennms.org/Interface/threshold/TrafficOUT/highThresholdExceeded"
 rearmedUEI="uei.opennms.org/Interface/threshold/TrafficOUT/highThresholdRearmed"
 filterOperator="or" expression="8 * ifHCOutOctets / 1000000 / ifHighSpeed * 100.0">
 <resource-filter field="snmpifalias">.* #$</resource-filter>
 </expression>
 </group>
3. You should have a "collectd-configuration.xml" file already. If not, here is a sample:
<?xml version="1.0" encoding="UTF-8"?>
<collectd-configuration threads="50">
 <package name="example1">
 <filter>IPADDR != '0.0.0.0'</filter>
 <include-range begin="1.1.1.1" end="254.254.254.254"/>
 <include-range begin="::1" end="ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff"/>
 <service name="SNMP" interval="300000" user-defined="false" status="on">
 <parameter key="collection" value="default"/>
 <parameter key="thresholding-enabled" value="true"/>
 </service>
 <service name="WMI" interval="300000" user-defined="false" status="off">
 <parameter key="collection" value="default"/>
 <parameter key="thresholding-enabled" value="true"/>
 </service>
 <service name="OpenNMS-JVM" interval="300000"
 user-defined="false" status="on">
 <parameter key="port" value="18980"/>
 <parameter key="retry" value="2"/>
 <parameter key="timeout" value="3000"/>
 <parameter key="protocol" value="rmi"/>
 <parameter key="urlPath" value="/jmxrmi"/>
 <parameter key="rrd-base-name" value="java"/>
 <parameter key="ds-name" value="opennms-jvm"/>
 <parameter key="friendly-name" value="opennms-jvm"/>
 <parameter key="collection" value="jsr160"/>
 <parameter key="thresholding-enabled" value="true"/>
 </service>
 </package>
<collector service="SNMP" class-name="org.opennms.netmgt.collectd.SnmpCollector"/>
 <collector service="WMI" class-name="org.opennms.netmgt.collectd.WmiCollector"/>
 <collector service="OpenNMS-JVM" class-name="org.opennms.netmgt.collectd.Jsr160Collector"/>
</collectd-configuration>
4. Here is a sample of "poller-configuration.xml" that you might already have:

<?xml version="1.0" encoding="UTF-8"?>
<poller-configuration xmlns="http://xmlns.opennms.org/xsd/config/poller"
 threads="30" nextOutageId="SELECT nextval('outageNxtId')"
 serviceUnresponsiveEnabled="false" xmlrpc="false" pathOutageEnabled="true">
 <node-outage status="on" pollAllIfNoCriticalServiceDefined="true">
 <critical-service name="ICMP"/>
 </node-outage>
 <package name="example1">
 <filter>IPADDR != '0.0.0.0'</filter>
 <include-range begin="1.1.1.1" end="254.254.254.254"/>
 <include-range begin="::1" end="ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff"/>
 <rrd step="300">
 <rra>RRA:AVERAGE:0.5:1:2016</rra>
 <rra>RRA:AVERAGE:0.5:12:1488</rra>
 <rra>RRA:AVERAGE:0.5:288:366</rra>
 <rra>RRA:MAX:0.5:288:366</rra>
 <rra>RRA:MIN:0.5:288:366</rra>
 </rrd>
 <service name="ICMP" interval="300000" user-defined="false" status="on">
 <parameter key="retry" value="2"/>
 <parameter key="timeout" value="3000"/>
 <parameter key="rrd-repository" value="/opt/opennms/share/rrd/response"/>
 <parameter key="rrd-base-name" value="icmp"/>
 <parameter key="ds-name" value="icmp"/>
 <parameter key="thresholding-enabled" value="true"/>
 </service>
<service name="SNMP" interval="300000" user-defined="false" status="on">
 <parameter key="port" value="161"/>
 <parameter key="oid" value=".1.3.6.1.2.1.1.2.0"/>
 </service>
</package>

<monitor service="ICMP" class-name="org.opennms.netmgt.poller.monitors.IcmpMonitor"/>
<monitor service="SMTP" class-name="org.opennms.netmgt.poller.monitors.SmtpMonitor"/>

</poller-configuration>
5. Change directory to Events and add following to "programmatic.events.xml":

<event>
 <uei>uei.opennms.org/Interface/threshold/TrafficIN/highThresholdExceeded</uei>
 <event-label>User-defined threshold event uei.opennms.org/Interface/threshold/TrafficIN/highThresholdExceeded</event-label>
 <descr>Threshold exceeded for %service% datasource %parm[ds]% on interface %interface%, parms: %parm[all]</descr>
 <logmsg dest="logndisplay">Threshold exceeded for %service% datasource %parm[ds]% on interface %interface%, parms: %parm[all]%</logmsg>
 <severity>Warning</severity>
 </event>
 <event>
 <uei>uei.opennms.org/Interface/threshold/TrafficIN/highThresholdRearmed</uei>
 <event-label>User-defined threshold event uei.opennms.org/Interface/threshold/TrafficIN/highThresholdRearmed</event-label>
 <descr>Threshold rearmed for %service% datasource %parm[ds]% on interface %interface%, parms: %parm[all]</descr>
 <logmsg dest="logndisplay">Threshold rearmed for %service% datasource %parm[ds]% on interface %interface%, parms: %parm[all]%</logmsg>
 <severity>Warning</severity>
 </event>
 <event>
 <uei>uei.opennms.org/Interface/threshold/TrafficOUT/highThresholdExceeded</uei>
 <event-label>User-defined threshold event uei.opennms.org/Interface/threshold/TrafficOUT/highThresholdExceeded</event-label>
 <descr>Threshold exceeded for %service% datasource %parm[ds]% on interface %interface%, parms: %parm[all]</descr>
 <logmsg dest="logndisplay">Threshold exceeded for %service% datasource %parm[ds]% on interface %interface%, parms: %parm[all]%</logmsg>
 <severity>Warning</severity>
 </event>
 <event>
 <uei>uei.opennms.org/Interface/threshold/TrafficOUT/highThresholdRearmed</uei>
 <event-label>User-defined threshold event uei.opennms.org/Interface/threshold/TrafficOUT/highThresholdRearmed</event-label>
 <descr>Threshold rearmed for %service% datasource %parm[ds]% on interface %interface%, parms: %parm[all]</descr>
 <logmsg dest="logndisplay">Threshold rearmed for %service% datasource %parm[ds]% on interface %interface%, parms: %parm[all]%</logmsg>
 <severity>Warning</severity>
 </event>
6. Restart Opennms:
service opennms stop
service opennms start

7. You can change some of the threshold setting from WebUI's Admin section:
 a. Under Operation, click Manage Thresholds/bandwidth-threshold, and select Edit
 b. Scroll to the right and click Edit

c. The value of 80.0 and 60.0 are percentage of the interface bandwidth. Trigger value is the interval of scan. The Reqular Expression controls which interface will be monitored and you will have to add a description to the interface and the description will need to be ended with a "#".

8. From Admin section, click on Configure Notifications, and then Add New Notification:

9. From the long list of Events, select the one you've added in step 5.

10. Assume you have reset of notification already configured, such as destination path. Here is a sample of notification message:

NetenGU's Notes

Monday, June 24, 2013

OpenNMS - Err_Disable Alert for Cisco

Wednesday, June 19, 2013

OpenNMS - Interface Bandwidth Threshold