EEM Applet

Our network monitoring program detected  ICMP latency from a few WS-C3560CG-8PC-S switches recently. The "show process cpu sort" shows normal output. However, the "show process cpu history" indicates CPU utilization was always 60% and spike over 90% from time to time. To capture which process is the culprit, Cisco recommended to use following EEM (Embedded Event Manager) applet:

event manager applet high-cpu

!

event snmp oid 1.3.6.1.4.1.9.9.109.1.1.1.1.3 get-type next entry-op gt entry-val 80 poll-interval 5 exit-time 500 maxrun 600

!

action 0.1 cli command "enable"

action 0.2 syslog msg "TAC - Capturing high cpu information to flash:"

action 0.3 cli command "term length 0"

action 1.1 cli command "show process cpu sorted | redirect flash:eem-cpu1.txt"

action 1.2 cli command "show interface | redirect flash:eem-interface1.txt"

action 1.3 cli command "show interface stats | redirect flash:eem-stat1.txt"

action 1.4 cli command "show ip traffic | redirect flash:eem-traffic1.txt"

action 2.1 cli command "show process cpu sorted | redirect flash:eem-cpu2.txt"

action 2.2 cli command "show interface | redirect flash:eem-interface2.txt"

action 2.3 cli command "show interface stats | redirect flash:eem-stat2.txt"

action 2.4 cli command "show ip traffic | redirect flash:eem-traffic2.txt"

What this applet will do is when CPU process is over 80%, it will write those show output to text files in flash. This applet did capture the "virtual exec" was running over 90% of CPU process. 

Just felt this is a cool utility to share. You can find more EEM samples here: http://www.cisco.com/en/US/docs/switches/datacenter/sw/5_x/nx-os/system_management/configuration/guide/sm_eem_events_and_examples.html

BTW, the root cause of the high CPU utilization is a bug in the IOS. Upgraded to c3560c405ex-universalk9-mz.122-55.EX3.bin seems bring down the utilization to 20%.