Switching Cisco Aironet AP between Autonomous Mode and Lightweight Mode

To use Wireless LAN Controllers manage older Cisco Aironet AP, such as Air-AP1231, you will need to convert it to lightweight mode. Occasionally, you will need to convert it back to autonomous mode and then reconvert it to lightweight mode if the AP constantly reboots itself and following message shows up in the console session:

*Mar  1 00:00:27.657: %LWAPP-3-CLIENTERRORLOG: LWAPP Crypto Init: could not start PKI session

*Mar  1 00:00:27.658: %LWAPP-3-CLIENTERRORLOG: Failed to initialize Crypto. Rebooting

*Mar  1 00:00:27.823: %SYS-5-RELOAD: Reload requested by CAPWAP CLIENT. Reload Reason: FAILED CRYPTO INIT.

*Mar  1 00:00:27.828: %LWAPP-5-CHANGED: CAPWAP changed state to DOWNXmodem file system is available.

Here is a list of software you will need to perform the tasks:

·         CiscoAironet-AP-to-LWAPP-Upgrade-Tool-v34.exe

·         TFTP server

·         LWAPP Recovery Image: c1200-rcvk9w8-tar.124-21a.JA2.tar

·         Supported IOS Image: c1200-k9w7-tar.123-7.JA1.tar

Here is the step to covert the AP to Lightweight mode:

1.       Install the Upgrade tool

2.       Connect the autonomous AP to network and get its IP. Make sure the AP is running the IOS image mentioned above.

3.       Create a text file with following content: “10.90.1.104,Cisco,Cisco,Cisco” without the quotation mark. Replace the IP address here with the AP’s IP. The sample file I used here is called “ap-file.txt”

4.       Launch the upgrading tool and fill in required field, where IP File is the text file contains AP’s IP; Select “Use Upgrade Tool TFTP Server” and points to the LWAPP image; The System IP Addr is your local machine’s IP address; Enter WLC’s IP, and credential that can add AP to controller; Select “Use Controller Time”; Click Start. The conversion process could take a few minutes. Click on Detailed Log for progress.

5.       Make sure your security policy on WLC allows the AP to join. Once it is done, you should be able to find the AP on your controller.

To convert the AP back to autonomous mode:

1.       Assign your machine with a static IP: 10.0.0.2/8, and connect it to a spare switch.

2.       Rename the IOS image to c1200-k9w7-tar.default , save it to the TFTP program’s root folder, and launch the TFTP program. Make sure “default” is the image extension – check your Folder Display Option.

3.       Plug the AP into the same VLAN port as your machine, hold the Mode button, and power on the AP.

4.       Wait the status LED turn to RED, not amber, and release the mode button. The AP will assign 10.0.0.1 to itself and download the IOS image.

Here is a reference doc from Cisco: 

http://www.cisco.com/en/US/docs/wireless/access_point/conversion/lwapp/upgrade/guide/lwapnote.html#wp160918

ASIC to switch port mapping / oversubscription

Interface/Module Connectivity Problems

Connectivity Problem or Packet Loss with WS-X6548-GE-TX and WS-X6148-GE-TX Modules used in a Server Farm

When you use either the WS-X6548-GE-TX or WS-X6148-GE-TX modules, there is a possibility that individual port utilization can lead to connectivity problems or packet loss on the surrounding interfaces. Especially when you use EtherChannel and Remote Switched Port Analyzer (RSPAN) in these line cards, you can potentially see the slow response due to packet loss. These line cards are oversubscription cards that are designed to extend gigabit to the desktop and might not be ideal for server farm connectivity. On these modules there is a single 1-Gigabit Ethernet uplink from the port ASIC that supports eight ports. These cards share a 1 Mb buffer between a group of ports (1-8, 9-16, 17-24, 25-32, 33-40, and 41-48) since each block of eight ports is 8:1 oversubscribed. The aggregate throughput of each block of eight ports cannot exceed 1 Gbps. Table 4 in the Cisco Catalyst 6500 Series 10/100- & 10/100/1000-Mbps Ethernet Interface Modules shows the different types of Ethernet interface modules and the supported buffer size per port.
Oversubscription happens due to multiple ports combined into a single Pinnacle ASIC. The Pinnacle ASIC is a direct memory access (DMA) engine that transfers packets between backplane switching bus and the network ports. If any port in this range receives or transmits traffic at a rate that exceeds its bandwidth or utilizes a large amount of buffers to handle bursts of traffic, the other ports in the same range can potentially experience packet loss. The buffer assignment on these modules is documented in Buffers, Queues & Thresholds on Catalyst 6500 Ethernet Modules.

For WS-X6748-GE-TX module, use following command to find out ASIC to switch port mapping:


Switch1#show interfaces capabilities module 3
GigabitEthernet1/3/1
  Model:                 WS-X6748-GE-TX
  Type:                  10/100/1000BaseT
  Speed:                 10,100,1000,auto
  Duplex:                half,full
.
.
.
  Ports-in-ASIC (Sub-port ASIC) : 1-24 (1-12) <------ First 24 ports on one fabric, of which, the first half of the ports maps to one ASIC and the other half maps to the second ASIC. Combined with the output below, 20G bandwidth for each of the two fabric on the module or 10G shared per 12 ports. 

.
.
.                          
  Remote switch uplink:  no
  Port-Security:         yes
  Dot1x:                 yes

Switch1#sh fabric status

 slot  channel speed module   fabric   hotStandby  Standby  Standby

                     status   status      support  module   fabric

    3        0   20G     OK       OK   Y(not-hot) 

    3        1   20G     OK       OK   Y(not-hot)

    4        0   20G     OK       OK   Y(not-hot)

    4        1   20G     OK       OK   Y(not-hot)

    5        0   20G     OK       OK          N/A

    5        1   20G     OK       OK          N/A

Use Cisco vWLC in VMWare Workstation

Cisco offers WLC virtual machine. Here are a few good online resources to get you started:
http://jeensern.blogspot.com/2012/09/cisco-virtual-wireless-lan-controller.html
http://www.labminutes.com/wlc0001_vwlc_7_3_vmware_installation
http://www.labminutes.com/blog0003_vwlc_7.3_installation_caveats
And here is the Cisco's deployment guide:
http://www.cisco.com/en/US/products/ps12723/products_tech_note09186a0080bd2d04.shtml

If you are using VMWare Workstation, you will need to download the latest VMWare OVF Tool and convert the ova file to virtual machine files.

Once you fire up the vWLC in VMware Workstation, you will see "Press any key to use this terminal as the default terminal.". If you have not configured other way to ssh or telnet to the VM, press any key here.

If you see screen "freeze" with "kernel direct mapping tables up to 100000000 @ 8000-d000", it is OK. The WLC is running fine and you can still access it though CLI or HTTPS.
Alternatively, you can

•  press any key at above stage, 
• press ESC key at following screen to get more booting options
• Or let it boot to primary image by default

You will see log in prompt soon:

Update:

Played with newer version of the vWLC and here is an update on the setup:

Device/Software List:

• PC with dual NICs
• WMware Workstation 11
• vWLC 8.0 and 8.3
• Cisco 3560 PoE Switch
• Cisco Thin AP

NIC configuration:

• The 01BottemTP is the first NIC, which has internet connection.
• The Local Area Connection 2 (Realtek PCIe GBE) is the second NIC and it supports VLANs.
• I created 3 VLANs using the second NIC's utility program and renamed them as VLAN11, 12 and 1 for easy identification purpose from original name Realtek Virtual Adapter 1, 2, & 3 in Windows 7. 

VMware Workstation Setup:

• In WMware Workstation Virtual Network, assign VMnet0 to Realtak Virtual Adapter 3.
• Assign VMnet1 to the second physical NIC.

Once the OVA file was imported to WMware, assign the first Network Adapter to VMnet0 and the second Network Adapter to VMNet1.

vWLC Setup:

The key point to initially configure the vWLC is highlight below:

• Select default for Service interface IP Address configuration
• Management Interface VLAN ID is 0

Switch Configuration:

Connect the PC's second NIC to a trunk port - Gi0/1

DHCP Relay on CheckPoint R75.20

In case of requesting IP address from a DHCP server sitting behind a different FW interface, DHCP relay needs to be configured. In following example, DHCP server is in internal network and corresponding firewall interface is eth0. The DHCP client is in a DMZ and the matching firewall interface is eth1.
Step 1: Enable DHCP Relay:

1. SSH to FW and run "sysconfig"
2. Select option 8: "DHCP Relay Configuration"
3. Define DHCP server IP address in "DHCP servers list"
4. Select both eth0 and eth1 interface in "Relay via interfaces"
5. Enable DHCP relay

Step 2: Configure Firewall Rules

1. Open SmartDashborad and create following node or network objects:
1. Server_DHCP - a node with DHCP server's IP address in the internal network
2. DHCP_255.255.255.255 - this is a node with 255.255.255.255 address
3. Network_X.X.X.X - this is the network of the DHCP scope
2. Create following three rules:
1. (source) Server_DHCP, (destination) Firewall object, (services) dhcp-relay && dhcp-rep-localmodule && dhcp-req-localmodule, Accept, Log
2. (source) Network_X.X.X.X, (destination) Server_DHCP, (services) dhcp-relay && dhcp-rep-localmodule && dhcp-req-localmodule, Accept, Log
3. (source) Any, (destination) DHCP_255.255.255.255, (services) dhcp-relay && dhcp-rep-localmodule && dhcp-req-localmodule, Accept, Log

dhcp-relay and dhcp-req-localmodule is UDP port 67 and dhcp-rep-localmodule is UDP 68. They should be predefined.

EEM Applet

Our network monitoring program detected  ICMP latency from a few WS-C3560CG-8PC-S switches recently. The "show process cpu sort" shows normal output. However, the "show process cpu history" indicates CPU utilization was always 60% and spike over 90% from time to time. To capture which process is the culprit, Cisco recommended to use following EEM (Embedded Event Manager) applet:

event manager applet high-cpu

!

event snmp oid 1.3.6.1.4.1.9.9.109.1.1.1.1.3 get-type next entry-op gt entry-val 80 poll-interval 5 exit-time 500 maxrun 600

!

action 0.1 cli command "enable"

action 0.2 syslog msg "TAC - Capturing high cpu information to flash:"

action 0.3 cli command "term length 0"

action 1.1 cli command "show process cpu sorted | redirect flash:eem-cpu1.txt"

action 1.2 cli command "show interface | redirect flash:eem-interface1.txt"

action 1.3 cli command "show interface stats | redirect flash:eem-stat1.txt"

action 1.4 cli command "show ip traffic | redirect flash:eem-traffic1.txt"

action 2.1 cli command "show process cpu sorted | redirect flash:eem-cpu2.txt"

action 2.2 cli command "show interface | redirect flash:eem-interface2.txt"

action 2.3 cli command "show interface stats | redirect flash:eem-stat2.txt"

action 2.4 cli command "show ip traffic | redirect flash:eem-traffic2.txt"

What this applet will do is when CPU process is over 80%, it will write those show output to text files in flash. This applet did capture the "virtual exec" was running over 90% of CPU process. 

Just felt this is a cool utility to share. You can find more EEM samples here: http://www.cisco.com/en/US/docs/switches/datacenter/sw/5_x/nx-os/system_management/configuration/guide/sm_eem_events_and_examples.html

BTW, the root cause of the high CPU utilization is a bug in the IOS. Upgraded to c3560c405ex-universalk9-mz.122-55.EX3.bin seems bring down the utilization to 20%.

Checkpoint Firewall lab with VMware

I am new to Checkpoint Firewall and just want to share my experience on setting up a Checkpoint Firewall lab with VMware workstation. This lab will have two management stations and three firewalls. you can configure the two management station and the first two firewall as HA later.

1. First, download Check_Point_R75.20.Splat.iso from Checkpoint site.
2. Second, create one VM in WMware Workstation and call it MGMT1. This is a Red Hat Linux 5 VM. You just need one virtual NIC for now.
3. Boot the VM with the iso file you downloaded and install the Security Platform. The detailed instruction can be found here: http://www.sysadmintutorials.com/installing-check-point-r75-secureplatform-tutorials/ and you will need to stop at step 11 of the second
4. Turn off the VM and use VMware to clone this VM as MGMT2, FW1, FW2, and FW3.
5. You will need to add two more virtual NICs (in different VMNet) to FW1 and FW2, and one more virtual NIC to FW2.
6. Here is the tricky or annonying part: all your VMs first NIC will have same MAC address. To change that, turn on each VM, login as admin with default password (admin). Enter "expert" at the prompt, and enter "admin" as initial password again. Using VI to open "/etc/sysconfig/netconf.C" and "/etc/sysconfig/netconf.C.keep" and modify the MAC address of the first NIC.
7. After the change, reboot the VM, log in as admin, and enter "ifconfig" to verify each NIC has unique MAC address. (I have spent some time to change the MAC address using VMware KB 507: http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=507, but that did not work. My colleague Larry helped me out on step 6 & 7 here.)
8. On MGMT1, enter "sysconfig" to install Security Management - continue the steps in the above tutorial. This will be the primary management station.
9. On MGMT2, follow the same step to install Security Management, except this will be the secondary management station.
10. One the rest of the VMs, use sysconfig to install Security Gateway - those will be the actual firewall.

Now you have a few raw machines to start learning Checkpoint FW. Have fun and good luck!

99% != 100%

Adding a new switch to existing network is relatively an easy task, perhaps even more so in a VTP transparent domain. Spanning tree concern is also straight forward in this case - just need to make sure the new switch has proper priority value so that it won’t become the new root. However, Murphy’s Law still applies and here is a network diagram of a recent case:

• AGG_SW1 and AGG_SW2 are aggregation layer switches and they are connected to Core switches (omitted here)
• SW3 – SW 5 are access layer switches
• SW6 is just added to SW5 (port 3)
• VLAN70 is defined on the core switches and need to be extended to SW6
• All the switches in a transparent VTP domain and root switch is at Core

Before SW6 was added, VLAN70 is verified on AGG_SW1, SW3 – 5, and it was allowed on the trunk ports. The port 3 of SW5 was an access port in VLAN70 and connected to a Kiosk machine. Because the Kiosk machine got correct IP and worked fine, it is logical to assume VLAN70 was propagated to SW5 just fine. I was about 99% sure that if SW6 connects to SW5 and trunk ports configured properly, VLAN70 should be working fine on the new switch. With that in mind, following steps were taken:

1. The Kiosk machine was disconnected from SW5.
2. The port3 was converted to a trunk port and SW6 was attached and properly configured.
3. The Kiosk machine was re-connected to an access port in VLAN 70 on SW6 as the access ports were run out on SW5.
4. The Kiosk machine was also verified working fine at its new port.

Simple enough?  

The only problem was a week later while I was out of office, end user reported VLAN70 on SW6 was not working. My colleague jumped in and found the cause - VLAN70 is pruned on port 2 of AGG_SW2 and SW6 lost the VLAN. After a little bit investigation, I found following:

• Configuration achieve shows VLAN pruning is inconsistent on port 2 of AGG_SW1 & 2 - on AGG_SW1, it was allowed and on AGG_SW2, it was pruned.
•  Port 2 on SW4 is in spanning tree blocked mode at the moment.

Based on all the facts, I suspect:

• There might be some spanning tree changes recently and layer 2 topology changed
• Or, the Kiosk might have been using wireless connection since day one, even though its Ethernet connection in VLAN70 is active – I still need to verify this.

Either way, VTP pruning and Spanning tree add complexity into a simple configuration task.  A 99% sure of a configuration, is not equal to a 100% working configuration.