Cisco 5508 WLC 5 min timeout bug

Ok, don't you love it when you've been struggling to troubleshoot something that just doesn't make any sense, then you start grasping at a straw and it starts coming together. So, we have 4 devices. Three existing 4404 cisco wireless lan controllers. I come along with a new and pretty 5508 to add to the bunch. Configure it so it has all the same vlans/ssids/settings/mobility groups/etc. as the others, bring up some access points on it, everything looks great.

So, i go log into the old controller, take a building worth of APs and add the new 5508 to the top of their "High Availability" list, let them swing over on their own. Walk over to the building, log in for a minute, everything looks great, project successful right?

Hour later I start hearing reports about people in that building on the 5508 getting kicked off every 5 minutes. I hurriedly swing all the APs back to the old 4404s and problem goes away.

Ok, not sure why the testing didn't bring this up but lets bring up a test AP at my desk. I stream video on my laptop beside me for 4 hours before deciding there isn't a timeout issue. I swing a single AP over to that problem building and walk over to test with my laptop, make sure there actually is a problem. Sure enough, every 5 minutes I have issues. Still says I'm connected on laptop and controller, but cannot ping gateway, cannot get anywhere. Reconnect/click repair on my wireless icon and I'm good to go (for another 5 minutes). It doesn't even ask me to authenticate again so I know the web authentication isn't getting timed out, BUT, I go ahead and change the idle timeout to several hours (since it is 300 seconds, I tried to avoid seeing 300 anywhere), and changed the user timeout to 8 hours. I peer through looking for any other 300 second timer that might be giving me issues. Go ahead and test, still nothing wrong at my desk with the test APs but still timing out in the problem building...

verify there are no ACLs that might be blocking or different for the problem building versus my test lab building.

Finally I start grasping at something, the APs that I am testing with are booting straight to the 5508. The APs swung over using high availability do not reboot but simply authenticate to the new controller.

The next time I swung the APs over in the production building I chose to reset AP after changing the primary high availability. To my delight i am now typing this on a non-timing out connection. I also verified there are identical software versions on all controllers, so there shouldn't be problems moving between them... I should probably let TAC know about this little bug.


*Before any tells me not to test on a production network, #1, I did test in the lab first and didn't find anything #2, it's spring break so there are only a handful of students on campus anyway, far from the thousands of wireless clients I would have on a typical school day*


PROBLEM CAME BACK TODAY- think I found the real problem now (and this time it makes sense). Two of my controllers were using the same AP multicast address!!! #facepalm APs aren't going to like getting updates from two different WLCs at the same time, periodic updates must come at a 5 minute interval. I think this will actually be the solution, will update if for some reason it is not.

ASA prompts

SO USEFUL!!! If you manage redundant ASA's, read @aconaway 's post on changing your prompt to include the active/standby and primary/secondary state of your firewall to the prompt. Click Here

You might be a network guy if:

You might be an Network Guy If

you know more ip addresses than phone numbers

You regularly mock TV shows for using technology that isn't part of the feature set available on the devices they have

You correct people who mix up Megabytes and Megabits

You waited eagerly for wireless N to be approved officially.

You can explain everything in your life using 7 layers

You tell people not to use TKIP because of it's security flaw

You think people should be able to do without DNS for a day, just use IP addresses...

You follow your wife around shopping retail stores and spend your time skimming the ceilings for their APs and mapping out a heat map of the store in your head

you know what TCP/IP stands for, not to mention DNS, HTTP, SNMP, BGP, OSPF, WPA, and DHCP - Sometimes you wonder if you know more acronyms than words

You've known what IPv6 was for years

cmd, telnet, and ssh are useful everyday tools, not just black boxes

Linus Torvalds comes up in everyday conversation

You know jokes about DHCP and LSAs

You cringe when you have to use a Gui to configure a switch or router

Your Amazon wish list consists of routers and ASA firewalls

Dealing with Tier 1 tech support makes you pull your hair out.

You have read the NSA's security best practices

The routing protocol in your house changes daily depending on what you have been reading

You know what a nibble is

You know what 1000 Terabytes is called

You can intelligently discuss how Egypt shut off their Internet to the country

Five types of OSPF Packets

These may be simple but are likely to come up again and again in studies, practice tests, and perhaps the real thing.

Name the five OSPF packet types:
Hello - Discovers neighbors/builds adjacencies
Database Description - Checks to see if databases are synchronized between neighbor routers.
Link-State request (LSR) - request a specific link state record(s) from another router
LSU - Sends the specific link state record(s) requested by the LSR
LSAck - Acknowledgment of all the above packet types

The reason OSPF needs an Ack packet is because it does not use TCP or UDP, the above packets are encapsulated directly into IP.


What does a Router do when it receives an LSU (link state update)

1. If LSA entree does not already exist: add entry to LSDB, send LSAck, Flood update to other routers, run SPF, updates routing table

2. If LSA already exists with same sequence number: ignores

3. If LSA already exists with but new has later sequence number: add entry to LSDB, send LSAck, Flood update to other routers, run SPF, updates routing table

4. If LSA already exists and new LSA has older Sequence Number: sends LSU to sender with newer update


What is a Drother?
A non-DR and non-BDR on a segment - still gets a two-way adjacency state.

How does a router verify Bi-directional communication?
Sees itself listed in Hello Packet of Neighbor

OSPF's multicast address - (KNOW THIS!!!!!, Not just for the test, but for every desktop support technician who wants to send multicast images over the first few multicast addresses!!!)
224.0.0.5 - used for hello packets


Hello and dead intervals must match to create an adjacency.
By default: hello - 10 sec / Dead - 4x hello

Once a DR and BDR have been selected, any future routers joining that segment will only form adjacencies with the DR and BDR, not any DROTHERS.

Comcast vs Level 3

Because tech blogs and twitter have EXPLODED on the subject, I thought I would add my useless two cents. For those who haven't heard, comcast is having a dispute with level 3 over whether level 3 should pay them to add more links into their comcast network.


Here are the facts.
1. Formerly Akamai (CDN - Content Delivery Network) was the primary deliverer of netflix streaming video.
2. Akamai was/is a customer of comcast and paid them for the bandwidth they used.
3. Level 3 was a carrier network that had a free agreement with comcast to trade a "relatively" equal (2:1) amount of traffic between their networks.
4. Level 3 has gotten into the CDN market.
5. Akamai lost the netflix account to limelight and Level 3, two separate CDNs
6. Level 3 communications != L3 communications
7. Level 3 expects the netflix streaming will increase the amount of traffic traveling onto comcast's network to increase to a 5:1 ratio (2.9 terabits/s increase)
8. Level 3 does not think it should have to pay, comcast does


I see this ultimatly as two businesses bickering over money, the usual, but I see two arguments, for and against.

Against Comcast:
I pay you (comcast) a lot of money to provide me 12Mb/s of whatever (legal) internet content I desire to upload or download. It is your end of the bargain to have enough bandwidth to other parts of the internet to provide that in a reliable fashion. You should be taking the money I pay you to make sure ALL your links to outside carriers will not be saturated if you expect I will desire content from that part of the internet. This has nothing to do with netflix, if comcast did not have enough bandwidth to support it's customers streaming from youtube, CNN, or something else, it should be comcast that needs to add bandwidth to let more content onto it's network, as comcast is the requester. The reason I choose to continue to pay comcast and not your competitor, is that I believe you have less over-saturation and can provide the content I want.
As a network engineer, if my end users need more bandwidth into my network from a specific entree point, isn't it my responsibility to increase that bandwidth?

Against Level 3:
You made an agreement with comcast to allow 2:1 bandwidth ingress/egress from that point of your network. By adding 2.9 Terabits/s of bandwidth, you have rendered the previous contract void and need to re-negotiate. Comcast does not wish to renew the contract without raising the price. Either pay up or find a different carrier/backbone network you can connect to which will make it's own connection with comcast.


Honestly, I don't see which is a better argument, possibly Comcast's. I really don't see that this has anything to do with "net neutrality" as Level 3 was claiming though.