Monday, December 27, 2010
Five types of OSPF Packets
Name the five OSPF packet types:
Hello - Discovers neighbors/builds adjacencies
Database Description - Checks to see if databases are synchronized between neighbor routers.
Link-State request (LSR) - request a specific link state record(s) from another router
LSU - Sends the specific link state record(s) requested by the LSR
LSAck - Acknowledgment of all the above packet types
The reason OSPF needs an Ack packet is because it does not use TCP or UDP, the above packets are encapsulated directly into IP.
What does a Router do when it receives an LSU (link state update)
1. If LSA entree does not already exist: add entry to LSDB, send LSAck, Flood update to other routers, run SPF, updates routing table
2. If LSA already exists with same sequence number: ignores
3. If LSA already exists with but new has later sequence number: add entry to LSDB, send LSAck, Flood update to other routers, run SPF, updates routing table
4. If LSA already exists and new LSA has older Sequence Number: sends LSU to sender with newer update
What is a Drother?
A non-DR and non-BDR on a segment - still gets a two-way adjacency state.
How does a router verify Bi-directional communication?
Sees itself listed in Hello Packet of Neighbor
OSPF's multicast address - (KNOW THIS!!!!!, Not just for the test, but for every desktop support technician who wants to send multicast images over the first few multicast addresses!!!)
224.0.0.5 - used for hello packets
Hello and dead intervals must match to create an adjacency.
By default: hello - 10 sec / Dead - 4x hello
Once a DR and BDR have been selected, any future routers joining that segment will only form adjacencies with the DR and BDR, not any DROTHERS.
Friday, December 3, 2010
Comcast vs Level 3
Here are the facts.
1. Formerly Akamai (CDN - Content Delivery Network) was the primary deliverer of netflix streaming video.
2. Akamai was/is a customer of comcast and paid them for the bandwidth they used.
3. Level 3 was a carrier network that had a free agreement with comcast to trade a "relatively" equal (2:1) amount of traffic between their networks.
4. Level 3 has gotten into the CDN market.
5. Akamai lost the netflix account to limelight and Level 3, two separate CDNs
6. Level 3 communications != L3 communications
7. Level 3 expects the netflix streaming will increase the amount of traffic traveling onto comcast's network to increase to a 5:1 ratio (2.9 terabits/s increase)
8. Level 3 does not think it should have to pay, comcast does
I see this ultimatly as two businesses bickering over money, the usual, but I see two arguments, for and against.
Against Comcast:
I pay you (comcast) a lot of money to provide me 12Mb/s of whatever (legal) internet content I desire to upload or download. It is your end of the bargain to have enough bandwidth to other parts of the internet to provide that in a reliable fashion. You should be taking the money I pay you to make sure ALL your links to outside carriers will not be saturated if you expect I will desire content from that part of the internet. This has nothing to do with netflix, if comcast did not have enough bandwidth to support it's customers streaming from youtube, CNN, or something else, it should be comcast that needs to add bandwidth to let more content onto it's network, as comcast is the requester. The reason I choose to continue to pay comcast and not your competitor, is that I believe you have less over-saturation and can provide the content I want.
As a network engineer, if my end users need more bandwidth into my network from a specific entree point, isn't it my responsibility to increase that bandwidth?
Against Level 3:
You made an agreement with comcast to allow 2:1 bandwidth ingress/egress from that point of your network. By adding 2.9 Terabits/s of bandwidth, you have rendered the previous contract void and need to re-negotiate. Comcast does not wish to renew the contract without raising the price. Either pay up or find a different carrier/backbone network you can connect to which will make it's own connection with comcast.
Honestly, I don't see which is a better argument, possibly Comcast's. I really don't see that this has anything to do with "net neutrality" as Level 3 was claiming though.
Wednesday, November 10, 2010
World Politics
Dilma Rousseff, Brazil's president-elect: "The last time there was a competitive devaluation of currencies it ended up where it did, in the second World War."
Personally a little uncomfortable with where our leadership is taking us, with Obama continuing to ignore (or often worse) our long time allies (Isreal, France, Germany, UK), unfathomable debt, and the Fed pursuing very shortsighted solutions, I think this world could be a very different place in the next few years.
(although my caveat is that I am not an economist, political scientist, or have any advanced education specifically relating to these issues, so take it with a grain of salt)
Friday, October 15, 2010
ASA access lists
Ok, look through my ACLs, make sure the DNS lines are still there, pasted correctly etc. Everything looks ok, so I run packet tracer and it ends with: blocked by access-list .... implicit deny.
Now, my only grudge about packet tracer is that it doesn't list WHICH ACL blocked it, but I'm pretty sure it should only be hitting the one. I try adding a ip any any on the end of the ACL... Still blocked. I try adding an permit ip any DNSIP to line 1 of the acl... Still blocked. Starting to want to pull my hair out, I know it should be hitting this ACL and I have to correct entries!
(Now this dmz is also my wireless network, so I'm doing some troubleshooting on the WLCs and laptops etc. to be sure but I can't imagine it is anything other than that ACL I just changed on the firewall. It had worked after changing the subnet mask on both the interface and nat pool...
So finally, I just sit back and decide to stop doing pipes and look at the entire show run. I crawl through my entire ASA config line by line until at the bottom I notice, the ACL for that dmz isn't applied.... Apparently when you remove all lines of the ACL it removes it from the interface (and fails closed to an implicit deny). So I re-add the ACL to the Interface and Magically everything is working again!
So - when you remove all lines of an ACL - the ASA also removes the
access-group dmzACL in interface DMZ
command from your running config as well.
Lesson learned!
Thursday, October 14, 2010
ASA 8.3
and of course the migration guide: http://www.cisco.com/en/US/docs/security/asa/asa83/upgrading/migrating.html to start seeing what all is changing.
Is there a reason Cisco now has us permit incoming traffic to the inside address rather than the public IP? No more Nat0??? I feel like I am going to need to learn natting all over again.
Friday, October 1, 2010
I love marketing
"Able to move the entire printed library of congress in a second" ... ok, so is that in txt files, pdfs, e-pubs. This one is a little better but once again, it sounds like a good bit but I have no idea how many books that actually is, how many gigs or terabytes you are moving.
322 Terabyte performance - Awww, now there is something I recognize, but of course they aren't going to mention any specifics beyond their biggest number! What exactly is that a measure of? How are you defining "performance"?
Aw well, I suppose if I was a sales guy I might not be able to do much better, it is why marketers should always have a tech with them.
Thursday, September 30, 2010
KISS Troubleshooting
%APF-4-REGISTER_IPADD_ON_MSCB_FAILED: apf_foreignap.c:1281 Could not Register IP Add on MSCB. MSCB still in init state.
The reason you will see this error is pretty simple. You are getting packets for or from a client (such as ARP requests) and the client doesn't have an IP address yet. Without an IP address, the "MSCB" is said to be in the init state.
%APF-1-CHANGE_ORPHAN_PKT_IP: apf_foreignap.c:
The registered IP address changed leaving packets "orphaned". Note, I saw this in regard to clients not being able to get DHCP and the changing IP address was 169. addresses. This is also a common error with mac clients trying to hold on to old IP addresses.
%DHCP-3-BIND_SRPORT_ERR: dhcp_support.c:374 Binding service port failed.
There is no info on cisco about this error (other than check the bug tool, which has nothing) and I thought this was where my problem lied. The controller was having problems binding the DHCP addresses to the clients. Well low and behold, I found it wasn't that the controller was unable the DHCP response, this message means that there was either no response or an error response. In my case I had run out of addresses, the entire /22 was full. So now I get to enlarge the network and DHCP pool.
So in conclusion, if your clients aren't getting IP addresses, before you dig through obscure bugs and error messages, make sure you pool isn't used up!
Tuesday, September 28, 2010
P@ssw0rd
Monday, September 27, 2010
Use twitter with Geotagging?
Click here for Josh's article.
New Internet Wiretap legislation
Well first off, this would appear to be a terrible idea. Forcing companies to create backdoors and master private keys is just adding new avenues for attack.
Perhaps this is just one sided journalism, but I noticed they were not able to find any tech or industry professionals to quote in support of this bill, only against. All the "for" quotes are by politicians. This is also an amazingly poor time to bring up something so fiercly unpopular just before November Elections.
Anyway, looking at the technical aspect:
Facebook, I'm sure they store every keystroke ever entered on their site for advertising purposes anyway, so they shouldn't have a problem.
Blackberry, I assume this only includes the encryption built into the default mail program. Users should be able to still encrypt their own messages with a key system if they have the technical knowhow, if blackberry was forced to ban personal encryption from their devices, well I would suspect every major corporation would ditch blackberry. Do you want any joe smoe at an ISP or blackberry corporate being able to view defense data being e-mailed between people at Lockheed Martin or Boeing? I assume this would mean government oversite committees would have to be created (and funded with our tax dollars) for every company dealing with government contracts and want to use a blackberry?
Skype: Ok, so how would this work. Skype is not like vonage or a regular phone company that sends everything through a central office, skype is peer to peer. After the call creation the users are talking to each other, not a skype server. So do wiretapped phones then get re-routed to skype servers? Wouldn't this be noticed and affect service? I'm sure people would make programs to alert them if Skype wasn't being routed directly to the end user.
Peer2Peer Messaging: Well, there is certainly a variety here, from IRC to ICQ to MSN Messenger to google to .... Microsoft and google might be able to keep track of messages fairly easy, but IRC? I can't imagine what it would take to get all the IRC servers around the world to listen to the FBI/NSA. IRC has been part of wars, coups, and rebellions without anyone being able to stop the flow of information. I doubt a US wiretap request for "fear someone might be a terrorist" is going to change that.
My personal oppinion (if you couldn't tell from the post) terrible idea, unpopular politically and socially, and a nightmare of new costs, both trying to implement something like this as well as the lawsuits/profit loss that would come from any new security holes that are created and used by malicious users. This could also destroy consumer confidence in online usages. (yes I just wrote usages because I was too lazy to think of a real word)
The governments only defense was: it worked out with cell phone companies despite fears. Well, lets just say that is comparing apples and oranges.
Cisco IOS Hints and Tricks: EIGRP myths debunked
Cisco IOS Hints and Tricks: EIGRP myths debunked: "Matthew Norwood performed a really thorough EIGRP research and unearthed a lot of myths around it, some of them coming from official documen..."
Thursday, September 23, 2010
Wireless Lan Controllers SNMP Traps
So I have an IP address and need to track the username to see what user snuck some bittorrent past my packetshaper block. I grep through all my syslog files and there are no IP addresses, even at the debug level. So now I had to set up an SNMP trap receiver on my linux box as well as syslog server to catch all the needed info.
I also might mention the SNMP traps are horribly ugly, probably purposely so that you will buy WCS or a cisco solution to make them readable. Oh well, it will be 100 years before we get the money for that!
Now if only I could get my logs to rotate properly...
Monday, September 20, 2010
Iperf - tool for every NetEngineer's Toolbox
Download the file applicable to your OS. You will need two installs, a source and a destination (Iperf will refer to these as server and client).
Pick one to be your server. Open a command window and browse to the Iperf file. Type: iperf -s
Done, you have an Iperf server running.
Go to machine 2, browse to where you placed the iperf file. To run a simple bandwidth test, just type: iperf -c x.x.x.x where x.x.x.x is the server’s IP address.
View the output, if it is a 100Mb link, you will probably see around 80-85 Mb/s speed and near a 100MB transfered. (If you are running fast, I have seen as quick as 98Mb/s, but that tends to be local to the same switch or nearby)
Now you can start playing with the extra switches. A common one might be:
iperf -c x.x.x.x -r -t 300 -i 2
here, we see -r will run both a regular test from the client to the server, but then reverse it and send traffic from the server to the client. Next we have -t 300 which tells iperf to run for 300 seconds, or 5 minutes. -i 2 informs the program to give you an output ever 2 second interval so you can see what is happening and see any spikes or slowdowns during the test rather than just a averaged output at the end.
Want to test udp traffic, you will need to add a -u on the end of the server command as well as the client command. That lets both sides of the link know to use UDP traffic rather than TCP.
If you are testing a production link, perhaps you don't want to use up the entire link and slow down everyones traffic. use the -b 10m command to set bandwidth to 10Mb/s.
-B x.x.x.x on both ends of the link will bind them to a multicast address, allowing you to look at your router and see the multicast joins and traffic flowing. Don't forget the -T for setting time to live numbers.
This data as pretty specific, and you can show your complainers/customers how well the network handles a solid gig of traffic, when their sluggish program using 50k of bandwidth is not a "slow network issue".
Pair this data with some colorful graphs showing how little they actually use your network and most people accept that the network isn't the problem.
Of course with your excellent customer service skills you have worded this all super politely and let them "arrive at their own conclusion" that there must be something else wrong than the network. Unless of course your gig interface is only able to push 2Mb/s of traffic, then you need to suck it up and figure out why your network is slow!
Thursday, September 16, 2010
Snort Struggles
Well snort is a headache of headaches. Attempting to install snort on a mac and get it to send to aanval is proving to be quite the feet. Mac tutorials for snort are … well lets just say none of them use even remotely the same method, and none seem to work. All instructing me to modify files that are not on the system anywhere or change lines of code that do not appear in the listed files… frustration ensues. On the bright side I am working with the server team to get me a Linux server, and the Linux instructions appear, at least right now, to be much much simpler. Although the fact that we already have a handful of mac minis to locate throughout the network as snort sensors means I will eventually have to get snort running on them… Oh well, a problem I can push off a few days. School starts Monday so other priorities trump IDS for the week.
To wet your appetite, I am about to implement some Cisco IP SLA monitoring, I can blog about how that goes, should be good times. Especially since HP procurve does not have any features that can do this, so I have to add cisco routers behind my procurves to support the needed feature...
Wednesday, September 8, 2010
OSPF packet types
Type 1 is a Hello Packet. This is how routers discover neighbors and build adjacencys between them.
The second type is a DBD or Database Description. This is used to synch the databases between two routers.
Third is an LSR, Link State Request. LSRs are used to request specific link-state records from a different router.
Fourth is an LSU or Link State Update. This is the link state records that were just requested in the LSR.
Fifth is the LSAck or Link State Acknowledgement. This is the reliable portion of the protocol, a LSAck is sent in response (acknowledged) to any of the above packets to let the neighbor know that the packet has been received.
Wednesday, August 25, 2010
Back from Vacation
Wednesday, July 21, 2010
Difference between Link-State and Distance Vector
Well, the biggest difference is the view of the network that each router has.
In a Distance Vector protocol, routers only see their neighboring routers, and only see the "distance" or cost passed to it from each "vector" or direction. In other words, a distance vector router sees a list of x routes with a cost on interface 1 and y routes with costs on interface 2.
Link-State protocols on the other hand use a bit more overhead and processing power. With a link state router, the entire (area or domain) topology is known by every router.
Picture it like this, Distance Vector is sitting at an intersection seeing the signs, turn left = 2000 miles to NY, Turn right = 1000 miles to portland, continue straight = 1700 miles to Dallas. All you have are distances and an arrow, you have no idea how straight or round-about the path may lead.
Link state is when you are looking at a map. You can see that turning left does indeed take you toward NY, but can see a slew of different roads and freeways that you can choose from. You make your own decision based on the entire topology of the road system.
The other difference between the two protocols is that link state sends advertisements only when a link changes, so if a router sees a link go up or down, it sends the changes.
Distance Vector sends regular announcements of it's paths and costs at a set time interval.
Wednesday, July 14, 2010
STP Topology Change Notifications
A. Well, the noticeable part is from the client's view. Portfast means the port does not move through the listening and learning states (30 seconds) which can mean delayed, slow, and even missed startup scripts.
The (arguably) more important point is the affect on your network. Every time a port comes out of blocking and moves to listening, learning, and forwarding, a topology change has occured. The switch sends a Topology change notification to the root switch and EVERY switch in your STP topology will shorten their cam table aging time to the forward delay time (default 15 seconds). This means every device in your STP network that isn't very chatty will have their mac flushed from the switch and need to be relearned. So moral of the story, use portfast on every client port to prevent a whole lot of spanning tree traffic and cam table flushes.
Q2. So describe this Topology Change Notification Process
A. Ok, a port on a switch changes state (we are assuming it is not in portfast mode), so a link comes up or a forwarding port goes down. The switch immediately sends a Topology Change Notice (TCN) out it's root port. (unless the root port was the link that just went down, then the switch will sit and wait to hear from the root or attempt to make itself root) All switches that receive the TCN forward it toward the root.
Once the root receives the TCN, first it sends an acknowledment to the originating switch, then it creates a packet with the Topology Change Bit Set, to notify all switches that there has been a topology change. This is flooded to all switches in the topology. When each switch sees the packet with the topology change bit set, it will lower it's bridge table aging time to the forward delay timer and pass on the topology change packet down the tree. The cam table flush is because the changing topology can mean mac addresses will be appearing on new interfaces. The switches don't yet know how the topology has changed, just that they need to flush any unused mac addresses to prepare for a change.
Ports will go through their regular STP states to create a new topology if one is required.
Q3. When does a port come out of blocking state?
A. If no BPDU (bridge protocol data units, how switches communicate STP) arrives within the max age time, OR if a port receives a BPDU with a better root "cost".
Remember, If a port is not a root port or a designated port, it will be in blocking.
Q4. How often are BPDUs sent
A. By default, 2 seconds
Thursday, July 8, 2010
Follow the Moon Strategy
The idea is that power consumption is, in many areas, cheaper at night/non-peak-usage times. So, you slide your virtual machines between datacenters every few hours all the way around the world to keep your power usage high only at "night".
To do this you need a good strategy to be able to keep an IP address moving all around the world between datacenters... (and the physical datacenters all around the world, not everyone has that!)
Local Area Mobility
Looking into it a little though I run into an old gem from the mid 90s (the whitepaper talks about how the new windows 95, with a DHCP client installed, can automatically get an IP address!). Local Area Mobility. Now this may not scale well, but for the problem of a few servers needing to be in the same subnet but a different physical location it works great. In fact it is so simple it blew my mind that I had never heard about or thought of such a thing.
The key is simply that routers use the longest prefix to determine a route. Solution - Host routes.
Local Area Mobility, when turned on, will listen and when a computer is turned on with a statically set IP address that doesn't match the interface's subnet, the router injects a host route (/32) into the routing table. It will also proxy arp requests so that clients in the same subnet (but different router) can still communicate. User attempts to connect to the server's IP, router looks up and finds the /32 hostroute preferred over the /24 or whatever and the packet is properly sent to the server.
I don't know why but that is still blowing my mind how simple it is.
http://www.cisco.com/en/US/products/ps6590/products_white_paper09186a00800a3ca5.shtml
Wednesday, July 7, 2010
Petition for an educational IOS version
Click Here or browse to http://etherealmind.com/petition/ to view and sign the petition.
Monday, June 28, 2010
EIGRP Equal and UnEqual cost load balancing
Friday, June 25, 2010
EIGRP Feasible Successor
Advertised distance is the cost that is being passed or advertised by the neighbor router. In my diagram, I am looking at how R1 can get to the 192.168.0.0 network. There are three paths, and each advertise a cost. The top neighbor is advertising 3000, the middle 2500, and the bottom 4100. These paths are added to the EIGRP topology table.
Feasible distance (FD) is once R1 has taken those advertised distances and added the local link cost. FD is used to pick the best route to the destination. In our diagram the lowest FD is the top route at 4000 (AD + Local Link Cost or 3000 + 1000). This route would be pulled from the topology table and inserted in our routing table.
EIGRP can also have a Feasible Successor (FS). This is a loop free route used as a "backup" immediatly if the successor (main route) goes down. By doing this immediatly, the network does not need to reconverge to choose a new best path. To qualify as a FS, the next hop router must have an ADVERTISED DISTANCE of less than the current successor's Feasible Distance.
In our diagram, the top route has been chosen as successor, so now we look at the other two options for feasible successor. Even though the bottom route would have the lowest FD total, it does not qualify for a feasible successor because the advertised distance of 4100 is higher than the successor's FD of 4000. Looking now at the middle route, the advertised distance, 2500, is lower than the successor's feasible distance of 4000. The middle route qualifies and will be chosen as the Feasible Successor.
Thursday, June 24, 2010
OSPF Authentication
In my example, I will use Fa0/0 as the area facing interface, and secret as my key. I used OSPF process 150... because it was a nice number.
First we have a clear text example. We go into our fa0/0 interface, set the key and area. Then we go into our ospf process and tell all of area 0 to use "authentication" which just means clear text here. Do this on each router and you are set to go.
R1(config)#interface FastEthernet0/0
R1(config-if)#ip ospf authentication-key secret
R1(config-if)#ip ospf 150 area 0
R1(config-if)#
R1(config-if)#
R1(config-if)#router ospf 150
R1(config-router)#area 0 authentication
Now we have nearly the same config but using an md5 (message digest version 5) key. Go into the interface and this time use the message-digest-key, give the key a number, say md5, and enter secret as our key.
**A useful thing to note, notice this is key 1. If our security policy said we needed to change keys every 90 days, we could create key 2 on all our routers, wait a minute and the routers will detect that they all have a newer matching key and start using it, then we can eliminate key one without ever losing a connection. If we just changed key 1, no matter how fast we could type, the neighbor adjacency would have to be re-established meaning our routing would be down momentarily.
R1(config)#interface FastEthernet0/0
R1(config-if)#ip ospf message-digest-key 1 md5 secret
R1(config-if)#ip ospf 150 area 0
R1(config-if)#
R1(config-if)#
R1(config-if)#router ospf 150
R1(config-router)#area 0 authentication message-digest
Finally, instead of setting authentication up for the whole area, we can set it individually on interfaces. Perhaps you only have two routers with an interface each that you want to authenticate because you have a switch between them that a malicious/misconfigured router could plug into and join your OSPF domain. We simply go into the interface and turn on authentication, choose a password, and join the area.
R1(config)#interface FastEthernet0/0
R1(config-if)#ip ospf authentication message-digest
R1(config-if)#ip ospf message-digest-key 1 md5 secret
R1(config-if)#ip ospf 150 area 0
Crossover T1 vs Cat5
I'll have to make myself a new cable or 2.
Network world has recently blogged about the best Routers and Switches for CCNP labs if anyone else is building a home lab.
Best Router for CCNP Prep in 2010 and Best Switch for CCNP Prep in 2010 and finally the full lab setup What to Buy?
Cat5 Cross over Cable
End 1
Pin – Description
1 – Green White
2 – Green
3 – Orange White
4 – Blue
5 – Blue White
6 – Orange
7 – White Brown
8 – Brown
End 2
Pin – Description
1 – Orange White
2 – Orange
3 – Green White
4 – Blue
5 – Blue White
6 – Green
7 – Brown White
8 – Brown
T1 Cross over Cable
End 1
Pin – Description
1 – Green White
2 – Green
3 – Orange White
4 – Blue
5 – Blue White
6 – Orange
7 – White Brown
8 – Brown
End 2
Pin – Description
1 – Blue
2 – Blue White
3 – Orange White
4 – Green White
5 – Green
6 – Orange
7 – Brown White
8 – Brown
*Note that this also means you lose your solid-stripe-solid-stripe pattern on one end.
Wednesday, June 23, 2010
OSPF area types Stub, Totally Stubby, Not so Stubby
Alright, I know I'm jumping around a lot here, but things come up in real life, in forums, in books, and in all the blogs I follow that gets me headed off on another track, so today I am talking about OSPF stub area types.
First, what are the types of OSPF Link State Advertisements? These are how OSPF routers communicate routes and route changes etc.
Type1 - Router - router announcing itself and lists links to other routers with metrics in the same area
Type2 - Network - lists what routers are connected by the broadcast segment.
Type3 - Net Summary - An Area Border Router summarizes one area, and sends it to other areas
Type4 - ASBR Summary - An ASBR summarizes external routes that are injected from another routing protocol
Type5 - AS External - Full network routes imported from other routing protocols
Type6 - Group Membership - A multicast advertisement that is not used in OSPFv3
Type7 - NSSA External - Routers in a not so stubby area (NSSA) send external routing information to the ABR to be redistributed (as type 5s to other areas)
There are 11 types, but these are the ones that affect stub areas for this discussion.
First, why use stub areas? Stub areas greatly minimize the traffic and routing table sizes of routers in these stub areas, increasing performance. (And area 0 can't be a stub of any sort)
First we start out with a stub area. A stub area allows Type 2 and Type 3 LSAs. Type 5 LSAs are not passed into the area. A default route is also passed in from the ABR.
Now, what does that mean, Type 2 and Type 3 LSAs allow the area routers to get inter and intra area routes. Routes in the same area are seen as usual, and the routes on the other side of your ABR is visible. Routes that are injected from other protocols, like static or EIGRP, are not advertised. To get to these, the router must use the default route.
**Remember - Stub= No type 5 LSAs and default route
Next we'll look at a Totally Stubby network. Only Type 2 LSAs and a default route are passed into the area (although technically the default route is a type 3... but thats the only one).
This means the routes inside the area are advertised, but all other destinations will use the default route. In other words, to a router in a Totally Stubby area, the ABR is the end of the world and everything not in it's own little area gets passed to the ABR to deal with.
**Remember - Totally Stubby - ONLY Type 2 LSAs and default route
Next we have Not-So-Stubby Areas. NSSAs have Type 2, Type 3, and Type 7 LSAs.
Type 7 LSAs are ONLY found in NSSAs. A NSSA is when you want your area to have external routes and be able to pass them to other areas, but not have other area's external routes imported into this area. In other words, local external routes are allowed, but not external routes from other areas. The ABR changes the Type 7 into a Type 5 when it passes it to other areas.
Cisco says: "NSSAs are more flexible than stub areas in that an NSSA can import external routes into the OSPF routing domain, thereby providing transit service to small routing domains that are not part of the OSPF routing domain."
**Remember - NSSA - Type 2, 3, and 7 LSAs with default route
Finally, we get to Totally Not-So-Stubby Area or officially called Not-So-Stubby Totally Stubby Area (seriously). This combines the attributes of a Totally Stubby and a NSSA area. First you have the Totally Stubby in which Type 3 and 4 advertisements are not passed into the area (Everything outside of the area is found through a default gateway), but you can still connect an external route to the area by Type 7 LSA.
Why use this? Perhaps you have a remote site that you want to connect back by default route only, but it also has a backup DSL or Cable modem for local internet.
**Remember - Totally NSSA - Only Type 2 and 7 LSAs and default route.
Monday, June 21, 2010
STP- Difference between Link Cost and Port Priority
VOCAB:
STP: Spanning Tree Protocol - Layer 2 loop avoidance protocol
BPDU - Briding Protocol Data Unit: Used by Spanning tree to communicate and determine the tree structure.
Spanning tree is a layer 2 protocol that prevents loops from forming. By sending BPDUs out all switch ports, a layer 2 device communicates with it's neighbors and a "tree" structure is developed. One device is determined as the root and all other switches pick a single lowest cost port, deemed the root port, to connect toward that root switch. All other ports that connect to the root can then be put into a blocking state. By doing this, a loop should never be able to develop.
To create the tree that fits your environment best, you can change all the settings. First set the switch priority to make sure the switch you want will be the root, then you can manipulate the path costs to determine the desired tree structure.
After the root switch is chosen, BPDUs are sent out from the root switch, starting with a cost of 0. As each switch recieves the BPDU, it adds the cost of the link the BPDU just traversed and floods it out the other ports. If a switch receives two or more BPDUs, the following is checked:
Lowest Root Bridge ID - part of choosing the root switch
Lowest Root Path Cost - The link "cost" to send traffic to the root switch as modified by all switches in the path
Lowest Sender Bridge ID - Used as a tie-breaker in the case of identical path costs
Lowest Sender Port ID - Used if all above methods are a tie
The lowest cost option is chosen as a root port. Ports connecting to another switch's root port are designated ports. All other ports leading back to the root can then be blocked.
I know that is a very quick overview of STP but I just wanted to focus on link cost vs port priority. I will likely delve further into other parts of STP at a later time.
So, why not just change path costs or just change bridge or port IDs to manage what your spanning tree looks like and what port is used as the root port. The answer lies in looking at the downstream switches rather than the switch you are configuring. If you have two equal cost connections to the root, you can change the cost of one to be higher or lower, manipulating the tree to your design. But you need to remember that path costs are added to the root path cost in each BPDU, bridge and port priorities are not. This means if you change the path cost to a notably lower cost, the switches further out on the branches will be offered a lower root path cost. If you instead choose your root port by changing the port priority or bridge priority, none of that information is carried along to the later switches.
Changing port/switch priorities can be a much administratively cleaner setup. If you have 50 100MB links, it would make sense for them all to have a cost of 19, or a network wide cost of your choosing rather than a mismatch of costs on each individual switch. That could make determining your path costs much more of a task.
Changing path costs can be useful in situations such as edge switches with uplinkfast. In order to ensure the switch is always chosen as a "leaf" node and is never traversed in a regular STP topology, the port costs are all increased by 3000. This is passed on to any switches further downstream making this switch very undesirable as part of the root path. This wouldn't work if the priorities were the only things increased as they are not passed on to other switches.
Friday, June 18, 2010
EIGRP BASICS - Split Horizon/Route Poisoning
Update packets contain route change information (sent reliably, needs ACK).
Query packets are sent when a router does not have a feasible successor which is a next hop router for a backup path (sent reliably, needs ACK).
Split Horizon is a way to prevent (minimize) routing loops. The logic is that if a router uses interface A to get to a destination, there is no reason to advertise that route out intA.
For example, if network 10.1.1.0/24's next hop router is located out fa0/0, then the router will not send update packets containing 10.1.1.0/24 out of fa0/0, only out fa0/1, fa0/2, etc.
If you think about it, if router A is your next hop to a network, why would you need to tell router A how to get to itself?
Split horizon is on by default on all interfaces.
If a router changes the interface it uses to get to a destination, it momentarily turns off split horizon and commits route poisoning. Route poisoning is when the router, in order to stop other routers from using its old route/cost that it just changed, floods out route unreachable packets. This causes other routers to flush that old route out of their routing tables so that the new route can now be successfully added.
Thursday, June 17, 2010
EIGRP Basics - Hello Packets
IGRP: Interior Gateway Routing Protocol
EIGRP: Enhanced Interior Gateway Routing Protocol
AS: Autonomous System - a group of routers share an AS so they know they are part of a routing group or system and should share routes and form relationships within that group. A router can be a member of multiple AS's but must inject routes between them if the two groups wish to share common routes.
Cisco loves to ask questions on it's own proprietary protocols, so don't skip over something just because it isn't the most common protocol out there. (although it looks like IGRP is getting phased out)
EIGRP Hello Packets:
EIGRP sends hello packets to the multicast address 224.0.0.10 in order to create and maintain Neighbor relationships. These are not "reliable" packets and so do not expect an acknowledgement. When a router receives an EIGRP hello packet, it first checks three items. Assuming it is configured with:
1. The same AS
2. The Primary IP address is in the same subnet
3. The metric-calculation constants match (how it determines route cost/metrics)
Then a neighbor adjacency relationship will be established.
Over a fast link (greater than T1) Hello packets are sent every 5 seconds (default). Over a slow link (T1 or slower) Hello packets are sent every 60 seconds (default).
Hello packets include the hold time. This is the time a router waits, and if it doesn't receive any EIGRP packets from the neighbor switch, flushes the routes and attempts to re-converge deciding the neighbor is down. Hold time, by default is 3 times the hello interval. This means fast links have a hold time of 15 seconds, while slow links are 180 seconds. Remember that changing the hello time does NOT automatically change the hold time, you need to remember to change both.
Issue the command show ip eigrp neighbors to see if your hello packets have formed an adjacency.
Monday, June 14, 2010
Floating Static Routes
Say you have a primary route, using a routing protocol such as OSPF and you want to input a manual static backup route. Well, the administrative distance of a static route is always 1, so your backup route would always be chosen over the primary OSPF route which has an administrative distance of 110.
Floating static routes are when you manually set the administrative distance of a static route to be higher than your routing protocol. This way the route "floats" above the primary route until that route goes down, only then will the static route be entered into your routing table. Simple concept with a fancy name.
Enter your static route with the command:
ip route [destination net] [subnet] [next hop] [Admin Distance]
ip route 172.16.1.0 255.255.255.0 192.168.1.1 112
Notice the administrative distance I chose was 112, higher than OSPF's 110, so that the OSPF route will be used while it is active. You can choose any admin distance up to 255.
Friday, June 11, 2010
Port Based Authentication (Dot1x auth)
Why do you need port authentication? Well suppose someone brings in a laptop from home full of virus', or a malicious user decides to unplug a work computer and use the network cable to plug in their own laptop to attack your network. (I know everyone has perfect, aware, tech savy, completely trusted employees and users, but this is just theoretical)
Wednesday, June 9, 2010
Cisco QOS trust boundaries
Vocab:
QOS - Quality of Service - Prioritizing packets so that time or delay sensitive applications move through the network first.
COS - Class of Service - Layer 2 QOS, also referred to as IP Precedence, what a switch uses to determin priorities (0-7)
DSCP - differentiated services code point - Layer 3 QOS, what routers use to determine qos priority. Six bits give you 63 levels, but I believe only 56 are usually used.
-Devices keep a map to translate COS and DSCP values, switching between the two as needed, but every device can have a different map, so a COS value of 5 is not always translated to DSCP 46.
Ok, since I got qos trust questions both times I took the BCMSN, I thought I had better cover it. You should be able to look at a diagram and write/match a config or vice versa.
Since VoIP phones have a jack for a PC in the back, the question of trusting QOS values comes into play. The options are to re-write all the QOS values on the switch, therby not trusting the phone or the PC. You can trust the phone's default QOS tag, or tell it what QOS tag to use, therby trusting the phone to act correctly. Last, if you have a PC plugged in the back of the phone that needs special QOS for an application, you can instruct the phone to pass on the PC's QOS tags without changing them, and trust everything.
Case one is to simply re-write all QOS values at the switch port. You can do this by choosing a cos value followed by the command instructing the switch to override all qos tags with the value you just chose.
mls qos cos [#]
mls qos cos override
Case two is the default. You can simply let the switch do it's thing, and it will trust the qos values of the phone, while the phone rewrites any qos values the PC may attempt to use. The other option that results in the same trust boundary is to instruct the phone to use a particular COS value. Even though the switch is instructing the phone to use a particular COS value, you are still trusting the phone to listen and act as instructed, hence the trust boundary after the phone.
mls qos trust cos
switchport priority extend cos [#]
Case three is not as common. This instructs the phone to not re-write any qos values on the packets coming in from the PC, and to pass them on as is so that the switch uses those qos values. Only extend the trust to the PC if the computer is under your control/management and has a specific need for an application to have qos priority, such as a soft phone.
switchport priority extend trust
Monday, June 7, 2010
Voice Vlans
Note that cisco phones expect CDP to be enabled on the switch ports as they use it for both choosing the correct PoE levels and to negotiate a special dot1Q(802.1Q) trunk.
So there are 4 modes to set up a switch port you expect to plug a phone into.
The Voice Vlan command is: switchport voice vlan [vlan-id | dot1p | untagged | none]
1. First you can just use a regular access port. (in voice vlans, it is referred to as none) In this mode, both the phone traffic and pc data both land on the same access vlan and there is no way to distinguish between the two. Two things to note, because the traffic is inter-mingled you have a security risk as well as having no ability to provide QOS (quality of service) priority to only the phone. Any QOS is applied to all traffic coming in that switch port. Specify switchport mode access to use this, or use default switchport voice vlan none command
2. Now we see the special 802.1Q trunk where CDP is required. The second mode is referred to as "untagged". Now cisco doesn't use the term untagged nearly as much as other network vendors, but when you create a dot1Q trunk, every packet entering the switch needs to have a vlan tag to specify what vlan number it belongs to. Any packets entering the trunk port without a vlan tag, is dumped into the untagged vlan, or as cisco calls it a native vlan. By default this is vlan 1, so you probably need to specify a untagged vlan for this method. Specify the native or untagged vlan with the interface command switchport trunk native vlan [vlan #]. Then you use the voice vlan command to instruct the phone to send it's voice packets without attaching a vlan tag. switchport voice vlan untagged
3. Third we have the dot1p mode. In this mode, you gain the qos abilities. The phone will actually tag it's own voice traffic with vlan 0, and send it with a 802.1p priority of 5 by default. (call control gets a priority of 3). The benefit of this mode is that you get QOS abilities without needing a separate voice vlan created on your switches and routers. switchport voice vlan dot1p
4. Fourth is the most common (and best in my opinion) the vlan-id option. Create a vlan on your routers and switches that will be used just for phones. The phone will now send voice packets tagged with your voice vlan ID to the switch, with qos priority, while the data packets are sent along untagged to the access vlan. The command is switchport voice vlan [vlanID]
Methods 2, 3, and 4 use a special dot1Q trunk that is negotiated through cdp messages and carries only up to 2 vlans for voice and pc traffic.
To set a different qos priority to the phone for it's voice packets, use the interface command switchport priority extend [cos value] (you can also extend trust to the pc with the same command, but since I know you will get QOS trust boundary questions on the BCMSN test, I will give it it's own post.)
To display voice vlan configurations for an interface, type show interface [interface id] or simply show running interface [interface id]
*CDP should always be enabled on an interface with a cisco phone plugged in for voice vlan, trust, power settings, and qos to be set up properly with the phone.
*voice vlans automatically enable spanning-tree portfast (and does not disable when your remove the voice vlan)
*voice vlans can cause conflicts with private vlans, do not enable them on the same port
TCP Sequence Numbers
Friday, June 4, 2010
ACLs to mitigate spoofed IP addresses
ACL (COMMON) ENTREE FORMULA:
Permit/Deny [protocol] [sourceIP] [InvertedSourceMask] [DestinationIP] [InvertedDestMask] eq [optionalPort#]
Ok, here is a subject that I have been going back and forth on in a forum. First we will use some logic. You have private IP addresses that are non routable on the internet (10.0.0.0 192.168.0.0 172.16.0.0 127.0.0.0 etc.), so we conclude that these addresses should never being entering the outside of your firewall.
Just looking at that angle, we can then put some deny statements on our OUTSIDE inbound interface (traffic from the internet) to deny them before we permit traffic to our web servers.
deny ip 10.0.0.0 0.255.255.255 any
deny ip 172.16.0.0 0.15.255.255 any
deny ip 192.168.0.0 0.0.255.255 any
permit tcp any [webserverIP] eq 80
Remembering that ACLs are read sequentially from the top, we have just denied anyone who tried to spoof their source IP as a private address from accessing our web server, and allowed everyone else on the internet in on port 80. We can rely on the implicit deny at the end to block all traffic to any other ports on our webserver.
Now what if there is a user inside your network trying to spoof IP addresses. I would suggest having ACLs on each of your routers. If building A has a subnet of 172.16.45.0/24, then you can apply an ACL such as the following to make sure ONLY addresses from inside that subnet are allowed out. (You need to allow it to any, unless you want to specify every website your users are allowed to visit... unlikely.)
permit ip 172.16.45.0 0.0.0.255 any
Unless of course you have a proxy server, then you can really tighten your ACL with:
permit tcp 172.16.45.0 0.0.0.255 [webProxyIP] 0.0.0.0 eq 8080
this allows ONLY your subnet (no other spoofed IPs) to access ONLY your proxy server ONLY on port 8080.
Basically, you just need to draw out your network, and write ACLs to make sure only those addresses that you have provided (DHCP or STATIC) are allowed to be used as source addresses. Also remember that private IP addresses should NEVER be entering your network from your ISP.
Thursday, June 3, 2010
LISP
LISP separates the routing locator from the endpoint identifier into two numbering spaces. Primarily, this means routers will have far smaller routing tables if the routing table only has to hold the route portion of addresses. The host portion of the address is encapsulated inside the route headed packet. This is also intended to make multi-homing far easier, aimed at medium to small sized businesses. Multi-homing is "load leveling" between separate ISPs.
LISP works between LISP and non LISP networks, LISP IPv4 to IPv6, and two LISP IPv6 sites can communicate overtop of an IPv4 ISP network between them (or IPv4 to IPv4 over a fully IPv6 network).
More information at lisp4.cisco.com
Facebook is one of the big beta testers at www.lisp4.facebook.com
Wednesday, June 2, 2010
Format Change
DAI (Dynamic Arp Inspection)
Next, I'll quickly explain ARP and CAM tables. Every switch stores a copy of what mac address(es) are located off each interface in a CAM table (content addressable memory). If the host hasn't sent or received a packet in a while (or is unplugged), the mac address will time-out and be removed from the cam table. To find that mac address next time a packet comes, the switch must broadcast (flood) out packets to all switch ports and wait for a response.
ARP is the address resolution protocol. Routers keep an ARP table, or a list of what IP addresses match what mac addresses. (view with the command: show IP arp) The router is then able to rewrite the destination mac address on incoming packets to the matching IP/MAC entree so that the layer 2 edge switches can find the host (since layer 2 switches ONLY understand mac addresses, know your OSI model).
For example: The diagram describes a regular ARP request. The router must flood out an ARP request to find the destination host's mac address, re-write the packet header and then is able to send the packet. (This ARP request/cam table fill, is why the first ping of a host that hasn't sent packets lately or is new to the network will always fail. The second or third will work correctly.)
Now, what if people didn't play so nicely. What if a malicious host on port 2 of the layer 2 switch responded to the ARP request before the real client saying THEY had mac address bbbb.1111.bbbb and IP address 192.168.5.3. The router would update it's ARP table to match 192.168.5.3 with mac bbbb.1111.bbbb, and the switch would include the mac in it's cam table for port 2. Any new packets sent to 192.168.5.3 would be sent to the malicious host, who can then forward it to the real host after assessing it's contents if it wishes. This attack is known as ARP spoofing or ARP poisening.
Enter Dynamic ARP inspection. DAI works similar to DHCP snooping, in that all ports need to be set as either trusted or untrusted. All packets entering an untrusted port are intercepted and checked, while trusted ports (switch uplinks) pass traffic without DAI checking anything. If a packet enters an untrusted port but doesn't match the information gathered either statically or in the DHCP snooping database, the packet is dropped and a log message created.
Commands: (Assuming you already have DHCP snooping running) turn on DAI on client vlans.
Switch(config)#ip arp inspection vlan vlan range
All switch ports in those vlans are now considered un-trusted. Make any switch uplinks or explicitly trusted ports trusted with the interface command
switch(config-if)#ip arp inspection trust
For static IPs that are not included in the DHCP snooping database, you can use an access-list. (Note that this is not a regular ACL as it includes a mac address and only the sender information. This is specifically an arp ACL)
arp access-list aclName
permit ip host senderIP mac host sender Mac Address [log]
(repeat this as needed for more static entrees)
exit
ip arp inspection filter aclName vlan Vlan range
One further thing you can do. By default this only checks the ARP reply's message body, you can also verify the ARP reply's packet header for further security.
ip arp inspection validate {[src-mac] [dst-mac] [ip]}
Tuesday, June 1, 2010
IP Source Guard
In general, IP addresses are assigned and used "on the honor system" with nothing in place to make sure clients are only using their own. Common uses are to disguise denial of service attacks. If a lot of requests could come from an IP address, but their is no return address to send the replies to, or to track down the offending computer.
Between vlans this might be simple to detect, if the 192.168.5.0 network is in vlan 5, then that network appearing out of any other vlan can be dropped. What about when the spoofed IP address is inside the proper vlan, the user is just using a random or stolen IP from inside it's own subnet?
IP source guard comes into play. Using the DHCP snooping database or statically entered bindings, all packets entering the switch verify that the port, IP address, and mac address all match the database or the packet is dropped.
Turn this feature on by entering each interface and adding:
Switch(config-if)#ip verify source [port-security]
(Port security can be added to verify the mac addresses).
show ip verify source will show you the status of your IP source guard configurations.
Friday, May 28, 2010
DHCP Snooping
Having a rogue DHCP server on your network is obviously a problem, so we can block this at the switch level. DHCP snooping actually keeps a table of DHCP binding information for each port (that you leave untrusted). First you enable it globally with ip dhcp snooping. Then you enable it per vlan with the ip dhcp snooping vlan number (you can use a range of vlan numbers here as well).
Next you need to manually set all your trusted interfaces. This means all your switch uplinks, as well as any DHCP servers. These ports must be trusted to allow DHCP responses to traverse the interface inbound (from the server to the switch).
This means regular clients should always be on an untrusted port so that they cannot act as an DHCP server.
Other options include rate limiting. Ip dhcp snooping limit rate rate will allow you to limit how many requests a client on an untrusted port can make as well. Set this on a per-interface basis.
DHCP snooping binding tables include MAC address, IP address, lease time, binding type,VLAN number, and interface information
Host Unreachable Reply
Well the reply is actually from the router, and the host is, as it says, unreachable. The router can't forward the ping, so it sends you a friendly reply. You can turn this off with the command no ip unreachables, and the router will silently let your packet drop and your ping time-out.
On a similar note, one major DOS attack is to send a flood of packets to unreachable address(es), causing the router to reply to each of these requests with an ip unreachable. If you want to still have the unreachables, you can rate limit responses. For instance:
Router(config)# ip icmp rate-limit unreachable 1000
This will limit the router to one reply per second.
Thursday, May 27, 2010
RIP with discontiguous subnets
So what if, as described in the below diagram, There are two different subnets being advertised that are not contiguous. Router A and C both know about separate 172.168 networks. The middle router will see identical route advertisements from both sides.
How do you resolve this? Well, use RIPv2, EIGRP, OSPF, or IS-IS (and don't summarize either of the 172.168 routes). This situation is just one of the reasons why classful routing is not used much any more, but watch out for a question like this on a CCNA test.
Wednesday, May 26, 2010
VTP
VTP - VLAN Trunking Protocol
Cisco uses VTP (other vendors have MVRP (standards based) for the same thing) to share vlans among switches. If you had 30 switches in a building, that all needed to have a a vlan added, logging into 30 switches to add the vlan is certainly more work than doing it on a single switch.
First you decide what switches you think will need to all share the same vlans, and add them to the VTP domain. Set the VTP password, and set all but one or two of the switches to client mode (not necessary but helps keep things simpler if you always make changes to a single switch). You can then log into any of the server mode switches and add or remove a vlan, and the change will be propagated to the entire domain.
Transparent mode, a third option, allows you to add or remove vlans (like server mode) but will never send out it's own database to propagate the network.
Now back to the original question, why would you not want to use VTP.
1. You are a control freak and want to make sure you control every change individually.
2. The switches all need different vlans, they share very few. Having 20 different VTP domains with only 4 or 5 switches per domain would be more work keeping the domains straight than simply logging in and making the changes individually.
3. Too Many Vlans - 2950 switches and before, I believe (I know 3500s for sure), only support 64 vlans. If VTP propagates more than 64 vlans to those switches, random vlans are chosen to not be added. If a vlan that that switch needs is missing, you have problems.
4. If you are adding a switch to a production network, ALWAYS ALWAYS change to an unused domain and delete vlan.dat to erase the vlans. Verify that show vtp stat shows a 0 (or lower than your production network) revision number or you will lose all your vlans.
Common misconceptions:
1. Switches in Client mode can't overwrite the config on one in server mode
The vlan database used is decided entirely upon the revision number. Every time you add or remove a vlan, the revision number is increased by one and "should" be higher than all the other switches. Because it is higher it is passed out as the new database. If you bring in a switch in client mode with the same domain name and higher revision number, the entire database of all the other switches is lost forever, including those in server mode.
2. Only Client mode switches receive updates
Client mode only means that you cannot add or remove vlans directly to that switches command line. Apart from that, Server and Client modes are identical, they all receive updates.
3. Switches in Transparent mode do not pass any VTP updates
Transparent mode means the switch will not pass any copies of it's own database, if you connect two switches to the transparent with different revision numbers, the transparent switch will pass that info between them.
Tuesday, May 25, 2010
Wireless Inter-Controller Roaming
lightweight access point - a "dumb" wireless device that basically just sends and receives traffic from the client and performs some simple encryption. Will not work without a controller to tell it what to do.
Controller - A device that controls lightweight access points, providing the SSIDs, authentication, routing, and security decisions.
Ok, so here is the problem. You have lightweight access points that talk to several different controllers. Now a client has a laptop or a phone, that has connected to your wireless network. They decide to walk around and the client moves out of range of one access point. The client's NIC drivers decide that the signal is too low and begin probing for a better signal, find one, and connect to a second AP. If the second AP is connected to the same controller, then there shouldn't be any problem. The roam from AP1 to AP2 is completed inside the controller with no data drop. Now say they continue walking to a different floor and need to move to AP3. AP3 is not only on a different controller, but a different subnet. Being on a different subnet means the client must request a new DHCP address, a long (say 15 second) process. During these 15 seconds the client's sessions are dropped, the phone call dies, the video stops streaming (your phone starts ringing).
Cisco's solution is a "mobility group". Under the Controller tab, you can expand "Mobility Management" and click mobility groups. This allows you to enter the Mac Address, IP, Mobility Group Name, and Multicast IP for the other controllers in your environment. By putting all (or all within a building or site) your controllers into a single mobility group, the controllers will constantly be talking over that multicast group to each other about what clients they have. When your client decides to move to a new controller, the original controller becomes an "anchor" controller.
Once the client has an anchor controller (C1) and has roamed to the new controller on a different subnet (C2) they send all their traffic through C2, who in turn passes everything to C1 for processing. This allows the client to retain their original IP address and avoid the connection drop that would have occured. C1 then passes the traffic out onto the wire (wired network), and all return traffic is sent to C1 and then passed to C2 and finally the client.
In older code versions, outbound traffic moved from client to C2, and was dropped out onto the wired network directly, and only return traffic proceded through both C1 and C2. The problem people found with this was when there was a firewall between the two controllers (eg. remote site) the outbound IP address was different than the return IP. I will save the firewall posts for another day, but simply put, a firewall will send a packet out, and automatically open a hole for the return packet. If that return packet is coming to a different IP, it doesn't fit in the waiting hole and gets dropped.
I know a drawing always helps me so I will try to provide them regularly. Here you can see the client before the roam on the left, sending traffic in and out of a single controller. After the roam, on the right, you can see traffic sent through both controllers in both directions.