Cisco Wireless Controllers: High Availability

A few gotcha's from bringing up a High Availability pair of Cisco 5508s.

1. Do not try to straight up follow Cisco's directions of upgrading from the boot up wizard
2. Bring up new interfaces in the same subnets as per directions etc. etc. 
3. First you have to upgrade both controllers to the same image (makes sense)
4. upgrade both controllers to the same FUS image
5. Manually set your HA Sku'd controller as secondary, even though cisco says it isn't necessary...
6. If you like the GUI, you won't be able to use it on the service port anymore, so delete your network route, and use it from the regular LAG interface management port.  If you are good with CLI then you can still use the service port
7. Enable SSO on your PRIMARY controller and let it reboot   (all of cisco's directions imply you should do this at the same time on both controllers, but that doesn't work)
8. Once the Primary controller is back up in SSO state, then you can enable SSO on the secondary controller, wait for it to reboot a couple times, and look for "ACTIVE HOT"
9. Tell your customers they now have sub-second failover and see them not care


Hopefully these can save someone a bit of troubleshooting


 

STP Enhancements Part 1

STP Enhancements Part 1 - BDPU Guard, BPDU Filter, BPDU Root Guard

BPDU Guard - BPDU is guarding your switch against BPDUs.  Wait, I thought BPDUs were a good thing?  Yes but only on ports uplinked to other switches, ports you expect to have BPDUs.

Why:  When you configure your switch to connect to hosts (PCs and Servers) you are probably using portfast.  If a user ever connected a switch, or some wires got crossed and a switch got plugged in to a host port, or a linux user enabled a software bridge etc. etc.... you could have a network loop 

What:  BPDU guard is a port setting that will shut down a port if it ever sees any BPDU

How:

Globally:

spanning-tree portfast edge bpduguard default

*make sure you disable bpduguard on the uplink ports you expect to have switches plug into: the global command affects all ports with portfast enabled*

Per Interface:

spanning-tree bpduguard enable

Recovery:  if BPDU guard triggers (sees a BPDU) it places the port in an error-disabled state, and the port is effectively shut down.  You can recover by (first removing the offending device), then logging into the switch, and issuing a shut command, followed by no shut.

If you would like the switch to be able to re-enable the port by itself, you can use error-disabled's regular methods:

errdisable recovery cause bpduguard

errdisable recovery interval 400

This will cause the switch to shut down the port for 400 seconds, then if it isn't receiving BPDUs, re-enable the interface.

*Note that BPDU Guard is triggered by BPDUs, so if a user plugs in a home/residential switch, it will not trigger as small home switches do not send BPDUs or participate in spanning tree at all.  This will not have any effect on home switches or hubs. (unless there is a loop and the cisco switch see's it's own BPDU sent back to itself, but then you already have a problem!)

BPDU Filter:

Why: Filter is slightly trickier as the global command and the interface command perform two different functions.  

1. First the interface method.  If you apply BPDU filtering to a specific interface, it effectively turns off spanning-tree.  

2. The global method is intended to save on bandwidth/proc overhead by not sending BPDUs out host ports.

What:

1. Interface Method: When turned on for a specific interface, that interface will ignore BPDUs sent to it, and it will not send any BPDUs.  It will continue forwarding traffic and not participate in spanning-tree.

2. In global mode: all interfaces in portfast mode will be put in filtering mode.  They will send out 10 BPDUs when the port comes up, and assuming no incoming BPDUs the switch will stop sending any BPDUs.  If a port ever receives a BPDU at any time, it will lose it's BPDU filter mode, removes the portfast status, and begins regular spanning tree listening/learning process.

How:

global mode

config)# spanning-tree portfast bpdufilter default

interface mode

(config-if)# spanning-tree bpdufilter enable

*note - enabling both BPDU guard and BPDU filtering on the same interface - BPDU filtering takes precedence and BPDU guard has no effect

Root Guard:

Why: ensure an interface will never be used as a root port, increasing STP stability.  Now at first it is tempting to think you should add root guard to all non-root ports but don't forget, you have a redundant topology for a reason.  You need your STP domain to be able to reconverge in the case of a failure and elect a new root bridge/use a different root port.  When you choose what ports get root guard, take into consideration any paths you might ever want your traffic to traverse and do not implement root guard on them.  The easiest and preferred way might be to just use this on edge/access ports and leave it off for all switch-to-switch connections, but perhaps your topology has a "leaf loop" off of your primary STP loop that you never want to use as an STP root, and would rather stop forwarding traffic than allow a switch out there to become root... maybe.

Anyway, root guard is designed so that if anyone plugs a new or non-corporate switch into your network, and it happens to have a low priority, it won't become root.

What:  If the root guard port receives a BPDU with superior priority to what the LOCAL switch is seeing (not the root's own priority) the port will move to a "root-inconsistent state" and stop forwarding traffic.  If the switch is removed or the priority is raised, the port will move out of root inconsistent state on it's own and begin forwarding traffic (after moving through listening/learning states).  No user intervention or config changes needed to unblock the port.

This feature does not have a global function, only a per-interface function.

config# int fa0/2

config-if)# spanning-tree guard root

*note - if you are just enabling this on host ports to protect your STP topology, just use BPDU guard instead.  The only useful time I can think of using root guard: if you want to create a policy that all un-used uplink ports have root guard configured to protect the topology.  When a new switch is added the root is protected, and once the switch is up and operational you can remove the root guard feature.

Spanning Tree Portfast

Portfast - didn't think I would have enough for a whole post, but there are a few items worth mentioning.


What is it - Typically, if a host (server, PC, etc.) plugs into a port, spanning tree will run to ensure there is not a network loop before allowing the host to talk.  If you are running the default, Cisco per-vlan spanning tree 802.1d, this takes 30 seconds for both listening and learning mode to run.

The problem - Thirty seconds is a long time to wait, especially when it means you are not getting DHCP responses and your computer is deciding that it must not be on a real network so it doesn't need to run all the network startup scripts.

The solution - configuring portfast on an interface tells that switch that only a host (nothing that could cause a loop) is connected.  The switch then allows the port to immediately transition to forwarding (but will loose it's portfast status if a BPDU is recieved)

The dangers - if you mis-configure this, and turn portfast on for an uplink to another switch, hub, etc. a loop could form and crash your network before spanning tree ever has a chance to prevent it.

The terms:  in rapid spanning tree protocol (802.1w) they are called "edge ports" but are still configured with the same portfast command. 

The commands:

interface fa0/1
spanning-tree portfast
exit

or

interface fa0/1
switchport host
 *This turns on portfast and also disables channeling and trunk negotiation*

or globally from privileged exec mode

#spanning-tree portfast default
*In the global form of portfast, immediate forwarding is enabled for all access ports (NOT ANY TRUNK PORTS) but you should still manually input the command no spanning-tree portfast for any ports that may be connected to other switches*


spanning-tree portfast trunk
*This command is to turn portfast on for a trunk.  You would typically have a trunk enabled for a host such as a server that needs multiple vlans, or a VoIP phone*


If a BPDU is ever recieved on an edge/portfast port, it loses it's portfast status AND sends a Topology Change Notification to all other switches in the STP domain.

Too Long

Sorry about no posts in forever, life/work have been crazy.

I did pass my CCNP-Route so I might have a few posts from notes to bring, but will probably start focusing on SWITCH topics.

Let me know if anyone has a topic request.

dt-lacp Distributed Trunking

Distributed Trunking -

Now everyone understands how two switches connected logically (stacked) can balance an aggregate link between them with a shared logical backplane, as cisco does, but what about HP who doesn't have stacked switches, where just a few higher end switches have the ability to do distributed trunking. A server (as it is not supported for switch to switch) can connect to two separate switches and create a single LACP trunk to both.

*note - the two switches must be directly connected, and it is a little unclear but it sounds like they have to have a dedicated link between them as the dt-trunk link, apart from the regular link. (now this does not have to be aggregated, could be a single gig connection, but as you will see in a moment it should be of equivalent bandwidth to each switch's lacp link.

Now this seems simple but what happens under the hood? With HP's usual lack of documentation it took me a while to dig up, but basically, create a non-negotiating trunk. (eg. lacp auto will not work), and connect the server. The server sends a frame to switch A which learns the mac and forwards it to the destination. Switch B also learns the mac as being on switch A. Switch B receives a frame and, having learned the mac as being on switch A, forwards the frame to switch A to send to destination.

So in essence, you are creating redundancy, but as for doubling bandwidth, ALL traffic will proceed through a single switch so both the inter-switch link and EACH switch's core uplink must be able to support the full server lacp bandwidth, for all it's links.

Eg. Server has 2gigs to switch A and 2gigs to switch B. Switch A needs at minimum 2gigs to switch B. Switch A and Switch B both need at minimum 4 gigs up to the uplink/core. (assuming you want the server to have 4gigs of bandwidth to the core) This is of course on top of any other traffic the switches will be forwarding from other devices.

Of course spanning tree is going to have to block a link somewhere anyway, so all traffic forwarding through a single switch may be the expected outcome anyway.

Cheers!

Oh ya, the config:

ProCurve Switch 1(config)# switch-interconnect a1
ProCurve Switch 2(config)# switch-interconnect a1
 ProCurve Switch 2(config)# trunk b20-b21 trk99 dt-lacp
ProCurve Switch 1(config)# trunk c10-c11 trk99 dt-lacp

 and of course if you needed an aggregate link between switches, replace a1 with trk5 and add whatever ports you needed to trk5 just like a regular trunk port.

  show lacp distributed is your show command to see the lacp status of BOTH switches

IP Source Guard

So, after saying I would post about IP source guard in my DHCP Snooping Post I neglected to post much of anything for six months BUT I'm back to fulfill my promise.

IP source guard is basically a dynamic per port access list that will be applied to make sure the device is using the IP address (and mac address if configured) that the computer received during it's DHCP handshake.


As mentioned in the previous article, DHCP snooping is going to keep track of what IP address the device gets from the server.  Then IP source guard ensures that address, stored in the dhcp snooping database, is the only IP allowed to communicate out the port, thereby defeating IP spoofing and some man in the middle attacks.

Picture a malicious user who connects to the network, and receives an IP address.  They then listen to broadcast traffic and ARPs and determine another user's IP/Mac address.  The malicious user could then either send out packets with the neighboring device's IP and pretend to be them, possibly bypassing ACLs that grant the attackee extra access.  Attack two could be that the malicious user simply responds to request's by using the gateway's IP address and appears to be legitimate traffic.

Both of these could be mitigated by IP source guard, as the malicious user, sending any packets out with an IP (or mac) not included in the dhcp snooping database will be dropped.

NOTE* If you configure source guard on a port that doesn't have an entree either learned dynamically by dhcp snooping or manually entered in the ip source binding table, all packets will be dropped

Source guard is not allowed on routed interfaces or etherchannels



Configuration:

Step one -
A. ensure dhcp snooping is enabled on the vlan you intend to use IP source guard on (or you have manually entered all IP/Mac bindings in the ip source binding table)
B. If you enable IP source guard on a trunk interface (multiple vlans) then all vlans are filtered, ensure dhcp snooping is enabled for ALL vlans on the trunk.

Step two -
Determine if some or all of your addresses need to be manually entered:


ip source binding mac-address vlan vlan-id ip-address inteface interface-id


Step three -
Determine if you just want to use source guard based on IP addresses, or both IP and Mac addresses.  If you want to check Mac addresses as well, ensure your DHCP server supports option 82.

interface interface-id

          For just checking IP addresses:

               ip verify source

          For both IP and Mac addresses:

               ip verify source port-security


Step Four -
Verify Config

show ip verify source [interface interface-id]

Verify Bindings

show ip source binding


Step Five -
Save if everything looks good

wri mem

(or if you are studying for an exam)
copy running-config startup-config

Curious what the Openflow stuff is you are hearing all over? Check Out Ivan Pepelnjak's links to recorded versions of OPENFLOW AND SOFTWARE DEFINED NETWORKING 101 type introduction session on the technology causing all the buzz. Interesting stuff, even if I am very very weary of taking the "brains" out of my switches and placing it in a controller or software situation, if it works well it could very well change the hole concept of networking as we know it.

OSPF Feasible Successors

OSPFv2 Loop-Free Alternate Fast Reroute - Once OSPF has calculated SPF and installed routes in the routing table, it calculates a successor route to destinations, much as EIGRP does. Greatly reduce your down time during re-convergence.

You can find it in the Cisco 15.1 release notes.

 Cisco - Yep
Juniper - Yep
HP - As usual, they probably will get around to it in a couple years...

DHCP-Snooping and Dynamic Arp Inspection

Q. What is DHCP Snooping?

A.
1. Switch will watch all dhcp requests and build a table of ip-mac bindings for each port.
2. Switch will drop all dhcp RESPONSES from untrusted ports. (eg. If an end user places a dhcp server on their PC or device and connects to your switch, they won't be able to hand out addresses)

Q. What ports should I make trusted or untrusted?

A. Any ports you expect a dhcp response to come from. This includes the port your DHCP server resides on, as well as switch uplink ports. Make everything else untrusted.

Q. What is option 82?

A. Option 82 adds switch and port information to the DHCP request, so that the dhcp server can log information about the exact location of the client. Useful for tracking down malicious end users.

Q. I am on a procurve and there is something about authorized servers.

A. Put in the IP address of any DHCP servers you have in your network as authorized.

Q. So what about the arp inspection stuff?

A.

Malicious use 1. Think of a malicious user who responds to an arp request for the default gateway's IP address with their own mac. Now all the traffic goes to the malicious user rather than the router. (man in the middle attack)

Malicious use 2. A malicious or malfunctioning computer can send out thousands of arp responses filling up arp tables

Malicious use 3. A malicious or malfunctioning computer can send out arp responses (or gratuitous arp) to false mac addresses breaking network communication.

Arp inspection uses the DHCP snooping binding table to ensure that any arp response from an end user is only for the IP and Mac address they have been properly given by the DHCP server.

Q. This sounds like great security enhancements, are there any "Gotcha's I should know about?"

A. Two of them I can think of.

#1 Say you love this idea and go turn on dhcp-snooping and ip arp inspection today. A computer arp's for the DG or another device but their IP is not in the table yet, they still have a day left on their DHCP lease before they need to renew it. Their arp is dropped and they can't talk to that device. Fix#1, enable DHCP-snooping AT LEAST a full DHCP lease time prior to enabling arp inspection.

#2 Similar to number one. So say you have dhcp-snooping going, and you enable arp inspection, and everything is working fantastic. Now your maintenance window comes along and you need to upgrade your ios and reboot the switch. Now you rebooted and the DHCP-binding table cleared and you are stuck with problem #1 again, until the DHCP leases are renewed clients can't ARP. Fix#2 There is a way to save the database to flash before you reload, or even better, store a copy of the DHCP-snooping table off the switch, on a tftp server.


Q. Configurations?

Cisco:

(config)# ip dhcp snooping
(config)# ip dhcp snooping vlan 20,25-28
(config)# ip dhcp snooping verify mac-address
(config)# interface GigabitEthernet 2/2
(config-if)# ip dhcp snooping trust (for DHCP server connected interfaces and switch uplinks *do all of them since spanning tree could re-converge and send requests up alternate paths!*)
(config)# ip dhcp snooping limit rate 10
(config)# ip dhcp snooping database tftp://10.1.1.1/file
(config)# show running-config dhcp
(config)# ip arp inspection vlan 20,25-28


Procurve:
dhcp-snooping
dhcp-snooping authorized-server #.#.#.#
dhcp-snooping authorized-server #.#.#.#
dhcp-snooping vlan 5 10 20 99

interface a4
dhcp-snooping trust

dhcp-snooping database file tftp:///

show dhcp-snooping
show dhcp-snooping stats
show dhcp-snooping binding

arp-protect vlan 5 10 20 99


**dhcp-snooping on many(most?) procurve switches breaks PXE booting/imaging, and as far as I can tell has only been fixed on a few switch models, if you use PXE booting, test this feature before rolling it out to a production environment**


I will look at IP source-guard  soon as it also uses the dhcp-snooping database

Thoughts? Suggestions? Accusations? Comments on my mental health?

Cisco Live Sessions

Cisco allowed virtual sessions free this year, and they are recorded so I am finally getting around to watching a few. I strongly recommend browsing the selection.


Some highlights from the QoS Design clinic I'm watching now.

Uncompressed HD video - 1.5 Gbps
compressed to - 3-5 Mbps (300:1)
Which is why a single packet dropped in 10,000 is visible to watchers.

This makes it 100 times more sensitive than voice and requires sub-millisecond jitter.


Suggests not just marking packets but also policing.

Eg. No voice endpoint will ever send more than 128kps of data, so if traffic is coming in a trusted voice port and getting tagged with your Voice QoS designation, either drop or re-classify it when it exceeds the 128kps threshold.

-Priority Queue should be less than 33% of link but best effort guaranteed 25%

EtherChannel QoS

On a Catalyst 4k and 6k cisco switches, apply INBOUND QoS policies to the Port-channel

On a Catalyst 2k and 3k, ingress policies are applied to the physical ports.

Egress Queuing is always applied to the member interfaces on all types.

Hopefully this will get cleared standardized in the future eh.

HP5400 blade failures

Every so often I find a dead blade or switch... (thanks for the job security HP). And looking back at the logs you'll see tombstone errors:

W 07/28/11 19:55:11 00374 chassis: Slot A Slave ROM Tombstone: 0x13000101
W 07/28/11 19:52:05 00374 chassis: Slot A Failed to boot-timeout-(ROM_ALIVE)
W 07/28/11 19:52:04 00374 chassis: Slot A Slave ROM Tombstone: 0x13000101
W 07/28/11 19:48:58 00374 chassis: Slot A Failed to boot-timeout-(ROM_ALIVE)
W 07/28/11 19:48:56 00374 chassis: Slot A Slave ROM Tombstone: 0x13000101
W 07/28/11 19:45:50 00274 chassis: (84) Slot A: Blade Crash detected - Available

And what is the fix? Well a reboot of course (if you are lucky). But luckily in a blade switch you can just reload that module. Type:

#reload module A (where A is the letter of the blade)


In probably 3/4 the cases this seems to fix it, and you may never have a problem with it again. In the other quarter, you can try this repeatedly but the only thing that works: call up HP and get a replacement.


If you are lucky enough to have it come up, don't expect it to come up right away either, it takes around two minutes to reload the blade, then you will probably still see:

#show log -r
I 07/28/11 21:25:52 00422 chassis: Slot A Ready
I 07/28/11 21:25:36 00376 chassis: Slot A Download Complete
I 07/28/11 21:25:34 00375 chassis: Slot A Downloading
W 07/28/11 21:25:28 00374 chassis: Slot A Failed to boot-timeout-(ROM_ALIVE)
W 07/28/11 21:25:27 00374 chassis: Slot A Slave ROM Tombstone: 0x13000101
W 07/28/11 21:22:21 00374 chassis: Slot A Failed to boot-timeout-(ROM_ALIVE)
W 07/28/11 21:22:20 00374 chassis: Slot A Slave ROM Tombstone: 0x13000101
W 07/28/11 21:19:14 00374 chassis: Slot A Failed to boot-timeout-(ROM_ALIVE)
W 07/28/11 21:19:12 00374 chassis: Slot A Slave ROM Tombstone: 0x13000101
W 07/28/11 21:16:06 00374 chassis: Slot A Failed to boot-timeout-(ROM_ALIVE)
W 07/28/11 21:16:05 00374 chassis: Slot A Slave ROM Tombstone: 0x13000101
I 07/28/11 21:13:00 02756 chassis: Slot A is powered up.
I 07/28/11 21:12:57 02755 chassis: Slot A is powered down.
I 07/28/11 21:12:45 02762 chassis: Request for "reload module A".


You might notice that it took 13 minutes before the switch decided to bring up the blade after the module reload, enough time for you to give up on it and start trekking out with a replacement. Cheers to those of you lucky enough to work with procurve gear!

MRTG for large environments

So, lots of sites try to tell you how to install MRTG to monitor your computer, and a few tell you how to monitor a router or two, but what if you have hundreds or thousands of interfaces you want to see traffic on? I prefer having them show up on multiple organized web pages rather than one giant 5000 graph long page.



Here is my guide to installing MRTG in a large network environment on Debian Linux

Step one - make sure all your target devices (routers switches etc.) have an snmp read only string that does NOT include special characters like ! $ or >. You just need to stick with lowercase uppercase and numbers to get cfgmaker to work.


On a cisco - add
snmp-server community SNMPSTRING123 ro
to add a read only string of SNMPSTRING123





sudo apt-get install apache2
sudo apt-get install snmpd
sudo apt-get install mrtg

make sure you add the snmp string to your /etc/snmp/snmpd.conf file (see some instructions by Clicking Here )


Now for the MRTG part.


mrtg installs into /etc/mrtg.cfg you should move it into /etc/mrtg/mrtg.cfg to keep things neat

Next create a directory at /var/www/mrtg to place your web pages in.

mkdir /var/www/mrtg

Now we need to make a config

gather all the switch or router IPs that you want on a SINGLE web page. This might be one 200 interface switch, or it might be a building worth.

run

sudo cfgmaker --global 'WorkDir: /var/www/mrtg' --global 'Options[_]: bits,growright' --output=/etc/mrtg/mrtgG1.cfg SNMPSTRING123@Router1IPAddress SNMPSTRING123@Router2IPAddress SNMPSTRING123@Router3IPAddress





Notice I called this mrtgG1.cfg for group 1, change this each time you set up a new web page and new group of switches/routers


next, using your favorite editor modify the new file to run as a daemon

vi /etc/mrtg/mrtgG1.cfg

add the lines

RunAsDaemon: Yes
Interval: 5


these set it to run as a daemon, every 5 minutes

next we can build the actual web page.

sudo indexmaker --output=/var/www/mrtg/Group1.html /etc/mrtg/mrtgG1.cfg

This will pull the interface info from the mrtgG1.cfg file and make a web page out of it.


Repeat these steps for group 2 and group 3 etc.

Next you will probably want an index page to link to the individual group pages. Nothing fancy for me, just create

vi /var/www/mrtg/index.html

add the regular html, a title, and a few links like this:
<A HREF="http://serverIP/mrtg/Group1.html"> Routers1-3 Interface Traffic </a>
<p>
<A HREF="http://ServerIP/mrtg/Group2.html"> Switches4-12 in Building ABC Traffic </a>
<p>

You can certainly get fancier but this is just a starting point.


By now you can browse to your index page and click a few of your links. You may or may not have graphs showing up yet. Sometimes MRTG needs to restart a few times to create all the necessary files.

The way most assured to work right now is to type

ps -e

Find the mrtg process, note the number next to it (say 12345) and type

kill 12345

Then start mrtg again with

sudo env LANG=C /usr/bin/mrtg /etc/mrtg/mrtgG1.cfg


*note, you can do this for each mrtg group (process) you created*


Ok, now wait ten minutes (remember it only polls the switches every 5) and see if you have some data showing up. Hopefully you do, if not try kill/restarting the process again.


Now if you do have graphs showing up and a nifty index page to surf to them it might seem like you are done but first you need to add it to init.d and startup. You don't want all mrtg to stop next time you reboot do you?


place the following file in your /home directory

vi /home/mrtgG1

---

(this config file originally created by iceflatline@gmail.com) at iceflatline.com and only slightly modified by me. The credit goes ENTIRELY to him.)





### BEGIN INIT INFO

# Provides: mrtg
# Required-Start:
# Required-Stop:
# Default-Start: 2 3 4 5
# Default-Stop: 0 1 6
# Short-Description: mrtg init script
# Description: This file is used to start, stop, restart,
# and determined status of the mrtg daemon.
# Author: iceflatline
### END INIT INFO

### START OF SCRIPT

set -e
# PATH should only include /usr/* if it runs after the mountnfs.sh script

PATH=/sbin:/usr/sbin:/bin:/usr/bin
DESC="mrtg"
NAME=mrtgG1
DAEMON=/usr/bin/$NAME
DAEMON_ARGS="/etc/mrtg/mrtgG1.cfg"
PIDFILE=/etc/$NAME.pid
SCRIPTNAME=/etc/init.d/$NAME

# Exit if the mrtg package is not installed

[ -x "$DAEMON" ] || exit 0

# Load the VERBOSE setting and other rcS variables

. /lib/init/vars.sh

# Define LSB log_* functions.
# Depend on lsb-base (>= 3.0-6) to ensure that this file is present.
. /lib/lsb/init-functions
# Function that starts the mrtg daemon
start()
{
env LANG=C start-stop-daemon --start --quiet --exec $DAEMON -- $DAEMON_ARGS
}
# Function that stops the mrtg daemon
stop()
{
start-stop-daemon --stop --quiet --retry=TERM/30/KILL/5 --pidfile $PIDFILE
}
case "$1" in
start)
log_daemon_msg "Starting $DESC"
start
case "$?" in
0) log_end_msg 0 ;;
1) log_end_msg 1 ;;
esac
;;
stop)
log_daemon_msg "Stopping $DESC"
stop
case "$?" in
0) log_end_msg 0 ;;
1) log_end_msg 1 ;;
esac
;;
restart|force-reload)
log_daemon_msg "Restarting $DESC"
stop
case "$?" in
0|1)
start
case "$?" in
0) log_end_msg 0 ;;
1) log_end_msg 1 ;;
esac
;;
esac
;;
status)
status_of_proc "$DAEMON" "$NAME"
;;
*)
echo "Usage: $SCRIPTNAME {start|stop|status|restart|force-reload}"
;;
esac
exit 0
### END OF SCRIPT

---

Note the two lines toward the top

NAME=mrtgG1
DAEMON_ARGS="/etc/mrtg/mrtgG1.cfg"

These will have to match whatever you called your .cfg and .pid files.




Now issue the following commands to make it executable and move it to the startup folder

cd /home
sudo chmod +x mrtgG1
sudo mv mrtgG1 /etc/init.d/


issue the following to add it to run levels for startup
sudo update-rc.d mrtgG1 defaults


test with
sudo /etc/init.d/mrtgG1 restart

repeat to make each .cfg file run at startup.



Hooray, you now have mrtg running, graphing the bandwidth used on a slew of switches, with separate groups running in separate daemons so your server can get through them all in the 5 minute window it has before it needs to start again, and they will start up again if you need to reboot the server. Hopefully this helped someone! Let me know any tweaks, if you know how to run the multiple .cfg files out of one startup file, or if I made a mistake somewhere!

RIPv2 passive interfaces

So the only RIP to be encountered on the ROUTE test appears to be passive interfaces.

So in EIGRP and OSPF passive interfaces do not send or receive routing updates, they do not participate in neighborships or the routing protocol.

RIPv2 on the other hand:
A passive interface does not SEND updates out this interface, it will still install RIP updates it receives.


R1#
router rip
version 2
passive interface s0/0/1


Router 1 will now receive routes from s0/0/1 but will not send any information out s0/0/1.

Multicast Vlan Feature

So, after upgrading to a new version of WLC code, 7.0.116.0 you might notice a new feature in your SSID configs, multicast vlan feature. I wasn't sure what it did so I looked it up and best as I can tell it goes like this.


Enabled per ssid (a handy checkbox or config wlan multicast interface wlan_id enable interface_name)

If you use the VLAN select feature, each client will be listening to the multicast stream on a different VLAN, and the upstream router must send a copy for each VLAN, and multiple copies of a multicast stream are sent out over the air. (not very good use of your air time, and negates some of the purpose of multicast).

Multicast optimization lets you create a multicast vlan, and the controller will make sure all multicast streams go out on the multicast vlan so that the upstream router only sees the single multicast entry for all the VLANs in your pool.

So clients on separate vlans will still only have one single multicast stream over the air.


Since I'm not currently using VLAN pooling I'm not 100% solid on how this would look in the real world but certainly seems a strong feature to have enabled if you were.

Layer 2 edge port security

I was asked the question "how I would secure switch ports" in an interview, and of course I blanked on most of it, but thought I would compile a list now that I'm not in the hot seat.

1. Not allow trunking - in cisco terms switchport mode access (if it needs to be a trunk, make sure the native vlan is not vlan 1, and you add switchport trunk allowed vlan x,y,z to only allow necessary vlans on the port.)

2. BPDU guard and/or root guard - do not allow your spanning tree to be hijacked. BPDU guard will block or restrict any port that someone attaches a spanning tree capable device to (eg. anything that sends a BPDU will shut down the port) while root guard will block it if anything off that port attempts to become your spanning tree root. This can be attached to your own switch uplinks/downlinks if you KNOW you never want the neighbor switch or any switches down that branch to ever be root.

3. no cdp enable - no sense broadcasting out to every client what switch, Management IP address, port, IOS version etc. that they are connected to.

4. DHCP snooping/ Dynamic ARP inspection - There are a few commands related to DHCP snooping that will give you a few benefits.
a. You can block any edge port that offers DHCP responses - blocking clients from running a DHCP server on your network.
b. You can tell what IP address has been dynamically assigned out a port, and if an ARP response is sent out not matching that address (eg. IP spoofing) block the port
c. limit how many DHCP requests are sent out per second
d. I think there is another command or two that can go with this for more features but I haven't studied switch since it was the BCMSN so it has been a year+

5. Port Security - limit what mac addresses are allowed to be used on a port
a. options here include statically set addresses, or sticky addresses so you don't have to type them in yourself and the switch saves the address first seen on the port.
b. You can then limit how many addresses are allowed in a port, (limit 1 etc.)
c. Obviously MAC addresses are easy to spoof if they unplugged a different device, but if it is just an open port, they aren't likely to guess a statically assigned mac address

6. shut down unused ports - I know, I'm terrible at this, but shut or disable ports that do not have a computer plugged into them. No sense letting someone walk into your IDF and plugging in any open port without you knowing about it.

7. Network Access Control - Bit more involved here - you can have it fairly simple where the machine and user need to match up with an AD or Radius authentication, to more advanced options where User A on Machine x at Time H is placed on vlan V, but if they are on a different machine or a different time, perhaps they are placed in a more restricted vlan since it isn't their normal working hours.
NAC also has options to verify the machine is up to date on patches and virus definitions before being placed on a vlan. Lots and Lots to think about before implementing NAC.

8. Black Hole Vlan - Just in-case you forget to shut the port down, assign all unused ports to a "Black Hole Vlan" or a vlan that is not routed or trunked so the user can't get anywhere if they do manage to connect.

9. VACLs, I typically do ACLs at the router, but feel free to assign VACLs to limit traffic WITHIN the vlans to further secure INTRA-VLAN traffic. (if for some terrible reason your switch management IP is on the client vlan, make sure you add a VACL and/or vty ACL to keep clients from accessing it!!!)



10. ??? I'm sure there is more, what have I missed? Feel free to comment, argue, or add more for typical edge port security.

EIGRP stub redistribution

Q. Will the following static route be redistributed?

ip route 10.1.1.0 255.255.255.0 10.2.2.1

router eigrp 100
redistribute static 1000 1 255 1 1500
eigrp stub


A. NO - eigrp stub default is summary and connected, redistributed static routes will NOT be advertised based on that stub command. The command eigrp stub static needs to be included as well

EIGRP Stub Networks

A stub router indicates in it's hello packet, to let other routers know it is a stub router.

Q. Why use a EIGRP Stub?

A.
1. Limit amount of convergence traffic sent over links
2. Tell other routers you will NOT be a transit path for any additional networks. (eg. if a router is in active because it lost it's path, do not query me because I won't help you)
3. Avoid possible Stuck-In-Active scenarios

Typically, only remote routers are set as stubs, and only if they are single router sites. When a route goes down, stub routers ARE NOT QUERIED to find an alternate path, hub (regular neighbor routers) answer on behalf of the stub router.

*It is useful to note, stub features do not prevent other routers from advertising routes to the stub router.

Cisco defines a stub router as one which "core transit traffic should not flow"


Configure a router as a stub with the commands

eigrp stub [receive-only|connected|static|summary]

Any combination (other than receive-only) can be placed after the stub command.

The default is both connected and summary.

receive-only restricts the router from sharing any of it's routes with another router. When would you use this? If a router only has one interface in use, perhaps you have a router in place for the sole purpose of providing DHCP, VoIP gateway, or other services.

The connected, static, and summary commands are fairly self explanitory. Note that these must be in place AND the network command or redistribute command must be in use. Adding connected does not automatically add all connected routes to EIGRP, it only allows EIGRP to advertise them to neighboring routers if it is already in the routing table.

If a true stub network is desired, the hub (regular) router is typically configured to send a default route to the spoke (stub) routers so that only a single route is passed to them.

EIGRP passive Interfaces

This may be more of a CCNA topic but lets look at passive interfaces. First we have a router. S0/0/1 is connected to another router, Fa0/0 is connected to a client vlan (no router should ever try to neighbor from here). You have two options, you could redistribute connected networks, (but this may have issues with the administrative distance being labeled as exterior EIGRP and being higher, and it is not as clean a way of doing it) or the preferred way of setting this up is using passive interfaces.

Q. Will a passive interface accept hellos?
A. No, passive interfaces do not send unicast or multicast hellos, and do not accept them either. The router will not neighbor off this interface.

This adds security/stability for your EIGRP domain, and lessens the CPU load on the router as it isn't creating hellos on a silent interface.

Q. How do I make an interface passive?
A. Two ways
Option 1:
router eigrp 1
passive-interface fa0/0

Option 2:
router eigrp 1
passive-interface default
no passive-interface s0/0/1

Unless you know you are going to place a neighbor off an interface, always make it passive. A malicious user (or un-knowledgeable Jr Tech) Could place a router off this interface (eg. off any access layer switch port) and affect your entire routing domain unless you use the passive interface command and prevent it.


It may be easier to use the second option, and make all interfaces passive unless you specifically turn them on, but make sure you make this change while consoled into the router, inputting that command remotely will likely turn your uplink interfaces passive and kill your routing/connection.

Q. How do I check if my interfaces are passive?
A. show ip eigrp interfaces will not show any passive interfaces, or show ip protocols will explicitly list all passive interfaces.

EIGRP Hello & Hold Timers

Lets take a brief look at EIGRP Hello and Hold Timers.

So, R1 and R2 want to form a Neighbor relationship.

Q. Do the timers need to match?
A. No, but if a hello timer is longer than it's hold timer, or slow links cause that you will have flapping links.


Q. How do I change the Hello and Hold Timers?
A. R1:
interface fa0/1
ip hello-interval eigrp 55 2 (where 55 is the AS number, 2 is the timer value in seconds)
ip hold-time eigrp 55 6 (cisco does not force you to make this 3 times the hello, but it is strongly recommended)


Q. So does this mean I need to make sure R2 uses a hello timer less than 6 to keep my links from flapping?
A. No, the hold time of six is sent to R2 to use. The value set on R2 will be sent to R1 to tell R1 how long to hold waiting for hello messages. The hold time set is the value actually used on the neighboring router.
If multiple routers are on this segment, they will all use this hold time while waiting for R1's next hello.

So you can interpret ip hold-time eigrp 55 6 as "tell all my neighbors with eigrp 55 to use a 6 second hold time when they listen for me".

R2 could use 10/30 as it's hello/hold-timers and the nieghborship would stay up.


Q. How can you verify your hello times?
A. show ip eigrp interface detail interface

Q. Does that show the hold timer as well?
A. No, you need to look at the show run or use show ip eigrp neighbors command and watch the countdown until it resets to it's highest value.

Q. How does the router send Hellos?
A. Routers using EIGRP send Hellos to multicast 224.0.0.10 attempting to find potential neighbors.


For general hello packet and neighboring information, read my hello packet post from last year