Thursday, July 28, 2011

HP5400 blade failures

Every so often I find a dead blade or switch... (thanks for the job security HP). And looking back at the logs you'll see tombstone errors:

W 07/28/11 19:55:11 00374 chassis: Slot A Slave ROM Tombstone: 0x13000101
W 07/28/11 19:52:05 00374 chassis: Slot A Failed to boot-timeout-(ROM_ALIVE)
W 07/28/11 19:52:04 00374 chassis: Slot A Slave ROM Tombstone: 0x13000101
W 07/28/11 19:48:58 00374 chassis: Slot A Failed to boot-timeout-(ROM_ALIVE)
W 07/28/11 19:48:56 00374 chassis: Slot A Slave ROM Tombstone: 0x13000101
W 07/28/11 19:45:50 00274 chassis: (84) Slot A: Blade Crash detected - Available

And what is the fix? Well a reboot of course (if you are lucky). But luckily in a blade switch you can just reload that module. Type:

#reload module A (where A is the letter of the blade)


In probably 3/4 the cases this seems to fix it, and you may never have a problem with it again. In the other quarter, you can try this repeatedly but the only thing that works: call up HP and get a replacement.


If you are lucky enough to have it come up, don't expect it to come up right away either, it takes around two minutes to reload the blade, then you will probably still see:

#show log -r
I 07/28/11 21:25:52 00422 chassis: Slot A Ready
I 07/28/11 21:25:36 00376 chassis: Slot A Download Complete
I 07/28/11 21:25:34 00375 chassis: Slot A Downloading
W 07/28/11 21:25:28 00374 chassis: Slot A Failed to boot-timeout-(ROM_ALIVE)
W 07/28/11 21:25:27 00374 chassis: Slot A Slave ROM Tombstone: 0x13000101
W 07/28/11 21:22:21 00374 chassis: Slot A Failed to boot-timeout-(ROM_ALIVE)
W 07/28/11 21:22:20 00374 chassis: Slot A Slave ROM Tombstone: 0x13000101
W 07/28/11 21:19:14 00374 chassis: Slot A Failed to boot-timeout-(ROM_ALIVE)
W 07/28/11 21:19:12 00374 chassis: Slot A Slave ROM Tombstone: 0x13000101
W 07/28/11 21:16:06 00374 chassis: Slot A Failed to boot-timeout-(ROM_ALIVE)
W 07/28/11 21:16:05 00374 chassis: Slot A Slave ROM Tombstone: 0x13000101
I 07/28/11 21:13:00 02756 chassis: Slot A is powered up.
I 07/28/11 21:12:57 02755 chassis: Slot A is powered down.
I 07/28/11 21:12:45 02762 chassis: Request for "reload module A".


You might notice that it took 13 minutes before the switch decided to bring up the blade after the module reload, enough time for you to give up on it and start trekking out with a replacement. Cheers to those of you lucky enough to work with procurve gear!

Wednesday, July 20, 2011

MRTG for large environments

So, lots of sites try to tell you how to install MRTG to monitor your computer, and a few tell you how to monitor a router or two, but what if you have hundreds or thousands of interfaces you want to see traffic on? I prefer having them show up on multiple organized web pages rather than one giant 5000 graph long page.



Here is my guide to installing MRTG in a large network environment on Debian Linux

Step one - make sure all your target devices (routers switches etc.) have an snmp read only string that does NOT include special characters like ! $ or >. You just need to stick with lowercase uppercase and numbers to get cfgmaker to work.


On a cisco - add
snmp-server community SNMPSTRING123 ro
to add a read only string of SNMPSTRING123





sudo apt-get install apache2
sudo apt-get install snmpd
sudo apt-get install mrtg


make sure you add the snmp string to your /etc/snmp/snmpd.conf file (see some instructions by Clicking Here )


Now for the MRTG part.


mrtg installs into /etc/mrtg.cfg you should move it into /etc/mrtg/mrtg.cfg to keep things neat

Next create a directory at /var/www/mrtg to place your web pages in.

mkdir /var/www/mrtg

Now we need to make a config

gather all the switch or router IPs that you want on a SINGLE web page. This might be one 200 interface switch, or it might be a building worth.

run

sudo cfgmaker --global 'WorkDir: /var/www/mrtg' --global 'Options[_]: bits,growright' --output=/etc/mrtg/mrtgG1.cfg SNMPSTRING123@Router1IPAddress SNMPSTRING123@Router2IPAddress SNMPSTRING123@Router3IPAddress





Notice I called this mrtgG1.cfg for group 1, change this each time you set up a new web page and new group of switches/routers


next, using your favorite editor modify the new file to run as a daemon

vi /etc/mrtg/mrtgG1.cfg

add the lines

RunAsDaemon: Yes
Interval: 5


these set it to run as a daemon, every 5 minutes

next we can build the actual web page.

sudo indexmaker --output=/var/www/mrtg/Group1.html /etc/mrtg/mrtgG1.cfg

This will pull the interface info from the mrtgG1.cfg file and make a web page out of it.


Repeat these steps for group 2 and group 3 etc.

Next you will probably want an index page to link to the individual group pages. Nothing fancy for me, just create

vi /var/www/mrtg/index.html

add the regular html, a title, and a few links like this:
<A HREF="http://serverIP/mrtg/Group1.html"> Routers1-3 Interface Traffic </a>
<p>
<A HREF="http://ServerIP/mrtg/Group2.html"> Switches4-12 in Building ABC Traffic </a>
<p>

You can certainly get fancier but this is just a starting point.


By now you can browse to your index page and click a few of your links. You may or may not have graphs showing up yet. Sometimes MRTG needs to restart a few times to create all the necessary files.

The way most assured to work right now is to type

ps -e

Find the mrtg process, note the number next to it (say 12345) and type

kill 12345

Then start mrtg again with

sudo env LANG=C /usr/bin/mrtg /etc/mrtg/mrtgG1.cfg


*note, you can do this for each mrtg group (process) you created*


Ok, now wait ten minutes (remember it only polls the switches every 5) and see if you have some data showing up. Hopefully you do, if not try kill/restarting the process again.


Now if you do have graphs showing up and a nifty index page to surf to them it might seem like you are done but first you need to add it to init.d and startup. You don't want all mrtg to stop next time you reboot do you?


place the following file in your /home directory

vi /home/mrtgG1




(this config file originally created by iceflatline@gmail.com) at iceflatline.com and only slightly modified by me. The credit goes ENTIRELY to him.)





### BEGIN INIT INFO

# Provides: mrtg
# Required-Start:
# Required-Stop:
# Default-Start: 2 3 4 5
# Default-Stop: 0 1 6
# Short-Description: mrtg init script
# Description: This file is used to start, stop, restart,
# and determined status of the mrtg daemon.
# Author: iceflatline
### END INIT INFO

### START OF SCRIPT

set -e
# PATH should only include /usr/* if it runs after the mountnfs.sh script

PATH=/sbin:/usr/sbin:/bin:/usr/bin
DESC="mrtg"
NAME=mrtgG1
DAEMON=/usr/bin/$NAME
DAEMON_ARGS="/etc/mrtg/mrtgG1.cfg"
PIDFILE=/etc/$NAME.pid
SCRIPTNAME=/etc/init.d/$NAME

# Exit if the mrtg package is not installed

[ -x "$DAEMON" ] || exit 0

# Load the VERBOSE setting and other rcS variables

. /lib/init/vars.sh

# Define LSB log_* functions.
# Depend on lsb-base (>= 3.0-6) to ensure that this file is present.
. /lib/lsb/init-functions
# Function that starts the mrtg daemon
start()
{
env LANG=C start-stop-daemon --start --quiet --exec $DAEMON -- $DAEMON_ARGS
}
# Function that stops the mrtg daemon
stop()
{
start-stop-daemon --stop --quiet --retry=TERM/30/KILL/5 --pidfile $PIDFILE
}
case "$1" in
start)
log_daemon_msg "Starting $DESC"
start
case "$?" in
0) log_end_msg 0 ;;
1) log_end_msg 1 ;;
esac
;;
stop)
log_daemon_msg "Stopping $DESC"
stop
case "$?" in
0) log_end_msg 0 ;;
1) log_end_msg 1 ;;
esac
;;
restart|force-reload)
log_daemon_msg "Restarting $DESC"
stop
case "$?" in
0|1)
start
case "$?" in
0) log_end_msg 0 ;;
1) log_end_msg 1 ;;
esac
;;
esac
;;
status)
status_of_proc "$DAEMON" "$NAME"
;;
*)
echo "Usage: $SCRIPTNAME {start|stop|status|restart|force-reload}"
;;
esac
exit 0
### END OF SCRIPT





Note the two lines toward the top

NAME=mrtgG1
DAEMON_ARGS="/etc/mrtg/mrtgG1.cfg"


These will have to match whatever you called your .cfg and .pid files.




Now issue the following commands to make it executable and move it to the startup folder

cd /home
sudo chmod +x mrtgG1
sudo mv mrtgG1 /etc/init.d/



issue the following to add it to run levels for startup
sudo update-rc.d mrtgG1 defaults


test with
sudo /etc/init.d/mrtgG1 restart

repeat to make each .cfg file run at startup.



Hooray, you now have mrtg running, graphing the bandwidth used on a slew of switches, with separate groups running in separate daemons so your server can get through them all in the 5 minute window it has before it needs to start again, and they will start up again if you need to reboot the server. Hopefully this helped someone! Let me know any tweaks, if you know how to run the multiple .cfg files out of one startup file, or if I made a mistake somewhere!

Tuesday, July 19, 2011

RIPv2 passive interfaces

So the only RIP to be encountered on the ROUTE test appears to be passive interfaces.

So in EIGRP and OSPF passive interfaces do not send or receive routing updates, they do not participate in neighborships or the routing protocol.

RIPv2 on the other hand:
A passive interface does not SEND updates out this interface, it will still install RIP updates it receives.


R1#
router rip
version 2
passive interface s0/0/1


Router 1 will now receive routes from s0/0/1 but will not send any information out s0/0/1.

Multicast Vlan Feature

So, after upgrading to a new version of WLC code, 7.0.116.0 you might notice a new feature in your SSID configs, multicast vlan feature. I wasn't sure what it did so I looked it up and best as I can tell it goes like this.


Enabled per ssid (a handy checkbox or config wlan multicast interface wlan_id enable interface_name)

If you use the VLAN select feature, each client will be listening to the multicast stream on a different VLAN, and the upstream router must send a copy for each VLAN, and multiple copies of a multicast stream are sent out over the air. (not very good use of your air time, and negates some of the purpose of multicast).

Multicast optimization lets you create a multicast vlan, and the controller will make sure all multicast streams go out on the multicast vlan so that the upstream router only sees the single multicast entry for all the VLANs in your pool.

So clients on separate vlans will still only have one single multicast stream over the air.


Since I'm not currently using VLAN pooling I'm not 100% solid on how this would look in the real world but certainly seems a strong feature to have enabled if you were.

Friday, July 15, 2011

Layer 2 edge port security

I was asked the question "how I would secure switch ports" in an interview, and of course I blanked on most of it, but thought I would compile a list now that I'm not in the hot seat.

1. Not allow trunking - in cisco terms switchport mode access (if it needs to be a trunk, make sure the native vlan is not vlan 1, and you add switchport trunk allowed vlan x,y,z to only allow necessary vlans on the port.)

2. BPDU guard and/or root guard - do not allow your spanning tree to be hijacked. BPDU guard will block or restrict any port that someone attaches a spanning tree capable device to (eg. anything that sends a BPDU will shut down the port) while root guard will block it if anything off that port attempts to become your spanning tree root. This can be attached to your own switch uplinks/downlinks if you KNOW you never want the neighbor switch or any switches down that branch to ever be root.

3. no cdp enable - no sense broadcasting out to every client what switch, Management IP address, port, IOS version etc. that they are connected to.

4. DHCP snooping/ Dynamic ARP inspection - There are a few commands related to DHCP snooping that will give you a few benefits.
a. You can block any edge port that offers DHCP responses - blocking clients from running a DHCP server on your network.
b. You can tell what IP address has been dynamically assigned out a port, and if an ARP response is sent out not matching that address (eg. IP spoofing) block the port
c. limit how many DHCP requests are sent out per second
d. I think there is another command or two that can go with this for more features but I haven't studied switch since it was the BCMSN so it has been a year+

5. Port Security - limit what mac addresses are allowed to be used on a port
a. options here include statically set addresses, or sticky addresses so you don't have to type them in yourself and the switch saves the address first seen on the port.
b. You can then limit how many addresses are allowed in a port, (limit 1 etc.)
c. Obviously MAC addresses are easy to spoof if they unplugged a different device, but if it is just an open port, they aren't likely to guess a statically assigned mac address

6. shut down unused ports - I know, I'm terrible at this, but shut or disable ports that do not have a computer plugged into them. No sense letting someone walk into your IDF and plugging in any open port without you knowing about it.

7. Network Access Control - Bit more involved here - you can have it fairly simple where the machine and user need to match up with an AD or Radius authentication, to more advanced options where User A on Machine x at Time H is placed on vlan V, but if they are on a different machine or a different time, perhaps they are placed in a more restricted vlan since it isn't their normal working hours.
NAC also has options to verify the machine is up to date on patches and virus definitions before being placed on a vlan. Lots and Lots to think about before implementing NAC.

8. Black Hole Vlan - Just in-case you forget to shut the port down, assign all unused ports to a "Black Hole Vlan" or a vlan that is not routed or trunked so the user can't get anywhere if they do manage to connect.

9. VACLs, I typically do ACLs at the router, but feel free to assign VACLs to limit traffic WITHIN the vlans to further secure INTRA-VLAN traffic. (if for some terrible reason your switch management IP is on the client vlan, make sure you add a VACL and/or vty ACL to keep clients from accessing it!!!)



10. ??? I'm sure there is more, what have I missed? Feel free to comment, argue, or add more for typical edge port security.

Thursday, July 14, 2011

EIGRP stub redistribution

Q. Will the following static route be redistributed?

ip route 10.1.1.0 255.255.255.0 10.2.2.1

router eigrp 100
redistribute static 1000 1 255 1 1500
eigrp stub


A. NO - eigrp stub default is summary and connected, redistributed static routes will NOT be advertised based on that stub command. The command eigrp stub static needs to be included as well

Wednesday, July 13, 2011

EIGRP Stub Networks

A stub router indicates in it's hello packet, to let other routers know it is a stub router.

Q. Why use a EIGRP Stub?

A.
1. Limit amount of convergence traffic sent over links
2. Tell other routers you will NOT be a transit path for any additional networks. (eg. if a router is in active because it lost it's path, do not query me because I won't help you)
3. Avoid possible Stuck-In-Active scenarios

Typically, only remote routers are set as stubs, and only if they are single router sites. When a route goes down, stub routers ARE NOT QUERIED to find an alternate path, hub (regular neighbor routers) answer on behalf of the stub router.

*It is useful to note, stub features do not prevent other routers from advertising routes to the stub router.

Cisco defines a stub router as one which "core transit traffic should not flow"


Configure a router as a stub with the commands

eigrp stub [receive-only|connected|static|summary]

Any combination (other than receive-only) can be placed after the stub command.

The default is both connected and summary.

receive-only restricts the router from sharing any of it's routes with another router. When would you use this? If a router only has one interface in use, perhaps you have a router in place for the sole purpose of providing DHCP, VoIP gateway, or other services.

The connected, static, and summary commands are fairly self explanitory. Note that these must be in place AND the network command or redistribute command must be in use. Adding connected does not automatically add all connected routes to EIGRP, it only allows EIGRP to advertise them to neighboring routers if it is already in the routing table.

If a true stub network is desired, the hub (regular) router is typically configured to send a default route to the spoke (stub) routers so that only a single route is passed to them.

Wednesday, July 6, 2011

EIGRP passive Interfaces

This may be more of a CCNA topic but lets look at passive interfaces. First we have a router. S0/0/1 is connected to another router, Fa0/0 is connected to a client vlan (no router should ever try to neighbor from here). You have two options, you could redistribute connected networks, (but this may have issues with the administrative distance being labeled as exterior EIGRP and being higher, and it is not as clean a way of doing it) or the preferred way of setting this up is using passive interfaces.

Q. Will a passive interface accept hellos?
A. No, passive interfaces do not send unicast or multicast hellos, and do not accept them either. The router will not neighbor off this interface.

This adds security/stability for your EIGRP domain, and lessens the CPU load on the router as it isn't creating hellos on a silent interface.

Q. How do I make an interface passive?
A. Two ways
Option 1:
router eigrp 1
passive-interface fa0/0

Option 2:
router eigrp 1
passive-interface default
no passive-interface s0/0/1

Unless you know you are going to place a neighbor off an interface, always make it passive. A malicious user (or un-knowledgeable Jr Tech) Could place a router off this interface (eg. off any access layer switch port) and affect your entire routing domain unless you use the passive interface command and prevent it.


It may be easier to use the second option, and make all interfaces passive unless you specifically turn them on, but make sure you make this change while consoled into the router, inputting that command remotely will likely turn your uplink interfaces passive and kill your routing/connection.

Q. How do I check if my interfaces are passive?
A. show ip eigrp interfaces will not show any passive interfaces, or show ip protocols will explicitly list all passive interfaces.

EIGRP Hello & Hold Timers

Lets take a brief look at EIGRP Hello and Hold Timers.

So, R1 and R2 want to form a Neighbor relationship.

Q. Do the timers need to match?
A. No, but if a hello timer is longer than it's hold timer, or slow links cause that you will have flapping links.


Q. How do I change the Hello and Hold Timers?
A. R1:
interface fa0/1
ip hello-interval eigrp 55 2 (where 55 is the AS number, 2 is the timer value in seconds)
ip hold-time eigrp 55 6 (cisco does not force you to make this 3 times the hello, but it is strongly recommended)


Q. So does this mean I need to make sure R2 uses a hello timer less than 6 to keep my links from flapping?
A. No, the hold time of six is sent to R2 to use. The value set on R2 will be sent to R1 to tell R1 how long to hold waiting for hello messages. The hold time set is the value actually used on the neighboring router.
If multiple routers are on this segment, they will all use this hold time while waiting for R1's next hello.

So you can interpret ip hold-time eigrp 55 6 as "tell all my neighbors with eigrp 55 to use a 6 second hold time when they listen for me".

R2 could use 10/30 as it's hello/hold-timers and the nieghborship would stay up.


Q. How can you verify your hello times?
A. show ip eigrp interface detail interface

Q. Does that show the hold timer as well?
A. No, you need to look at the show run or use show ip eigrp neighbors command and watch the countdown until it resets to it's highest value.

Q. How does the router send Hellos?
A. Routers using EIGRP send Hellos to multicast 224.0.0.10 attempting to find potential neighbors.


For general hello packet and neighboring information, read my hello packet post from last year