Large amount of HSRP traffic?

[Jonteponte]

Hi,

Our organization has recently begun analysizing our network traffic and we have discovered that a large amount of outgoing traffic is HSRP (Hot Standby Router Protocol) packages. With large amount I mean about 2-3Gb daily. The network consist of all Cisco equipment on our side and we're connected to our "ISP" with a 10Mb line, we however don't have any knowledge about the equipment that connects us, that part is invisble to us. We have not initially configured our network, it was done by consultants, but we are now trying to get a better hold on how the network is built.

So, does anyone have any idea on if this is normal or is something very wrong?

[Kevin Dorrell]

This certainly is not normal, and certainly the HSRP should not be getting out of your LAN. Even then, each VLAN would normally expect to see a small Hello packet every couple of seconds, and no more than that.

So that makes me ask: what is it about the traffic that leads to the conclusion that it is HSRP?

Kevin Dorrell

Luxembourg

[Jonteponte]

We have Webtrend Firewall Suite setup with our Firewall for analyzing the traffic, but of course I assume it only classifies the traffic type from the port number used so I can't be 100% sure that it is actually HSRP. Any suggestions on how I could check this?

[Kevin Dorrell]

HSRP uses UDP port 1985, but is always a multicast to 224.0.0.2. It is quite possible that you have something else using UDP 1985 that is being mis-identified as HSRP. Here is a link to the specification of HSRP:

http://www.ietf.org/rfc/rfc2281.txt

I think you will have to trace the traffic itself, and see where it is coming from and where it is going. Unless you can get something more from your firewall logs.

Kevin Dorrell

Luxembourg

[Jonteponte]

Nr 1 on top external adresses used is:

all-routers.mcast.net (224.0.0.2) and the traffic is identified as hsrp

Is that enough to safely assume that it is indeed hsrp traffic?

[Kevin Dorrell]

I guess so, but the volume of traffic is colossal. And I still cannot see why the traffic should be finding its way out of your LAN. Normally the 224 addresses don't get beyond a single link, not least because HSRP packets are sent with a TTL=1. You seem to have a veritable storm of HSRP.

I did a quick calculation of the traffic volume of HSRP with default settings, including the MAC and IP headers, and I make it about 1.5 Mb per day per router per VLAN. ((14 b MAC, 20 b IP, 20 b HSRP) x seconds-in-a-day/3). You need an awful lot of HSRP groups to generate your level of traffic.

Do you have any matching source address that might give you a clue where it is coming from?

Kevin Dorrell

Luxembourg

[Jonteponte]

Yes, I know where it's coming from, it's coming from 2 switches. Most of our switches are part of a cluster. I'm not 100% sure of the correct terms here, but it's coming from the command switch and its backup.

Could this be an issue of a faulty configuration? Should I post the running-config from the command switch perhaps?

[Kevin Dorrell]

Yes please. I think there is something strange going on. Are they 3550 multilayer switches?

Kevin Dorrell

Luxembourg

[Jonteponte]

They are of the model "Catalyst 2950", not sure what layers they work on (link to switch information page: http://www.cisco.com/en/US/products/hw/switches/ps628/index.html )

Running-config on command switch:

Current configuration : 3321 bytes

!

version 12.1

no service pad

service timestamps debug uptime

service timestamps log datetime

no service password-encryption

service sequence-numbers

!

hostname 216-1005-A

!

clock timezone Lokal 0

ip subnet-zero

no ip finger

ip domain-name

ip name-server 10.68.12.13

ip name-server 10.68.12.129

cluster standby-group MHC-CMS

cluster enable MHC 0

cluster member 1 mac-address 0004.c1c9.82c0

cluster member 2 mac-address 000a.b78e.4ec0

cluster member 3 mac-address 0007.84fd.2c80

cluster member 4 mac-address 000a.f4c5.c180

cluster member 5 mac-address 0050.8071.adc0

cluster member 6 mac-address 00d0.58ae.93c0

cluster member 7 mac-address 000a.b78e.5000

cluster member 8 mac-address 0001.4222.6200

cluster member 9 mac-address 0004.c076.ef80

cluster member 11 mac-address 0001.4222.6f00

cluster member 12 mac-address 000e.8352.ffc0

!

cluster discovery hop-count 7

!

!

interface Port-channel1

!

interface Port-channel2

!

interface Port-channel6

!

interface Port-channel4

!

interface Port-channel5

!

interface Port-channel3

!

interface FastEthernet0/1

description mhbackup

channel-group 1 mode auto

!

interface FastEthernet0/2

description mhbackup

channel-group 1 mode auto

!

interface FastEthernet0/3

description mhnas

channel-group 2 mode auto

!

interface FastEthernet0/4

description mhnas

channel-group 2 mode auto

!

interface FastEthernet0/5

description mhnas

channel-group 2 mode auto

!

interface FastEthernet0/6

description mhnas

channel-group 2 mode auto

!

interface FastEthernet0/7

description mhnnt04

!

interface FastEthernet0/8

description mhts06

!

interface FastEthernet0/9

description mhweb

channel-group 5 mode auto

!

interface FastEthernet0/10

description mhweb

channel-group 5 mode auto

!

interface FastEthernet0/11

description mhts01

!

interface FastEthernet0/12

description mhprint

!

interface FastEthernet0/13

description mhts06

!

interface FastEthernet0/14

description mhts02

duplex full

speed 100

!

interface FastEthernet0/15

description mhts03

duplex full

speed 100

!

interface FastEthernet0/16

!

interface FastEthernet0/17

description mhad01

channel-group 4 mode auto

!

interface FastEthernet0/18

description mhad01

channel-group 4 mode auto

!

interface FastEthernet0/19

description mhtesec

channel-group 6 mode auto

!

interface FastEthernet0/20

description mhtesec

channel-group 6 mode auto

!

interface FastEthernet0/21

description mhsql

channel-group 3 mode auto

!

interface FastEthernet0/22

description mhsql

channel-group 3 mode auto

!

interface FastEthernet0/23

description mhts05

!

interface FastEthernet0/24

description mhts05

!

interface GigabitEthernet0/1

!

interface GigabitEthernet0/2

!

interface Vlan1

ip address 10.68.12.69 255.255.252.0

no ip redirects

no ip route-cache

standby priority 150 preempt delay sync 0

standby name MHC-CMS

standby ip 10.68.12.72

!

!

ip default-gateway 10.68.12.1

ip http server

!

ip access-list extended CMP-NAT-ACL

snmp-server engineID local xxxxx

snmp-server community xxx

snmp-server community xxx

snmp-server community xxxx

snmp-server location Serverrum Vvre

!

line con 0

transport input none

line vty 0 4

password xxxxx

login

line vty 5 15

password xxxxx

login

!

end

[Jonteponte]

Noone with the faintest idea of why this could be happening? Could it have something to do with the cluster configuration?

[7rbowenii]

Actually, I can't think of a single reason why someone would configure HSRP on a 2950. There is no value added (that I know of). If the HSRP address assigned to the switches (10.68.12.72) isn't the default gateway of any hosts on your network, remove that part of the configuration from your switches. If that is an address used as the default gateway for some of your hosts, chagne the default gatreway to 10.8.12.1 as I believe (from the configuration you posted) that the .1 address belongs to a router in your network somewhere. I don't mind getting into more detail with you over this, but turning this "feature" off will resolve the problem that you are faced with now.

[Jonteponte]

Thank you, that sounds very hopeful! :)

I was under the impression that the .72 IP was a IP assigned to the entire Switch Cluster, the thing called MHC-CMS in the configuration file. Of course, I have no real idea what the use would be except that I am able to enter the .72 IP in my browser and start the CMS.

But just to be clear, I can remove the "standby ip 10.68.12.72" line and that alone would remove the HSRP configuration and solve my problem?

[7rbowenii]

I am working on a test config. Don't remove anything yet...

-Bo

[Jonteponte]

Any luck with thast test config?

[7rbowenii]

Sorry for not getting back to you sooner. I attempted to set up a lab utilizing 2950s. However, I was unable to recreate the problem. It maybe that I don't have enough clustering resources to create the same volume fo traffic.