[Routing] - Routing issue at Cisco 2951 router

[luiz.alexandre.paiva]

We have a hosting/housing supplier that provides us connecivity through two links and we are using HSRP.

Las week after rebooting our router, it stopped reaching specifically the HSRP address at the suppliers routers, but still can reach the real addresses at the equipments:

RCURST08#ping  10.216.16.190

Type  escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.216.16.190,  timeout is 2 seconds:
U.U.U
Success rate is 0 percent  (0/5)
RCURST08#
RCURST08#ping 10.216.16.189

Type  escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.216.16.189,  timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip  min/avg/max = 1/1/4 ms
RCURST08#ping 10.216.16.188

Type  escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.216.16.188,  timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip  min/avg/max = 1/1/4 ms
RCURST08#

Our router informations:

Anterior:

ip route  10.216.17.0 255.255.255.0 10.216.16.190 name TIVIT

Atual:

ip route  10.216.17.0 255.255.255.0 10.216.16.188 name TIVIT

RCURST08#sh ip route  10.216.17.0
Routing entry for 10.216.17.0/24
  Known via "static",  distance 1, metric 0
  Routing Descriptor Blocks:
  *  10.216.16.188
      Route metric is 0, traffic share count is  1
RCURST08#

RCURST08#sh ip route  10.216.16.190
Routing entry for 10.216.16.176/28
  Known via "connected",  distance 0, metric 0 (connected, via interface)
  Routing Descriptor  Blocks:
  * directly connected, via BVI11
      Route metric is 0, traffic  share count is 1
RCURST08#
RCURST08#sh ip route 10.216.16.188
Routing  entry for 10.216.16.176/28
  Known via "connected", distance 0, metric 0  (connected, via interface)
  Routing Descriptor Blocks:
  * directly  connected, via BVI11
      Route metric is 0, traffic share count is  1
RCURST08#sh ip route 10.216.16.189
Routing entry for  10.216.16.176/28
  Known via "connected", distance 0, metric 0 (connected,  via interface)
  Routing Descriptor Blocks:
  * directly connected, via  BVI11
      Route metric is 0, traffic share count is  1
RCURST08#

RCURST08#sh run int bvi11
Building  configuration...

Current configuration : 211  bytes
!
interface BVI11
description Intercon_Softway_TIVIT Vlan.11
ip address 10.172.65.101  255.255.255.252 secondary
ip address 10.251.127.209 255.255.255.252  secondary
ip address 10.216.16.177  255.255.255.240
!
end

RCURST08#

RCURST08#sh int bvi11
BVI11 is up, line protocol is up
  Hardware is BVI, address is 588d.095d.35a2 (bia 0000.0000.0000)
  Description: Intercon_Softway Vlan.11
  Internet address is 10.216.16.177/28
  MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 5000 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 2d20h, output never, output hang never
  Last clearing of "show interface" counters never
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/0 (size/max)
  5 minute input rate 234000 bits/sec, 118 packets/sec
  5 minute output rate 193000 bits/sec, 116 packets/sec
     22953711 packets input, 6005157828 bytes, 0 no buffer
     Received 0 broadcasts, 0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
     24760494 packets output, 3518436390 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 unknown protocol drops
     0 output buffer failures, 0 output buffers swapped out
RCURST08#

RCURST08#sh run int g0/2
Building configuration...Current configuration : 79 bytes
!
interface GigabitEthernet0/2
no ip address
duplex auto
speed auto
!
endRCURST08#sh run int g0/2.11
Building configuration...Current configuration : 80 bytes
!
interface GigabitEthernet0/2.11
encapsulation dot1Q 11
bridge-group 11
endRCURST08#sh int g0/2      
GigabitEthernet0/2 is up, line protocol is up
  Hardware is PQ3_TSEC, address is 588d.095d.35a2 (bia 588d.095d.35a2)
  MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation 802.1Q Virtual LAN, Vlan ID  1., loopback not set
  Keepalive set (10 sec)
  Full-duplex, 1000Mb/s, media type is RJ45
  output flow-control is unsupported, input flow-control is unsupported
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 00:00:00, output 00:00:00, output hang never
  Last clearing of "show interface" counters never
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 252000 bits/sec, 160 packets/sec
  5 minute output rate 217000 bits/sec, 140 packets/sec
     30357690 packets input, 6516602290 bytes, 0 no buffer
     Received 6369564 broadcasts, 0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 4446773 multicast, 0 pause input
     0 input packets with dribble condition detected
     24955712 packets output, 3662033203 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     3 unknown protocol drops
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 pause output
     0 output buffer failures, 0 output buffers swapped out
RCURST08#
RCURST08#sh int g0/2.11
GigabitEthernet0/2.11 is up, line protocol is up
  Hardware is PQ3_TSEC, address is 588d.095d.35a2 (bia 588d.095d.35a2)
  MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation 802.1Q Virtual LAN, Vlan ID  11.
  Keepalive set (10 sec)
  ARP type: ARPA, ARP Timeout 04:00:00
  Last clearing of "show interface" counters never
RCURST08#

RCURST08#sh ver
Cisco IOS Software, C2951 Software (C2951-UNIVERSALK9-M), Version 15.0(1)M4, RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2010 by Cisco Systems, Inc.
Compiled Thu 28-Oct-10 18:58 by prod_rel_teamROM: System Bootstrap, Version 15.0(1r)M6, RELEASE SOFTWARE (fc1)RCURST08 uptime is 3 days, 10 hours, 19 minutes
System returned to ROM by reload at 01:42:04 UTC Fri May 6 2011
System restarted at 01:43:29 UTC Fri May 6 2011
System image file is "flash0:c2951-universalk9-mz.SPA.150-1.M4.bin"
Last reload type: Normal Reload
Last reload reason: Reload CommandThis product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.htmlIf you require further assistance please contact us by sending email to
export@cisco.com.Cisco CISCO2951/K9 (revision 1.1) with 1003520K/45056K bytes of memory.
Processor board ID FTX1447A0MH
4 Gigabit Ethernet interfaces
4 Serial(sync/async) interfaces
1 terminal line
1 Virtual Private Network (VPN) Module
DRAM configuration is 72 bits wide with parity enabled.
255K bytes of non-volatile configuration memory.
500472K bytes of ATA System CompactFlash 0 (Read/Write)
License Info:License UDI:-------------------------------------------------
Device#   PID                   SN
-------------------------------------------------
*0        CISCO2951/K9          FTX1447A0MH    Technology Package License Information for Module:'c2951'----------------------------------------------------------------
Technology    Technology-package          Technology-package
              Current       Type          Next reboot 
-----------------------------------------------------------------
ipbase        ipbasek9      Permanent     ipbasek9
security      securityk9    Permanent    securityk9
uc            None          None          None
data          None          None          NoneConfiguration register is 0x2102RCURST08#

Detail is that we have HSRP at our internal firewalls that communicates with this router through other interface and is fine.

If someone has some experience or idea of what could be happenning?

Rds,

Luiz

[Calin C.]

Hello,

DId you checked with your provider that the HSRP configuration is fine on their side?

A "show standby brief" on their side could clarify if everything is fine.

How is your physical connection with the provider done? Two interfaces bridged? I see only one interface and one subinterface in your output.

Cheers,

Calin

[Richard Burts · Accepted solution]

Luiz

It would perhaps be helpful if you would post the output of show arp from the router. From the symptoms you describe I suspect that there are arp entries for 188 and 189 but not for 190.

HTH

Rick

Sent from Cisco Technical Support iPhone App

[luiz.alexandre.paiva]

Hi,

At supplier they say that is everything ok, and at my router here is the show ARP:

RCURST08#sh arp
Protocol  Address          Age (min)  Hardware Addr   Type   Interface
Internet  10.172.65.101           -   588d.095d.35a2  ARPA   BVI11
Internet  10.172.65.102         120   000d.bcb9.0c60  ARPA   BVI11
Internet  10.216.16.177           -   588d.095d.35a2  ARPA   BVI11
Internet  10.216.16.178          35   000d.bcb9.0c60  ARPA   BVI11
Internet  10.216.16.179          25   000d.bcb9.0c60  ARPA   BVI11
Internet  10.216.16.180          21   000d.bcb9.0c60  ARPA   BVI11
Internet  10.216.16.183          19   000d.bcb9.0c60  ARPA   BVI11
Internet  10.216.16.188         120   001e.be17.0818  ARPA   BVI11
Internet  10.216.16.189         120   0024.97ea.8088  ARPA   BVI11
Internet  10.216.16.190          62   000d.bcb9.0c60  ARPA   BVI11
Internet  10.251.127.209          -   588d.095d.35a2  ARPA   BVI11
Internet  10.251.127.210        120   0064.405b.d6b0  ARPA   BVI11
Internet  172.25.121.9            -   588d.095d.35b8  ARPA   GigabitEthernet2/0
Internet  172.25.248.113        158   0000.0c07.ac06  ARPA   GigabitEthernet0/0
Internet  172.25.248.114         88   0030.8040.6aa2  ARPA   GigabitEthernet0/0
Internet  172.25.248.115        142   0030.8047.d9a2  ARPA   GigabitEthernet0/0
Internet  172.25.248.116        218   00e0.1e8d.fab0  ARPA   GigabitEthernet0/0
Internet  172.25.248.117          -   588d.095d.35a0  ARPA   GigabitEthernet0/0
Internet  172.25.248.118        222   0016.c724.4521  ARPA   GigabitEthernet0/0
RCURST08#

[Richard Burts · Accepted solution]

Luiz

Thanks for posting the output of show arp as I asked. I am surprised to see that there is an arp entry for the .190 address. I notice that the same MAC address is also in the arp table for addresses 178, 179, 180, 183. Do you know anything about those addresses? Are those addresses pingable?

HTH

Rick

[luiz.alexandre.paiva]

I'm looking but they belong to the range and are not active.

Other strange thing is about the Int BVI11 that is showing MAC as 0:

BVI11 is up, line protocol is up
  Hardware is BVI, address is 588d.095d.35a2 (bia 0000.0000.0000)

I've tried to  use the MAC address command with the one of Giga inteface, but it seems not to recognize:

RCURST08#sh int g0/2.11
GigabitEthernet0/2.11 is up, line protocol is up
  Hardware is PQ3_TSEC, address is 588d.095d.35a2 (bia 588d.095d.35a2)

Could this be a signal of problem?

Rds,

Luiz

[luiz.alexandre.paiva]

These are information at SP main router that answers for HSRP:

Renault-Principal#sho arp

Protocol  Address          Age (min)  Hardware Addr   Type   Interface

Internet  10.216.16.177          18   588d.095d.35a2  ARPA   FastEthernet0/0

Internet  10.216.16.190           -   0000.0c07.ac01  ARPA   FastEthernet0/0

Internet  10.216.16.188           -   001e.be17.0818  ARPA   FastEthernet0/0

Internet  10.216.16.189           1   0024.97ea.8088  ARPA   FastEthernet0/0

Renault-Principal#sho standby

FastEthernet0/0 - Group 1

  State is Active

    131 state changes, last state change 5w1d

  Virtual IP address is 10.216.16.190

  Active virtual MAC address is 0000.0c07.ac01

    Local virtual MAC address is 0000.0c07.ac01 (v1 default)

  Hello time 3 sec, hold time 10 sec

    Next hello sent in 1.836 secs

  Preemption enabled, delay min 10 secs

  Active router is local

  Standby router is 10.216.16.189, priority 95 (expires in 7.208 sec)

  Priority 100 (default 100)

    Track interface Serial0/0/0 state Up decrement 10

  IP redundancy name is "hsrp-Fa0/0-1" (default)

     10.0.0.0/8 is variably subnetted, 5 subnets, 5 masks

C       10.216.16.176/28 is directly connected, FastEthernet0/0

S       10.216.17.0/24 is directly connected, Serial0/0/0

S       10.221.0.0/16 [1/0] via 10.216.9.41

C       10.216.9.41/32 is directly connected, Serial0/0/0

C       10.216.9.40/30 is directly connected, Serial0/0/0

S*   0.0.0.0/0 [1/0] via 10.216.16.177

[Richard Burts · Accepted solution]

Luiz

Thank you for the additional information. It is quite helpful. It shows what is preventing access to the .190 address though it does not show what the underlying cause of the problem is.

First let us look at the arp table from your previous post:

Internet  10.216.16.178          35   000d.bcb9.0c60  ARPA   BVI11
Internet  10.216.16.179          25   000d.bcb9.0c60  ARPA   BVI11
Internet  10.216.16.180          21   000d.bcb9.0c60  ARPA   BVI11
Internet  10.216.16.183          19   000d.bcb9.0c60  ARPA   BVI11
Internet  10.216.16.190          62   000d.bcb9.0c60  ARPA   BVI11

The important entry is the one for 190 but I include the others because they show the same MAC address. So your router believes that the MAC address associated with .190 is 000d.bcb9.0c60.

Then let us look at the entry on the ISP router:

Internet  10.216.16.190           -   0000.0c07.ac01  ARPA   FastEthernet0/0

so on the ISP router the MAC address is 0000.0c07.ac01. This is a normal MAC address for HSRP and it is different from the MAC that your router is using.

So your router can not access .190 because it is learning the wrong MAC address for 190.

It is not clear from what we know so far what is causing the wrong MAC to be learned. Is 000d.bcb9.0c60 by any chance the MAC associated with a firewall or something like that?

HTH

Rick

[luiz.alexandre.paiva]

The point is that they are connected at layer 2, at same Vlan in a 2950 switch.

[Richard Burts · Accepted solution]

Luiz

Some device other than the ISP router is responding to the ARP request for the .190 address and this is what is causing the problem. If you can find what device has MAC  000d.bcb9.0c60 then you may be able to fix this problem.

If the connections are in a 2950 switch then you should be able to look into the mac address table and find what port has MAC  000d.bcb9.0c60 and then find what device is connected on that switch port.

HTH

Rick

[luiz.alexandre.paiva]

Using the sh mac-address at switch found this address at one other supplier's router, that ha been working for the last 3 years without problem en know decided to call attentiom. Apparentally no bug or problem or even config mistake.

Good case to study deeper.