Asymmetrical Routing + ARP + HSRP issue

nicholas.pace · ‎08-26-2011

Good morning,

I have 2 CRS-1 Routers configured for HSRP which act as default gateway to 4 Cisco 2511 network consoles. In this configuration, we have asymmetrical routing setup, so inbound traffic goes from R1 ----- towards cisco 2511 network consoles (4 of th em) which hang off a single 3750. When the 2511''s access the internet, they travel over the same layer 2 circuit but towards R2 and this is the active router within HSRP.

My problem!!

When the 4 network consoles are rebooted, they arp to the subnet and everything works fine howver, when the arp time-out expires on the routers, communication is lost to the consoles and arp is never re-learned from the routers. I check the debug on the consoles and can see the arp requests come in from the routers, but I think when it sends back its request, to goes to R2 (which isnt the one requesting arp.. its R1 --- remember traffic flow).

When we lose communication to the 2511 and I console into it, its arp table looks fine but it can't ping any of the HSRP Ips.

I know that I could put in static arps for these consoles but this is not an option in our network.

I'n very stumped here and would appreciate any assistance.

nicholas.pace · ‎08-26-2011

Additionally:

If I clear the mac address table on any of the network consoles, I will re-learn all of the same mac addresses and connectivity will be restored until the arp time expires on the routers.

Also:

If I plug a network console directly into the MPLS switch (green area of diagram) all of these issues go away. So it seems that the 3750 switch could be the cause of this issue.

Jon Marshall · ‎08-26-2011

Nicholas

Can you post visio as jpg.

Is the 3750 only connected to one router ?

Which link is blocking with STP ? or are you running HSRP via the 3750 ?

Jon

nicholas.pace · ‎08-26-2011

Hi Jon,

The routers are connected via Alcatel MPLS (only using layer 2). STP is not enabled on any of these devices and layer 2 redundancy is handled by alcatel. the 3750 is connected to both routers via mpls. hsrp is not enabled on the 3750, and the 3750 is providing only layer 2 comunication.

Jon Marshall · ‎08-26-2011

Nicholas

I don't think the issue is the arp request being sent back to R2 when R1 arps out simply because the arp request from R1 should have the source mac-address of R1's interface and as this is all L2 then the 2511 will send the arp reply to R1. At least it should do as you say all links are forwarding ie. there is no blocking.

R2 only becomes relevant when the destination is not on the same vlan because packets then need to be sent to the HSRP active gateway.

I'm not familiar with Alcatel but is the port connected to the 3750 configured any differently than the port connected directly to the 2511 ?

You say there is no blocking because there is no STP but what stops L2 loops in this network ? Whatever it is i would look there first.

Jon

ameya_oke · ‎08-26-2011

Nicolas,

Can you share config of a 2511 router, 3750 and vlan info on these(If vlans are configured)

It seems to be a layer 2 issue.

Question1:

(I believe all switches 3750,2511,Alcatels are in one subnet)

Have you configured ip default-gateway 64.59.159.225 on all 2511 routers?

Question 2:

Also if that is the case and correct me if i am wrong the traffic flow when you are taking console is....

Forward Pathof Console:

Packet Enters on R1 -> Alcatel Switches -> 3750 -> 2511

Reverse Path:

2511 -> 3750 -> Alcatel Switches -> R2 -> 'X'

Now the packet has to go to R1 for reverse path reachability ,right??

Also please elaborate on packet flow from this point.

Do u have Direct connectivity between R1 and R2?

Ameya

CCNP

nicholas.pace · ‎08-26-2011

Hi Ameya,

I really think this is a problem with the 3750. Reason being is that when I take it out of the picture, I have no problems. The routers do have a direct connection between one another.

The switches here (3750, Alcatel) are on the same layer 2 network. They are unaware of the subnet. All of my routing happens on the CRS's.

Another thing to note is... if I keep all cfgs the same and clear arp on the 2511s, it works until the arp time-out occurs on the routers. I see this on the router after the timeout

R2_#show arp | i 64.59.159.233

64.59.159.233 - 0000.0000.0000 Deleted ARPA TenGigE0/3/0/2.1

2511#show arp

Protocol Address Age (min) Hardware Addr Type Interface

Internet 64.59.159.230 47 000f.2349.e5c2 ARPA Ethernet0

Internet 64.59.159.225 0 0000.0c07.ac0f ARPA Ethernet0

Internet 64.59.159.226 0 5475.d023.9a12 ARPA Ethernet0

Internet 64.59.159.227 0 5475.d024.1a12 ARPA Ethernet0

Internet 64.59.159.233 - 00b0.64fd.2bf7 ARPA Ethernet0

Internet 64.59.159.234 9 0013.5f06.6024 ARPA Ethernet0

Internet 64.59.159.235 47 00b0.64fd.216d ARPA Ethernet0

2511#ping 64.59.159.225

Type escape sequence to abort.

Sending 5, 100-byte ICMP Echos to 64.59.159.225, timeout is 2 seconds:

.....

Success rate is 0 percent (0/5)

Jon Marshall · ‎08-26-2011

Nicholas

As i said with arp the HSRP active gateway is largely irrelevant.

Is there anything special configured on the 3750 ?

What about the Alcatel switch that the 3750 connects to ?

Jon

nicholas.pace · ‎08-26-2011

the port connecting to mpls is setup in a port channel as such:

3750#

interface GigabitEthernet1/0/1

switchport trunk encapsulation dot1q

switchport mode trunk

speed nonegotiate

channel-group 1 mode active

end

interface Port-channel1

switchport trunk encapsulation dot1q

switchport mode trunk

speed nonegotiate

end

To answer this question: "You say there is no blocking because there is no STP but what stops L2 loops in this network ? Whatever it is i would look there first."

Our MPLS switches have their own built in redundancy to accomidate for looping. the vlan for this layer 2 circuit doesn't seem to be looped. The same layer 2 circuit is used when I plug my network console direct to the mpls bypassing the 3750 and it works.

Is there anything on the 3750 that I should check?

Thanks for the info on the active gateway Jon, I was getting a little confused by that.

Jon Marshall · ‎08-26-2011

Nicholas

Perhaps you can post the 3750 config ?

It may come down to capturing packets ie. removing the arp entry on R1 for one of the 2511 and then pinging from R1 and spanning a port on the 3750 to a device running wireshark to see exactly what is happening with the packets.

Jon

nicholas.pace · ‎08-26-2011

Current configuration : 17642 bytes

!

! Last configuration change at 11:43:11 PDT Fri Aug 26 2011 by npace

! NVRAM config last updated at 11:43:14 PDT Fri Aug 26 2011 by npace

!

version 12.2

no service pad

service timestamps debug datetime msec localtime show-timezone

service timestamps log datetime msec localtime show-timezone

service password-encryption

service unsupported-transceiver

!

hostname switch3750

!

boot-start-marker

boot-end-marker

!

no logging console

enable secret 5 xxxxxxxxxxxxxxxxxxxxxx

!

username admin privilege 15 password 7 xxxxxxxxxxxxxxxxxxxxxx

username user password 7 xxxxxxxxxxxxxxxxxxxxxxxxxxxxx

!

aaa new-model

!

aaa authentication login default group tacacs+ local

aaa authentication login local-login local

aaa authorization exec default group tacacs+ local

aaa authorization exec login-local local

aaa authorization commands 0 default group tacacs+ local if-authenticated

aaa authorization commands 1 default group tacacs+ local if-authenticated

aaa authorization commands 15 default group tacacs+ local if-authenticated

aaa accounting exec default start-stop group tacacs+

aaa accounting commands 0 default start-stop group tacacs+

aaa accounting commands 1 default start-stop group tacacs+

aaa accounting commands 15 default start-stop group tacacs+

aaa accounting system default start-stop group tacacs+

!

aaa session-id common

clock timezone PST -8

clock summer-time PDT recurring

switch 1 provision ws-c3750-48ts

system mtu routing 1500

vtp domain WAMU_1

vtp mode transparent

authentication mac-move permit

ip subnet-zero

no ip domain-lookup

ip domain-name mgmt.mlb.inet

!

crypto pki trustpoint TP-self-signed-592045440

enrollment selfsigned

subject-name cn=IOS-Self-Signed-Certificate-592045440

revocation-check none

rsakeypair TP-self-signed-592045440

!

crypto pki certificate chain TP-self-signed-592045440

certificate self-signed 01

3082024E 308201B7 A0030201 02020101 300D0609 2A864886 F70D0101 04050030

30312E30 2C060355 04031325 494F532D 53656C66 2D536967 6E65642D 43657274

69666963 6174652D 35393230 34353434 30301E17 0D313130 32303731 39343533

365A170D 32303031 30313030 30303030 5A303031 2E302C06 03550403 1325494F

532D5365 6C662D53 69676E65 642D4365 72746966 69636174 652D3539 32303435

34343030 819F300D 06092A86 4886F70D 01010105 0003818D 00308189 02818100

B72FA97D E0FBCA50 0F6649D5 E280BFA4 B9A98004 B0D630C6 EC2E9D21 16B7EA76

F01B1B1D C2CB260F 6A74437C A15953E8 465092C6 FAAE8837 30E68AF4 DEE791E2

1F2B44E4 71B9DD5E 2249C6E5 B854B8A8 9708CD30 43B22F5E B802F676 98A83309

64CF1303 99B2D0EB C958C80C F426EF95 27AEE5B9 D5F898F0 B20D23D7 4E34B3D3

02030100 01A37830 76300F06 03551D13 0101FF04 05300301 01FF3023 0603551D

11041C30 1A821873 776D312E 616E2E76 632E6D67 6D742E6D 6C622E69 6E657430

1F060355 1D230418 30168014 4EA71434 9F4DA1CD BECD656A DC4A70DE 0D360551

301D0603 551D0E04 1604144E A714349F 4DA1CDBE CD656ADC 4A70DE0D 36055130

0D06092A 864886F7 0D010104 05000381 81008048 A9AEB618 FDB55A22 9A398811

32456FB9 9E876412 9151C5CD 8D56D6F0 ADDADCEE B39C4EA3 FF27C30D DDF9C5DE

E40ED928 EE49913D 829DA3AC 9CE9DB04 A2E83697 B14BAFBB B65ADFAC 7A1B5760

BACEA05E B58AF3EC 301341DC 7DBF22CC 87367A68 5D18C93D 2D6ACD6E 03A7CE23

BEA0ED76 FA56E545 01D1A2EB FB6E2A34 A280

quit

!

errdisable recovery cause udld

errdisable recovery cause bpduguard

errdisable recovery cause security-violation

errdisable recovery cause channel-misconfig (STP)

errdisable recovery cause pagp-flap

errdisable recovery cause dtp-flap

errdisable recovery cause link-flap

errdisable recovery cause sfp-config-mismatch

errdisable recovery cause gbic-invalid

errdisable recovery cause l2ptguard

errdisable recovery cause psecure-violation

errdisable recovery cause port-mode-failure

errdisable recovery cause dhcp-rate-limit

errdisable recovery cause mac-limit

errdisable recovery cause vmps

errdisable recovery cause storm-control

errdisable recovery cause inline-power

errdisable recovery cause arp-inspection

errdisable recovery cause loopback

errdisable recovery cause small-frame

!

spanning-tree mode pvst

spanning-tree etherchannel guard misconfig

spanning-tree extend system-id

no spanning-tree vlan 1-4094

!

vlan internal allocation policy ascending

!

vlan 7

!

vlan 207

!

vlan 433

!

vlan 447

!

vlan 531

!

vlan 532

C

!

vlan 867

!

vlan 900

!

vlan 1443

!

interface Port-channel1

switchport trunk encapsulation dot1q

switchport mode trunk

speed nonegotiate

!

interface FastEthernet1/0/1

switchport access vlan 867

switchport mode access

!

interface FastEthernet1/0/2

switchport access vlan 867

switchport mode access

!

interface FastEthernet1/0/3

switchport access vlan 867

switchport mode access

!

interface FastEthernet1/0/4

switchport access vlan 867

switchport mode access

!

interface FastEthernet1/0/5

switchport access vlan 867

switchport mode access

!

interface FastEthernet1/0/6

switchport access vlan 867

switchport mode access

!

interface FastEthernet1/0/7

switchport access vlan 867

switchport mode access

!

interface FastEthernet1/0/8

switchport access vlan 867

switchport mode access

!

interface FastEthernet1/0/9

switchport access vlan 867

switchport mode access

!

interface FastEthernet1/0/10

switchport access vlan 867

switchport mode access

!

interface FastEthernet1/0/11

switchport access vlan 867

switchport mode access

!

interface FastEthernet1/0/12

switchport access vlan 867

switchport mode access

!

interface FastEthernet1/0/13

d

switchport access vlan 867

switchport mode access

speed 10

duplex half

!

interface FastEthernet1/0/14

switchport access vlan 867

switchport mode access

speed 10

duplex half

!

interface FastEthernet1/0/15

switchport access vlan 531

switchport mode access

speed 100

duplex full

!

interface FastEthernet1/0/16

switchport access vlan 531

switchport mode access

speed 100

duplex full

!

interface FastEthernet1/0/17

switchport access vlan 531

switchport mode access

speed 100

duplex full

!

interface FastEthernet1/0/18

switchport access vlan 531

switchport mode access

speed 100

duplex full

!

interface FastEthernet1/0/19

switchport access vlan 531

switchport mode access

speed 100

duplex full

!

interface FastEthernet1/0/20

switchport access vlan 531

switchport mode access

speed 100

duplex full

!

interface FastEthernet1/0/21

switchport access vlan 531

switchport mode access

speed 100

duplex full

!

interface FastEthernet1/0/22

switchport access vlan 531

switchport mode access

speed 100

duplex full

!

interface FastEthernet1/0/23

switchport access vlan 532

switchport mode access

speed 100

duplex full

!

interface FastEthernet1/0/24

switchport access vlan 7

switchport mode access

speed 10

duplex half

!

interface FastEthernet1/0/25

switchport access vlan 7

switchport mode access

speed 100

duplex full

!

interface FastEthernet1/0/26

switchport access vlan 7

switchport mode access

speed 100

duplex full

!

interface FastEthernet1/0/27

switchport access vlan 7

switchport mode access

speed 10

duplex half

!

interface FastEthernet1/0/28

switchport access vlan 7

switchport mode access

speed 100

duplex full

!

interface FastEthernet1/0/29

switchport access vlan 7

switchport mode access

speed 100

duplex full

!

interface FastEthernet1/0/30

switchport access vlan 447

switchport mode access

speed 10

duplex half

!

interface FastEthernet1/0/31

switchport access vlan 447

switchport mode access

speed 10

duplex half

!

interface FastEthernet1/0/32

switchport access vlan 447

switchport mode access

speed 10

duplex half

!

interface FastEthernet1/0/33

switchport access vlan 447

switchport mode access

speed 10

duplex half

!

interface FastEthernet1/0/34

switchport access vlan 433

switchport mode access

!

interface FastEthernet1/0/35

switchport access vlan 207

switchport mode access

!

interface FastEthernet1/0/36

switchport access vlan 1443

switchport mode access

speed 100

duplex full

!

interface FastEthernet1/0/37

switchport access vlan 531

switchport mode access

speed 100

duplex full

!

interface FastEthernet1/0/38

switchport access vlan 7

switchport mode access

speed 10

duplex full

!

interface FastEthernet1/0/39

switchport access vlan 7

switchport mode access

speed 100

duplex full

!

interface FastEthernet1/0/40

switchport access vlan 531

switchport mode access

speed 100

duplex full

!

interface FastEthernet1/0/41

switchport access vlan 531

switchport mode access

speed 100

duplex full

!

interface FastEthernet1/0/42

switchport access vlan 531

switchport mode access

speed 100

duplex full

!

interface FastEthernet1/0/43

switchport access vlan 531

switchport mode access

speed 100

duplex full

!

interface FastEthernet1/0/44

switchport access vlan 447

switchport mode access

speed 100

duplex full

!

interface FastEthernet1/0/45

switchport access vlan 531

switchport mode access

speed 100

duplex full

!

interface FastEthernet1/0/46

switchport access vlan 531

switchport mode access

speed 100

duplex full

!

interface FastEthernet1/0/47

switchport access vlan 531

switchport mode access

speed 100

duplex full

!

interface FastEthernet1/0/48

switchport access vlan 900

switchport mode access

!

interface GigabitEthernet1/0/1

description Gi1/0/1>mpls<< | GE 2/2/16 >>

switchport trunk encapsulation dot1q

switchport mode trunk

speed nonegotiate

channel-group 1 mode active

!

interface GigabitEthernet1/0/2

description Gi1/0/2>mplm1an.vc>

switchport trunk encapsulation dot1q

switchport mode trunk

speed nonegotiate

channel-group 1 mode active

!

interface GigabitEthernet1/0/3

shutdown

!

interface GigabitEthernet1/0/4

shutdown

!

interface Vlan1

no ip address

shutdown

!

interface Vlan7

ip address 172.21.55.201 255.255.255.0

!

interface Vlan447

no ip address

!

ip default-gateway 172.21.55.1

ip classless

no ip http server

ip http secure-server

!

ip tacacs source-interface Vlan7

!

ip sla enable reaction-alerts

logging facility local2

logging 10.63.113.254

no cdp run

!

line con 0

login authentication local-login

line vty 5 15

!

ntp clock-period 36028861

ntp server 172.21.112.59

ntp server 172.21.16.59

end

Jon Marshall · ‎08-26-2011

Nicholas

There is nothing obvious in the switch config. As it is acting purely as L2 and there are no redundant links i'm not convinced the issue is with the switch.

I think you are going to have to span the trunk port and see what happens when you try and ping from R1 after clearing the arp cache for one of the 2511 devices.

Jon

ameya_oke · ‎08-26-2011

Hi Jon,

What if we enable cdp on 2511 and 3750 and test traceroute mac from 2511,will it will help the cause??

We can check those frames on wireshark.

Ameya

Jon Marshall · ‎08-26-2011

Ameya

My understanding of the problem is not actually with the 2511 but with the CRS router R1. The mac-address times out and then when it arps again the request does not get back to R1 for some reason.

I think we need to understand what happens to the arp request at the 3750/2511.

Jon

nicholas.pace · ‎08-26-2011

Yup I agree with ya Jon. Thanks to both of you for helping out here. I will follow up on this thread when I have a chance to capture packets.

Thanks again