arp table entries from non existing layer3 vlan interface

[Michael KARL]

hello community

i have a strange behavior in my switch environment..

i have a 3750x switch stack which is the core switch in my network

there are some 2960s client switches with port-channel connected to that core switch

the core switch have different vlan interfaces. vlan1 for workstations and servers, vlan506 for management.

the client switches only have an management vlan interface (id506) - the native vlan1 is shutdown.

as i say, servers and workstations are located in vlan1. when i connect from a workstation, which have an ip address from vlan1, to the client switch and take an configuration backup using tftp on that workstation, the client switch insert an arp entry for this workstation - located in vlan1 - where the client switch dosent have an ip address..

when i troubleshoot this problem, i first see the mac address from the core switch vlan 506 interface and the ip address from the client in the arp table.

after few minutes the switch change the mac address to the real mac of the client..

this is strange because of arp.. the switch should not have arp entries from a layer3 interface in which he dosent have an ip adress.. am i right??

thank you in advance for your help, best regards and stay happy!

michael

[Michael KARL]

ok, for better understanding, i post a snippet of the configuration (core and client switch)

core switch

***********************************

version 15.2

no service pad

service timestamps debug datetime localtime

service timestamps log datetime localtime

service password-encryption

!

hostname coreswitch

!

boot-start-marker

boot-end-marker

!

logging buffered 30000

no logging console

!

aaa new-model

!

!

aaa authentication login default local

aaa authentication enable default none

aaa authentication dot1x default group radius

aaa authorization network default group radius

aaa accounting dot1x default start-stop group radius

!

!

!

!

!

!

aaa session-id common

clock timezone MET 1 0

clock summer-time MET-DST recurring last Sun Mar 2:00 last Sun Oct 2:00

switch 1 provision ws-c3750x-48

switch 2 provision ws-c3750x-48

switch 3 provision ws-c3750x-48

switch 4 provision ws-c3750x-48

switch 5 provision ws-c3750x-48

system mtu routing 1500

ip routing

!

!

ip dhcp snooping vlan 1,504

no ip dhcp snooping information option

ip domain-name domain.net

ip device tracking

!

!

!

dot1x system-auth-control

!

spanning-tree mode rapid-pvst

spanning-tree extend system-id

spanning-tree vlan 1-1000 priority 24576

!

!

!

!

!

errdisable recovery cause dhcp-rate-limit

!

!

!

!

vlan internal allocation policy ascending

!

!

!

!

interface Loopback0

description RID

ip address 192.168.254.102 255.255.255.255

!

interface Port-channel1

switchport trunk encapsulation dot1q

switchport trunk native vlan 1001

switchport mode trunk

ip dhcp snooping trust

!

interface GigabitEthernet1/1/1

switchport trunk encapsulation dot1q

switchport trunk native vlan 1001

switchport mode trunk

channel-group 1 mode active

ip dhcp snooping trust

!

interface GigabitEthernet2/1/1

switchport trunk encapsulation dot1q

switchport trunk native vlan 1001

switchport mode trunk

channel-group 1 mode active

ip dhcp snooping trust

!

interface Vlan1

ip address 172.26.253.60 255.255.0.0 secondary

ip address 172.26.253.56 255.255.0.0

standby 1 ip 172.26.254.254

standby 1 priority 115

standby 1 preempt

standby 1 authentication md5 key-string 7 xxxxx

ip ospf authentication message-digest

ip ospf message-digest-key 1 md5 7 xxxxx

ip ospf 1 area 11

!

interface Vlan506

ip address 172.25.6.1 255.255.255.0

standby 1 ip 172.25.6.254

standby 1 priority 115

standby 1 preempt

standby 1 authentication md5 key-string 7 xxxxxxx

!

!

ip forward-protocol nd

no ip http server

ip http secure-server

!

ip route 0.0.0.0 0.0.0.0 10.25.254.243

ip route 172.25.10.0 255.255.254.0 172.26.255.242

ip route 172.25.12.0 255.255.254.0 172.26.255.242

!

***********************************************************

client switch

*******************************************************

version 15.2

no service pad

service timestamps debug datetime localtime

service timestamps log datetime localtime

service password-encryption

!

hostname client-switch

!

boot-start-marker

boot-end-marker

!

logging buffered 30000

aaa new-model

!

!

aaa authentication login default local

aaa authentication enable default none

aaa authentication dot1x default group radius

aaa authorization network default group radius

aaa accounting dot1x default start-stop group radius

!

!

!

!

!

!

aaa session-id common

clock timezone MET 1 0

clock summer-time MET-DST recurring last Sun Mar 2:00 last Sun Oct 2:00

switch 1 provision ws-c2960s-48ts-l

switch 2 provision ws-c2960s-48ts-l

switch 3 provision ws-c2960s-24ts-l

!

!

ip dhcp snooping vlan 1,504

ip dhcp snooping

ip domain-name domain.net

ip device tracking

!

!

dot1x system-auth-control

!

spanning-tree mode rapid-pvst

spanning-tree extend system-id

!

!

!

!

!

errdisable recovery cause dhcp-rate-limit

!

!

!

!

vlan internal allocation policy ascending

!

interface Port-channel1

switchport trunk native vlan 1001

switchport mode trunk

ip dhcp snooping trust

!

interface FastEthernet0

no ip address

shutdown

!

interface GigabitEthernet1/0/49

switchport trunk native vlan 1001

switchport mode trunk

channel-group 1 mode active

ip dhcp snooping trust

!

interface GigabitEthernet2/0/49

switchport trunk native vlan 1001

switchport mode trunk

channel-group 1 mode active

ip dhcp snooping trust

interface Vlan1

no ip address

!

interface Vlan506

ip address 172.25.6.8 255.255.255.0

!

ip default-gateway 172.25.6.254

no ip http server

ip http secure-server

********************************************

[Michael KARL]

hello together

today i tried the "no ip redirect" (on core and edge switch), but no affect.

once i made a backup, the client switch add the ip in his arp table..

could it be in interaction with dot1x?

regards - michael

[Richard Burts]

Michael

I can not absolutely rule out the possibility that dot1x is causing this, but I doubt that it is.

Can you tell us the IP of the workstation and perhaps post the output of the arp table which has the workstation address in it? This might give us some clue about what is happening. Also can you post the output from the client switch of the command show ip interface vlan 506?

Am I correct in understanding that you are taking a backup of the client switch? That you have connected to the client switch and issue the command copy run tftp where the tftp server is the workstation connected to the client switch on vlan 1? This would mean that the management interface of the client switch is looking for the address of the workstation. I have seen situations where some Catalyst switches will arp for remote destinations. I wonder if that is the case here?

If on the client switch you run debug for arp we can perhaps tell whether the switch tries first to the core and then to the workstation or whether it just immediately sends the local arp. And if on the client switch you run debug for ip icmp then perhaps we can tell whether ip redirects are playing a role in this.

HTH

Rick

[Michael KARL]

hi Rick

i agree about dot1x..

you understand my problem correctly, thats happend...

additional, i have installed prtg on the workstation where i have the tftp server installed..

on the client switches i have configured a krone job, which create the backup every day.. after done this job, the switch is unreachable from this workstation - as i am alerted from prtg..

maybe its a case like this here...

when i run debug arp i can see that the switch first tries the core switch - because of this i think the client switch adds the mac address of the layer3 interface of the core switch in his arp table..

is the debug ip icmp hi cpu performancy?

its an productive environment..

regards - michael

following the outputs (i have made some censored in mac address..):

first i clear the arp table in client switch - now it looks so:

*******************************************

MSW10K-01#clear arp 172.26.1.169

MSW10K-01#sh arp

Protocol  Address          Age (min)  Hardware Addr   Type   Interface

Internet  172.25.6.21           49   000b.xxxx.8fff  ARPA   Vlan506

Internet  172.25.6.75           30   5017.xxxx.e841  ARPA   Vlan506

Internet  172.25.6.90            -   4c00.xxxx.3841  ARPA   Vlan506

Internet  172.25.6.95          137   5017.xxxx.cf41  ARPA   Vlan506

Internet  172.25.6.96           27   7c95.xxxx.bbc1  ARPA   Vlan506

Internet  172.25.6.97          224   5017.xxxx.d9c1  ARPA   Vlan506

Internet  172.25.6.253          50   0008.xxxx.fc04  ARPA   Vlan506

Internet  172.25.6.254          90   0000.xxxx.ac02  ARPA   Vlan506

*******************************************

then i create a backup from the client switch, and then i get an arp entry from the workstation (172.26.1.169 with mac address 000b.xxxx.8fff from the layer3 interface of the core switch) where the tftp server is installed - but i dont have a layer3 interface in this subnet..

*******************************************

MSW10K-01#copy run tftp

Address or name of remote host []? 172.26.1.169

Destination filename [msw10k-01-confg]? MSW10K-01.cfg

!!

21793 bytes copied in 10.712 secs (2034 bytes/sec)

MSW10K-01#sh arp

Protocol  Address          Age (min)  Hardware Addr   Type   Interface

Internet  172.25.6.21           49   000b.xxxx.8fff  ARPA   Vlan506

Internet  172.25.6.75           30   5017.xxxx.e841  ARPA   Vlan506

Internet  172.25.6.90            -   4c00.xxxx.3841  ARPA   Vlan506

Internet  172.25.6.95          137   5017.xxxx.cf41  ARPA   Vlan506

Internet  172.25.6.96           27   7c95.xxxx.bbc1  ARPA   Vlan506

Internet  172.25.6.97          224   5017.xxxx.d9c1  ARPA   Vlan506

Internet  172.25.6.253          50   0008.xxxx.fc04  ARPA   Vlan506

Internet  172.25.6.254          90   0000.xxxx.ac02  ARPA   Vlan506

Internet  172.26.1.169            0   0000.xxxx.ac02  ARPA   Vlan506

MSW10K-01#

*******************************************

and the next strange thing is that after few minutes the arp entry "172.26.1.169 0000.xxxx.ac02 Vlan506" change to "172.26.1.169 0050.xxxx.7193 Vlan1" - the real mac address from the workstation where the tftp server is installed..

*******************************************

MSW10K-01#sh arp

Protocol  Address          Age (min)  Hardware Addr   Type   Interface

Internet  172.25.6.21           49   000b.xxxx.8fff  ARPA   Vlan506

Internet  172.25.6.75           30   5017.xxxx.e841  ARPA   Vlan506

Internet  172.25.6.90            -   4c00.xxxx.3841  ARPA   Vlan506

Internet  172.25.6.95          137   5017.xxxx.cf41  ARPA   Vlan506

Internet  172.25.6.96           27   7c95.xxxx.bbc1  ARPA   Vlan506

Internet  172.25.6.97          224   5017.xxxx.d9c1  ARPA   Vlan506

Internet  172.25.6.253          50   0008.xxxx.fc04  ARPA   Vlan506

Internet  172.25.6.254          90   0000.xxxx.ac02  ARPA   Vlan506

Internet  172.26.1.169            0   0050.xxxx.7193  ARPA   Vlan1

*******************************************

MSW10K-01#sh ip int vl 506

Vlan506 is up, line protocol is up

  Internet address is 172.25.6.8/24

  Broadcast address is 255.255.255.255

  Address determined by non-volatile memory

  MTU is 1500 bytes

  Helper address is not set

  Directed broadcast forwarding is disabled

  Multicast reserved groups joined: 224.0.0.251

  Outgoing access list is not set

  Inbound  access list is not set

  Proxy ARP is enabled

  Local Proxy ARP is disabled

  Security level is default

  Split horizon is enabled

  ICMP redirects are always sent

  ICMP unreachables are always sent

  ICMP mask replies are never sent

  IP fast switching is enabled

  IP Flow switching is disabled

  IP CEF switching is disabled

  IP Null turbo vector

  IP Null turbo vector

  IP multicast fast switching is enabled

  IP multicast distributed fast switching is disabled

  IP route-cache flags are Fast

  Router Discovery is disabled

  IP output packet accounting is disabled

  IP access violation accounting is disabled

  TCP/IP header compression is disabled

  RTP/IP header compression is disabled

  Probe proxy name replies are disabled

  Policy routing is disabled

  Network address translation is disabled

  BGP Policy Mapping is disabled

  Input features: MCI Check

  Output features: Check hwidb

[Richard Burts]

Michael

Thank you for the additional information. I find it especially interesting and informative that at first there is an ARP entry that shows the IP address of the workstation is associated with VLAN 506 and is the MAC used for HSRP. So this indicates that initially the client switch is doing what we expect and is sending its traffic for the workstation using the core switch as the next hop. But then it changes and the client switch associates the IP of the workstation with VLAN 1 and uses the MAC of the workstation. To me that suggests that ip redirect is probably the explanation.

I would think that debug ip icmp would not be so very intensive. If you are concerned about that you might consider using debug ip packet with an access list. You could configure an extended access list that permits just icmp redirects, perhaps access-list 199. And then you can use the command debug ip packet 199. The result is that debug for ip packet only reports when it sees redirect traffic. That should be somewhat less impact than just debug ip icmp (though I am not convinced that the difference would be significant).

HTH

Rick

[paul driver]

Hello

Just to confirm

Your tftp server doesn't reside in either vlan 1 - 506 or is this a typo?

Res
Paul

Sent from Cisco Technical Support iPad App

[Michael KARL]

hi Paul

the tftp server is located in vlan1

and i have tried to add the "no ip redirect" - but no matter..

best regards - michael

[paul driver]

Hello

Forgive me for some reason I was looking at the svi for vlan 1 and seeing 24 bit ranges - think I need stronger glasses!

Where did you apply the no Icmp redirect command?

Res
Paul

Sent from Cisco Technical Support iPad App

[Michael KARL]

hi Paul

no worries :-)

i applied it on the core switch in if vlan1 and if vlan506 and as well on the client switch in if vlan1 (which is shutdown) and if vlan506

best regards - michael

[Richard Burts]

Would you post the output of show ip interface for both vlan 1 and vlan 506 on the core (and maybe on the client as well)?

HTH

Rick

[Michael KARL]

sure, here are the outpus:

core:

MSWVSS-01#sh ip inter vl 1

Vlan1 is up, line protocol is up

  Internet address is 172.26.253.60/16

  Broadcast address is 255.255.255.255

  Address determined by non-volatile memory

  MTU is 1500 bytes

  Helper address is not set

  Directed broadcast forwarding is disabled

  Multicast reserved groups joined: 224.0.0.2

  Outgoing access list is not set

  Inbound  access list is not set

  Proxy ARP is disabled

  Local Proxy ARP is disabled

  Security level is default

  Split horizon is enabled

  ICMP redirects are never sent

  ICMP unreachables are always sent

  ICMP mask replies are never sent

  IP fast switching is enabled

  IP Flow switching is disabled

  IP CEF switching is enabled

  IP CEF switching turbo vector

  IP Null turbo vector

  IP multicast fast switching is enabled

  IP multicast distributed fast switching is disabled

  IP route-cache flags are Fast, CEF

  Router Discovery is disabled

  IP output packet accounting is disabled

  IP access violation accounting is disabled

  TCP/IP header compression is disabled

  RTP/IP header compression is disabled

  Probe proxy name replies are disabled

  Policy routing is disabled

  Network address translation is disabled

  BGP Policy Mapping is disabled

  Input features: MCI Check

  WCCP Redirect outbound is disabled

  WCCP Redirect inbound is disabled

  WCCP Redirect exclude is disabled

MSWVSS-01#sh ip inter vl506

Vlan506 is up, line protocol is up

  Internet address is 172.25.6.1/24

  Broadcast address is 255.255.255.255

  Address determined by non-volatile memory

  MTU is 1500 bytes

  Helper address is not set

  Directed broadcast forwarding is disabled

  Multicast reserved groups joined: 224.0.0.2

  Outgoing access list is not set

  Inbound  access list is not set

  Proxy ARP is disabled

  Local Proxy ARP is disabled

  Security level is default

  Split horizon is enabled

  ICMP redirects are never sent

  ICMP unreachables are always sent

  ICMP mask replies are never sent

  IP fast switching is enabled

  IP Flow switching is disabled

  IP CEF switching is enabled

  IP CEF switching turbo vector

  IP Null turbo vector

  IP multicast fast switching is enabled

  IP multicast distributed fast switching is disabled

  IP route-cache flags are Fast, CEF

  Router Discovery is disabled

  IP output packet accounting is disabled

  IP access violation accounting is disabled

  TCP/IP header compression is disabled

  RTP/IP header compression is disabled

  Probe proxy name replies are disabled

  Policy routing is disabled

  Network address translation is disabled

  BGP Policy Mapping is disabled

  Input features: MCI Check

  WCCP Redirect outbound is disabled

  WCCP Redirect inbound is disabled

  WCCP Redirect exclude is disabled

client:

MSW03K-01#sh ip int vl1

Vlan1 is administratively down, line protocol is down

  Internet protocol processing disabled

MSW03K-01#

MSW03K-01#sh ip int vl506

Vlan506 is up, line protocol is up

  Internet address is 172.25.6.8/24

  Broadcast address is 255.255.255.255

  Address determined by non-volatile memory

  MTU is 1500 bytes

  Helper address is not set

  Directed broadcast forwarding is disabled

  Multicast reserved groups joined: 224.0.0.251

  Outgoing access list is not set

  Inbound  access list is not set

  Proxy ARP is enabled

  Local Proxy ARP is disabled

  Security level is default

  Split horizon is enabled

  ICMP redirects are always sent

  ICMP unreachables are always sent

  ICMP mask replies are never sent

  IP fast switching is enabled

  IP Flow switching is disabled

  IP CEF switching is disabled

  IP Null turbo vector

  IP Null turbo vector

  IP multicast fast switching is enabled

  IP multicast distributed fast switching is disabled

  IP route-cache flags are Fast

  Router Discovery is disabled

  IP output packet accounting is disabled

  IP access violation accounting is disabled

  TCP/IP header compression is disabled

  RTP/IP header compression is disabled

  Probe proxy name replies are disabled

  Policy routing is disabled

  Network address translation is disabled

  BGP Policy Mapping is disabled

  Input features: MCI Check

  Output features: Check hwidb

[Richard Burts]

Michael

Thank you for the additional information. It does confirm that on the core switch both vlan 1 and vlan 506 have disabled both proxy arp and icmp redirects. It also shows that on the client switch that proxy arp is enabled on vlan 506. Would you disable proxy arp on that vlan interface and see what happens.

HTH

Rick

[Michael KARL]

hi Rick

Same Problem.. :-(

Michael

MSW10K-01#sh ip int vl 506

Vlan506 is up, line protocol is up

  Internet address is 172.25.6.8/24

  Broadcast address is 255.255.255.255

  Address determined by non-volatile memory

  MTU is 1500 bytes

  Helper address is not set

  Directed broadcast forwarding is disabled

  Multicast reserved groups joined: 224.0.0.251

  Outgoing access list is not set

  Inbound  access list is not set

  Proxy ARP is disabled

  Local Proxy ARP is disabled

  Security level is default

  Split horizon is enabled

  ICMP redirects are never sent

  ICMP unreachables are always sent

  ICMP mask replies are never sent

  IP fast switching is disabled

  IP Flow switching is disabled

  IP CEF switching is disabled

  IP Null turbo vector

  IP Null turbo vector

  IP multicast fast switching is disabled

  IP multicast distributed fast switching is disabled

  IP route-cache flags are No CEF, No Distributed

  Router Discovery is disabled

  IP output packet accounting is disabled

  IP access violation accounting is disabled

  TCP/IP header compression is disabled

  RTP/IP header compression is disabled

  Probe proxy name replies are disabled

  Policy routing is disabled

  Network address translation is disabled

  BGP Policy Mapping is disabled

  Input features: MCI Check

  Output features: Check hwidb

[Richard Burts]

Michael

Thanks. Clearly proxy arp is now disabled. Could you post the output of ipconfig /all from the workstation?

HTH

Rick

[Michael KARL]

hi Rick

Surly, here it is:

Physical Address. . . . . . . . . : 00-50-xxxx-71-93

DHCP Enabled. . . . . . . . . . . : No

Autoconfiguration Enabled . . . . : Yes

IPv4 Address. . . . . . . . . . . : 172.26.1.169(Preferred)

Subnet Mask . . . . . . . . . . . : 255.255.0.0

Default Gateway . . . . . . . . . : 172.26.254.254

DNS Servers . . . . . . . . . . . : 172.26.1.20

                                    172.26.1.25

NetBIOS over Tcpip. . . . . . . . : Enabled

best regards - michael