Hi,As u mention above:There

CapUcisco · ‎05-16-2014

2 switches

switch 1 (3550) does the routing with OSPF

vlan 74 with ACL on

vlan 100

vlan 200

switch 2 (3750) is a layer 2 switch with SVIs

vlan 74 with ACL on

vlan 100

vlan 200

There is a Layer 2 trunk between the two switches with native Vlan 100

The problem is:

When a packed coming from vlan 100 in switch 1 with a destination of vlan 200 on switch 2 the route of the packet goes via vlan 74 on switch 2 first and it hits the ALC and therefor it gets dropped.

Why is that packet traversing vlan 74 first.

It should go vlan 100 (sw1) to vlan 200 (sw2) directly

Am I wrong?

manish arora · ‎05-16-2014

Yes it should go from vlan 100 to vlan 200 given everything is configured correctly :-). Would you please post "show ip route" from both sw1 & sw2 ?

Manish

CapUcisco · ‎05-16-2014

The actual vlan # are:

Vlan74                 10.7.4.10       YES NVRAM up                    up
Vlan562                10.56.16.10     YES NVRAM up                    up
Vlan572                10.57.16.10     YES NVRAM up                    up
Vlan582                10.58.16.10     YES NVRAM up                    up
Vlan592                10.59.16.10     YES NVRAM up                    up
Vlan602                10.60.16.10     YES NVRAM up                    up
Vlan612                10.61.16.10     YES NVRAM up                    up
Vlan622                10.62.16.10     YES NVRAM up                    up

Vlan 74 is the only one with an ACL on it.

switch 1

show ip route

Gateway of last resort is 172.19.56.45 to network 0.0.0.0

     172.19.0.0/16 is variably subnetted, 24 subnets, 2 masks
O       172.19.253.64/32 [110/3] via 172.19.56.45, 1d17h, GigabitEthernet0/2
                         [110/3] via 172.19.56.41, 1d17h, GigabitEthernet0/1
O       172.19.252.80/30 [110/2] via 172.19.56.41, 1d17h, GigabitEthernet0/1
O       172.19.252.84/30 [110/2] via 172.19.56.45, 1d17h, GigabitEthernet0/2
O       172.19.252.88/30 [110/2] via 172.19.56.41, 1d17h, GigabitEthernet0/1
O       172.19.252.92/30 [110/2] via 172.19.56.45, 1d17h, GigabitEthernet0/2
O       172.19.253.59/32 [110/3] via 172.19.56.45, 1d17h, GigabitEthernet0/2
                         [110/3] via 172.19.56.41, 1d17h, GigabitEthernet0/1
O       172.19.253.56/32 [110/2] via 172.19.56.41, 1d17h, GigabitEthernet0/1
O       172.19.253.57/32 [110/2] via 172.19.56.45, 1d17h, GigabitEthernet0/2
O       172.19.253.62/32 [110/3] via 172.19.56.45, 1d17h, GigabitEthernet0/2
                         [110/3] via 172.19.56.41, 1d17h, GigabitEthernet0/1
C       172.19.253.63/32 is directly connected, Loopback0
O       172.19.253.60/32 [110/3] via 172.19.56.45, 1d17h, GigabitEthernet0/2
                         [110/3] via 172.19.56.41, 1d17h, GigabitEthernet0/1
O       172.19.253.61/32 [110/3] via 172.19.56.45, 1d17h, GigabitEthernet0/2
                         [110/3] via 172.19.56.41, 1d17h, GigabitEthernet0/1
O       172.19.56.4/30 [110/2] via 172.19.56.45, 1d17h, GigabitEthernet0/2
O       172.19.56.0/30 [110/2] via 172.19.56.41, 1d17h, GigabitEthernet0/1
O       172.19.56.12/30 [110/2] via 172.19.56.45, 1d17h, GigabitEthernet0/2
O       172.19.56.8/30 [110/2] via 172.19.56.41, 1d17h, GigabitEthernet0/1
O       172.19.56.20/30 [110/2] via 172.19.56.45, 1d17h, GigabitEthernet0/2
O       172.19.56.16/30 [110/2] via 172.19.56.41, 1d17h, GigabitEthernet0/1
O       172.19.56.28/30 [110/2] via 172.19.56.45, 1d17h, GigabitEthernet0/2
O       172.19.56.24/30 [110/2] via 172.19.56.41, 1d17h, GigabitEthernet0/1
O       172.19.56.36/30 [110/2] via 172.19.56.45, 1d17h, GigabitEthernet0/2
O       172.19.56.32/30 [110/2] via 172.19.56.41, 1d17h, GigabitEthernet0/1
C       172.19.56.44/30 is directly connected, GigabitEthernet0/2
C       172.19.56.40/30 is directly connected, GigabitEthernet0/1
     10.0.0.0/8 is variably subnetted, 27 subnets, 2 masks
C       10.7.4.0/23 is directly connected, Vlan74
O       10.58.40.0/21 [110/9] via 172.19.56.45, 1d17h, GigabitEthernet0/2
                      [110/9] via 172.19.56.41, 1d17h, GigabitEthernet0/1
O       10.59.40.0/21 [110/9] via 172.19.56.45, 1d17h, GigabitEthernet0/2
                      [110/9] via 172.19.56.41, 1d17h, GigabitEthernet0/1
O       10.56.40.0/21 [110/9] via 172.19.56.45, 1d17h, GigabitEthernet0/2
                      [110/9] via 172.19.56.41, 1d17h, GigabitEthernet0/1
O       10.57.40.0/21 [110/9] via 172.19.56.45, 1d17h, GigabitEthernet0/2
                      [110/9] via 172.19.56.41, 1d17h, GigabitEthernet0/1
O       10.62.40.0/21 [110/9] via 172.19.56.45, 1d17h, GigabitEthernet0/2
                      [110/9] via 172.19.56.41, 1d17h, GigabitEthernet0/1
O       10.60.40.0/21 [110/9] via 172.19.56.45, 1d17h, GigabitEthernet0/2
                      [110/9] via 172.19.56.41, 1d17h, GigabitEthernet0/1
O       10.61.40.0/21 [110/9] via 172.19.56.45, 1d17h, GigabitEthernet0/2
                      [110/9] via 172.19.56.41, 1d17h, GigabitEthernet0/1
C       10.58.16.0/21 is directly connected, Vlan582
C       10.59.16.0/21 is directly connected, Vlan592
C       10.56.16.0/21 is directly connected, Vlan562
C       10.57.16.0/21 is directly connected, Vlan572
C       10.62.16.0/21 is directly connected, Vlan622
C       10.60.16.0/21 is directly connected, Vlan602
C       10.61.16.0/21 is directly connected, Vlan612
O       10.58.24.0/21 [110/9] via 172.19.56.45, 1d17h, GigabitEthernet0/2
                      [110/9] via 172.19.56.41, 1d17h, GigabitEthernet0/1
O       10.59.24.0/21 [110/9] via 172.19.56.45, 1d17h, GigabitEthernet0/2
                      [110/9] via 172.19.56.41, 1d17h, GigabitEthernet0/1
O       10.56.24.0/21 [110/9] via 172.19.56.45, 1d17h, GigabitEthernet0/2
                      [110/9] via 172.19.56.41, 1d17h, GigabitEthernet0/1
O       10.60.24.0/21 [110/9] via 172.19.56.45, 1d17h, GigabitEthernet0/2
                      [110/9] via 172.19.56.41, 1d17h, GigabitEthernet0/1
O       10.58.0.0/21 [110/9] via 172.19.56.45, 1d17h, GigabitEthernet0/2
                     [110/9] via 172.19.56.41, 1d17h, GigabitEthernet0/1
O       10.59.0.0/21 [110/9] via 172.19.56.45, 1d17h, GigabitEthernet0/2
                     [110/9] via 172.19.56.41, 1d17h, GigabitEthernet0/1
O       10.56.0.0/21 [110/9] via 172.19.56.45, 1d17h, GigabitEthernet0/2
                     [110/9] via 172.19.56.41, 1d17h, GigabitEthernet0/1
O       10.57.0.0/21 [110/9] via 172.19.56.45, 1d17h, GigabitEthernet0/2
                     [110/9] via 172.19.56.41, 1d17h, GigabitEthernet0/1
O       10.58.8.0/21 [110/9] via 172.19.56.45, 1d17h, GigabitEthernet0/2
                     [110/9] via 172.19.56.41, 1d17h, GigabitEthernet0/1
O       10.59.8.0/21 [110/9] via 172.19.56.45, 1d17h, GigabitEthernet0/2
                     [110/9] via 172.19.56.41, 1d17h, GigabitEthernet0/1
O       10.56.8.0/21 [110/9] via 172.19.56.45, 1d17h, GigabitEthernet0/2
                     [110/9] via 172.19.56.41, 1d17h, GigabitEthernet0/1
O       10.57.8.0/21 [110/9] via 172.19.56.45, 1d17h, GigabitEthernet0/2
                     [110/9] via 172.19.56.41, 1d17h, GigabitEthernet0/1
S*   0.0.0.0/0 [1/0] via 172.19.56.45
               [1/0] via 172.19.56.41

switch 1

show ip inter br

Interface              IP-Address      OK? Method Status                Protocol
Vlan74                 10.7.4.10       YES NVRAM up                    up
Vlan562                10.56.16.10     YES NVRAM up                    up
Vlan572                10.57.16.10     YES NVRAM up                    up
Vlan582                10.58.16.10     YES NVRAM up                    up
Vlan592                10.59.16.10     YES NVRAM up                    up
Vlan602                10.60.16.10     YES NVRAM up                    up
Vlan612                10.61.16.10     YES NVRAM up                    up
Vlan622                10.62.16.10     YES NVRAM up                    up
GigabitEthernet0/1     172.19.56.42    YES NVRAM up                    up
GigabitEthernet0/2     172.19.56.46    YES NVRAM up                    up
Loopback0              172.19.253.63   YES NVRAM up                    up

switch2

show ip route

Default gateway is not set

Host Gateway Last Use Total Uses Interface
ICMP redirect cache is empty

This switch is Layer 2 trunked to sw 1

switch 2

show ip inter br

Interface              IP-Address      OK? Method Status                Protocol
Vlan74                 10.7.4.11       YES NVRAM up                    up
Vlan562                10.56.16.11     YES NVRAM up                    up
Vlan572                10.57.16.11     YES NVRAM up                    up
Vlan582                10.58.16.11     YES NVRAM up                    up
Vlan592                10.59.16.11     YES NVRAM up                    up
Vlan602                10.60.16.11     YES NVRAM up                    up
Vlan612                10.61.16.11     YES NVRAM up                    up
Vlan622                10.62.16.11     YES NVRAM up                    up

Layer 2 Etherchannel between the 2 switches:

interface Port-channel45
description Ma112sw1 Trunk
switchport trunk encapsulation dot1q
switchport trunk native vlan 562
switchport mode trunk

manish arora · ‎05-17-2014

Just labbed it up and I am unable to replicate your problem. Can you shed more light in to your issue with reference to correct vlans as in your config ? Also, any traceroutes showing that your switch is taking an extra hop before going to a different vlan would be good along with ACL logs when packets hit the access-group on vlan 74 would be helpful in finding the issue.

Also, can you double check that same vlans are present in both switches in the "Show vlan brife" or if using vtp , the switches are in sync ?

Manish

CapUcisco · ‎05-17-2014

Both switches have these vlans, (VTP is in transparent mode)

Vlan74, 562, 572, 582, 592, 602, 612, 622

Vlan74 is the only one that has an ACL attached to its SVI

The native Vlan on both sides of the trunk is Vlan 562

This is a trace from sw1 Vlan 582 SVI to sw2 Vlan 592 SVI

Ma112sw1#trace
Protocol [ip]:
Target IP address: 10.59.16.11
Source address: 10.59.16.10
Numeric display [n]:
Timeout in seconds [3]:
Probe count [3]:
Minimum Time to Live [1]:
Maximum Time to Live [30]:
Port Number [33434]:
Loose, Strict, Record, Timestamp, Verbose[none]:
Type escape sequence to abort.
Tracing the route to ma112sw2-vlan592.capilanou.ca (10.59.16.11)

1 ma112sw2-vlan592.capilanou.ca (10.59.16.11) 0 msec * 12 msec
Ma112sw1#

This is as expected.

Now watch what happens with this trace mac from switch 1 between the last 2 hosts in the arp table, hosts attached to the same switch 1 in vlan 592

Ma112sw1#sh arp | i 592
Internet 10.59.16.1              -   0000.0c07.acc0 ARPA   Vlan592
Internet 10.59.16.3              0   442b.0391.479f ARPA   Vlan592
Internet 10.59.16.2              0   442b.03d8.b602 ARPA   Vlan592
Internet 10.59.16.11           110   44d3.ca6f.b645 ARPA   Vlan592
Internet 10.59.16.10             -   000c.316a.aa80 ARPA   Vlan592
Internet 10.59.20.21           101   000a.9c10.75f3 ARPA   Vlan592
Internet 10.59.20.22             0   0002.3102.224d ARPA   Vlan592
Internet 10.59.20.25             6   00e0.d812.6d0a ARPA   Vlan592
Internet 10.59.20.24             9   0020.85fb.690b ARPA   Vlan592
Internet 10.59.20.101            0   0018.badc.c514 ARPA   Vlan592
Internet 10.59.20.100            0   0017.e095.0ef8 ARPA   Vlan592
Internet 10.59.20.110            4   0019.ee00.029b ARPA   Vlan592
Ma112sw1#sh mac add
Ma112sw1#sh mac address-table | i 0ef8
592    0017.e095.0ef8    DYNAMIC     Fa0/44
Ma112sw1#sh mac address-table | i 029b
592    0019.ee00.029b    DYNAMIC     Fa0/43
Ma112sw1#trace mac 0017.e095.0ef8 0019.ee00.029b
Source 0017.e095.0ef8 found on Ma112sw1
1 Ma112sw1 (10.7.4.10) : Fa0/44 => Fa0/43
Destination 0019.ee00.029b found on Ma112sw1
Layer2 trace completed.

This line 1 Ma112sw1 (10.7.4.10) : Fa0/44 => Fa0/43

indicates that traffic is flowing via 10.7.4.10 which is the SVI of vlan 74 on sw1 (this is the vlan that has the ACL)

Now the ACL log:

this is the config of SVI on vlan 74

Ma112sw1#sh run interface vlan 74
Building configuration...

Current configuration : 363 bytes
!
interface Vlan74
description 10.7.4-CSU-Maple
ip address 10.7.4.10 255.255.254.0
ip access-group Filter-CSU-Network in
ip helper-address 204.239.154.50
ip helper-address 204.239.154.189
no ip redirects
ip directed-broadcast
ip pim sparse-mode
ip ospf cost 7
standby name vlan74
standby 74 ip 10.7.4.1
standby 74 priority 95
standby 74 preempt
end

and this is the SVI of vlan 592

Ma112sw1#sh run interface vlan 592
Building configuration...

Current configuration : 340 bytes
!
interface Vlan592
description 10.59.16-Maple-1xx-Mgmt
ip address 10.59.16.10 255.255.248.0
ip helper-address 204.239.154.50
ip helper-address 204.239.154.189
no ip redirects
ip directed-broadcast
ip pim sparse-mode
ip ospf cost 7
standby name vlan592
standby 192 ip 10.59.16.1
standby 192 priority 95
standby 192 preempt
end

%SEC-6-IPACCESSLOGP: list Filter-CSU-Network denied tcp 10.59.16.11(23) -> 204.239.152.212(60198), 1 packet

From my computer 204.239.152.212 I'm trying to initiate a telnet session to ma112sw1 with ip 10.59.16.11 this is on vlan 592 without any ACL as you can see above

But still the ACL applied to vlan 74 is dropping the packet

CapUcisco · ‎05-17-2014

Even more interesting.

A layer 2 trace between two hosts attached to vlan 592 on sw1 reveals:

Ma112sw1#sh arp | i 592
Internet 10.59.16.1              -   0000.0c07.acc0 ARPA   Vlan592
Internet 10.59.16.3              0   442b.0391.479f ARPA   Vlan592
Internet 10.59.16.2              0   442b.03d8.b602 ARPA   Vlan592
Internet 10.59.16.11             8   44d3.ca6f.b645 ARPA   Vlan592
Internet 10.59.16.10             -   000c.316a.aa80 ARPA   Vlan592
Internet 10.59.20.21           116   000a.9c10.75f3 ARPA   Vlan592
Internet 10.59.20.22            15   0002.3102.224d ARPA   Vlan592
Internet 10.59.20.25            11   00e0.d812.6d0a ARPA   Vlan592
Internet 10.59.20.24             4   0020.85fb.690b ARPA   Vlan592
Internet 10.59.20.101            0   0018.badc.c514 ARPA   Vlan592
Internet 10.59.20.100            0   0017.e095.0ef8 ARPA   Vlan592
Internet 10.59.20.110            8   0019.ee00.029b ARPA   Vlan592
Ma112sw1#
Ma112sw1#
Ma112sw1#
Ma112sw1#trace mac 0017.e095.0ef8 0019.ee00.029b
Source 0017.e095.0ef8 found on Ma112sw1
1 Ma112sw1 (10.7.4.10) : Fa0/44 => Fa0/43
Destination 0019.ee00.029b found on Ma112sw1
Layer2 trace completed.

address 10.7.4.10 is the SVI of vlan 74 on sw1 (this is the vlan with the ACL on)

CapUcisco · ‎05-17-2014

I also get this logs on sw1

%FM-3-UNLOADING: Unloading input vlan label 2 feature from all TCAMs
41w5d: %FM-3-UNLOADING: Unloading input vlan label 4 feature from all TCAMs
41w5d: %FM-3-UNLOADING: Unloading input vlan label 4 feature from all TCAMs

manish arora · ‎05-17-2014

Logs on SW1 suggests ACL is consuming more memory than allotted by the Cisco.

I can see your issue now and its really baffling, can you please test this out and post results :

#traceroute mac ip 10.59.20.100 10.59.20.110

Also, Can you please run the similar traceroute mac's ( with both source & dest on SW2 only) on the SW2 ( 3750) which is much newer than 3550 and see if the results are same or different ?

Thanks

CapUcisco · ‎05-17-2014

Ma112sw1#sh ip inter br | e unass
Interface              IP-Address      OK? Method Status                Protocol
Vlan74                 10.7.4.10       YES NVRAM up                    up
Vlan562                10.56.16.10     YES NVRAM up                    up
Vlan572                10.57.16.10     YES NVRAM up                    up
Vlan582                10.58.16.10     YES NVRAM up                    up
Vlan592                10.59.16.10     YES NVRAM up                    up
Vlan602                10.60.16.10     YES NVRAM up                    up
Vlan612                10.61.16.10     YES NVRAM up                    up
Vlan622                10.62.16.10     YES NVRAM up                    up
GigabitEthernet0/1     172.19.56.42    YES NVRAM up                    up
GigabitEthernet0/2     172.19.56.46    YES NVRAM up                    up
Loopback0              172.19.253.63   YES NVRAM up                    up
Ma112sw1#
Ma112sw1#
Ma112sw1#sh arp | i 592
Internet 10.59.16.1              -   0000.0c07.acc0 ARPA   Vlan592
Internet 10.59.16.3              0   442b.0391.479f ARPA   Vlan592
Internet 10.59.16.2              0   442b.03d8.b602 ARPA   Vlan592
Internet 10.59.16.11           120   44d3.ca6f.b645 ARPA   Vlan592
Internet 10.59.16.10             -   000c.316a.aa80 ARPA   Vlan592
Internet 10.59.20.21           111   000a.9c10.75f3 ARPA   Vlan592
Internet 10.59.20.22            10   0002.3102.224d ARPA   Vlan592
Internet 10.59.20.25            14   00e0.d812.6d0a ARPA   Vlan592
Internet 10.59.20.24             6   0020.85fb.690b ARPA   Vlan592
Internet 10.59.20.101            0   0018.badc.c514 ARPA   Vlan592
Internet 10.59.20.100            0   0017.e095.0ef8 ARPA   Vlan592
Internet 10.59.20.110            0   0019.ee00.029b ARPA   Vlan592
Ma112sw1#
Ma112sw1#
Ma112sw1#trace
Ma112sw1#traceroute
Protocol [ip]:
Target IP address: 10.59.20.110
Source address: 10.59.20.100
% Invalid source address
Ma112sw1#traceroute
Protocol [ip]:
Target IP address: 10.59.20.110
Source address: 10.59.16.10
Numeric display [n]:
Timeout in seconds [3]:
Probe count [3]:
Minimum Time to Live [1]:
Maximum Time to Live [30]:
Port Number [33434]:
Loose, Strict, Record, Timestamp, Verbose[none]:
Type escape sequence to abort.
Tracing the route to ma000-mtr1.capilanou.ca (10.59.20.110)

1 ma000-mtr1.capilanou.ca (10.59.20.110) 0 msec 4 msec 0 msec
Ma112sw1#

Trace mac on sw2

Ma112sw2#sh arp | i 562
Internet 10.56.20.35             1   90e6.bac2.7950 ARPA   Vlan562
Internet 10.56.20.44             0   d4be.d9a4.f503 ARPA   Vlan562
Internet 10.56.20.43             2   d4be.d9a5.c2a4 ARPA   Vlan562
Internet 10.56.20.41             2   90b1.1ca2.bde0 ARPA   Vlan562
Internet 10.56.16.1            120   0000.0c07.aca2 ARPA   Vlan562
Internet 10.56.16.10            27   000c.316a.aa80 ARPA   Vlan562
Internet 10.56.16.11             -   44d3.ca6f.b642 ARPA   Vlan562
Ma112sw2#
Ma112sw2#
Ma112sw2#
Ma112sw2#
Ma112sw2#trace mac
Ma112sw2#sh mac add | i f503
562    d4be.d9a4.f503    DYNAMIC     Gi1/0/23
Ma112sw2#sh mac add | i c2a4
562    d4be.d9a5.c2a4    DYNAMIC     Gi1/0/21
Ma112sw2#trace mac d4be.d9a4.f503 d4be.d9a5.c2a4
Unable to send a l2trace request to 10.58.23.246. Timed out
Layer2 trace aborted.
Ma112sw2#
Ma112sw2#sh arp | ii 23.246
                   ^
% Invalid input detected at '^' marker.

Ma112sw2#sh arp | i 23.246
Internet 10.58.23.246            4   c8f9.f968.aed0 ARPA   Vlan582
Ma112sw2#sh mac add | i aed0
562    c8f9.f968.aed0    DYNAMIC     Gi1/0/23
582    c8f9.f968.aed0    DYNAMIC     Gi1/0/23
Ma112sw2#

Odd, 10.58.23.246 is a Cisco phone

manish arora · ‎05-19-2014

Ok, The "Traceroute mac xxx xxx" command shows the smallest vlan IP add as a switch id in the output, in your case it's showing VLAN74, if you create another VLAN for example VLAN55 with smaller IP Subnet than vlan 74 , it will start showing that in the output of your "traceroute mac" command , as far as ACL log being hit when you try to telnet from PC than please run a traceroute from your PC and see if vlan74 shows up as one of the next hop in that path or not.

Manish

CapUcisco · ‎05-20-2014

actually a trace from my pc to a pc in vlan 562 on sw2 does not show touching vlan 74

But a trace from sw2 back to my pc shows the route hits the 10.7.4.10 with is the SVI on sw1 with the ACL applied

10.59.16.11 is the SVI of vlan 592 on sw2

Ma112sw2#traceroute
Protocol [ip]:
Target IP address: 204.239.152.212
Source address: 10.59.16.11
Numeric display [n]:
Timeout in seconds [3]:
Probe count [3]:
Minimum Time to Live [1]:
Maximum Time to Live [30]:
Port Number [33434]:
Loose, Strict, Record, Timestamp, Verbose[none]:
Type escape sequence to abort.
Tracing the route to vhutanu1.cds.capilanou.ca (204.239.152.212)

1 *
10.7.4.10 !A *

syed kazim abbas · ‎05-17-2014

Hi,

As u mention above:

There is a Layer 2 trunk between the two switches with native Vlan 100

but your configuration show: your native vlan 562, may be native vlan mismatch on both sides or one switch have command switchport access vlan 74

interface Port-channel45

description Ma112sw1 Trunk
switchport trunk encapsulation dot1q
switchport trunk native vlan 562
switchport mode trunk

HTH

"please rate if post helpful"

CapUcisco · ‎05-17-2014

Sorry, I started my first post with different Vlan numbers.

The actual vlan numbers are in the config example below

Both trunks have native vlan 562

Odd inter vlan routing