Hi -
One of the more odd issues I've encountered with Cisco and I have a TAC case open now. Here's the issue:
IP Transit via static route
||
Cisco 6509E Sup720 BXL w/ 3x WS-X6748-GE-TX <> Cisco 6509E Sup720 BXL w/ 3x WS-X6748-GE-TX
||
Internal hosts on 10.200.4.0/22 network
When i'm on the 6509, I can source ping from the 10.200.4.2 VLAN (VLAN 44) IP to external Internet just fine. We can check nat translations and see this, so we know nat is set up ok from that subnet:
core1-22002#ping 4.2.2.2 source vlan 44
!!!!!
core1-22001#sh ip nat translations | i 4.2.2.2
icmp 141.136.100.122:1024 10.200.4.2:124 4.2.2.2:124 4.2.2.2:1024
Here is the ACL being referenced for NAT rule:
Extended IP access list 118
10 permit ip 10.200.0.0 0.0.255.255 any log (9 matches) << matches coming from source ping tests
However, when a server connected to VLAN 44 (10.200.4.x/22) tries to get out, it fails without NAT ever trying to set up the translation and doesn't add log match to entry. I should say that these hosts can ping across subnets internally, reach the public VLAN gateway, and reach any other IP on the 6509 or directly connected subnets.
So I kept saying to myself, its almost as if the VLAN is just dropping the packet. So on the phone with Cisco TAC, after about 1 hour of verification on config (everything was correct), he modified the configuration from this:
interface Vlan44
ip address 10.200.4.2 255.255.252.0
ip nat inside
to this:
core1-22002(config)#ip access-l ex test
core1-22002(config-ext-nacl)#10 permit ip host 10.200.4.36 any log
core1-22002(config-ext-nacl)#20 permit ip any any
core1-22002(config)#int vlan 44
core1-22002(config-if)#ip access-g test in
and BOOM, it works
Extended IP access list test
10 permit ip host 10.200.4.36 any log (12 matches)
20 permit ip any any
Eventually we figure out ANY ACL permit statement that matches works, as long as the "log" command is entered on ACL AND ACL is applied directly to that VLAN interface!
The Cisco TAC tells me that the 'log' command forces process switching and basically means that CEF is not working in this case, so the only way packets can forward out of the interface to the outside world is to use process switching instead of fast switching/CEF. Obviously something is wrong here and no way am I putting our large amount of PPS prod network on something that has to touch the CPU/higher level resources every time we need to forward a packet.
Has anyone ever seen this? It seems like an IOS change may be in order, fix it, but wanted to run it by here first. Both 6509's (connected via 2x 10G port-channel) display the exact same behavior. TAC punted me to switching group from routing group and of course I'm still waiting to hear back. Thanks in advance - Jason
--------------------------------------
CONFIG:
core1-22001#sh ver
Cisco IOS Software, s72033_rp Software (s72033_rp-ADVIPSERVICESK9_WAN-M), Version 12.2(33)SXI9, RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2012 by Cisco Systems, Inc.
Compiled Fri 24-Feb-12 21:38 by prod_rel_team
ROM: System Bootstrap, Version 12.2(17r)S4, RELEASE SOFTWARE (fc1)
core1-22001#sh module
Mod Ports Card Type Model Serial No.
--- ----- -------------------------------------- ------------------ -----------
1 48 CEF720 48 port 10/100/1000mb Ethernet WS-X6748-GE-TX SAD10450C3P
2 48 CEF720 48 port 10/100/1000mb Ethernet WS-X6748-GE-TX SAL08311AL6
3 48 CEF720 48 port 10/100/1000mb Ethernet WS-X6748-GE-TX SAL08311ALU
5 2 Supervisor Engine 720 (Active) WS-SUP720-3BXL SAD110400EG
6 0 Unknown (Other) Unknown Unknown
7 4 CEF720 4 port 10-Gigabit Ethernet WS-X6704-10GE SAD104803EN
8 16 SFM-capable 16 port 1000mb GBIC WS-X6516-GBIC SAL0730HEKR
9 16 SFM-capable 16 port 1000mb GBIC WS-X6516-GBIC SAL0739M7TA
Mod MAC addresses Hw Fw Sw Status
--- ---------------------------------- ------ ------------ ------------ -------
1 0019.aacc.6692 to 0019.aacc.66c1 2.4 12.2(14r)S5 12.2(33)SXI9 Ok
2 0013.60ff.4e50 to 0013.60ff.4e7f 2.0 12.2(14r)S5 12.2(33)SXI9 Ok
3 0013.60ff.2920 to 0013.60ff.294f 2.1 12.2(14r)S5 12.2(33)SXI9 Ok
5 0013.807b.49e8 to 0013.807b.49eb 5.3 8.4(2) 12.2(33)SXI9 Ok
6 0000.0000.0000 to 0000.0000.0000 0.0 Unknown Unknown Unknown
7 001a.2f00.1380 to 001a.2f00.1383 2.4 12.2(14r)S5 12.2(33)SXI9 Ok
8 0008.7dc9.ce50 to 0008.7dc9.ce5f 5.5 Unknown 12.2(33)SXI9 PwrDown
9 000d.bc3b.da60 to 000d.bc3b.da6f 5.5 6.3(1) 12.2(33)SXI9 Ok
Mod Sub-Module Model Serial Hw Status
---- --------------------------- ------------------ ----------- ------- -------
1 Centralized Forwarding Card WS-F6700-CFC SAL14502G4C 4.1 Ok
2 Centralized Forwarding Card WS-F6700-CFC SAD074104SA 1.1 Ok
3 Centralized Forwarding Card WS-F6700-CFC SAL09444N7C 2.0 Ok
5 Policy Feature Card 3 WS-F6K-PFC3BXL SAD110302YT 1.8 Ok
5 MSFC3 Daughterboard WS-SUP720 SAD110200P3 2.6 Ok
7 Centralized Forwarding Card WS-F6700-CFC SAL1249C4MD 4.1 Ok
Mod Online Diag Status
---- -------------------
1 Pass
2 Pass
3 Pass
5 Pass
6 Not Applicable
7 Pass
8 Not Applicable
9 Pass
core1-22001#
core1-22001#sh run
Building configuration...
Current configuration : 18013 bytes
!
! Last configuration change at 21:44:54 UTC Wed Jul 17 2013 by pubmatic_ops
!
upgrade fpd auto
version 12.2
service timestamps debug uptime
service timestamps log uptime
no service password-encryption
service counters max age 5
!
hostname core1-22001
!
boot-start-marker
boot system flash disk0:s72033-advipservicesk9_wan-mz.122-33.SXI9.bin
boot-end-marker
!
logging console warnings
enable secret 5
!
username
aaa new-model
!
!
aaa authentication login default local
!
!
!
aaa session-id common
logging event link-status default
!
!
!
ip ssh time-out 60
ip ssh authentication-retries 5
no ip domain-lookup
ip domain-name pubmatic.com
ip name-server 208.67.222.222
ip name-server 8.8.8.8
ipv6 mfib hardware-switching replication-mode ingress
udld enable
no mls acl tcam share-global
mls netflow interface
mls cef error action freeze
!
!
!
!
!
!
!
!
!
!
spanning-tree mode rapid-pvst
spanning-tree extend system-id
diagnostic bootup level minimal
access-list 24 permit 10.200.0.0 0.0.255.255 log
access-list 118 permit ip 10.200.0.0 0.0.255.255 any log
!
redundancy
main-cpu
auto-sync running-config
mode sso
!
ip access-list extended inside_nat_acl
permit ip 10.200.1.0 0.0.0.255 198.47.127.0 0.0.0.255 log
ip access-list extended test
permit ip any any log
!
vlan internal allocation policy ascending
vlan access-log ratelimit 2000
!
!
!
!
!
interface Loopback0
no ip address
!
interface Loopback198
no ip address
!
interface Port-channel2
switchport
switchport trunk encapsulation dot1q
switchport mode trunk
!
interface Port-channel16
switchport
switchport access vlan 11
switchport mode access
interface GigabitEthernet1/48
description test port
switchport
switchport access vlan 22
switchport mode access
interface GigabitEthernet2/47
switchport
switchport access vlan 44
switchport mode access
spanning-tree portfast edge
!
interface GigabitEthernet3/1
no ip address
!
interface GigabitEthernet3/2
no ip address
!
interface GigabitEthernet3/3
no ip address
!
interface GigabitEthernet3/4
no ip address
!
interface GigabitEthernet3/5
no ip address
!
interface GigabitEthernet3/6
no ip address
!
interface GigabitEthernet3/7
no ip address
!
interface GigabitEthernet3/8
no ip address
!
interface GigabitEthernet3/9
no ip address
!
interface GigabitEthernet3/10
description xcp22006-pub
switchport
switchport access vlan 100
switchport mode access
!
interface TenGigabitEthernet7/1
ip address 141.136.102.122 255.255.255.252
ip nat outside
!
interface TenGigabitEthernet7/2
switchport
switchport trunk encapsulation dot1q
switchport mode trunk
channel-group 2 mode active
!
interface TenGigabitEthernet7/3
switchport
switchport access vlan 100
switchport mode access
!
interface TenGigabitEthernet7/4
switchport
switchport trunk encapsulation dot1q
switchport mode trunk
channel-group 2 mode active
!
interface Vlan1
no ip address
!
interface Vlan2
ip address 10.200.0.1 255.255.255.0 secondary
ip address 10.200.0.2 255.255.255.0
no ip redirects
ip nat inside
standby 1 name pub-ams-net1
!
interface Vlan11
ip address 10.200.1.2 255.255.255.0
ip access-group test in
ip nat inside
standby 11 ip 10.200.1.1
!
interface Vlan22
ip address 10.200.2.1 255.255.255.0 secondary
ip address 10.200.2.2 255.255.255.0
no ip redirects
ip nat inside
!
interface Vlan44
ip address 10.200.4.2 255.255.252.0
ip access-group test in
ip nat inside
standby 44 ip 10.200.4.1
!
interface Vlan100
ip address 198.47.127.1 255.255.255.0
ip nat outside
!
interface Vlan101
ip address 10.200.254.2 255.255.255.0
!
interface Vlan999
ip address 10.200.99.1 255.255.255.0
!
ip nat pool private_outbound 198.47.127.118 198.47.127.120 prefix-length 24
ip nat inside source list 118 interface TenGigabitEthernet7/1 overload
ip classless
ip forward-protocol nd
ip route 0.0.0.0 0.0.0.0 141.136.102.121
!
!
no ip http server
no ip http secure-server
!
!
!
!
control-plane
!
!
dial-peer cor custom
!
!
!
!
line con 0
exec-timeout 0 0
line vty 0 4
transport input ssh
!
!
end
core1-22001#
