ZBFW Intra zone traffic not working

[Elton Babcock]

I am having an issue on one of our 2811 routers where I can't get traffic between interfaces within the same zone to flow. I know this should happen by default and that's why it is so confusing.

One of the interfaces is fastethernet0/0.1 which is internal LAN And the others are tunnel interfaces using IPSEC tunnel protection back to the main datacenter. By design one tunnel is preferred over the other by using OSPF costing. Due to this there doesn't seem to be any asymmetric routing.

I inter zone traffic working just fine by defining the policy and zone pair. It is just when I enable another zone on our internal LAN interfaces it stops passing traffic. Just to note I do have this working on our LAB 2811 router running the same IOS version.

Any recommendations would be helpful. I have a case open with TAC but they aren't figuring it out. So now I'm calling the experts.

Thanks in advance. Elton

Sent from Cisco Technical Support iPhone App

[Julio Carvajal]

Hello Elton,

What version are you running??

Anything higher than 15.0(1)M?

If yes I have something that you could do

Rate all of the helpful posts!!!

Regards,

Jcarvaja

Follow me on http://laguiadelnetworking.com

[Elton Babcock]

No I am not running version 15 on this router. I am however running 15 on a 2911 router with this setup working just fine.

BRK-C2811-T1#show ver
Cisco IOS Software, 2800 Software (C2800NM-ADVIPSERVICESK9-M), Version 12.4(24)T1, RELEASE SOFTWARE (fc3)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2009 by Cisco Systems, Inc.
Compiled Fri 19-Jun-09 15:13 by prod_rel_team

ROM: System Bootstrap, Version 12.3(8r)T7, RELEASE SOFTWARE (fc1)

BRK-C2811-T1 uptime is 6 weeks, 5 days, 19 hours, 21 minutes
System returned to ROM by power-on
System restarted at 14:59:46 SummerTime Wed Oct 23 2013
System image file is "flash:c2800nm-advipservicesk9-mz.124-24.T1.bin"

[Julio Carvajal]

Hello Elton,

Okey, Then time to run some logs

ip inspect log drop-pkt

Then recreate the issue and provide

show logging | include x.x.x.x (Source of Traffic)

Also provide us more detail bud

What are the interfaces having the issue doing

Whats the traffic being denied

A diagram will be great

Rate all of the helpful posts!!!

Regards,

Jcarvaja

Follow me on http://laguiadelnetworking.com

[Elton Babcock]

I have already done some testing using the "ip inspect log drop-pkt" command the other day. This is what I am seeing as soon as I enable the zone even on just one of the interfaces.

Dec  4 20:45:15: %FW-6-DROP_PKT: Dropping tcp session 10.69.17.172:9123 10.27.19.254:48074  due to  policy match failure with ip ident 0

Dec  4 20:45:46: %FW-6-DROP_PKT: Dropping tcp session 10.69.16.1:53312 10.27.19.10:139  due to  policy match failure with ip ident 0

The funny thing is I never see as much traffic as I would expect and I don't see my ICMP traffic dropping to a server in the LAN. However it times out as soon as I enable the zone.

From what I can tell all traffic is dropping. TCP,UDP and ICMP traffic all fails as soon as I enable to the zones on the interfaces.

Elton

[Julio Carvajal]

We cannot do anything with only that info.

We are missing so much information buddy.... Configs, diagrams,etc

Rate all of the helpful posts!!!

Regards,

Jcarvaja

Follow me on http://laguiadelnetworking.com

[Elton Babcock]

Here is the sanitized configuration. The zone that I am trying to apply is "LAN".

I would like to apply it to all of the tunnel interfaces along with the fastethernet0/0.1 interface. This is working on another 2811 router.

Thanks again for the assistance.

!

version 12.4

service timestamps debug datetime localtime

service timestamps log datetime localtime

service password-encryption

!

hostname ****************

!

boot-start-marker

boot-end-marker

!

logging message-counter syslog

logging buffered 16384 informational

enable secret 5 ******************************

!

aaa new-model

!

!

aaa authentication login default group tacacs+ local

aaa authorization exec default group tacacs+ if-authenticated

!

!

aaa session-id common

clock timezone est -5

clock summer-time SummerTime recurring

!

dot11 syslog

ip source-route

!

ip traffic-export profile CAPTURE mode capture

  bidirectional

  incoming access-list CAPTURE_IN

  outgoing access-list CAPTURE_OUT

  length 512

!

!

ip cef

ip dhcp excluded-address 192.168.43.33 192.168.43.37

!

ip dhcp pool CREDIT_CARD_SCANNERS

   network 192.168.43.32 255.255.255.224

   default-router 192.168.43.33

   dns-server 4.2.2.2 8.8.4.4

   lease 2

!

!

no ip domain lookup

ip multicast-routing

ip inspect log drop-pkt

ip inspect name incoming tcp router-traffic

ip inspect name incoming udp router-traffic

login on-failure log every 3

no ipv6 cef

ntp server 10.69.16.1

!

multilink bundle-name authenticated

!

!

!

!

isdn switch-type basic-ni

!

!

!

!

!

!

!

!

!

!

!

!

!

!

!

!

!

!

!

voice-card 0

!

!

crypto pki trustpoint TP-self-signed-218647659

enrollment selfsigned

subject-name cn=IOS-Self-Signed-Certificate-218647659

revocation-check none

rsakeypair TP-self-signed-218647659

!

!

crypto pki certificate chain TP-self-signed-218647659

certificate self-signed 03

  30820242 308201AB A0030201 02020103 300D0609 2A864886 F70D0101 04050030

  30312E30 2C060355 04031325 494F532D 53656C66 2D536967 6E65642D 43657274

  69666963 6174652D 32313836 34373635 39301E17 0D313130 36303831 38303833

  395A170D 32303031 30313030 30303030 5A303031 2E302C06 03550403 1325494F

  532D5365 6C662D53 69676E65 642D4365 72746966 69636174 652D3231 38363437

  36353930 819F300D 06092A86 4886F70D 01010105 0003818D 00308189 02818100

  F9FF373A F00F58CF F4C6E6B1 C7676D6E EBD0D2D1 E239FAAA 42BD4335 B779D873

  A2D654FA 04F47F90 CCC79596 B3D5B719 D3994E6E 43B05D4D 4419D92C F8EC6149

  5094F9AB 7CB11EFA 5E72B723 A04D2999 BB43A8B8 11314E45 CA26BA77 909A63AA

  64A95D75 411C5141 026AA11A EA27724F A6832EBF A0C5DD7B A1E48803 4B8C0585

  02030100 01A36C30 6A300F06 03551D13 0101FF04 05300301 01FF3017 0603551D

  11041030 0E820C42 524B2D43 32383131 2D543130 1F060355 1D230418 30168014

  CA02D9F0 3B1772EE BECCFD40 888CD35B 4BF00440 301D0603 551D0E04 160414CA

  02D9F03B 1772EEBE CCFD4088 8CD35B4B F0044030 0D06092A 864886F7 0D010104

  05000381 810077C0 3260CF10 8652CE8D 6B0DE3F8 9BD87870 51087020 E00CC56B

  F01EBC1C F6DE78D9 D309E3D6 B63B713C 80FEE77B CEA7AD0D 3CA587B3 26912CC8

  EADA52D9 74698936 B8196FE0 120071EA B9F4CF3C 14D9E67C 34A0EA61 192BF856

  F77B5034 D45834CE D38D241A B1B08694 C786FAAF 9833D6DD DDF00562 F4839A51

  7ECEE3C1 BC06

        quit

!

!

username ************************** privilege 15 secret 5 ***********************************

archive

log config

  hidekeys

!

!

crypto isakmp policy 1

authentication pre-share

crypto isakmp key ***************** address *****************

crypto isakmp key **************** address *********************

crypto isakmp key ************* address **********************

crypto isakmp key ******************* address *********************

crypto isakmp keepalive 120 periodic

!

!

crypto ipsec transform-set TRANSFORM-AES esp-aes esp-sha-hmac

crypto ipsec transform-set TRANSFORM-AES-TRAN esp-aes esp-sha-hmac

mode transport require

!

crypto ipsec profile PROFILE-DMVPN

set transform-set TRANSFORM-AES

!

crypto ipsec profile PROFILE-DMVPN-TRAN

set transform-set TRANSFORM-AES-TRAN

!

!

!

!

!

track 1 ip sla 1 reachability

!

track 10 interface FastEthernet0/1 line-protocol

!

class-map type inspect match-any CC_SCAN_TRAFFIC_CLASS

match access-group name CC_SCAN_OUT

class-map type inspect match-all BBDBU-CMAP

match access-group name BBDBU

!

!

policy-map type inspect CC_SCAN_TRAFFIC_POLICY

class type inspect CC_SCAN_TRAFFIC_CLASS

  inspect

class class-default

  drop log

policy-map type inspect BBDBU-PMAP

class type inspect BBDBU-CMAP

  pass

class class-default

  drop log

!

zone security internet

zone security CC_SCAN_LAN

zone security LAN

zone-pair security self-to-internet source self destination internet

service-policy type inspect BBDBU-PMAP

zone-pair security internet-to-self source internet destination self

service-policy type inspect BBDBU-PMAP

zone-pair security CC_SCAN-TO-INTERNET source CC_SCAN_LAN destination internet

service-policy type inspect CC_SCAN_TRAFFIC_POLICY

!

!

!

!

interface Tunnel1

description Broadband backup circuit

bandwidth 256

ip address 10.69.7.111 255.255.255.0

ip mtu 1400

ip pim sparse-mode

ip nhrp authentication ****************

ip nhrp map 10.69.7.1 *********************

ip nhrp network-id **************

ip nhrp holdtime 300

ip nhrp nhs 10.69.7.1

ip nhrp server-only

ip ospf authentication-key 7 *******************

ip ospf network broadcast

ip ospf cost 130

ip ospf priority 0

tunnel source FastEthernet0/1

tunnel destination ********************

tunnel key ********************

tunnel protection ipsec profile PROFILE-DMVPN-TRAN

!

interface Tunnel2

description Backup Tunne2

bandwidth 512

ip address 10.69.10.111 255.255.255.0

ip mtu 1400

ip pim sparse-mode

ip nhrp authentication **************

ip nhrp map 10.69.10.1 ********************

ip nhrp network-id **************

ip nhrp holdtime 300

ip nhrp nhs 10.69.10.1

ip nhrp server-only

ip ospf authentication-key 7 ********************

ip ospf network broadcast

ip ospf priority 0

tunnel source FastEthernet0/1

tunnel destination ********************

tunnel key *********************

tunnel path-mtu-discovery

tunnel protection ipsec profile PROFILE-DMVPN-TRAN

!

interface Tunnel16

description mGRE TUNNEL FOR NYe0008981

bandwidth 1500

ip address 10.69.4.111 255.255.255.0

ip mtu 1400

ip flow ingress

ip pim sparse-mode

ip nat outside

ip nhrp authentication ****************

ip nhrp map 10.69.4.1 *********************

ip nhrp network-id ***************

ip nhrp holdtime 300

ip nhrp nhs 10.69.4.1

ip nhrp server-only

ip virtual-reassembly

ip ospf network broadcast

ip ospf cost 120

ip ospf priority 0

tunnel source Serial0/0/0

tunnel destination ******************

tunnel key ******************

tunnel protection ipsec profile PROFILE-DMVPN-TRAN

!

interface Tunnel17

description mGRE TUNNEL FOR NYe0008981

bandwidth 1450

ip address 10.69.8.111 255.255.255.0

ip mtu 1400

ip flow ingress

ip pim sparse-mode

ip nhrp authentication *******************

ip nhrp map 10.69.8.1 ****************

ip nhrp network-id **************

ip nhrp holdtime 300

ip nhrp nhs 10.69.8.1

ip nhrp server-only

ip ospf network broadcast

ip ospf cost 125

ip ospf priority 0

tunnel source Serial0/0/0

tunnel destination *****************

tunnel key ****************

tunnel protection ipsec profile PROFILE-DMVPN-TRAN

!

interface FastEthernet0/0

description PARENT INTERFACE

no ip address

ip flow ingress

ip traffic-export apply CAPTURE size 10000000

duplex auto

speed auto

!

interface FastEthernet0/0.1

description DEFAULT VLAN

encapsulation dot1Q 1 native

ip address 10.27.19.1 255.255.255.0

ip helper-address 10.69.16.7

ip pim sparse-mode

ip tcp adjust-mss 1344

ip traffic-export apply CAPTURE size 10000000

ip policy route-map PBR

ip ospf priority 0

!

interface FastEthernet0/0.10

description INITIAL VLAN

encapsulation dot1Q 10

ip traffic-export apply CAPTURE size 10000000

!

interface FastEthernet0/0.20

description AUTH-FAIL VLAN

encapsulation dot1Q 20

ip traffic-export apply CAPTURE size 10000000

shutdown

!

interface FastEthernet0/0.43

description CREDIT_CARD_SCANNERS

encapsulation dot1Q 43

ip address 192.168.43.33 255.255.255.224

ip nat inside

ip virtual-reassembly

zone-member security CC_SCAN_LAN

ip traffic-export apply CAPTURE size 10000000

!

interface FastEthernet0/0.98

description Remediation Vlan

encapsulation dot1Q 98

ip address 10.69.243.1 255.255.255.248

ip access-group Remediation in

ip helper-address 10.69.252.7

ip inspect incoming out

ip traffic-export apply CAPTURE size 10000000

ip ospf priority 0

!

interface FastEthernet0/0.99

description GUEST VLAN

encapsulation dot1Q 99

ip traffic-export apply CAPTURE size 10000000

!

interface FastEthernet0/0.666

description VENDOR VLAN

encapsulation dot1Q 666

ip traffic-export apply CAPTURE size 10000000

interface FastEthernet0/1
mtu 1492
ip address 192.168.1.47 255.255.255.0 secondary
ip address ************************** ip flow ingress
ip nat outside
ip virtual-reassembly
zone-member security internet
duplex auto
speed auto
!
interface Serial0/0/0
ip address **************************

ip flow ingress
encapsulation ppp
no fair-queue
service-module t1 remote-alarm-enable
service-module t1 fdl both
no cdp enable
!
interface BRI0/2/0
no ip address
encapsulation ppp
shutdown
dialer pool-member 1
isdn switch-type basic-ni
isdn point-to-point-setup
isdn spid1 71878317920101 7831792
isdn spid2 71878340300101 7834030
no cdp enable
!
interface Async0/1/0
no ip address
encapsulation slip
!
interface Dialer1
description T-1 Site ISDN Backup
ip address 192.168.103.38 255.255.255.0
encapsulation ppp
no ip route-cache cef
no ip route-cache
dialer pool 1
dialer idle-timeout 120 either
dialer load-threshold 32 either
dialer-group 1
no peer default ip address
no cdp enable
ppp multilink
!
router ospf 1
router-id 10.27.19.1
log-adjacency-changes
area 48 stub
network 10.27.19.0 0.0.0.255 area 48
network 10.69.4.0 0.0.0.255 area 48
network 10.69.7.0 0.0.0.255 area 48
network 10.69.8.0 0.0.0.255 area 48
network 10.69.10.0 0.0.0.255 area 48
network 10.69.243.0 0.0.0.7 area 48
!
ip forward-protocol nd
no ip forward-protocol udp domain
no ip forward-protocol udp time
no ip forward-protocol udp netbios-ns
no ip forward-protocol udp netbios-dgm
no ip forward-protocol udp tacacs
ip forward-protocol udp bootpc
ip route 198.203.191.83 255.255.255.255 ******************** track 1
ip route 198.203.192.245 255.255.255.255 *************** track 1
ip route 198.203.192.20 255.255.255.255 ****************** track 1
ip route 8.8.4.4 255.255.255.255 ***************** track 1
ip route 4.2.2.2 255.255.255.255 ******************* track 1
ip route 8.8.8.8 255.255.255.255 ********************** track 10
ip route 0.0.0.0 0.0.0.0 Dialer1 200
ip route 10.48.9.254 255.255.255.255 *****************
ip route 10.48.32.101 255.255.255.255 *****************
ip route 10.48.32.102 255.255.255.255 *****************
ip route 161.11.124.78 255.255.255.255 ******************

ip route 173.226.250.130 255.255.255.255 **************

ip route 204.89.170.126 255.255.255.255 ****************

no ip http server
no ip http secure-server
!
!
ip pim rp-address 10.69.31.1
ip nat pool CC_DMV_POOL 10.27.19.253 10.27.19.253 prefix-length 24
ip nat inside source route-map CC_BB_NAT interface FastEthernet0/1 overload
ip nat inside source route-map CC_DMV_NAT pool CC_DMV_POOL overload
ip tacacs source-interface FastEthernet0/0.1
!
ip access-list extended BBDBU
permit esp host *****************************

permit udp host **************************

permit gre host *******************************
permit udp host ****************************

permit gre host **************************

permit esp host ***********************

permit ip host **************************
permit ip host *****************************
permit icmp any host 8.8.8.8 echo
permit icmp host 8.8.8.8 any echo-reply
ip access-list extended BRK
permit ip 10.27.19.0 0.0.0.255 host 10.69.31.128
ip access-list extended CAPTURE_IN
permit ip host 10.27.19.10 host 10.69.66.108
ip access-list extended CAPTURE_OUT
permit ip host 10.69.66.108 host 10.27.19.10
ip access-list extended CC_SCAN_OUT
permit icmp 192.168.43.32 0.0.0.31 host 8.8.8.8
permit udp 192.168.43.32 0.0.0.31 host 8.8.8.8 eq domain
permit tcp 192.168.43.32 0.0.0.31 host 8.8.8.8 eq domain
permit tcp 192.168.43.32 0.0.0.31 host *************************

permit tcp 192.168.43.32 0.0.0.31 host **************************

permit tcp 192.168.43.32 0.0.0.31 host **************************

permit udp 192.168.43.32 0.0.0.31 host 4.2.2.2 eq domain
permit udp 192.168.43.32 0.0.0.31 host 8.8.4.4 eq domain
permit tcp 192.168.43.32 0.0.0.31 host 4.2.2.2 eq domain
permit tcp 192.168.43.32 0.0.0.31 host 8.8.4.4 eq domain
ip access-list extended Remediation
permit ip 10.69.240.0 0.0.15.255 host 10.69.252.7 log
permit icmp 10.69.240.0 0.0.15.255 10.69.66.0 0.0.0.255 log
permit tcp any host 10.69.16.182 eq 443 log
permit tcp any host 10.69.17.38 eq 8444 log
permit udp any any eq bootps
deny   ip any any
ip access-list extended VTY
permit tcp 10.69.66.0 0.0.0.255 any eq telnet log
permit tcp 10.69.66.0 0.0.0.255 any eq 22 log
permit tcp 10.69.31.0 0.0.0.255 any eq 22 log
permit tcp 10.69.31.0 0.0.0.255 any eq telnet log
permit tcp 10.48.32.96 0.0.0.7 any eq telnet log
permit tcp 10.48.32.96 0.0.0.7 any eq 22 log
permit tcp 1.11.1.0 0.0.0.255 any eq telnet log
permit tcp 1.11.1.0 0.0.0.255 any eq 22 log
deny   ip any any
!
ip sla 1
icmp-echo 8.8.8.8 source-interface FastEthernet0/1
timeout 7000
threshold 7000
frequency 10
ip sla schedule 1 life forever start-time now
logging 10.69.27.129
access-list 1 permit 10.69.66.11
access-list 1 remark SNMP Managers
access-list 1 permit 10.69.31.97
access-list 1 permit 10.69.31.100
access-list 1 permit 10.69.31.101
access-list 1 permit 10.69.66.59
access-list 1 permit 10.69.66.108
access-list 1 permit 10.69.16.223
access-list 1 permit 10.69.30.242
access-list 1 permit 10.69.16.250
access-list 1 permit 10.69.19.229
access-list 1 permit 10.69.16.150
access-list 1 permit 10.69.27.129
access-list 4 permit 10.69.31.148
access-list 4 permit 10.69.31.149
access-list 4 permit 10.69.31.150
access-list 4 permit 10.69.31.151
access-list 101 deny   ospf any any
access-list 101 permit ip any any
dialer-list 1 protocol ip list 101
!
!
!
!
route-map CC_DMV_NAT permit 10
match ip address CC_SCAN_OUT
match interface Tunnel16
!
route-map PBR permit 10
description BRK
match ip address BRK
set ip next-hop 10.69.7.1
!
route-map CC_BB_NAT permit 10
match ip address CC_SCAN_OUT
match interface FastEthernet0/1
!
!
snmp-server community ******************
snmp-server community *****************

snmp-server community ******************

snmp-server location **********************

snmp-server enable traps snmp coldstart warmstart
snmp-server enable traps tty
snmp-server enable traps flash insertion removal
snmp-server enable traps envmon
snmp-server enable traps config
snmp-server enable traps syslog

tacacs-server host 10.69.31.18 timeout 10
tacacs-server host 10.69.31.17
tacacs-server directed-request
tacacs-server key 7 ********************
control-plane
!
!
!
!
mgcp fax t38 ecm
mgcp behavior g729-variants static-pt
!
!
!
!
!
banner login ^C************************************
Unauthorized Entry To This Device Is

        STRICTLY PROHIBITED
************************************^C
!
line con 0
exec-timeout 30 0
logging synchronous
line aux 0
line 0/1/0
exec-timeout 60 0
modem InOut
modem autoconfigure discovery
transport input all
stopbits 1
speed 115200
flowcontrol hardware
line vty 0 4
access-class VTY in
exec-timeout 30 0
password 7 *********************
logging synchronous
transport input ssh
!
scheduler allocate 20000 1000
end

[Julio Carvajal]

Hello Elton,

The other router is running the exact same OS right???

You have no idea how many issues I have seeing like this and to be honest this are the worst as traffic should not be even considered by the firewall.

There are so many bugs about Tunnel interfaces and ZBFW.

I would recommend going to version 15 and if the issue happen you always have the chance to configure an Intra-Zone policy to make it happen

I mean there is nothing we can do at this version level to change the behavior, this is certanly a bug behavior!

Rate all of the helpful posts!!!

Regards,

Jcarvaja

Follow me on http://laguiadelnetworking.com

[Elton Babcock]

Yes the other router is running the exact same IOS version. Here is the 2 differnet show versions.

Working:

LAB-C2811-T1#show ver
Cisco IOS Software, 2800 Software (C2800NM-ADVIPSERVICESK9-M), Version 12.4(24)T1, RELEASE SOFTWARE (fc3)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2009 by Cisco Systems, Inc.
Compiled Fri 19-Jun-09 15:13 by prod_rel_team

ROM: System Bootstrap, Version 12.3(8r)T7, RELEASE SOFTWARE (fc1)

LAB-C2811-T1 uptime is 32 weeks, 1 day, 14 hours, 30 minutes
System returned to ROM by power-on
System restarted at 07:59:59 SummerTime Mon Apr 29 2013
System image file is "flash:c2800nm-advipservicesk9-mz.124-24.T1.bin"

Not Working:

BRK-C2811-T1#show ver
Cisco IOS Software, 2800 Software (C2800NM-ADVIPSERVICESK9-M), Version 12.4(24)T1, RELEASE SOFTWARE (fc3)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2009 by Cisco Systems, Inc.
Compiled Fri 19-Jun-09 15:13 by prod_rel_team

ROM: System Bootstrap, Version 12.3(8r)T7, RELEASE SOFTWARE (fc1)

BRK-C2811-T1 uptime is 6 weeks, 6 days, 7 hours, 24 minutes
System returned to ROM by power-on
System restarted at 14:59:46 SummerTime Wed Oct 23 2013
System image file is "flash:c2800nm-advipservicesk9-mz.124-24.T1.bin"

It makes me feel a little better that you have seen multiple issues but I just don't understand why one router would work and one wouldn't when they are running the same IOS and basically the same config.

I am going to see how TAC does to diagnose the problem but like you said it most likley is a bug. I really need to get this working as I currently dont have any failover to our other ISP for a specific subnet on this router that needs to be firewalled.

I'm not sure if I have the option to upgrade to version 15. We haven't tested it at all on any of our 2811 routers at all and i'm not sure if we are currently running enough RAM to support it. I would need to check.

Elton

[Julio Carvajal]

Hello Elton,

Yeah, I used to work for TAC (Like 3 months ago) and I always liked to grab the ZBFW/CBAC cases.

And this is certanly a bug. I do not have access to the Database that I used to have to look for a bug so we basically hit a wall here hehe.

Just remember to rate all of the helpful posts and provide the Troubleshoot steps TAC provides.

Regards,

Jcarvaja

Follow me on http://laguiadelnetworking.com