Routing Issue

[waqas.arshad]

Hi,

I have a situation where i am trying to access from 172.16.226.0/24 172.18.156.2 but traffic is not leaving my switch outing interface. I have another subnet 172.18.158.2 and i can ping it successfully. 

172.18.158.2 is accessible via default routing. I don't see any specific entry for this subnet. 

debaswco01#show ip route
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2
i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
ia - IS-IS inter area, * - candidate default, U - per-user static route
o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP
a - application route
+ - replicated route, % - next hop override, p - overrides from PfR

Gateway of last resort is 172.16.100.4 to network 0.0.0.0

S* 0.0.0.0/0 [1/0] via 172.16.100.4
10.0.0.0/8 is variably subnetted, 12 subnets, 3 masks
C 10.1.1.0/30 is directly connected, TenGigabitEthernet1/2/2
L 10.1.1.2/32 is directly connected, TenGigabitEthernet1/2/2
C 10.1.1.4/30 is directly connected, TenGigabitEthernet2/2/2
L 10.1.1.6/32 is directly connected, TenGigabitEthernet2/2/2
C 10.10.201.0/24 is directly connected, Vlan201
L 10.10.201.1/32 is directly connected, Vlan201
C 10.10.202.0/24 is directly connected, Vlan202
L 10.10.202.1/32 is directly connected, Vlan202
C 10.16.1.0/24 is directly connected, Vlan1
L 10.16.1.1/32 is directly connected, Vlan1
L 10.16.1.2/32 is directly connected, Vlan1
O 10.16.2.0/24 [110/41] via 10.1.1.5, 7w0d, TenGigabitEthernet2/2/2
[110/41] via 10.1.1.1, 7w0d, TenGigabitEthernet1/2/2
172.16.0.0/16 is variably subnetted, 91 subnets, 3 masks
C 172.16.1.0/24 is directly connected, Vlan601
L 172.16.1.1/32 is directly connected, Vlan601
C 172.16.2.0/24 is directly connected, Vlan602
L 172.16.2.1/32 is directly connected, Vlan602
C 172.16.3.0/24 is directly connected, Vlan603
L 172.16.3.1/32 is directly connected, Vlan603
C 172.16.4.0/24 is directly connected, Vlan604
L 172.16.4.1/32 is directly connected, Vlan604
C 172.16.5.0/24 is directly connected, Vlan605
L 172.16.5.1/32 is directly connected, Vlan605
C 172.16.6.0/24 is directly connected, Vlan606
L 172.16.6.1/32 is directly connected, Vlan606
C 172.16.7.0/24 is directly connected, Vlan607
L 172.16.7.1/32 is directly connected, Vlan607
C 172.16.8.0/24 is directly connected, Vlan608
L 172.16.8.1/32 is directly connected, Vlan608
C 172.16.9.0/24 is directly connected, Vlan609
L 172.16.9.1/32 is directly connected, Vlan609
C 172.16.10.0/24 is directly connected, Vlan610
L 172.16.10.1/32 is directly connected, Vlan610
C 172.16.12.0/24 is directly connected, Vlan612
L 172.16.12.1/32 is directly connected, Vlan612
C 172.16.14.0/24 is directly connected, Vlan614
L 172.16.14.1/32 is directly connected, Vlan614
C 172.16.16.0/24 is directly connected, Vlan616
L 172.16.16.1/32 is directly connected, Vlan616
C 172.16.17.0/24 is directly connected, Vlan617
L 172.16.17.1/32 is directly connected, Vlan617
C 172.16.18.0/24 is directly connected, Vlan618
L 172.16.18.1/32 is directly connected, Vlan618
C 172.16.19.0/24 is directly connected, Vlan619
L 172.16.19.1/32 is directly connected, Vlan619
C 172.16.20.0/24 is directly connected, Vlan620
L 172.16.20.1/32 is directly connected, Vlan620
C 172.16.22.0/24 is directly connected, Vlan622
L 172.16.22.1/32 is directly connected, Vlan622
C 172.16.23.0/24 is directly connected, Vlan623
L 172.16.23.1/32 is directly connected, Vlan623
C 172.16.24.0/24 is directly connected, Vlan624
L 172.16.24.1/32 is directly connected, Vlan624
C 172.16.25.0/24 is directly connected, Vlan625
L 172.16.25.1/32 is directly connected, Vlan625
C 172.16.26.0/24 is directly connected, Vlan626
L 172.16.26.1/32 is directly connected, Vlan626
C 172.16.31.0/24 is directly connected, Vlan631
L 172.16.31.1/32 is directly connected, Vlan631
C 172.16.32.0/24 is directly connected, Vlan632
L 172.16.32.2/32 is directly connected, Vlan632
C 172.16.33.0/24 is directly connected, Vlan633
L 172.16.33.1/32 is directly connected, Vlan633
C 172.16.34.0/24 is directly connected, Vlan634
L 172.16.34.1/32 is directly connected, Vlan634
C 172.16.35.0/24 is directly connected, Vlan635
L 172.16.35.1/32 is directly connected, Vlan635
C 172.16.36.0/23 is directly connected, Vlan636
L 172.16.36.1/32 is directly connected, Vlan636
C 172.16.40.0/24 is directly connected, Vlan640
L 172.16.40.1/32 is directly connected, Vlan640
C 172.16.42.0/24 is directly connected, Vlan642
L 172.16.42.1/32 is directly connected, Vlan642
C 172.16.50.0/24 is directly connected, Vlan11
L 172.16.50.1/32 is directly connected, Vlan11
C 172.16.52.0/24 is directly connected, Vlan652
L 172.16.52.1/32 is directly connected, Vlan652
C 172.16.53.0/24 is directly connected, Vlan653
L 172.16.53.1/32 is directly connected, Vlan653
C 172.16.54.0/24 is directly connected, Vlan654
L 172.16.54.1/32 is directly connected, Vlan654
C 172.16.100.0/24 is directly connected, Vlan100
L 172.16.100.1/32 is directly connected, Vlan100
C 172.16.133.0/24 is directly connected, Vlan705
L 172.16.133.1/32 is directly connected, Vlan705
C 172.16.134.0/24 is directly connected, Vlan707
L 172.16.134.1/32 is directly connected, Vlan707
C 172.16.151.0/24 is directly connected, Vlan151
L 172.16.151.1/32 is directly connected, Vlan151
C 172.16.152.0/23 is directly connected, Vlan152
L 172.16.152.1/32 is directly connected, Vlan152
C 172.16.154.0/23 is directly connected, Vlan154
L 172.16.154.1/32 is directly connected, Vlan154
C 172.16.156.0/23 is directly connected, Vlan156
L 172.16.156.1/32 is directly connected, Vlan156
C 172.16.200.0/24 is directly connected, Vlan200
L 172.16.200.1/32 is directly connected, Vlan200
C 172.16.201.0/24 is directly connected, Vlan702
L 172.16.201.1/32 is directly connected, Vlan702
O 172.16.220.0/23 [110/41] via 10.1.1.5, 7w0d, TenGigabitEthernet2/2/2
[110/41] via 10.1.1.1, 7w0d, TenGigabitEthernet1/2/2
O 172.16.222.0/23 [110/41] via 10.1.1.5, 7w0d, TenGigabitEthernet2/2/2
[110/41] via 10.1.1.1, 7w0d, TenGigabitEthernet1/2/2
O 172.16.224.0/23 [110/41] via 10.1.1.5, 7w0d, TenGigabitEthernet2/2/2
[110/41] via 10.1.1.1, 7w0d, TenGigabitEthernet1/2/2
O 172.16.226.0/24 [110/41] via 10.1.1.5, 7w0d, TenGigabitEthernet2/2/2
[110/41] via 10.1.1.1, 7w0d, TenGigabitEthernet1/2/2
O 172.16.227.0/24 [110/41] via 10.1.1.5, 7w0d, TenGigabitEthernet2/2/2
[110/41] via 10.1.1.1, 7w0d, TenGigabitEthernet1/2/2
debaswco01#
debaswco01#
debaswco01#ping 172.18.158.2
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.18.158.2, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 28/32/44 ms
debaswco01#
debaswco01#
debaswco01#ping 172.18.156.2
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.18.156.2, timeout is 2 seconds:
.....
Success rate is 0 percent (0/5)

debaswco01#show ip route 172.18.158.2
% Network not in table

anyone can help me to resolve the issue?

Regards,

Warshad

[Harold Ritter]

Hi @waqas.arshad ,

The acl only allows ping from 172.16.226.21,22,23 to 172.18.156.21. This is why you can't ping or traceroute from debaswdata300 or debaswco01. Please change the acl to allow icmp from 172.16.226.2 and 172.16.100.1 if you want to be able to ping from these two devices. 

Regards,

[MHM Cisco World]

hitcnt=0 !! why all ACL hitcnt equal 0.
there is something make traffic not reach FW at all.

[Harold Ritter]

Hi @MHM Cisco World ,

The acl entries that he is showing are for 172.16.226.21,22 and 23 to 172.18.156.21 and he is pinging from 172.16.100.1 and 172.16.226.2. This would explain the hitcnt being equal to zero.

Regards,

[waqas.arshad]

Hi Harold,

Now i have added 172.16.100.1 in the rule but still i am getting no response. In ACL i can see the hitcnt=5.

debafwin001/pri/act# show access-list INSIDE | i 172.18.156.2
access-list INSIDE line 224 extended permit icmp host 172.16.226.21 host 172.18.156.21 (hitcnt=0) 0xd66b7f13
access-list INSIDE line 224 extended permit icmp host 172.16.226.22 host 172.18.156.21 (hitcnt=0) 0x64871058
access-list INSIDE line 224 extended permit icmp host 172.16.226.23 host 172.18.156.21 (hitcnt=0) 0x5b47f8f2
access-list INSIDE line 224 extended permit tcp host 172.16.226.21 host 172.18.156.21 range 8080 8180 (hitcnt=0) 0xc4f9ddb7
access-list INSIDE line 224 extended permit tcp host 172.16.226.22 host 172.18.156.21 range 8080 8180 (hitcnt=0) 0x1db1010a
access-list INSIDE line 224 extended permit tcp host 172.16.226.23 host 172.18.156.21 range 8080 8180 (hitcnt=0) 0x9b57b65e
access-list INSIDE line 224 extended permit tcp host 172.16.226.21 host 172.18.156.21 eq www (hitcnt=0) 0xc0e096cd
access-list INSIDE line 224 extended permit tcp host 172.16.226.22 host 172.18.156.21 eq www (hitcnt=0) 0x70d0a587
access-list INSIDE line 224 extended permit tcp host 172.16.226.23 host 172.18.156.21 eq www (hitcnt=0) 0x736ebc21
access-list INSIDE line 224 extended permit tcp host 172.16.226.21 host 172.18.156.21 eq https (hitcnt=0) 0x7dc5dd9f
access-list INSIDE line 224 extended permit tcp host 172.16.226.22 host 172.18.156.21 eq https (hitcnt=0) 0x5ef3afdd
access-list INSIDE line 224 extended permit tcp host 172.16.226.23 host 172.18.156.21 eq https (hitcnt=0) 0xfaeb2e21
access-list INSIDE line 224 extended permit icmp host 172.16.100.1 host 172.18.156.21 (hitcnt=5) 0x0b6c99c7
access-list INSIDE line 224 extended permit tcp host 172.16.100.1 host 172.18.156.21 range 8080 8180 (hitcnt=0) 0xd1a44b14
access-list INSIDE line 224 extended permit tcp host 172.16.100.1 host 172.18.156.21 eq www (hitcnt=0) 0x36c9f1ac
access-list INSIDE line 224 extended permit tcp host 172.16.100.1 host 172.18.156.21 eq https (hitcnt=0) 0xc26ad204

Regards,

Waqas

[Harold Ritter]

Hi @waqas.arshad ,

We are making progress. The traffic is now passing through the FW.  Can you validate that 172.18.156.21 is alive, because you started this thread pinging 172.18.156.2 and now you shifted to 172.18.156.21. Can you validate that this address is responding to ping from outside the FW.

Also, I would recommend that you look at the rules for 172.18.158.0/24 and apply similar rules for 172.16.158.0/24. 

Regards,

[waqas.arshad]

Hi Harold,

I can ping both addresses from FW. From FW to onward i don't see any communication issue.

debafwin001/pri/act# ping 172.18.156.21
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.18.156.21, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 30/30/30 ms
debafwin001/pri/act#
debafwin001/pri/act#
debafwin001/pri/act#
debafwin001/pri/act# ping 172.18.156.2
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.18.156.2, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 20/28/30 ms

Regards,

Arshad

[Harold Ritter]

Can you run the following commands on the FW:

packet-tracer input inside icmp 172.16.100.1 8 0 172.18.156.21 detail

packet-tracer input outside icmp 172.18.156.21 0 0 172.16.100.1 detail

Regards,

[waqas.arshad]

Sure, Here is the ouput.

debafwin001/pri/act# packet-tracer input inside icmp 172.16.100.1 8 0 172.18.1$

Phase: 1
Type: INPUT-ROUTE-LOOKUP
Subtype: Resolve Egress Interface
Result: ALLOW
Config:
Additional Information:
Found next-hop 172.31.255.15 using egress ifc outside

Phase: 2
Type: ACCESS-LIST
Subtype: log
Result: DROP
Config:
access-group INSIDE in interface inside
access-list INSIDE extended deny ip any any log notifications
Additional Information:
Forward Flow based lookup yields rule:
in id=0x7f7171bfe220, priority=13, domain=permit, deny=true
hits=1381012415, user_data=0x7f71714aaf00, cs_id=0x0, use_real_addr, flags=0x0, protocol=0
src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any, dscp=0x0, nsg_id=none
input_ifc=inside, output_ifc=any

Result:
input-interface: inside
input-status: up
input-line-status: up
output-interface: outside
output-status: up
output-line-status: up
Action: drop
Drop-reason: (acl-drop) Flow is denied by configured rule, Drop-location: frame 0x000055d5d35080c7 flow (NA)/NA

debafwin001/pri/act# packet-tracer input outside icmp 172.18.156.21 0 0 172.16$

Phase: 1
Type: ACCESS-LIST
Subtype:
Result: ALLOW
Config:
Implicit Rule
Additional Information:
Forward Flow based lookup yields rule:
in id=0x7f716f129b20, priority=1, domain=permit, deny=false
hits=159926561102, user_data=0x0, cs_id=0x0, l3_type=0x8
src mac=0000.0000.0000, mask=0000.0000.0000
dst mac=0000.0000.0000, mask=0100.0000.0000
input_ifc=outside, output_ifc=any

Phase: 2
Type: ROUTE-LOOKUP
Subtype: No ECMP load balancing
Result: ALLOW
Config:
Additional Information:
Destination is locally connected. No ECMP load balancing.
Found next-hop 172.16.100.1 using egress ifc inside

Phase: 3
Type: ACCESS-LIST
Subtype:
Result: DROP
Config:
Implicit Rule
Additional Information:
Forward Flow based lookup yields rule:
in id=0x7f7234c1fc90, priority=11, domain=permit, deny=true
hits=27974992, user_data=0x6, cs_id=0x0, use_real_addr, flags=0x0, protocol=0
src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any, dscp=0x0, nsg_id=none
input_ifc=outside, output_ifc=any

Result:
input-interface: outside
input-status: up
input-line-status: up
Action: drop
Drop-reason: (acl-drop) Flow is denied by configured rule, Drop-location: frame 0x000055d5d35080c7 flow (NA)/NA

Regards,

Arshad

[Harold Ritter]

Hi @waqas.arshad ,

The packet_tracer output still indicates that the packet for the flow 172.16.100.1 -> 172.18.156.21 is dropped because it hits the "deny ip any any" entry.

Phase: 2
Type: ACCESS-LIST
Subtype: log
Result: DROP
Config:
access-group INSIDE in interface inside
access-list INSIDE extended deny ip any any log notifications

There is not much I can do without seeing the complete INSIDE acl.

Regards,

[MHM Cisco World]

can you draw the topology please ??

[waqas.arshad]

Hi Everyone,

The issue has been resolved. I allowed the icmp for 172.16.226.2 which is gateway for host 172.16.226.21 and it started working. 

I would like to say  Thank you to @Harold Ritter @MHM Cisco World for their time and efforts.

Regards,

Waqas Arshad

[MHM Cisco World]

You are so welcome. 
one last thing can I see the output of 
show ip cef 172.18.158.2 <<-

[waqas.arshad]

@MHM Cisco World 

debaswco01#show ip cef 172.18.156.2
0.0.0.0/0
nexthop 172.16.100.4 Vlan100

[Harold Ritter]

You are very welcome @waqas.arshad . Thanks for the feedback 

[MHM Cisco World]

thanks, It now OK, the cef build L2 adj. 
thanks again for sharing detail.