Routing Issue

[waqas.arshad]

Hi,

I have a situation where i am trying to access from 172.16.226.0/24 172.18.156.2 but traffic is not leaving my switch outing interface. I have another subnet 172.18.158.2 and i can ping it successfully. 

172.18.158.2 is accessible via default routing. I don't see any specific entry for this subnet. 

debaswco01#show ip route
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2
i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
ia - IS-IS inter area, * - candidate default, U - per-user static route
o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP
a - application route
+ - replicated route, % - next hop override, p - overrides from PfR

Gateway of last resort is 172.16.100.4 to network 0.0.0.0

S* 0.0.0.0/0 [1/0] via 172.16.100.4
10.0.0.0/8 is variably subnetted, 12 subnets, 3 masks
C 10.1.1.0/30 is directly connected, TenGigabitEthernet1/2/2
L 10.1.1.2/32 is directly connected, TenGigabitEthernet1/2/2
C 10.1.1.4/30 is directly connected, TenGigabitEthernet2/2/2
L 10.1.1.6/32 is directly connected, TenGigabitEthernet2/2/2
C 10.10.201.0/24 is directly connected, Vlan201
L 10.10.201.1/32 is directly connected, Vlan201
C 10.10.202.0/24 is directly connected, Vlan202
L 10.10.202.1/32 is directly connected, Vlan202
C 10.16.1.0/24 is directly connected, Vlan1
L 10.16.1.1/32 is directly connected, Vlan1
L 10.16.1.2/32 is directly connected, Vlan1
O 10.16.2.0/24 [110/41] via 10.1.1.5, 7w0d, TenGigabitEthernet2/2/2
[110/41] via 10.1.1.1, 7w0d, TenGigabitEthernet1/2/2
172.16.0.0/16 is variably subnetted, 91 subnets, 3 masks
C 172.16.1.0/24 is directly connected, Vlan601
L 172.16.1.1/32 is directly connected, Vlan601
C 172.16.2.0/24 is directly connected, Vlan602
L 172.16.2.1/32 is directly connected, Vlan602
C 172.16.3.0/24 is directly connected, Vlan603
L 172.16.3.1/32 is directly connected, Vlan603
C 172.16.4.0/24 is directly connected, Vlan604
L 172.16.4.1/32 is directly connected, Vlan604
C 172.16.5.0/24 is directly connected, Vlan605
L 172.16.5.1/32 is directly connected, Vlan605
C 172.16.6.0/24 is directly connected, Vlan606
L 172.16.6.1/32 is directly connected, Vlan606
C 172.16.7.0/24 is directly connected, Vlan607
L 172.16.7.1/32 is directly connected, Vlan607
C 172.16.8.0/24 is directly connected, Vlan608
L 172.16.8.1/32 is directly connected, Vlan608
C 172.16.9.0/24 is directly connected, Vlan609
L 172.16.9.1/32 is directly connected, Vlan609
C 172.16.10.0/24 is directly connected, Vlan610
L 172.16.10.1/32 is directly connected, Vlan610
C 172.16.12.0/24 is directly connected, Vlan612
L 172.16.12.1/32 is directly connected, Vlan612
C 172.16.14.0/24 is directly connected, Vlan614
L 172.16.14.1/32 is directly connected, Vlan614
C 172.16.16.0/24 is directly connected, Vlan616
L 172.16.16.1/32 is directly connected, Vlan616
C 172.16.17.0/24 is directly connected, Vlan617
L 172.16.17.1/32 is directly connected, Vlan617
C 172.16.18.0/24 is directly connected, Vlan618
L 172.16.18.1/32 is directly connected, Vlan618
C 172.16.19.0/24 is directly connected, Vlan619
L 172.16.19.1/32 is directly connected, Vlan619
C 172.16.20.0/24 is directly connected, Vlan620
L 172.16.20.1/32 is directly connected, Vlan620
C 172.16.22.0/24 is directly connected, Vlan622
L 172.16.22.1/32 is directly connected, Vlan622
C 172.16.23.0/24 is directly connected, Vlan623
L 172.16.23.1/32 is directly connected, Vlan623
C 172.16.24.0/24 is directly connected, Vlan624
L 172.16.24.1/32 is directly connected, Vlan624
C 172.16.25.0/24 is directly connected, Vlan625
L 172.16.25.1/32 is directly connected, Vlan625
C 172.16.26.0/24 is directly connected, Vlan626
L 172.16.26.1/32 is directly connected, Vlan626
C 172.16.31.0/24 is directly connected, Vlan631
L 172.16.31.1/32 is directly connected, Vlan631
C 172.16.32.0/24 is directly connected, Vlan632
L 172.16.32.2/32 is directly connected, Vlan632
C 172.16.33.0/24 is directly connected, Vlan633
L 172.16.33.1/32 is directly connected, Vlan633
C 172.16.34.0/24 is directly connected, Vlan634
L 172.16.34.1/32 is directly connected, Vlan634
C 172.16.35.0/24 is directly connected, Vlan635
L 172.16.35.1/32 is directly connected, Vlan635
C 172.16.36.0/23 is directly connected, Vlan636
L 172.16.36.1/32 is directly connected, Vlan636
C 172.16.40.0/24 is directly connected, Vlan640
L 172.16.40.1/32 is directly connected, Vlan640
C 172.16.42.0/24 is directly connected, Vlan642
L 172.16.42.1/32 is directly connected, Vlan642
C 172.16.50.0/24 is directly connected, Vlan11
L 172.16.50.1/32 is directly connected, Vlan11
C 172.16.52.0/24 is directly connected, Vlan652
L 172.16.52.1/32 is directly connected, Vlan652
C 172.16.53.0/24 is directly connected, Vlan653
L 172.16.53.1/32 is directly connected, Vlan653
C 172.16.54.0/24 is directly connected, Vlan654
L 172.16.54.1/32 is directly connected, Vlan654
C 172.16.100.0/24 is directly connected, Vlan100
L 172.16.100.1/32 is directly connected, Vlan100
C 172.16.133.0/24 is directly connected, Vlan705
L 172.16.133.1/32 is directly connected, Vlan705
C 172.16.134.0/24 is directly connected, Vlan707
L 172.16.134.1/32 is directly connected, Vlan707
C 172.16.151.0/24 is directly connected, Vlan151
L 172.16.151.1/32 is directly connected, Vlan151
C 172.16.152.0/23 is directly connected, Vlan152
L 172.16.152.1/32 is directly connected, Vlan152
C 172.16.154.0/23 is directly connected, Vlan154
L 172.16.154.1/32 is directly connected, Vlan154
C 172.16.156.0/23 is directly connected, Vlan156
L 172.16.156.1/32 is directly connected, Vlan156
C 172.16.200.0/24 is directly connected, Vlan200
L 172.16.200.1/32 is directly connected, Vlan200
C 172.16.201.0/24 is directly connected, Vlan702
L 172.16.201.1/32 is directly connected, Vlan702
O 172.16.220.0/23 [110/41] via 10.1.1.5, 7w0d, TenGigabitEthernet2/2/2
[110/41] via 10.1.1.1, 7w0d, TenGigabitEthernet1/2/2
O 172.16.222.0/23 [110/41] via 10.1.1.5, 7w0d, TenGigabitEthernet2/2/2
[110/41] via 10.1.1.1, 7w0d, TenGigabitEthernet1/2/2
O 172.16.224.0/23 [110/41] via 10.1.1.5, 7w0d, TenGigabitEthernet2/2/2
[110/41] via 10.1.1.1, 7w0d, TenGigabitEthernet1/2/2
O 172.16.226.0/24 [110/41] via 10.1.1.5, 7w0d, TenGigabitEthernet2/2/2
[110/41] via 10.1.1.1, 7w0d, TenGigabitEthernet1/2/2
O 172.16.227.0/24 [110/41] via 10.1.1.5, 7w0d, TenGigabitEthernet2/2/2
[110/41] via 10.1.1.1, 7w0d, TenGigabitEthernet1/2/2
debaswco01#
debaswco01#
debaswco01#ping 172.18.158.2
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.18.158.2, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 28/32/44 ms
debaswco01#
debaswco01#
debaswco01#ping 172.18.156.2
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.18.156.2, timeout is 2 seconds:
.....
Success rate is 0 percent (0/5)

debaswco01#show ip route 172.18.158.2
% Network not in table

anyone can help me to resolve the issue?

Regards,

Warshad

[MHM Cisco World]

I see same issue before, 
I will run lab and check how we  can solve it 

[waqas.arshad]

So you dont see any issue in ACls? Let me know with your lab testing results.

Regards,

Warshad

[MHM Cisco World]

show ip cef 0.0.0.0 internal 
show adjacency detail

can share the above command  

[waqas.arshad]

Here are the output of the requested commands. Please find the attached file for show adjacency details. 

debaswco01#show ip cef 0.0.0.0 internal
0.0.0.0/32, epoch 13, flags [rcv], refcnt 6, per-destination sharing
sources: Spc
feature space:
Broker: linked, distributed at 4th priority
subblocks:
Special source: receive
ifnums: (none)
path list 275FAE58, 7 locks, per-destination, flags 0x41 [shble, hwcn]
path 275FB120, share 1/1, type receive, for IPv4
receive
output chain:
receive

Regards,

Warshad

[Harold Ritter]

Hi @waqas.arshad ,

This is the wrong command You should do "show ip ref 0.0.0.0 0.0.0.0 detail"

Regards,

[waqas.arshad]

Here is the requested output.

debaswco01#show ip cef 0.0.0.0 0.0.0.0 detail
0.0.0.0/0, epoch 13, flags [default route]
recursive via 172.16.100.4
attached to Vlan100

Regards,

Warshad

[Harold Ritter]

Hi @waqas.arshad ,
As others mentioned, it definitely looks like the issue is with the FW rules not allowing the ping to 172.18.156.2.
You can run the following command on the FW to proof the FW is the culprit (assuming the FW inside interface is the one with address 172.16.100.4):

packet-tracer input inside icmp 172.16.100.1 8 0 172.18.156.2 detail

This command should tell you exactly what is going on on the FW.

You can also run the command with 172.18.158.2, which should give you a positive result, as ping to that destination works.

packet-tracer input inside icmp 172.16.100.1 8 0 172.18.158.2 detail

Regards,

[MHM Cisco World]

friend, 
first can I ask you did you run clear adjacency ?
I think you have issue with CEF !! how I know ?

the CEF table have number. each time the CEF table is update this number is increase, this number is epoch, 
the epoch as I see in show adjacency is equal to 0 but the epoch of show ip cef internal or show ip cef x/x/x/x detail is 13
this mismatch I think is cause of issue. 

[waqas.arshad]

Hi,

I did not run clear adjacency command. Here is the output of packet-tracer input inside icmp 172.16.100.1 8 0 172.18.156.2 detail as suggested by @Harold Ritter . According to output traffic is dropped by Firewall.

Phase: 1
Type: INPUT-ROUTE-LOOKUP
Subtype: Resolve Egress Interface
Result: ALLOW
Config:
Additional Information:
Found next-hop 172.31.255.15 using egress ifc outside

Phase: 2
Type: ACCESS-LIST
Subtype: log
Result: DROP
Config:
access-group INSIDE in interface inside
access-list INSIDE extended deny ip any any log notifications
Additional Information:
Forward Flow based lookup yields rule:
in id=0x7f7171bfe220, priority=13, domain=permit, deny=true
hits=1374977023, user_data=0x7f71714aaf00, cs_id=0x0, use_real_addr, flags=0x0, protocol=0
src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any, dscp=0x0, nsg_id=none
input_ifc=inside, output_ifc=any

Result:
input-interface: inside
input-status: up
input-line-status: up
output-interface: outside
output-status: up
output-line-status: up
Action: drop
Drop-reason: (acl-drop) Flow is denied by configured rule, Drop-location: frame 0x000055d5d35080c7 flow (NA)/NA

I allowed ICMP is in acl but still ping is not successful. 

Regards,

Warshad

[Harold Ritter]

Hi @waqas.arshad ,

 The message states that ACL INSIDE rejects the ICMP request. Did you allow ICMP in the INSIDE acl? If so, can you show the entry you added in the acl? 

Phase: 2
Type: ACCESS-LIST
Subtype: log
Result: DROP
Config:
access-group INSIDE in interface inside
access-list INSIDE extended deny ip any any log notifications

Regards,

[waqas.arshad]

@Harold Ritter 

debafwin001/pri/act# show access-list INSIDE | i 172.18.156.21
access-list INSIDE line 224 extended permit icmp host 172.16.226.21 host 172.18.156.21 (hitcnt=0) 0xd66b7f13
access-list INSIDE line 224 extended permit icmp host 172.16.226.22 host 172.18.156.21 (hitcnt=0) 0x64871058
access-list INSIDE line 224 extended permit icmp host 172.16.226.23 host 172.18.156.21 (hitcnt=0) 0x5b47f8f2
access-list INSIDE line 224 extended permit tcp host 172.16.226.21 host 172.18.156.21 range 8080 8180 (hitcnt=0) 0xc4f9ddb7
access-list INSIDE line 224 extended permit tcp host 172.16.226.22 host 172.18.156.21 range 8080 8180 (hitcnt=0) 0x1db1010a
access-list INSIDE line 224 extended permit tcp host 172.16.226.23 host 172.18.156.21 range 8080 8180 (hitcnt=0) 0x9b57b65e
access-list INSIDE line 224 extended permit tcp host 172.16.226.21 host 172.18.156.21 eq www (hitcnt=0) 0xc0e096cd
access-list INSIDE line 224 extended permit tcp host 172.16.226.22 host 172.18.156.21 eq www (hitcnt=0) 0x70d0a587
access-list INSIDE line 224 extended permit tcp host 172.16.226.23 host 172.18.156.21 eq www (hitcnt=0) 0x736ebc21
access-list INSIDE line 224 extended permit tcp host 172.16.226.21 host 172.18.156.21 eq https (hitcnt=0) 0x7dc5dd9f
access-list INSIDE line 224 extended permit tcp host 172.16.226.22 host 172.18.156.21 eq https (hitcnt=0) 0x5ef3afdd
access-list INSIDE line 224 extended permit tcp host 172.16.226.23 host 172.18.156.21 eq https (hitcnt=0) 0xfaeb2e21

Regards,

Warshad

[Harold Ritter]

Hi @waqas.arshad ,

The address you are trying to ping is 172.18.156.2 or 172.18.156.21? In any case, I do not see any entry that would allow the source 172.16.100.1 (debaswco01) to neither 172.18.156.2 or 172.18.156.21. This is why you can't ping from the router to these two addresses.

Regards,

[waqas.arshad]

Dear Harold,

Thank you for your reply. 172.16.100.1 or 10.1.1.2 (debaswco01) is core L3 switch and it is connected to another switch debaswdata300 and source 172.16.226.21 is located behind this switch. If do ping from debaswdata300 it is not successful. In the traceroute you can traffic is going to 172.16.100.1 or 10.1.1.2 after this it is blocked and that's why i am doing testing from 10.1.1.2 because traffic is getting blocked after this hope. You can also see successful ping to 172.18.158.2. 

172.18.156.21 is host address and 172.18.156.2 is the vlan interface ip address. Source host is 172.16.226.21 and destination host is 172.18.156.21 I hope now you have clear picture.

debaswdata300# traceroute 172.18.156.2
traceroute to 172.18.156.2 (172.18.156.2), 30 hops max, 40 byte packets
1 10.1.1.2 (10.1.1.2) 0.566 ms 0.417 ms 0.694 ms
2 * * *
3 * * *
4 * * *
5 * * *
6 * * *
7 * * *

debaswdata300# ping 172.18.158.2 source 172.16.226.2
PING 172.18.158.2 (172.18.158.2) from 172.16.226.2: 56 data bytes
64 bytes from 172.18.158.2: icmp_seq=0 ttl=248 time=29.166 ms
64 bytes from 172.18.158.2: icmp_seq=1 ttl=248 time=28.889 ms
64 bytes from 172.18.158.2: icmp_seq=2 ttl=248 time=29.354 ms
64 bytes from 172.18.158.2: icmp_seq=3 ttl=248 time=28.964 ms
64 bytes from 172.18.158.2: icmp_seq=4 ttl=248 time=29.895 ms

--- 172.18.158.2 ping statistics ---
5 packets transmitted, 5 packets received, 0.00% packet loss
round-trip min/avg/max = 28.889/29.253/29.895 ms
debaswdata300#
debaswdata300#
debaswdata300#
debaswdata300# ping 172.18.156.2 source 172.16.226.2
PING 172.18.156.2 (172.18.156.2) from 172.16.226.2: 56 data bytes
Request 0 timed out
Request 1 timed out
Request 2 timed out
Request 3 timed out
Request 4 timed out

--- 172.18.156.2 ping statistics ---
5 packets transmitted, 0 packets received, 100.00% packet loss

debaswdata300# traceroute 172.18.156.2
traceroute to 172.18.156.2 (172.18.156.2), 30 hops max, 40 byte packets
1 10.1.1.2 (10.1.1.2) 0.566 ms 0.417 ms 0.694 ms
2 * * *
3 * * *
4 * * *
5 * * *
6 * * *
7 * * *

debaswdata300# show ip int brief
IP Interface Status for VRF "default"(1)
Interface IP Address Interface Status
Vlan1 10.16.2.4 protocol-up/link-up/admin-up
Vlan220 172.16.220.2 protocol-up/link-up/admin-up
Vlan222 172.16.222.2 protocol-up/link-up/admin-up
Vlan224 172.16.224.2 protocol-up/link-up/admin-up
Vlan226 172.16.226.2 protocol-up/link-up/admin-up
Vlan227 172.16.227.2 protocol-up/link-up/admin-up
Eth1/33 10.1.1.1 protocol-up/link-up/admin-up

let me know if you have any questions.

Regards,

Warshad

[Harold Ritter]

Hi @waqas.arshad ,

Thanks for the information. The action is still the same. You need to update you FW rule to allow communication between the source and the destination. You mentioned earlier that you had added to the acl, but you didn't provide the line you added. Can you please provide the line you added? 

Also you need to be careful, as you have a "deny ip any any" at the end of the INSIDE acl, so if you just add a new entry it will go at the end of the acl and it won't work, because the traffic will hit the 'deny ip any any" first. 

access-list INSIDE extended deny ip any any log notifications

Regards,

[waqas.arshad]

Dear Harlod,

Thank you for explanation. Here are rules in which i allowed ICMP between source and destination. In line 224 i allowed communication from different hosts of 172.16.226.0 subnet. 

The deny statement have entry in line 289 and icmp is allowed in line 224 so deny statement should not have any impact. Let me know if you have any questions.

debafwin001/pri/act# show access-list | I 172.18.156.21
access-list INSIDE line 224 extended permit icmp host 172.16.226.21 host 172.18.156.21 (hitcnt=0) 0xd66b7f13
access-list INSIDE line 224 extended permit icmp host 172.16.226.22 host 172.18.156.21 (hitcnt=0) 0x64871058
access-list INSIDE line 224 extended permit icmp host 172.16.226.23 host 172.18.156.21 (hitcnt=0) 0x5b47f8f2
access-list INSIDE line 224 extended permit tcp host 172.16.226.21 host 172.18.156.21 range 8080 8180 (hitcnt=0) 0xc4f9ddb7
access-list INSIDE line 224 extended permit tcp host 172.16.226.22 host 172.18.156.21 range 8080 8180 (hitcnt=0) 0x1db1010a
access-list INSIDE line 224 extended permit tcp host 172.16.226.23 host 172.18.156.21 range 8080 8180 (hitcnt=0) 0x9b57b65e
access-list INSIDE line 224 extended permit tcp host 172.16.226.21 host 172.18.156.21 eq www (hitcnt=0) 0xc0e096cd
access-list INSIDE line 224 extended permit tcp host 172.16.226.22 host 172.18.156.21 eq www (hitcnt=0) 0x70d0a587
access-list INSIDE line 224 extended permit tcp host 172.16.226.23 host 172.18.156.21 eq www (hitcnt=0) 0x736ebc21
access-list INSIDE line 224 extended permit tcp host 172.16.226.21 host 172.18.156.21 eq https (hitcnt=0) 0x7dc5dd9f
access-list INSIDE line 224 extended permit tcp host 172.16.226.22 host 172.18.156.21 eq https (hitcnt=0) 0x5ef3afdd
access-list INSIDE line 224 extended permit tcp host 172.16.226.23 host 172.18.156.21 eq https (hitcnt=0) 0xfaeb2e21