Solved: 2921 router and 3750g L3 config help - Page 2

jasonafernandez · ‎11-17-2016

I have a Cisco 2921 router setup at home on my cable internet. It's configured with NAT outside and inside. I have a uplink connected to a Cisco 3750g switch. The switch is configured as layer 3, ip routing enabled, dhcp for vlans, using external dns. I have 3 vlans enabled.

Configurations can be found here:
2921 router http://pastebin.com/CRsDBufc
3750g PoE switch http://pastebin.com/h4mG17xg

My problem is from Windows or Linux PC's behind the switch cannot traceroute. All returns are asterisks. I can ping out fine by hostname and IP address and receive replies.

Can anyone help me understand what is happening? Do I need ACL's? Does my switch port to router need "no switchport" instead of VLAN? Are my ip route's incorrect? I've tried a bunch of things but no avail.

I really appreciate everyones help and teaching me.

jasonafernandez · ‎12-03-2016

Hi Rez,

I think I have a bug in router IOS. While Windows ICMP traceroute does not work, running a nmap traceroute does work. But when I tell nmap to run a UDP based traceroute, my 2921 router crashes and reboots. Totally reproducible every time. Maybe I should try other router IOS versions?

First result success using ICMP.

PS C:\Utility\nmap-7.31> .\nmap.exe -sn --traceroute google.com

Starting Nmap 7.31 ( https://nmap.org ) at 2016-12-03 12:02 Pacific Standard Time
Nmap scan report for google.com (172.217.5.78)
Host is up (0.012s latency).
rDNS record for 172.217.5.78: lax17s15-in-f14.1e100.net

TRACEROUTE (using proto 1/icmp)
HOP RTT ADDRESS
1 1.00 ms 192.168.101.1
2 1.00 ms 192.168.100.1
3 14.00 ms 142.254.236.173
4 15.00 ms tge0-10-0-1.vnnzca2401h.socal.rr.com (76.167.27.133)
5 15.00 ms agg11.vnnycajz02r.socal.rr.com (72.129.14.98)
6 16.00 ms agg29.tustcaft01r.socal.rr.com (72.129.13.2)
7 23.00 ms bu-ether16.tustca4200w-bcr00.tbone.rr.com (66.109.6.64)
8 17.00 ms 0.ae3.pr1.lax10.tbone.rr.com (107.14.19.56)
9 17.00 ms 216.156.65.225.ptr.us.xo.net (216.156.65.225)
10 11.00 ms 207.88.14.212.ptr.us.xo.net (207.88.14.212)
11 10.00 ms 207.88.13.25.ptr.us.xo.net (207.88.13.25)
12 20.00 ms 216.0.6.50
13 15.00 ms 209.85.245.245
14 11.00 ms 108.170.237.115
15 16.00 ms lax17s15-in-f14.1e100.net (172.217.5.78)

Nmap done: 1 IP address (1 host up) scanned in 2.36 seconds

Second attempt trying to use UDP, router crashes instantly and I have the following.

PS C:\Utility\nmap-7.31> .\nmap.exe -sU --traceroute google.com

Starting Nmap 7.31 ( https://nmap.org ) at 2016-12-03 12:02 Pacific Standard Time
Nmap scan report for google.com (172.217.5.78)
Host is up (0.0096s latency).
rDNS record for 172.217.5.78: lax17s15-in-f14.1e100.net
Not shown: 999 open|filtered ports
PORT STATE SERVICE
28122/udp filtered unknown

TRACEROUTE (using port 28122/udp)
HOP RTT ADDRESS
1 1.00 ms 192.168.101.1
2 0.00 ms 192.168.101.1
3 ... 19
20 1.00 ms 192.168.101.1
21 ... 30

Nmap done: 1 IP address (1 host up) scanned in 43.32 seconds
PS C:\Utility\nmap-7.31>

I can share a crashdump, but it doesn't tell me much other than software error and a system restart.

2921-RTR#sh ver
Cisco IOS Software, C2900 Software (C2900-UNIVERSALK9-M), Version 15.6(3)M0a, RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2016 by Cisco Systems, Inc.
Compiled Sun 25-Sep-16 10:18 by prod_rel_team

ROM: System Bootstrap, Version 15.0(1r)M16, RELEASE SOFTWARE (fc1)

2921-RTR uptime is 9 minutes
System returned to ROM by bus error at PC 0x0, address 0x0 at 12:02:40 PST Sat Dec 3 2016
System image file is "flash:/c2900-universalk9-mz.SPA.156-3.M0a.bin"
Last reload type: Normal Reload
Last reload reason: bus error at PC 0x0, address 0x0

This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to
export@cisco.com.

Cisco CISCO2921/K9 (revision 1.0) with 446464K/77824K bytes of memory.
Processor board ID xxxxxxxxxxxxx
3 Gigabit Ethernet interfaces
1 terminal line
DRAM configuration is 64 bits wide with parity enabled.
255K bytes of non-volatile configuration memory.
250880K bytes of ATA System CompactFlash 0 (Read/Write)

License Info:

License UDI:

-------------------------------------------------
Device# PID SN
-------------------------------------------------
*1 CISCO2921/K9 xxxxxxxxxxxxxx

Suite License Information for Module:'c2900'

--------------------------------------------------------------------------------
Suite Suite Current Type Suite Next reboot
--------------------------------------------------------------------------------
FoundationSuiteK9 None None None
securityk9
datak9

AdvUCSuiteK9 None None None
uck9
cme-srst
cube

Technology Package License Information for Module:'c2900'

------------------------------------------------------------------------
Technology Technology-package Technology-package
Current Type Next reboot
------------------------------------------------------------------------
ipbase ipbasek9 Permanent ipbasek9
security None None None
uc None None None
data datak9 RightToUse datak9

Configuration register is 0x2102

2921-RTR#

jasonafernandez · ‎12-05-2016

I'm going to try and simplify things. Change config around to router on a stick, SVI's, DHCP from router, L2 switching. See how that works.

Reza Sharifi · ‎12-05-2016

The router crashing most likely is caused by an IOS issue. Before you make any changes, try loading different version of IOS on the router and test again.

HTH

jasonafernandez · ‎12-05-2016

Hi Reza,

Yesterday I did just that, dropped down to 3 prior versions. All resulted in same symptoms.

My ramblings...

I cannot explain why it works sometimes via WiFi but never on LAN. My CCNA experience isn't there just yet. I probably need ACL's and to configure loopback interfaces to get this working.

This can also be a NAT/L3 issue, FAD by Cisco. Last hop always resolves. I can only hypothesize if I had static public addresses this would be working?

Wireshark and span port to further investigate?

Why does nmap traceroute work? Using random port, icmp or tcp able to resolve all hops, LAN and Wifi, always works.

The IOS crash defiantly a bug. Reading release notes there are a lot of IOS crash bugs in v15 I'm running, not sure where this one falls under. Unfortunately my lab equipment is eBay and I don't carry any SmartNet.

I appreciate all your efforts so far, will keep you posted. Good practice in any outcome.

jasonafernandez · ‎12-08-2016

Hi Reza,

To update, after some further testing, a clean install of Windows 10 resolved the issue. Cisco config was sound. I am currently running router on a stick config, but will return back to L3 switch config.

As for the nmap UDP traceroute bug which causes router to crash, that is still a problem. I have reported to Cisco PSIRT for further review.

Appreciate all your help.

Carlos Villagran · ‎11-17-2016

Hi Jason,

Is the setup like this?

PC----3750----2921---INTERNET

In which VLAN or port is the PC connected and what is the IP address of the PC?

Regards,

JC

jasonafernandez · ‎11-17-2016

Hi JC,

Yes you're correct with the setup. PC is in switch port 1, vlan 10. PC has a dhcp address of 192.168.101.102.

IPv4 Address. . . . . . . . . . . : 192.168.101.102(Preferred)
Subnet Mask . . . . . . . . . . . : 255.255.255.0
Default Gateway . . . . . . . . . : 192.168.101.1

Gateway is vlan interface on 3750.

Carlos Villagran · ‎11-17-2016

Jason,

I see that configuration is fine, you are able to ping anything correct, right?

What are tracerouting? Can you try to trace route ip address: 192.168.100.1 ?

You should stop seeing the asterisks.

Regards,

jasonafernandez · ‎11-17-2016

Internally I can traceroute fine. It's only to internet destinations I cannot. If I debug the icmp, it reports TTL expired or to short.

PS C:\Users\jason> tracert 192.168.100.1

Tracing route to 192.168.100.1 over a maximum of 30 hops

1 * * * Request timed out.
2 <1 ms <1 ms <1 ms 192.168.100.1

Trace complete.

Carlos Villagran · ‎11-17-2016

Jason,

Can you share the output of what you get when you traceroute an internet address?

Please be aware that it may be expected to not see anything in the internet trace since many providers or routers do not answer to traceroute packets for security reasons.

Regards!

jasonafernandez · ‎11-17-2016

In the past, different router, layer 2 switch, I can trace this all the way out. I am going to try Reza's recommendation, just haven't had a chance to yet.

PS C:\Users\jason> tracert google.com

Tracing route to google.com [74.125.25.138]
over a maximum of 30 hops:

1 * * * Request timed out.
2 * * * Request timed out.
3 * * * Request timed out.
4 * * * Request timed out.
5 * * * Request timed out.
6 * * * Request timed out.
7 * * * Request timed out.
8 * * * Request timed out.
9 * * * Request timed out.
10 * * * Request timed out.
11 * * * Request timed out.
12 * * * Request timed out.
13 * * * Request timed out.
14 * * * Request timed out.
15 * * * Request timed out.
16 * * * Request timed out.
17 * * * Request timed out.
18 39 ms 40 ms 40 ms pa-in-f138.1e100.net [74.125.25.138]

Trace complete.