Switch that SolarWinds Cannot Reach

[Patrick McHenry]

Hi,

I have a switch that solarwinds is reporting down and from the switch I also cannot ping SolarWinds from the managment address of the switch.

The switch is not currently routing but when I do an sh ip route:

sh ip route
Default gateway is not set

Host               Gateway           Last Use    Total Uses  Interface
172.20.8.211       10.3.1.254            0:00       6096264  Vlan103

I get this entry. It's not configured. The 172.20.8.211 is Solarwinds and the 10.3.1.254 is the default route so this is why the switch is not pinging out of the vlan but how do I get rid of this entry?

Thanks, Pat

[Giuseppe Larosa]

Hello Patrick,

the switch is telling that default gateway is not set

if IP routing is disabled you need to configured the ip default-gateway using

conf t

ip default-gateway  x.x.x.x

where x.x.x.x is the address of a router device in the same IP subnet as the management IP address of the switch, this is just like when we set the IP default gateway on a PC

Edit:

also check that from the IP subnet of the solarwinds server there is connectivity to the IP subnet where the switch management address is, because the issue may be on the return path.

Hope to help

Giuseppe

[Patrick McHenry]

I was thinking the same and added the default route but still I'm not able to ping Solarwinds. How can I get rid of the host route that I see when I do a sh ip route.

FYI - I can ping Solarwinds from other switches on that management VLAN - so I don't think it is the Solarwinds side.

Thnaks, Pat.

[Giuseppe Larosa]

Hello Patrick,

when IP routing is disabled an ip default route is not effective!

you need to set a default-gateway as I have explained in previous post.

Conversely if you enable ip routing you can use a default route

so the following options are available:

ip routing

ip route 0.0.0.0 0.0.0.0 10.3.1.254

OR

no ip routing

ip default-gateway 10.3.1.254

If Solariswind can ping other hosts in same IP subnet the return path is fine.

When IP routing is disabled the IP routing table output is totally different.

Hope to help

Giuseppe

[Patrick McHenry]

I added the default-gateway but it did not help. So maybe routing is enabled and I should disable it?

Thanks, Pat.

[Patrick McHenry]

I just pinged google and now it shows in the ip route show command:

173.194.43.37      10.3.1.254            0:02            39  Vlan103

172.20.8.211       10.3.1.254            0:00       6103860  Vlan103

Is there something on the switch that is caching these routes?

Thanks, Pat

[Giuseppe Larosa]

Hello Patrick,

yes see it as a sort of ARP cache more then an IP routing table actually it should use the default gateway in all cases and it provides a packet counter for each host destination.

Edit:

given the different output in show ip route my guess is that IP routing is disabled. At this point verify the ARP entry, check if the MAC address of the default gateway is learned correctly  ( I don't see anything else that can be wrong)

Hope to help

Giuseppe

[John Blakley]

Patrick,

Do a "show ip redirects" and see if there are entries in there. If there are, you can clear them by typing "clear ip redirect". Then disable redirects on the router's interface that leads toward this switch.

HTH,

John

[Patrick McHenry]

I saw the ip redirects and cleared them. Then I could ping Solarwinds. Thanks. What was making these ip redirects get cached?

Thank you, Pat.

[John Blakley · Accepted solution]

I had the same issue at one point with Solarwinds and some devices. It's not an SW issue, but redirects are enabled by default on all Cisco switches/routers that I'm aware of. What I'd recommend is to disable icmp redirects on all of your devices. Redirects happen when a router thinks it knows of a better path for you and then it hands the path to you. It's stored locally in cache and the device goes directly to the path instead of through the router. This would cause a problem until the cache is cleared out. It's almost like a host file on a computer.

http://www.cisco.com/en/US/tech/tk365/technologies_tech_note09186a0080094702.shtml

HTH,

John

[Patrick McHenry]

Is this something I should disable on the core router VLAN Management interface? I see that it is not disabled:

interface Vlan103

description AP-Datacenter vlan

ip address 10.3.1.1 255.255.255.0

end

I disabled ip redirects on the access layer switches but, I'm a little scared to disable it on the Core as it is in production. Could this be helping some devices find default route to the Internet. I would think not as there is a default route on the Core.If ip redirects are not disabled on the Core switch could it be sending redirects even though it has a default route?

Hope that question isn't too confusing.

Thanks, Pat

[John Blakley]

Personally, I disable it everywhere.

"If ip redirects are not disabled on the Core switch could it be sending redirects even though it has a default route?"

I don't think so. If the default next hop isn't responding for some reason, but you have another default route or floating route that enters into the routing table, it can redirect its traffic there and then pass the next hop to the device. The device, workstations also, can enter this into their cache/routing table as being the best path to get to the destination. Because the table is static, when the true next hop for the switch comes back up, devices that didn't have their cache updated is still routed to the switch for that destination, but the device that did have the cache updated goes directly to the other hop bypassing the default route.

I have cellular devices at our branches that are used as a failover. One example that I can give is that our routers have a floating static route for default traffic to go to the cell device. Solarwinds uses icmp for some devices at branches. I have Cisco APs that had caching enabled. One day our circuit went down and Solarwinds polled the AP. The route was different than normal because the incoming traffic is directed to the cell card which then routes to the router, through the switch, to the AP. The AP updated itself to say the best way to get to my Solarwinds server was to go out the cell device. When the circuit came up, I noticed that when the tunnel that the cell card built would go down, so would the access point. I was able to ping the AP from my desk, but not from the solarwinds server. I figured out that the AP had updated its cache with the cell device being the next hop so it never traversed the router until I cleared out cache.

HTH,

John

[Patrick McHenry]

It sounds like to me from what you are saying is that if the route to SolarWinds had gone down at sometime through maintennance or gear failure, the core switch could have given the access layer switch the default route to get to solarWinds.. And what you are saying is that this route wouldn't timeout like an arp entry? Does this sound right?

Thnaks, Pat.

[John Blakley]

Yes, if the route that Solarwinds takes had changed, the core switch handed over a better route to the access switch in response to when Solarwinds was polling. From what I saw, it didn't time out at all.

[Patrick McHenry]

OK Thanks - so, your recommendation would be to turn off ip redirects on the management vlan interface. There are some servers that need access to the Internet on that subnet but they should be fine since there is a default route to the ISP.

Thanks, again. Pat