ip redirects - which direction?

[Kevin Dorrell]

I am troubleshooting an issue with CPU utilization on a 3750X stack. The show controllers cpu-interface tells me that the icmp queue counter is growing quite fast ... about 5000 per second. I read that this is a queue for ICMP redirect messages.

Now, I know what ICMP redirects are about, and how they are supposed to work. What I need to know is what would be the effect of the no ip redirects command on the SVI of the switch? Which direction of traffic does it apply to? Would it be:

1. If I receive a packet, and I know a better router that could handle it, but I will not send a redirect to tell the host so,
   
2. If I forward a packet, and receive a redirect, then I will not take the redirection into account,
   
3. Both of the above,
   
4. None of the above.
   

Thanks in advance.

Kevin Dorrell

Luxembourg

[Jon Marshall]

Hi Kevin

Long time no speak. Hope you are well.

I'm always reluctant to answer your questions as you know a lot more than me so i feel like i am walking into a trap

As far as i know ICMP redirects are only for end devices ie. it would be 1) in the examples you listed above. I don't believe routers use ICMP redirects themselves because they only use their IP routing table to forward traffic. So if a router received an ICMP redirect from another router or L3 device (it can't be a host as they do not send redirects) it should be silently ignored and definitely not used to update the routing table.

So for your case it would only be if there were multiple L3 gateways on the same vlan and the hosts were configured to use the wrong gateway ie. the L3 device  it arrived on had a route to the destination back out of the same interface pointing to another L3 gateway on the same IP subnet.

Jon

[Richard Burts]

Kevin

I agree with Jon that it is good to see you back on the forum. I also agree with him that the answer is 1.

HTH

Rick

[Kevin Dorrell]

Hi Rick.  Good to hear from you.

Yes, I suppose (1) does make sense, because if a router took any account of redirect messages, then it would have to do it by installing host routes.  The old "code 0" for "network redirect" was obsolete even when Doug Comer wrote his excellent book.  (My copy is 1991.)

So it looks like there is a host that has the wrong gateway configured.  I wonder which one.

Thanks.

Kevin.

[Kevin Dorrell]

Hi Jon.

Yes, it is true I have not been so active on the forum recently.  But no ... no trap ... it is a genuine case.

I am trying to pluck up the courage to do a debug ip icmp.  It is a pretty important router, and 5000 packets per second is a lot of debug, but at least it will give me a handle on where they are coming from and why.  I guess I need two console windows, one with a no debug all ready prepared but no term mon, and the other to do the term mon and debug commands.

There are indeed multiple (two) gateways on the VLAN, and two HSRP groups, one with preference on each router.  But they are both directly connected to all the other VLANs, so there should be no question of a redirect.  In fact, I seem to remember that at one time when you configured HSRP, it would do no ip redirects automatically.  But it seems these switches don't do that.

It's strange how the most simple-looking protocols sometimes raise the most unexpectedly subtle questions.  I have always said, for example, that ARP has one of the most misunderstood behaviours.

Anyway, thanks for the reply, and I'll post something when I have found out what is happening.

Kevin

[Richard Burts]

Kevin

Here are some suggestions which might get you what you want with less impact than debug ip icmp.

1) is it possible that you could use an access list on the interface which would (permit or deny as you choose) icmp redirect and use the log option to get information about what generated the redirect?

2) you might try debug with an access list (ex debug ip packet 199 where access list 199 does a permit for icmp redirect).

3) you might try configuring netflow ingress on the interface and see whether the redirects show up in the netflow data.

You remember correctly that original behavior with HSRP was to automatically disable redirects. But for some reasons they changed that behavior.

HTH

Rick

[Kevin Dorrell]

Thanks Rick.  Unfortunately I think that the ACL approach is out of the question because I am not even sure which VLAN this is happening on, and I have more than 50 to choose from. However, I do have HSRP on every one of those VLANs, so I guess I can afford to risk the lesser-used of the two stacks.

Thanks

Kevin

[Joseph W. Doherty]

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

You remember correctly that original behavior with HSRP was to automatically disable redirects. But for some reasons they changed that behavior.

Yes, they did that because HSRP, in the later IOSs, is "smarter".

The reason HSRP often had the automatic disablement of redirects because most of the smaller routers didn't support mHSRP and you didn't want the redirect to a real interface IP on another router.

I believe the newer HSRP code "knows" if the redirect can use a HSRP IP and if so, it will issue the redirect.  If the redirect cannot use another HSRP address, I believe the redirect is suppressed.

[daniel.dib]

I would suggest saving the logs to buffer instead of outputting it on TTY. I imagine that it would be more effective CPU wise but I'm not sure.

I wish there was a command like debug max-cpu 40 to set it to use maximum 40% of the CPU and if it exceeds kill all the debugs. Maybe it's difficult to implement it but it sure would be helpful.

Might be worth to put a reload in statement just in case you lose control over the debug.

Richard had some good suggestions but you might have to enable Netflow on a lot of interfaces. You could enable it on a few at a time though until you find the culprit.

Daniel Dib
CCIE #37149

Please rate helpful posts.

[Richard Burts]

Daniel

You bring up an interesting point which is not discussed very much in the forums and so I would like to re-enforce it. You are absolutely correct that the most high impact thing about running debug is attempting to send output to the console. And the most important thing that you could do to minimize the impact of debug is to make sure that the console will not receive debug output. (some people would advocate no logging console. but I believe that is a bit extreme. it is sufficient to set the logging level for the console to informational.)

You are correct that debug logs to logging buffered is the lowest impact way to run debug, followed by term mon and logging to your remote access session.

And scheduling a reload is always a good suggestion when you are about to do something that might have impacts like deubg or significant config changes.

HTH

Rick

[Kevin Dorrell]

Back to the original question.  I did promise I would report back when I found out what was going on.  I have tracked down my high CPU utilization.  I had a VLAN with two subnet ranges on it, one global and one private as a secondary, and a gateway presence in each range.  I did this because the global address subnet was full and I needed to expand the VLAN.  This worked OK, and when the hosts needed to talk between the different subnets on the VLAN, they bounced off the SVI.

Except that .... for each packet that got bounced off the SVI, there was a CPU interrupt.  It seems that the hardware did not like the fact that the egress interface was the same as the ingress, and so punted up to the CPU to see if it could generate an ip redirect.  It did this even if the source and destination were on different subnets.  What is even more interesting is that the punt happens even if you have no ip redirects.  If I do a debug of the cpu-queue icmp, I see all these packets but with something like "ip redirect disabled by interface".

Does anyone know a geek-knob to stop this interrupt happening?

I learned somethong today.

Kevin Dorrell

Luxembourg

P.S.  I am astounded that the forum has censored the colloqualism I just used!  I hear that expression enough during Cisco-Live presentations.  I never even occurred to me that it might be misinterpreted.

[Joseph W. Doherty]

Disclaimer

The  Author of this posting offers the information contained within this  posting without consideration and with the reader's understanding that  there's no implied or expressed suitability or fitness for any purpose.  Information provided is for informational purposes only and should not  be construed as rendering professional advice of any kind. Usage of this  posting's information is solely at reader's own risk.

Liability Disclaimer

In  no event shall Author be liable for any damages whatsoever (including,  without limitation, damages for loss of use, data or profit) arising out  of the use or inability to use the posting's information even if Author  has been advised of the possibility of such damage.

Posting

Does the 3750 accept the interface command, ip fast-cache same-interface?  If so, have you tried it?

[daniel.dib]

There is a session at Cisco Live on troubleshooting 3750, however it says:

Receives a copy of the traffic for which an ICMP packet needs to be generated. Hardware forwarding of the packet still occurs

However I guess that like you said the CPU is still interrupted and at that packet rate it becomes an issue for you.

I assume that show processes cpu shows ICMP using a lot of resources as well?

Unfortunately I don't know how to solve it. Platforms like 7600 used to have ip route-cache same-interface for these kind of situations but I can't find it in the 3750 command reference.

Daniel Dib
CCIE #37149

Please rate helpful posts.

[Kevin Dorrell]

Thanks Daniel.  I did find ip route-ache same-interface on my 3750X, but it did not seem to make any difference to the CPU.  As you say, it may be forwarding the packet in hardware, but it is still interrupting the CPU to see if it needs to generate a redirect.  I think I shall have to make two VLANs, one per subnet.

Kevin Dorrell

Luxembourg

[Jon Marshall]

Kevin

P.S.  I am astounded that the forum has censored the colloqualism I just used!  I hear that expression enough during Cisco-Live presentations.  I never even occurred to me that it might be misinterpreted

Yes, i've been caught out a few times with this as well. Whoever is responsible for which words you can and cannot use is a very sensitive soul

If the command Jossph and Daniel provided doesn't work then other than creating a new SVI for the secondary range i'm not sure there is a lot you can do about it. I did find an interesting thread on another fourm about this where they were describing exactly the same symptomps and nothing they configured stopped it happening.

Jon