L3 - switches - HSRP - asymmetric routing?

[Mariusz00001]

Hi,

I need an expert to provide some real-world feedback, please.

When I worked as a network engineer, in one company 99% of typologies looked like this:

There were of course more VLANs but I want to show one (let's call it VAN9) - 192.168.9.0/24

When I joined that company, they had hundreds of clients running this topology and all worked very well, it was many, many years ago (a lot of stretched vlans, QinQ, and so on).

My question is: this topology will cause asymmetric routing.

- 192.168.9.100 wants to browse the Internet (e.g. 1.1.1.1)

- The packet is sent to the default gateway which is HSRP (192.168.9.1).

- L3 switch (3560) sends it to the firewall 192.168.9.254

- Firewall sends it to the Internet

- Messages comes back: the firewall will bypass 192.168.9.1 and send it directly to the client 192.168.9.100

My questions

1.  is this really a big deal? It was designed by two CCIEs back in the day

2. I know that to fix it there should be a separate subnet/link for routing between a firewall and L3 switch. Is it better to use SVI or 'no switchport'? I know Cisco says that SVIs are slower but Jeremy from packetlife did an experiment and SVIs were faster.

Thanks!

[andrew.butterworth]

If you have ICMP redirects disabled on the L3 switch SVI interfaces, then what you describe is the behaviour.  If you have ICMP redirects enabled (default) on the L3 switches, then the L3 switch will send an ICMP redirect to the client to inform it there is a better gateway to send the packet to.  The client should consume this redirect and create a /32 host route entry for 1.1.1.1 to go via 192.168.9.254 (on a Windows client from a command prompt look at the local routing table 'route print' and there should be lots of local short life /32 entries in there).

However, the client may disregard this redirect if there is a host firewall running on it or some other local policy to ignore ICMP redirects, and continue to send the packet at L2 to the HSRP MAC address.

Its a bad design.  ICMP redirects are there to provide a workaround for this bad design.  You should have a transit subnet between the firewall and the L3 switches - typically a /29 to cater for the physical and virtual IPv4 addresses of the four devices.  The firewall and switches need to know the local routing topology - i.e. the firewall needs to know the next-hop to reach the client subnets.  For a small setup like this, you can usually get away with summary addresses for the routing on the firewall (10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16 etc) and a default on the L3 switches.  If its a bigger local setup with other L3 switches in the mix then an IGP would be best.

There is a document that someone wrote years ago with the title 'ICMP redirects are ba'ad, mkay?', an homage to Mr Mackey on Southpark. See if you can find it.  I had a quick search, but didn't manage to get a copy of the file.

[Joseph W. Doherty]

---

@andrew.butterworth wrote:

If you have ICMP redirects enabled (default) on the L3 switches, then the L3 switch will send an ICMP redirect to the client to inform it there is a better gateway to send the packet to.

---

BTW, many years ago (prior to IOS 12.x), HSRP would disable ICMP redirects.  This was due to the earlier HSRP redirecting you to another router's physical addressing, and if that router failed, you would lose connectivity to destinations now using the redirected to router, even if the original HSRP active router has another (backup) path to the destination.

---

@andrew.butterworth wrote:

Its a bad design.  ICMP redirects are there to provide a workaround for this bad design.

---

Andrew, could you elaborate why this design is "bad" and why ICMP redirects are a "workaround"?

I would be first to agree, there's no advantage to such a design, using L3 switches, beyond, perhaps, not needing another subnet for transit.  If the ICMP directs are a concern, they can be disabled, as they are unnecessary.  (Actually, with HSRP, I'm unsure it would redirect to the FW, as doing so could potentially recreate the same issue that earlier HSRP had with itself, you could be pinning the redirected traffic to a physical IP/MAC.)

Oh, if you do find your ICMP document, please provide a link to it.

[andrew.butterworth]

Think of it like this.  We're all in a room (L2 domain).  We can all see and hear each other.  We are all throwing stuff at each other, some of us want to throw stuff out of the door and get stuff thrown back at us from people outside the door (L3 routing).  Only one of us is at the door to be able to facilitate this in and out the door stuff (the L3 router for the subnet).

We are told to throw stuff that we want to get out of the door at one person who isn't at the door (bad design bit), but they know where the door is. They accept the 1st thing you send, but tell you where the door is (ICMP redirect), hoping you'll stop throwing stuff to them to throw out of the door on your behalf.

At LAN speeds it all becomes a bit irrelevant, as we won't perceive much of an impact.  It's just wrong though, mkay?

[Joseph W. Doherty]

Andrew, thank you for the reply.

"It's just wrong though, mkay?"

Sorry. I'm not convinced.  I suspect you want to avoid redirects, possibly too much so without due allowance for other considerations.

For example, consider we have some site with a couple of subnets and two off site connections that go to two different sites, but all other destinations are reachable by either path, although which path is better depends on actual destination.

If we have but one router, all is fine, but for redundancy we decide to add a second router.

We certainly can run HSRP, without redirects, so all external destination traffic hits just the active gateway, and it sends traffic to the second router that has a better path to a destination.  So, without redirects, we're adding needless loading (processing and port) to the active gateway router and needless latency to the traffic being sent to the other router.  Interestingly, as OP asked about asymmetric routing, some traffic sent out from the active router may return on the standby router because what was ECMP on the single router is different with two routers.

Redirects, preclude all but first packet being subject to the needless gateway funnel.

Of, course, there are alternatives, like having another router in front of the WAN routers, but it can then become a single point of failure.

If all the hosts ran the same IGP, as the routers, they could optimally direct their traffic.

Now, perhaps unclear, the above was about the best you could do, decades ago, using routers and L2 switches.  L3 switches was a game changer.

With L3 switches, especially stackable L3 switches, a design like OP is poor in comparison to what could be done instead.  (Perhaps, so poor, some may label it as a "bad" design, but to me, "bad" implies something that should never be done, which I don't believe is the case for just avoiding hosts getting redirects.)

ICMP redirects have a purpose, which is why ICMP provides them.  However, they may not be the best choice in all situations.

[andrew.butterworth]

Then use L3 switches to terminate your user VLANs and on the L3 transit network between the L3 switches and the WAN routers run an IGP.

[Joseph W. Doherty]

"The use L3 switches to terminate your user VLANs and on the L3 transit network between the L3 switches and the WAN routers run an IGP."

Use to commonly do that.  Further, for branch sites, the L3 switch in question was usually a stack so no individual stack member was a single point of failure for off site traffic.

Andrew, sorry, I still don't see why OP design is (fundamentally) "bad".  That design has its own pros and cons, as do other possible designs.

Again, you seemed exceedingly concerned with ICMP redirects, but they can be disabled, and very possibly, as this design is from many, many years ago, HSRP (then) may have disabled redirects.

In fact, if this design was using LAN hubs and/or L2 switches, and small routers, like 2500s, it, then, may have been a great design.

Today, with L3 switches, newer designs are likely better, but, also again, that design may still have value in some corner cases.  That's why rather than just describe it as "bad" perhaps it might describe it as "obsolete".

Basically, any design should be considered for why it's best, or not, for all network goals.

[Joseph W. Doherty]

"Is this really a big deal?"

Only if there's a need for the return traffic to transit the L3 switch.  If there is such a need, then you have the issue of transiting the correct L3 switch.

This design is also from many, many years ago, probably when you had actual routers rather than L3 switches, i.e. in the days of "switch when you can, route when you must".  The goal may have been, avoid router transit to improve performance.

For your 2nd set of questions, could you provide references.

If one finds noticable performance differences between SVIs and routed ports (for same traffic load), I suspect it's platform and/or IOS dependent.  I would be very cautious in taking some particular benchmark as a generization.

BTW, many take asymmetrical routing as an evil.  It's not.  What's evil is sometimes we need to avoid it, due to some transit stateful need.  By design, IP doesn't care about path, just packets can be sent from source to destination, so as to provide transit redundancy.

[Mariusz00001]

"Only if there's a need for the return traffic to transit the L3 switch"

I really appreciate that comment. I think that is the reason I have never seen any real issues with that design. These were L3 switches so any other SVI/VLAN would not be impacted because they would be in a different subnet, so that issue applies to traffic "local" to L3 switches and no other routing is performed.

SVI vs routed - that was Jeremy from packetlife. He did stress it was just his small test and he said nobody should take that a final benchmark.

So my question is still valid:

L3 switch ----- firewalll

Routed port or SVI? According to "the book", that should be a routed port because this is P2P link

[Joseph W. Doherty]

---

@Mariusz00001 wrote:

So my question is still valid:

L3 switch ----- firewalll

Routed port or SVI? According to "the book", that should be a routed port because this is P2P link

---

Well, as one Cisco SE told me years ago, a L3 switch's routed port was actually an internal "hidden" VLAN port.  I cannot verify if that's true, but it make sense.  A port is a port.  How it's allowed to be used, is based on logic.  For example, what's the difference between an access port, an access port with a voice VLAN and a trunk port?  Add to that mix, a "routed port".

Yes, the book recommends to use a routed port for a p2p link, as it's a bit more efficient, as that port can suppress some low level packets, like those for DTP.  It's also what expected, for a p2p link, it also more closely mimics a true router port.

Interestingly, I been working another posting where the OP was asking how to select between SVI and routed ports, as the former seems well able to handle any needs of the latter.

[andrew.butterworth]

I would never design a network where as a fundamental part of the design, there is a requirement for ICMP redirects. Client networks shouldn't have more than a default gateway - yes this can be a VIP and yes, the active VIP might not be the correct forwarder for some external routes and if that is the case, then ensure there is a transit between the two or more physical routers/L3 switches and run an IGP.

As I said, at LAN speeds and with modern hardware the user won't perceive any difference, but under the hood its operating differently.

Its honestly up to you, if you think ICMP redirects are a good thing.  Personally, I'd avoid them at all costs.

[Joseph W. Doherty]

". . . if you think ICMP redirects are a good thing."

I never said that, I just don't think they are a "bad" thing.  To me, they're just another feature of IP, which I would consider using if I thought they would be worthwhile to do so.

What I was hoping, as you clearly believe they are "bad", was why you consider them so "bad".

"I would never design a network where as a fundamental part of the design, there is a requirement for ICMP redirects."

(Oops, hit post button by accident. . .)

Well, it's fortunate you weren't designing networks last century.  ; )

Very unlikely you would need to intentionally use them now a days, using later equipment, but in ye olden times, when a small router might be unable to run a full T-1 at wire speed, if you even had a full T-1 circuit, getting the most out of such is where using ICMP redirects might have been critical.

I noticed OP not only noted this design was from many, many years ago, but was developed by a couple of CCIEs.  For some reason, those CCIEs, then, didn't see using ICMP redirects as so bad as they shouldn't ever be used in a design.

BTW, the only reason I make a fuss on this point, because when I or others were designing 3 or 4 decades ago, it was a very different environment.

What I consider "bad" is not considering all your design options and/or not using good judgment in how to use them to best accomplish your goals.

[andrew.butterworth]

Hopefully I'm not breaking any rules by attaching this document written by Stephen Gill back in 2002.

[Joseph W. Doherty]

Thank you!  An interesting read.

Not discounting the accuracy of what's described, at least at time it was published, but some of the "bad" arguments are "straw men".

For example, a FW sending a redirect to a host, but then subsequent traffic bypasses the FW which needs to see it because it maintains state.  Yea, well duh, then why is the FW sending redirects?!  (NB: OP is redirecting to the FW, not the converse.)

Or another example, host may add a lot of host route entries, due to redirects.  Yea, and how many concurrent destinations does an user edge host typically have?  Now, if you want to talk about server hosts, well, that's more likely to be an issue, but for server hosts, they often are treated just a bit differently from user hosts.

Mention was also made ICMP redirects could be used for various attacks against the host.  True.  But you're exposed to that unless host refuses to process them at all.  And if the latter is the case, well then ICMP redirects is off the table (for additional security).  (Of course, other useful features, which can be used as attach vectors, have also been disabled.  Personally, I found packet source routing useful in some debugging scenarios, but generally it too is no longer allowed.)

That paper has:

Based on the information presented so far it would seem that ICMP
redirects serve quite a useful purpose. In fact, they do! Yet there are
several reasons to steer clear from redirects, all of them relating to
network performance, consistency, reliability, and security.

"steer clear" isn't quite the same as "never" use.

Anyway, to put it another way, yes there are potential issues intentionally using ICMP directs, and not to make light of them, but with any design, you need to consider the whole thing.

That paper is worth reading, but IMO, it's placing too much emphasis on potential pitfalls, overlooking "cost" to avoid these potential pitfalls.

However, again, I'm not recommending using ICMP directs, and further, think it's unlikely, especially now a days, that using them would be a "best" (and "best" is very much subjective) design, but I like to keep my options open.

Andrew, also again, thanks for providing that attachment, I very much appreciate the information it provided.

@Mariusz00001 Although I'm "uncomfortable" with Andrew's use of words like "bad" and "never", his concern over using ICMP redirects is well founded.  Again, if this design is many, many years old, it was a different environment then too.  At that time, it may have been a great design, but today, little reason to continue to use it, and often good reasons to stop using it.

The paper that Andrew provided, although, IMO, overstates the potential issues, isn't without merit.

So, again, for your first question, isn't a "big deal", I don't think so, but conversely, is it a "good" or "best" deal (today)?  Most likely not.  Such a design should be reassessed.

As to your second question, SVI vs. routed port, again, for p2p I would lean toward a routed port for the reasons noted in my earlier replies, but I don't believe it's a deal breaker whichever you use.