cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
795
Views
6
Helpful
9
Replies

How to discover if L2 device is between L3 devices?

Hello.

INTENT: I need to troubleshoot ASA interface symptom...  "5543139 input errors, 0 CRC, 0 frame, 5543139 overrun, 0 ignored, 0 abort"

My strategy is to investigate the adjacent device's interface settings. It is currently unknown if there exists a l2 switch in-between the next hop L3 device.

My understanding is that "sh arp" and also "sh mac-address" gives me data about the next L3 device.

QUESTION:

How do I discover if an L2 device exists between L3 devices?

Thank you.

2 Accepted Solutions

Accepted Solutions

As both @Flavio Miranda and @Ramblin Tech basically implied, it can be next to impossible without someone doing a visual inspection (no one at said datacenter capable of doing this and/or that you trust?), especially, as Jim mentions, if the interloping device truly wants to conceal itself.

Laugh - in one part of the network I was maintaining, I had a fiber link, where one side had SX optics and the other side LX optics!  Huh???

Turned out there was an optical switch in between.  Unknown to me, at the time, our company also maintained, in some portions of it, its own optical network.  (Don't know how much you know about optical networks, but they deal with passing "light" about.)  Point is, except for this clue, i.e. different optical transceivers on either end of "link", although there was an optical network between my sites, it was "invisible".

Oh, love Jim's suggestion of using TDR (for copper).

Regarding pulling a fiber connection, and see what happens on other side, reminds me, initially we found our fiber "p2p links" running across an optical network, didn't physically drop our interface when there was a drop in the transit optical network.  I.e. we had to wait on timers, on devices, to detect lost loss of connectivity.  Then, we found out, if we reminded the optical team, they could configure their equipment to "drop" our end connections if they did have an end-to-end path break (they might also have redundancy in paths, so logically, you didn't lose end-to-end connectivity, although latency might change).  So, as Jim described, an optical loss of signal might not be fool proof either.

BTW, cannot say about ASAs, but overrun ingress errors, on Cisco routers, usually are due to the device cannot process the ingress at the rate it's being delivered.

This might be caused by two reasons.  Physically, the device cannot physically keep up with the ingress rate by moving received frames out of the hardware receive buffer fast enough.  This is unusual.  Or, physically, the device cannot physically "process" received frames at the ingress rate, so it doesn't bother to clear them from the ingress queue/buffers.  This is a subtle difference.

In the former, the platform is just incapable of physically dealing with the ingress rate

In the latter, much depends on what the ingress traffic contains, and the "work" needed to processed the received traffic.

Simple example, if an ingress interface has an ACL with 10,000 ACEs.  In one case, all traffic gets a match on the 1st ACE, in another case, all traffic hits the final ACE.  On many software based routers, same traffic, same ACL, but which ACE gets matched, may significantly impact throughput capacity.

So, is this problem recent?  Any change to FW rule set?  Any change to what rules traffic mix is hitting on now vs. "before"?

View solution in original post

Leo Laohoo
Hall of Fame
Hall of Fame

@jmaxwellUSAF wrote:
How do I discover if an L2 device exists between L3 devices?

How many MAC addresses appear on the interfaces?

View solution in original post

9 Replies 9

Hi @jmaxwellUSAF 

  For Firewall CDP is not available, which would be the best way for find directly connected devices.  On this case, the safe way would be check locally.  You can also check topology if you have one.

Ramblin Tech
Spotlight
Spotlight

If there is a benign device, it might have L2 control protocols enabled like CDP, LLDP, Link OAM (802.3Ah), or CFM (802.1ag) enabled. You could try turning on those protocols on your L3 devices (if supported on them) to see if they establish a link-peering relationship directly with each other or with some other device in the middle (ie, the benign L2 switch).

If your concern is with a malevolent device, you must assume that it will pass through L2CP transparently, and cannot be tricked into giving away its position. In that case, visually tracing the cable will expose an interloper. If that is not practical, perhaps a time-domain reflectometer could be used to ensure cables are the expected length (ie, no interloper device where you cannot see it).

If this is an optical connection, the simplest method might be to just pull the transmit fiber from one end and see if the receiver at the far-end experiences LoS. Again, if you are concerned with a malevolent device, you cannot assume it will fall for this trick as it may be able to propagate LoS from one interface to another.

Disclaimer: I am long in CSCO

Thank you for your reply.

ASA devices, for security reasons, do not support CDP or LLDP. Though I did not mention that an ASA was involved here in my original post.

The intent is to discover the L2 device without traveling to the datacenter cage for direct inspection.

As both @Flavio Miranda and @Ramblin Tech basically implied, it can be next to impossible without someone doing a visual inspection (no one at said datacenter capable of doing this and/or that you trust?), especially, as Jim mentions, if the interloping device truly wants to conceal itself.

Laugh - in one part of the network I was maintaining, I had a fiber link, where one side had SX optics and the other side LX optics!  Huh???

Turned out there was an optical switch in between.  Unknown to me, at the time, our company also maintained, in some portions of it, its own optical network.  (Don't know how much you know about optical networks, but they deal with passing "light" about.)  Point is, except for this clue, i.e. different optical transceivers on either end of "link", although there was an optical network between my sites, it was "invisible".

Oh, love Jim's suggestion of using TDR (for copper).

Regarding pulling a fiber connection, and see what happens on other side, reminds me, initially we found our fiber "p2p links" running across an optical network, didn't physically drop our interface when there was a drop in the transit optical network.  I.e. we had to wait on timers, on devices, to detect lost loss of connectivity.  Then, we found out, if we reminded the optical team, they could configure their equipment to "drop" our end connections if they did have an end-to-end path break (they might also have redundancy in paths, so logically, you didn't lose end-to-end connectivity, although latency might change).  So, as Jim described, an optical loss of signal might not be fool proof either.

BTW, cannot say about ASAs, but overrun ingress errors, on Cisco routers, usually are due to the device cannot process the ingress at the rate it's being delivered.

This might be caused by two reasons.  Physically, the device cannot physically keep up with the ingress rate by moving received frames out of the hardware receive buffer fast enough.  This is unusual.  Or, physically, the device cannot physically "process" received frames at the ingress rate, so it doesn't bother to clear them from the ingress queue/buffers.  This is a subtle difference.

In the former, the platform is just incapable of physically dealing with the ingress rate

In the latter, much depends on what the ingress traffic contains, and the "work" needed to processed the received traffic.

Simple example, if an ingress interface has an ACL with 10,000 ACEs.  In one case, all traffic gets a match on the 1st ACE, in another case, all traffic hits the final ACE.  On many software based routers, same traffic, same ACL, but which ACE gets matched, may significantly impact throughput capacity.

So, is this problem recent?  Any change to FW rule set?  Any change to what rules traffic mix is hitting on now vs. "before"?

Leo Laohoo
Hall of Fame
Hall of Fame

@jmaxwellUSAF wrote:
How do I discover if an L2 device exists between L3 devices?

How many MAC addresses appear on the interfaces?

From sh arp, There are many mac-addresses on the relevant interface. This seems to me to be evidence of a switch in between.

Do you agree?


@jmaxwellUSAF wrote:
There are many mac-addresses on the relevant interface. This seems to me to be evidence of a switch in between.

Take those MAC addresses and shove them into an OUI Lookup Tool.  One of those MACs could, potentially, be the model of the switch.  

This is a clever Idea. Many devices were cisco devices.

I think the fact that there exist many devices in the "sh arp" data reasonably implies that there is a switch on the other end.

Thank you.


@jmaxwellUSAF wrote:

I think the fact that there exist many devices in the "sh arp" data reasonably implies that there is a switch on the other end.


Indeed, it likely does.

Sorry, I, and possible the others, except Leo (?), assumed you had a p2p connection, and was trying to determine if there was a "hidden" switch in the path.  So, BTW, what's the subnet allocation on the FW interface?  Something larger than a /30?  If so, that too would be a clue there might be other devices on that interface's network.

It's a bit unusual to have many hosts on an "outside" interface of a FW.  There are some common exceptions, like there might be devices bypassing this FW, like VPN devices.

Another exception, might be, is this facility (logically) "multi-tenant"?  I.e. could the other hosts be other FWs?

Lastly, possibly some hosts are "publicly" accessible, although, when possible, it's considered "better" to place such devices, in a DMZ segment, off another interface of the FW.  This so a FW can limit the types of external traffic that can reach them, from the "outside".

Regardless, back to your OP, how do you see the presence of other hosts, on the FW's outside interface (discounting the whole Internet?) a factor in the interface overrun errors?

Review Cisco Networking for a $25 gift card