Cisco's Official Stance on SPAN

[Odysseu$]

I have heard from many different companies that call TAC, that TAC says Cisco does not recommend to turn SPAN on for anything other than troubleshooting.  I have seen this being said on all switching platforms.  Today someone from a very large company told me they called TAC today, and TAC repeated this message again, the company is using the Catalyst 9K platform which according to the documentation SPAN is done in hardware.

I cannot find anywhere in any of the SPAN documentation that Cisco ever states not to use SPAN for security tools.  As a matter of fact, the way you ingest traffic into Cisco Cybervision is by using SPAN.

So, can someone from Cisco please go on the record and explain Cisco's official stance of using SPAN.  Can it be used to feed other security tools or TAPs?  This issue confuses Cisco customers and this should be clarified.

My gut tells me that this is just another way TAC tries to close cases without any real troubleshooting.

[Joseph W. Doherty]

I don't speak for Cisco, but possibly two reasons for TAC making such suggestions, first it's easy to oversubscribe a SPAN port, in certain situations.  Second, even when supported in hardware, doesn't necessarily mean there aren't capacity limitations that you might bump into, i.e. not all hardware has the same capacity doing the same thing.  (Consider SPAN hardware is likely not designed to provide the best possible performance especially for a "troubleshooting" feature.  [Possibly much like Ethernet management ports being unable to sustain port's wire-rare.])

[Odysseu$]

I appreciate the reply.  And I want to be specific that this topic is for Local SPAN and not RSPAN or ERSPAN. 
Oversubscription can be argued for any port that does aggregation.  Since the destination port of a local SPAN is just another port I would think oversubscription would be the same as any other port.

I know that some limitations can come in place even when the switching is in hardware.  Things such as sourcing by a VLAN rather than a port would require using the rewrite queue and could add to the use of common resources like the TCAM.

But I do not see why there would be an impact when monitoring by sourcing an interface and destined to another interface.  At Cisco Live! in the cat9k architecture lecture there was considerable time spent on how the UA asic handles SPAN in hardware.  It behoves me why so much time and money would be put into developing this feature just to have TAC say that it is not recommended to be used except in limited cases.

[Joseph W. Doherty]

"It behoves me why so much time and money would be put into developing this feature just to have TAC say that it is not recommended to be used except in limited cases."

Ah, Jim's (@Ramblin Tech) reply, captures much of what I was going to say in this reply.

So, I'll cut to the quick.

As Jim mentioned, large corporations, like Cisco, have multiple divisions/BUs/whatevers, each with their own goals.  Those designing newer platform chose what they want the hardware to do well.  In your case, yes, some, or possible all, Catalyst 9Ks would handle (local) SPAN just fine (excluding issues you're not going to circumvent, like port oversubscription), and be touted as a sales feature.  (BTW, with specific 9Ks, much might depend on actual UADP version variant, along the rest of the hardware architecture.)

TAC, though, is in the business of supporting Catalyst 9Ks along with other supported Catalysts, and perhaps is but recommending "the lowest common denominator". 

Don't forget, maintenance based on fixed price is most profitable, the less you provide.  Whereas for sales, you want to sell products touting how wonderful the products are.

I'm sure, Cisco, much like any business, isn't immune to such considerations.

In other words, it's not totally unexpected, TAC would downplay using SPAN, while some Cisco Live session would tout, it slices, it dices, it SPANs, and it's now lemon scented too.

Unfortunately, this dichotomy is pretty much a business norm.

Romans understood this thousands of years ago with their caveat emptor.

Also in large businesses, often the right hand doesn't know what the left hand is doing.  For instance, how well is 1st or 2nd tier educated in the capabilities of different hardware to perform different functions.

[Ramblin Tech]

Usually when TAC says "unsupported" or "not recommended" it is because the Business Unit/Entity that produced that product did not specifically test that use-case. Without a BU/BE dev-test, unit test, or systems integration test program to discover scalability limits and corner cases, TAC will be hesitant to go out on a limb on their own and say that they will support untested configurations.

It is also very unlikely that you will get a blanket, definitive answer in this forum from a Cisco employee that covers all Cisco products capable of SPANing, given the shear number of routers and switches that support SPAN. I am not sure you will find anyone lower than Jonathan Davidson (direct report to Chuck Robbins) that can make definitive statements about all Cisco switches and routers across all BUs/BEs. I would suggest you reach out to your Cisco account team or Advanced Services team (if you have AS) to talk to them about what it would take to officially support a "permanent" SPAN session for your particular use-case.

[Odysseu$]

I do understand the politics and makeup of Cisco, for full disclosure I worked for Cisco for 15 years and at one point Chuck was my operational director.  I would imagine that the PM for the Cat9k in the UABU would want to ensure the world that they can use this feature without fear of lack of support from Cisco. 

And that is the perception from the customers, that if they turn on SPAN and an issue occurs (whether it is SPAN related or not) the defacto answer is turn of SPAN irregardless of the business impact of taking that action.  As previously mentioned SPAN is used by Cisco products, and advocated in those products (see cybervision installation guide where they advocate turning on ERSPAN which IS processed by the CPU and shared resources and no mention of doing a traffic engineering exercise to ensure you aren't going through a highly constricted device like a firewall)

As someone that still bleeds teal, it is an issue and I want to bring this to a public forum so it can be discussed.  

Recently a very large company I work with used the lack of TAC supported SPAN as a one of many bullet points in selecting a competitive switching platform to replace 80 plants switching infrastructure.  This wasn't the primary driver but it was brought up several times by the OT team as they require SPAN for their OT security solution.

[Ramblin Tech]

Hi @Odysseu$, 24-year SE here, until I RIFtired last year. Chuck was a channels RM in the Atlanta office when I started there. 

If the opportunity is big enough, BUs can be quite flexible when it comes to supporting “creative” configs, so this lost deal falls on the account team for not realizing what was important to the customer and getting that support road-mapped. The account SEs can register their customers’ feature requests in the Aha system and then AM/SE (and their managers) can work the business-case angle with the PMs to get on the roadmap.

I assume you are with a channel partner now; if so, you might express to your channel SE the importance of permanent SPAN sessions and ask if an Aha Idea can be submitted to support them. As an aside, I never heard anything like this on SPAN from the XR BU (MIG, or whatever they are called now). The idea that SPAN sessions should not be left on indefinitely might be just an enterprise Catalyst thing.