I'm trying to monitor all the traffic from the first 22 ports of 10 switches in a research lab environment (Cisco 2960X-24TD, Cisco 3650-48TD, CGS2520-24TC). I'm struggling to find a good solution to aggregate all the SPAN traffic and deliver it to multiple destination ports. In this lab environment, it's important that none of the mirrored packets are dropped. We have in excess of 100 devices that communicate across the network. I cannot rely upon taps between switches because I would miss traffic between devices that exist on the same switch. Installing taps between every device and switch isnt' feasible due to scale and the duplication of data when trying to aggregate it all.
This led me to using RSPAN and only observing the ingress traffic. I tried using RSPAN, but as our traffic has scaled up we've noticed packet loss and timeouts on both normal traffic and RSPAN traffic. Removal of the RSPAN traffic from the trunk solves this problem, so I attempted using duplicate trunks and using the VLAN priority to load the RSPAN traffic onto one trunk and the normal network traffic onto the other. This configuration still results in intermittent network packet loss/timeouts on the both normal traffic and the RSPAN. Removing the RSPAN VLAN from the allowed vlan again stops this problem.
I find it hard to believe this would be a backplane or forwarding issue on the 3650, unless SPAN/RSPAN is handled as forwarding instead of switched traffic.
The network structure has a Catalyst 3650 as the hub, with duplicate gig trunks to each Catalyst 2960 and CSG2520. The RSPAN VLAN is 900.
Each trunk port configured as:
switchport trunk allowed vlan 1-499, 900-1099
switchport mode trunk
And each 2960 and CSG2520 having the following monitored session:
monitor session 1 source interface Gi1/0/1 - 22 rx (the 2520s being the fa instead)
monitor session 1 destination remote vlan 900
The 3650 has the following monitored session:
monitor session 1 source remote vlan 900
monitor session 1 destination interface Gi1/0/25 (with the intention to have 6-8 destination ports with exact duplicate data for contrasting different network monitoring solutions)
Looking for ideas on how to resolve my issue or another configuration that would be more ideal. Would using traditional ingress SPAN ports connected to a separate switch (not network connected) to aggregate data the ingress data and then taking the ingress SPAN of that switch to local destination ports be more successful or ideal?
Thanks for the assistance.
