C3KX-SM-10G Capabilities?

[bakaneko59]

We have a small network which is completely serviced by a single StackWise stack of Catalyst 3750G and 3750E switches.  There are no external connections.  We recently purchased a 3750X switch with the C3KX-SM-10G Service Module for expansion, and for it's ability to allow Netflow monitoring of our network.  However, I apparently misinterpreted the capabilities of the service module from the information I used to order it.  I was hoping to be able to monitor all of my port-to-port on-switch traffic with Netflow, but I now understand that only the SFP ports on the service module itself can be monitored.  We have nothing connected to those ports as we do not have a distributed switch network (once again, only a single stack).

Is there any kind of workaround for this that would allow me to use Netflow to monitor my on-switch traffic?  I was thinking or maybe just connecting one of the SFP ports directly to my monitoring server, but there doesn't appear to be a 1000BaseT copper SFP module available for it, and even then the port-to-port traffic wouldn't really be travelling on the SFP.  Did I waste my money buying the module?

Thanks,

Bill

[Marvin Rhoads]

Netflow is usually supported on routers as it depends on having a visibility of the Layer 3 conversations.

Do you have multiple VLANs on your switch and is it acting a the router betwen the subnets serviced on those VLANs? If so, you should be able to collect and export Netflow statistics of those inter-VLAN flows.

You will not - either with the service module or any other Netflow collector - see intra-VLAN conversations as they do not go via any Layer 3 interface.

[bakaneko59]

Yes, I have multiple VLANs - my servers are on one, iSCSI on another, and workstations on a third, and the switch is routing between them.  Port-to-port may have been an unfortunate term for me to use.  It is the traffic between ports on different VLANs that I wish to be able to monitor, even though the ports are on the same switch stack.

So we should be able to get these flows even without it going through the SFP ports?  When we try to configure the flow, all the examples I see include commands similar to this:

interface TenGigabitEthernet1/1/2

ip flow ingress

ip flow egress

and

interface TenGigabitEthernet1/1/2

ip flow monitor flowrecordname1 layer2-switched input

ip flow monitor flowrecordname2 output

always referring to the 10GB ports.  That seems to indicate that the input and output can only be read from the SFP 10GB ports.   In particular, I'm trying to follow the information shown on this site - http://www.plixer.com/blog/network-traffic-analysis/3kx-netflow-configuration-catalyst-3750-netflow-support/.  Is that just the way this guy wrote it?  Could I use any of the ports?

We're trying to use Solarwinds Netflow Monitor (eval mode at present - will purchase if we get it working right) to do the analysis from a dedicated (virtual) server.

Any further help is welcome.

[Marvin Rhoads]

Good question. I'm not sure about the answer and I don't have a 3750X with the service module available to test it out on. Since you do, go ahead and give it a try. Please let us know what you find out.

I was reading about the 'smartlog' feature when researching this. If you have access-lists separating the VLANs, you may be able to leverage that.

[bascheew]

I have confirmed that you can do NetFlow over the gigabit ports, so you don't need to use a 10 gig port.  I tried using @cweinhold's config but it didn't work for some reason in our environment.   I made a step by step post that shows the config that worked for us.  Hope it helps!

[cweinhold]

Marvin,

There are two tricks.

The first is to force locally-switched traffic to the service module ports. Cisco has a white paper that describes two ways of doing this. Remote SPAN seems to be the best if you aren't planning on using the uplinks, though it does sacrifice netflow visibility into physical port and vlan information.

The second trick is to keep the ten gig port to go up, so that traffic gets forwarded. The simplest way to do this is to install a 1-meter twinax cable between the two uplink ports (p/n SFP-H10GB-CU1M, list price $150). [Updated 4-27-2012 because the Twinax cable does not work] To do this you will need a pair of fiber SFP's -- either GLC-SX-MM for 1G or SFP-10G-SR for 10GE -- and a short fiber run between them. You may be able to get away with a single SFP and a fiber loopback cable.

Now, before you go and create a 10 Gbps loop in your network, make sure that one port is configured to carry the RSPAN vlan, and the other port is set to drop every packet it receives. I.e., something like this:

vlan 950

remote-span

vlan 951

shutdown

interface ten 1/1

switchport trunk encapsulation dot1q

switchport trunk allowed vlan 950

switchport mode trunk

switchport nonegotiate

ip flow monitor myFlowMonitor layer2-switched output

no cdp enable

int ten 1/2

no cdp enable

switchport mode access

switchport nonegotiate

switcport access vlan 951

I've tested this and it works fine.

-Craig

[Jimmy Symoens]

but how incredibly dirty and inconvenient is this?

Can't grasp that there is no decent way to do this...

[cweinhold]

Update: I was able to make this work with a single 10G SFP+ (p/n SFP-10G-SR) and a multimode fiber loopback cable (about $25 from cdw.com).The configuration looks like this:

vlan 950
 remote-span

mac access-list extended blackhole-l2
 deny any any

ip access-list extended blackhole-l3
 deny ip any any

interface TenGigabitEthernet1/1
 switchport trunk encapsulation dot1q
 switchport trunk allowed vlan 950
 switchport mode trunk
 switchport nonegotiate
 ip flow monitor  layer2-switched output
 ip access-group blackhole-l3 in
 mac access-group blackhole-l2 in
 no cdp enable
 spanning-tree bpduguard enable

monitor session 1 source vlan [list of active vlans]
monitor session 1 destination remote vlan 950

With this config, one of the 10G ports is still available for server or uplink use.

-Craig

[jpoplawski]

Craig,

Is there any kind of CPU hit doing this?  So I either need to rip/replace 3750x's or buy a $2,500 module per site to get Netflow visibility.  I also have a Layer 3 port (no switchport) can I add that to the SPAN as well?

[cweinhold]

I don't know about performance. The main switch handles the SPAN session and probably also does the UDP Netflow export. The SM is responsible for all Netflow processing, so it shouldn't impact the main switch at all.

You should be able to add anything to the SPAN session, including routed interfaces..

-Craig