NIC teaming, load balanced swtiches and sub switches.

[Nick Bridgens]

I have a number of servers (yes, I know I ought to virtuaslise them all, but...) running Server 2016. They each have 4 or 8 NICS. The NIC's are teamed. All server ethernet cables currently go to a single 3750G.

PCs are connected to assorted different 3750G's around the site with single ethernet connections from those 3750G's to the central 3750G.

All switches are vanilla with no changes to the default port settings.

I want some resilience on the central switch, so if I add a second 3750G, can I simply split the teamed NIC's between the two switches on a 50 50 basis? Do I cable the two 3750's together, and how?

Do I then connect the PC's 3750's to BOTH central switches (dual ethernet cable)? Again, do I need to configure anyhhing on either the twin central 3750's or the 'slave' 3750?

Or do I need to find some instruction on how to use Cisco switches properly?

Nick Bridgens

[Jon Marshall]

Server wise yes you can split the team across two 3750G switches but you would have to connect the 3750s with stack cables so they become a stacked pair. 

PC wise you don't generally bother with dual NICs, you just accept if the NIC or switch goes you lose that PC or the PCs connected to the switch. 

Jon

[Nick Bridgens]

Does stacking the switches mean that if one of the two central switches fail, things just carry on as normal, with just reduced performance?

Obviously, anything only connected to failed one of the pair, will lose connectivity (and we only have one router connection)

The PCs are all single NIC but each switch they are connected to, is connected to the central one with a single ethernet cable. So if that is the central switch that dies, ALL those PCs die. Can we dual connect the 'remote' 3750's to the central 3750 stacked pair (so SW1/P23 to SW2/P24 and SW1/P24 to SW3/P24) or does that create a loop? Obviously is SW1 (the one the PC's are connected to dies) all the PC's die too!

[Jon Marshall]

Yes, if you stack the switches then you would get reduced performance (potentially) but as long as devices were dual connected to both switches you should still have network connectivity. 

This also applies to your other switches ie. as long as your central 3750s are stacked then you can connect the other switches to both stacked switches using an etherchannel (probably also a trunk) and the same applies in terms of redundancy etc. 

You will not create a loop because the stacked switches are logically seen as one switch. 

Jon

[Nick Bridgens]

Think I understand...

so the 2 ethernet cables coming from the PC 3750 connect to the two central 3750's but as an etherchannel connection (have to work out how to do that - from what I understand it is a bit like NIC teaming!).

Presumably that also means that when everything is running fine I have a 2Gb connection from the PC switch to the Central switch stack..?

Win Win situation !

Thank you very much

[Jon Marshall · Accepted solution]

It's pretty straightforward to setup etherchannel but come back here if you have any issues. 

All of the above works as long as you use the stack cables to interconnect your central switches. 

Jon

[Nick Bridgens]

Just getting my brain around the improvements I am about to make! More a case of thinking out loud..

The central server pool is to have twin 3750G's joined with stack cables. The servers each run a minimum of 4 NIC's in a single team and the wires are phyically split between the two switches - so NIC1 (server 1), with four cables, will be connected to switch A ports 1 and 2 and switch B ports 3 and 4. NIC2 (server 2), with four cables, will be connected to switch A ports 5 and 6 and switch B ports 7 and 8. etc. This means that if one of the two central switches die, we only lose performance. Given the quantity of ports on the stack (under 50% utilised) we would also be able to consolidate onto one switch as a temporary measure. When all is running smoothly, we have 4Gb connections into each server from the central switch.

The PCs are (mostly) connected to a single 3750G. We have a few other pools of PC's in different buildings, each connected to their own 3750G but low volume - three or four machines. IP phones and cameras are connected to 3750POE switches (one at each of the 3750G locations). So each small comms cab currently has two switches in it...

We have multiple CAT5 available from each 'remote' cabinet to the central cabinet. From what you have already told me, we can take a pair of wires from a remote 3750G to each switch in the central 3750G stack, using Etherchannel. This will give us a 2Gb connection between the remote and the centre, and if a central switch fails, it still works at the remote end.

If a remote switch fails though, we have nothing from that office (either no phones or no pc's!).

I know there are schools of thought about daisy chaining a PC through the phone, but our Cisco 504G's all have 10/100 RJ45 connections. This would mean the PC's running at 10/100 (so a built in bottleneck) and also pointless having gigabit POE switches when the connected devices don't run at that speed.

If I double up (stack) the main remote 3750G (it's got upward of 32 PC's connected to it, printers are connected to the 3750POE as speed is not a priority), I am not going to get any better performance - it will still only have a two wire / 2Gb connection to the centre, but it means that if a switch fails, only half will lose their connection, and we have the option of temporary consolidation? But the etherchanel link from the remote stack to the central stack would need to be doubled up to give resilience (ie four wires - two physical switches to each of two physical switches) and that would give the added bonus of a 4Gb link? Is my thinking correct on that?

Finally - lots of posts here refer to VLANs as being the way forward. We don't use VLANS - probably because I don't understand them! My (limited) understanding is that it is set at a port level on the 3750, but does that mean if a port is set to VLAN 'Phone' and I plug a laptop into it... it won't work? Given we only have a total of about 120 devices (about 25% in each category) is it worth the hassle?

[Jon Marshall]

Just seen this reply. 

Yes to the 4Gb uplink from other switches. 

Vlans are a way of segregating devices into their own broadcast domain and usually their own IP subnet but if you only have that many devices you may not need to use them at the moment. 

If you did to decide to use them then your switch interconnects would probably need to be trunks to carry traffic for multiple vlans. 

If you want to discuss more feel free to ask. 

Jon

[Nick Bridgens]

Thanks much

[Joseph W. Doherty]

Just to add some information . . .

When you Etherchannel Cisco ports, you often have a choice of hashing algorithms. The "wrong" choice will only use one link, i.e. you would receive no effective additional bandwidth, but you would still have a "hot" spare. A "good" choice, for the hashing algorithm, will distribute traffic equally across the links. NB: a single flow will only use one link. By default, I recall the 3750G uses src-mac (?), but I also recall is may offer src-dest-IP. If the latter is offered, it's generally a "good" choice.

BTW, there are a few L3 "gotcha" with 3750G stacks, if the stack master fails, where switch over can take longer than might otherwise be possible with an alternative configuration. However, if you're only doing L2, none might apply.

Also BTW, way back we found some Cisco VoIP phones, their internal switch, didn't really allow a PC to even achieve anywhere near 100 Mbps.

Lastly, in the 3750G, one of their models, the one with all SFP ports, has internal resources that better allow the switch to act in a small core or distribution role. To take full advantage of that, you had to run a SDM template unique to that model, and could not mix it, in a stack, with other models (while using the "special" SDM templates).

For 3750Gs, I would recommend, if running a 12.2 version, the last release in the (55) series.

[Nick Bridgens]

Thanks for that ...

The 24 port 3750Gs are running 12.2(55) SE12 and the 48 port 3750POEs 12.2(55) SE11. I seem to recall the ROM size means they cannot take anything higher. The 48 port 3750Gs are 15.0(2)SE11.

Never tested daisy chained a PC through a 504Gs for speed, but I did find at my previous place that some NIC's would not work with them at all. Same phone, same port, different PC (or put in a replacement NIC - usually RTL8139) and all was fine.

You mentioned Layer 3 or Layer 2. We have a single LAN (/22) with no VLANS (apart from the default VLAN1) and anything can see anything else. From what I gather, L3 is where you have multiple VLANS. I am not aware of having 'configured' anything on the switches to say they are L2 or L3 (they just worked out of the box....). I notice, looking at the GUI for the current central switch, that a couple of ports are set as "TRUNK" so perhaps I need to revisit them all and fix their settings.

Regards

Nick

[Joseph W. Doherty]

Yes, some of the earlier 3750s only had a 16MB flash, and cannot contain the later IOS images. (Of course, if you load from the net . . .)

If you're running no VLANs (other than the default VLAN 1), you might be all L2. However, what's hosting the gateway? (Assuming you have one.)

If, or when, you start to use multiple VLANs, and each has a SVI on a 3750 stack, then you have speed of failover considerations if the stack master dies.

[Nick Bridgens]

Not sure what you mean by 'load from the net'- I assume it is a work around to allow you to run later OS's?

Gateway is a Cisco 1941 (to give us NAT) connected to another Cisco provided by the ISP with a 100Mb leased line.

We have a way to go I think before we need VLANS.

CNA is currently quite keen on using smart ports - and shows quite a few of the interswitch links as "trunk"..... I think it is going to be clear it all out and start afresh !

[Joseph W. Doherty]

"Load from net" is an "old" feature, where a boot statement can load, and boot into, an IOS from a TFTP server.