Pair of CIsco 4507R+E each with a single WS-X45-SUP7-E / 1 1GbE Uplink

[vboyd]

I have a customer that has a pair of 4507R+E Chassis each with a single WS-X45-SUP7-E.  Issue is the SUP's were connected via a single 1GbE Uplink (ouch).  In the interim to provide some redundancy, I install another 1GbE GBIC and the pair are up now.  Even with 2x1GbE uplinks they are getting hammered and one of them is reporting massive input and CRC errors.  I need to upgrade to a pair of 10GbE GBIC's, or if the switch's will support it, 4x10GbE uplinks 

I don't believe i can swap the 1GbE GBIC's out one at a time as the Po won't form with different speed GBIC's

I don't believe i can pull them both out then insert the 10GbE GBIC's as that would result in a dual active situation = no fun

I can't down the either switch's as many of the IDF's only have single connection to either one of the 4507's = no fun

Since each 4507 is not operating in stand alone redundancy mode could i use the x/3/3 and x/3/4 interfaces to create an additional Po?  then assuming it sync's up, pull the 1GbE GBIC and replace them with 10GbE GBIC's?  If i recall correctly if the 4507's are running dual SUP's x/3/3 and x/3/4 are disabled?

Any advice or help would be greatly appreciated!

interface TenGigabitEthernet1/3/1

description link to FIN-DCSW2

switchport mode trunk

switchport nonegotiate

no lldp transmit

no lldp receive

no cdp enable

channel-group 1 mode on

service-policy output VSL-Queuing-Policy

interface TenGigabitEthernet1/3/2  <-This one was disconnected

description link to FIN-DCSW2

switchport mode trunk

switchport nonegotiate

no lldp transmit

no lldp receive

no cdp enable

channel-group 1 mode on

service-policy output VSL-Queuing-Policy

interface TenGigabitEthernet2/3/1

description link to FIN-DCSW1

switchport mode trunk

switchport nonegotiate

no lldp transmit

no lldp receive

no cdp enable

channel-group 2 mode on

service-policy output VSL-Queuing-Policy

interface TenGigabitEthernet2/3/2<-This one was disconnected

description link to FIN-DCSW1

switchport mode trunk

switchport nonegotiate

no lldp transmit

no lldp receive

no cdp enable

channel-group 2 mode on

service-policy output VSL-Queuing-Policy

FIN-DC01#sh redun
Redundant System Information :

------------------------------
Available system uptime = 23 weeks, 1 day, 1 hour, 11 minutes
Switchovers system experienced = 0
Standby failures = 0
Last switchover reason = none

Hardware Mode = Duplex
Configured Redundancy Mode = Stateful Switchover
Operating Redundancy Mode = Stateful Switchover
Maintenance Mode = Disabled
Communications = Up

Current Processor Information :
------------------------------
Active Location = slot 1/3
Current Software state = ACTIVE
Uptime in current state = 23 weeks, 1 day, 1 hour, 8 minutes
Image Version = Cisco IOS Software, IOS-XE Software, Catalyst 4500 L3 Switch Software (cat4500e-UNIVERSALK9-M), Version 03.11.04.E RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2021 by Cisco Systems, Inc.
Compiled Mon 08-Mar-21 15:37 by prod
BOOT = bootflash:cat4500e-universalk9.SPA.03.11.04.E.152-7.E4.bin,12;
Configuration register = 0x2102

Peer Processor Information :
------------------------------
Standby Location = slot 2/3
Current Software state = STANDBY HOT
Uptime in current state = 23 weeks, 1 day, 1 hour, 7 minutes
Image Version = Cisco IOS Software, IOS-XE Software, Catalyst 4500 L3 Switch Software (cat4500e-UNIVERSALK9-M), Version 03.11.04.E RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2021 by Cisco Systems, Inc.
Compiled Mon 08-Mar-21 15:37 by pr
BOOT = bootflash:cat4500e-universalk9.SPA.03.11.04.E.152-7.E4.bin,12;
Configuration register = 0x2102

FIN-DCSW1#sh cef state
CEF Status:
RP instance
common CEF enabled
IPv4 CEF Status:
CEF enabled/running
dCEF enabled/running
CEF switching enabled/running
universal per-destination load sharing algorithm, id 95764BE6
IPv6 CEF Status:
CEF disabled/not running
dCEF disabled/not running
universal per-destination load sharing algorithm, id 95764BE6
RRP state:
I am standby RRP: no
RF Peer Presence: yes
RF Peer Comm reached: yes
RF Peer Config done: yes
RF Progression blocked: never
Redundancy mode: sso(3)
CEF NSF sync: enabled/running

CEF ISSU Status:
FIBHWIDB broker
Slot(s): 13 (0x2000) (grp 0x691D8540) - Nego compatible.
FIBIDB broker
Slot(s): 13 (0x2000) (grp 0x691D8540) - Nego compatible.
FIBHWIDB Subblock broker
Slot(s): 13 (0x2000) (grp 0x691D8540) - Nego compatible.
FIBIDB Subblock broker
Slot(s): 13 (0x2000) (grp 0x691D8540) - Nego compatible.
Adjacency update
Slot(s): 13 (0x2000) (grp 0x691D8540) - Nego compatible.
IPv4 table broker
Slot(s): 13 (0x2000) (grp 0x691D8540) - Nego compatible.
IPv6 table broker
Slot(s): 13 (0x2000) (grp 0x691D8540) - Nego compatible.
CEF push
Slot(s): 13 (0x2000) (grp 0x691D8540) - Nego compatible.

[Leo Laohoo]

There will be an outage.  Guaranteed.

IF the customer is OK to upgrade the firmware of both line cards to 3.11.4, then down time to upgrade to 10 Gbps is nothing.

NOTE:  Do not skimp on the fibre optic patch cord.  Do not "re-use" the same patch cord used for the 1 Gbps link.  Get new ones and get ones from reputable patch cord vendors.

[vboyd]

Hey Leo thank you for the reply!  i kinda figured an outage would be a thing...  how would you recommend i proceed to limit the downtime?

They appear to be running 3.11.4   BOOT = bootflash:cat4500e-universalk9.SPA.03.11.04.E.152-7.E4.bin,12;

I see this as the latest version? unless i missed something?

CAT4500E Universal Image

cat4500e-universal.SPA.03.11.04.E.152-7.E4.bin

22-Mar-2021

186.16 MB

[Leo Laohoo]

3.11.4 is the "last" firmware for the Sup7E/LE.

The above statement is "officially" correct but "unofficially" incorrect.  Please read THIS.

[vboyd]

and noted on the optical cable....while it's tough to say never....if it's not my fibre optical patch cord, that i purchased, I let the customer plug it in...

[Leo Laohoo]

First things first, the 10 Gbps optics.  Have you tested them?  Plug them into a spare port.  Put them into a VLAN that is not found in the VLAN DB, say VLAN 1234.  Connect them up and enable the ports.  Do the ports go up/up?  Give it 10 minutes in this state.  Are there any line errors?  

Next, I never use fibre optic patch cord.  Especially one that has line error on them.  

For 10 Gbps, make sure the fibre optic cables are OM5 (multi-mode) or OS2 (single-mode).

[vboyd]

i'm going to swap out the fibre optic cable on the SFP throwing the errors as i didn't see the customer take it out of the bag....they simply handed it to me...

This is what i'm seeing

TenGigabitEthernet1/3/1 is up, line protocol is up (connected)
Hardware is Ten Gigabit Ethernet Port, address is f872.eae0.ad40 (bia f872.eae0.ad40)
Description: link to MCH-4507SW2
MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 2/255, rxload 122/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 1000Mb/s, link type is auto, media type is 1000BaseSX
input flow-control is on, output flow-control is on
ARP type: ARPA, ARP Timeout 04:00:00
Last input never, output never, output hang never
Last clearing of "show interface" counters 1d22h
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: Class-based queueing
Output queue: 0/40 (size/max)
5 minute input rate 478692000 bits/sec, 50579 packets/sec
5 minute output rate 9522000 bits/sec, 2956 packets/sec
9877421526 packets input, 10076353313070 bytes, 0 no buffer
Received 866022372 broadcasts (581114388 multicasts)
0 runts, 0 giants, 0 throttles
4994948 input errors, 4994802 CRC, 0 frame, 0 overrun, 0 ignored
0 input packets with dribble condition detected
1718838684 packets output, 1506114716750 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier
0 output buffer failures, 0 output buffers swapped out

[vboyd]

the "second" GBIC I installed with my cable is clean

TenGigabitEthernet1/3/2 is up, line protocol is up (connected)
Hardware is Ten Gigabit Ethernet Port, address is f872.eae0.ad41 (bia f872.eae0.ad41)
Description: link to MCH-4507SW2
MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 3/255, rxload 103/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 1000Mb/s, link type is auto, media type is 1000BaseSX
input flow-control is on, output flow-control is on
ARP type: ARPA, ARP Timeout 04:00:00
Last input never, output never, output hang never
Last clearing of "show interface" counters 1d22h
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: Class-based queueing
Output queue: 0/40 (size/max)
5 minute input rate 404538000 bits/sec, 55350 packets/sec
5 minute output rate 14269000 bits/sec, 2030 packets/sec
7953238931 packets input, 7777795453226 bytes, 0 no buffer
Received 833118127 broadcasts (534494564 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 input packets with dribble condition detected
211345610 packets output, 171858370526 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier
0 output buffer failures, 0 output buffers swapped out

[vboyd]

but both are getting hammered!  90% their vSAN environment is not happy, packet loss and latency 

[Leo Laohoo]

4500 is not designed to handle traffic from servers. 

The ports will need QoS configured into them or else the 4500 port buffers will get overwhelm and cause packets to get dropped.  

[vboyd]

i'm sorry i wasn't clear on that....those ports on the SUP7 are not handling server traffic....they are connected to the other 507...and I do have QoS on place on those links

class-map match-any VSL-MGMT-PACKETS
match access-group name VSL-MGMT
class-map match-any VSL-DATA-PACKETS
match any
class-map match-any VSL-L2-CONTROL-PACKETS
match access-group name VSL-DOT1x
match access-group name VSL-BPDU
match access-group name VSL-CDP
match access-group name VSL-LLDP
match access-group name VSL-SSTP
match access-group name VSL-GARP
class-map match-any VSL-L3-CONTROL-PACKETS
match access-group name VSL-IPV4-ROUTING
match access-group name VSL-BFD
match access-group name VSL-DHCP-CLIENT-TO-SERVER
match access-group name VSL-DHCP-SERVER-TO-CLIENT
match access-group name VSL-DHCP-SERVER-TO-SERVER
match access-group name VSL-IPV6-ROUTING
class-map match-any VSL-MULTIMEDIA-TRAFFIC
match dscp af41
match dscp af42
match dscp af43
match dscp af31
match dscp af32
match dscp af33
match dscp af21
match dscp af22
match dscp af23
class-map match-any VSL-VOICE-VIDEO-TRAFFIC
match dscp ef
match dscp cs4
match dscp cs5
class-map match-any VSL-SIGNALING-NETWORK-MGMT
match dscp cs2
match dscp cs3
match dscp cs6
match dscp cs7
!
policy-map VSL-Queuing-Policy
class VSL-MGMT-PACKETS
bandwidth percent 5
class VSL-L2-CONTROL-PACKETS
bandwidth percent 5
class VSL-L3-CONTROL-PACKETS
bandwidth percent 5
class VSL-VOICE-VIDEO-TRAFFIC
bandwidth percent 30
class VSL-SIGNALING-NETWORK-MGMT
bandwidth percent 10
class VSL-MULTIMEDIA-TRAFFIC
bandwidth percent 20
class VSL-DATA-PACKETS
bandwidth percent 20
class class-default
bandwidth percent 5

mac access-list extended VSL-BPDU
permit any 0180.c200.0000 0000.0000.0003
mac access-list extended VSL-CDP
permit any host 0100.0ccc.cccc
mac access-list extended VSL-DOT1x
permit any any 0x888E
mac access-list extended VSL-GARP
permit any host 0180.c200.0020
mac access-list extended VSL-LLDP
permit any host 0180.c200.000e
mac access-list extended VSL-MGMT
permit any 0022.bdcd.d200 0000.0000.00ff
permit 0022.bdcd.d200 0000.0000.00ff any
mac access-list extended VSL-SSTP
permit any host 0100.0ccc.cccd

[vboyd]

any thoughts on how to proceed limiting any downtime?

[Leo Laohoo]

---

@vboyd wrote:
any thoughts on how to proceed limiting any downtime?

---

How much downtime is "time"?  

Depending on the config, moving connections from 1Gbps to 10 Gbps can either be simple or complicated.  

IF the config is just a plain Layer 2, the outage will depend entirely on how behaved STP is.  If it's Layer 3, that's a different case.

Have I done it before?  Yes, I have.  When we moved connections from 1 Gbps to 10 Gbps a few years ago.  Recently, we've been moving our sites from 10 Gbps to 40 Gbps using the same methods.

[vboyd]

you mentioned in an earlier post that if they went to v 3.11.4 downtime would be nothing?  

This is layer 3....

since each 4507 only has one SUP7, can't i simply create two more port channels using the remaining two 10GbE ports on each SUP7?  once those come up...wait until it sync's then pull the 1GbE links from the first two 10GbE ports and swap them?  then i would have 4x10GbE uplinks between the two 4507's.   I understand that if a single 4507 has dual SUP7's that the last two 10GbE ports on each SUP are disabled...but that is not the case in this configuration...

thoughts? 

[Leo Laohoo]

---

@vboyd wrote:
you mentioned in an earlier post that if they went to v 3.11.4 downtime would be nothing? 

---

How many minutes does a chassis stay down while upgrading?  Four minutes and 45 seconds for non-VSS and up to seven minutes and 10 for a VSS.  And I am not even talking about ROMMON upgrade which will take an outage over 20 minutes (per supervisor card). 

If the business can take a hit (aka downtime), then switching from 1 Gbps to 10 Gbps is nothing.  

---

@vboyd wrote:
This is layer 3....

---

Passive-Interface always gets me.  Don't forget that.  

---

@vboyd wrote:
since each 4507 only has one SUP7, can't i simply create two more port channels using the remaining two 10GbE ports on each SUP7?  once those come up

---

Yes, this is possible.  But always get an agreement for a downtime because of convergence.  

This is a Layer 3 so this means the IP address will cross over to the new Layer 3 Etherchannel.  Duplicate IP address will not be an issue because the ports are down.  When the links go up to 10 Gbps, covergence will take it's wee time.  

And that is when downtime will occur.