Hello,
Just recently I refreshed a complete network where old 3550s and 6500 (SUP2) was switched to 2960X access and 6880-X-LE core. Ever since the refresh we're seeing really wierd response times and users are reporting the network as slow at times.The topology is pretty basic; 2x6880-X-LE core (VSS) and 2960X (single up to 3 switches in a stack) with dual 10G etherchannel-uplinks towards the core.
The logical setup is a little bit different becuse we are preparing for a migration from normally routed core to MPLS VPN. As such the configuration for MPLS is completed on the device but the routes are leaked locally to the old IPv4 core until all the sites have updated core equipment that supports MPLS VPN.
Looking at the CPU of the 6880-X-LE I find the slcp process (VSS) running really high plus there are spikes every now and then:
2222222221111122222111112222222222333331111122222111112222
2222555556666633333666663333300000111116666655555666663333
100
90
80
70
60
50
40
30 ***** ***** *****
20 **********************************************************
10 **********************************************************
0....5....1....1....2....2....3....3....4....4....5....5....
0 5 0 5 0 5 0 5 0 5
CPU% per second (last 60 seconds)
3333333333332222522222222332333233322222223323233322333223
1043420105114499699698569109110812598899880080621068080790
100
90
80
70
60 *
50 *
40 * * * *
30 ************ **#*****************************************
20 ##########################################################
10 ##########################################################
0....5....1....1....2....2....3....3....4....4....5....5....
0 5 0 5 0 5 0 5 0 5
CPU% per minute (last 60 minutes)
* = maximum CPU% # = average CPU%
5576556666565556766666567575656666666566665666556556666565556567667766
3506790362745380444136920707590321618722679271782992402874763916889230
100
90
80 * *
70 ** * * * * * * * * ** * * *****
60 ************ ******************************************* ************
50 **********************************************************************
40 **********************************************************************
30 **********************************************************************
20 ######################################################################
10 ######################################################################
0....5....1....1....2....2....3....3....4....4....5....5....6....6....7.
0 5 0 5 0 5 0 5 0 5 0 5 0
CPU% per hour (last 72 hours)
* = maximum CPU% # = average CPU%
Interrupt processing is 0% so all processing dedicated control plane processing. Total of 8 instances of OSPF is running (old VRF-Lite infrastructure) and BGP for the new MPLS VPN which is prepared:
CPU utilization for five seconds: 27%/0%; one minute: 21%; five minutes: 21%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
108 120571588 177074071 680 15.75% 12.55% 12.65% 0 slcp process
603 22238784 3375191 6588 6.23% 3.08% 2.88% 0 Lif stats RP tas
602 22289352 1620456 13754 4.00% 2.79% 2.81% 0 Lif stats hw rea
143 3863428 827054 4671 0.31% 0.39% 0.41% 0 OIR Process
924 2185836 58781876 37 0.31% 0.21% 0.22% 0 Port manager per
907 2682572 2521666 1063 0.23% 0.33% 0.33% 0 Env Poll
1013 84384 63259 1333 0.15% 0.02% 0.00% 0 l2_fwd_mac_oob_s
.
.
Pinging the 6880-X-LE or any device attached to it replies with an average 3ms locally, 15-20ms from remote.
Any ideas?