09-19-2019 01:35 AM
Hi All,
There is an delay in executing each command in the cisco nexus switch 3064 it takes 3-5 sec to respond.Kindly provide your inputs to fix this issue.
Model : Cisco Nexus 3064
Vesrion : 6.0(2)U6(10)
09-19-2019 05:41 AM
Hi PdhineshK and thanks for posting to the forums!
I'd love to help you troubleshoot this issue but I'll need a little more information to narrow it down. Are you using the console port to access the switch or a VTY line? If you're using the console port, please reseat the cable as the switch, the adapter, and at the computer, then re-initiate your serial connection. Honestly I get a lot of errors with asychnronous cable output that are solved with a simple re-seat.
If you're using a VTY line, do you have dedicated bandwidth for your SSH/Telnet session into the switch? Are you using another method to access the switch?
Lastly, can you run the command "Show process cpu history" and share the output here? If you can't share the output, are you facing any extremely high CPU utilization (great than 70%) that could be slowing down the response time of you switch?
Please let me know and I'm sure I can help you troubleshoot the issue.
Thanks! Have a great day.
-Zac
09-19-2019 06:37 PM
Thanks for your inputs.
I am using ssh to login the switch but not sure how to check the dedicated bandwidth allocated for SSH.As per your suggestions i have checked the CPU hostro below are the output.
RHPROD123A# Show process cpu history
111111 1411111 1111121211111131111 11111111111111111111111 1
404031741034688315320916345530851691114219743312695342358382
100
90
80
70
60
50
40 #
30 # #
20 # ## # #### ## ### # ## ### ##
10 ############################################################
0....5....1....1....2....2....3....3....4....4....5....5....
0 5 0 5 0 5 0 5 0 5
CPU% per second (last 60 seconds)
# = average CPU%
324722222222462222212247222222225722222211372212222227222221
032014206306487625190450541700340100502399700584504583422718
100
90
80
70 * * * * * *
60 * * * * * *
50 * * ** ** * *
40 ** ** ** ** ** *
30 * ** * ***** * *** * ** * ** * * *** *
20 **##****#****#*********#*********#*********#*********#******
10 ############################################################
0....5....1....1....2....2....3....3....4....4....5....5....
0 5 0 5 0 5 0 5 0 5
CPU% per minute (last 60 minutes)
* = maximum CPU% # = average CPU%
777777776777776777777777777777777777777777777777777777777777777777777777
343556449238637357536369442432245653577623658753126307381645658139532434
100
90
80 *** ** *** * ** *** **** ***** * * * * **** **
70 ************************************************************************
60 ************************************************************************
50 ************************************************************************
40 ************************************************************************
30 ************************************************************************
20 ****************####*******************#####*#******************####**#*
10 ########################################################################
0....5....1....1....2....2....3....3....4....4....5....5....6....6....7.
0 5 0 5 0 5 0 5 0 5 0 5 0
CPU% per hour (last 72 hours)
* = maximum CPU% # = average CPU%
RHPROD123A#
09-20-2019 05:04 AM
PdhineshK,
Your CPU utilization does look a little high, but if I understand correctly that there is a constant delay entering any command into the Nexus CLI, then it is probably not your CPU utilization. There are 2 likely causes, and we'll go over the most likely first.
1) Congestion on the network. I'm not sure how many hops to get to your Nexus switch, or if the Nexus is bogged down itself. Any dedicated SSH traffic (which you probably do want to allocate some bandwidth to) would be shown in the running config as a class map. With the show run command you should see some output that looks like this:
Policy-map LLQ
Class-map MANAGEMENT
Priority 2000 (kbps)
You can also run show policy-map interface gigabitethernet x/x and look for anything related to management or low latency cueing. If you don't have any classmaps I would recommend adding at least one service class to reduce cueing time and see if that solves your problem (these commands are approximate as i don't have a Nexus in front of me to test them exactly):
Nexus(config)#: class-map MANAGEMENT
Nexus(config-cmap)#: match type ssh
Nexus(config)#: policy-map LLQ
Nexus(config-pmap)#: class MANAGEMENT
Nexus(config-pmap-c)#: priority percent 5 (this command will allocate a minimum of 5% of your bandwidth to SSH traffic regardless of other network traffic. It's extremely unlikely you'd ever hit 5% with simple ssh traffic, honestly 1% is pretty slim, so you can choose your own value here or use a kbps measurement)
Nexus(config)#: interface gigabitethernet 0/0 (obviously select your trunk link here)
Nexus(config-if)#:Service-policy output LLQ
Those commands will at least make sure that your SSH traffic doesn't have any obstacles between itself and the way back to your terminal. You may wish to also apply a LLQ service policy in the direction of the trunk link to ensure that all your management traffic gets priority movement across the network.
The other thing I would check right off would be the CoPP status of your device. If you aren't familiar with Control Plane Policing (CoPP) it's essentially just a security policy to keep your CPU from getting overwhelmed or taken down by a DOS attack (by limiting the number of packets that can travel to the CPU). It's unlikely it's interfering, but I would check the policy anyway since your only real latency complaint is with control plane commands (configuration via ssh). Use the following commands to the CoPP and let me know what the output is:
Nexus#: show copp status
Nexus#: show policy-map interface control-plane
I hope some of this information helps you hunt down the problem, and I'll be happy to keep troubleshooting with you once we have some more output. Thanks for posting, good luck!
-Zac
09-20-2019 06:44 AM
Thank you very much for the inputs.
We are facing this delay after we upgrade the switch IOS from 5.x to 6.x.Not sure this upgrade causing the delay.
I checked in the Cisco forum they mentioned when upgrade 5.x to 6.x by default copp policy not be applied.
Does the copp policy causing this delay??
If copp need to be applied let me know the steps to apply the copp policy.
09-20-2019 08:56 AM
Hi,
Does the copp policy causing this delay??
No, COPP does not cause this. I have also seen this in Nexus when SNMP is enable. If you have it enabled, For testing, can you disable it and test again?
HTH
09-22-2019 08:51 PM
Hi ,
i don't see the below configuration in the switch
Policy-map LLQ
Class-map MANAGEMENT
Priority 2000 (kbps
Please find the below Copp policy output.
SGPROD123A# show copp status
Last Config Operation: None
Last Config Operation Timestamp: None
Last Config Operation Status: None
Policy-map attached to the control-plane: copp-system-policy
SGPROD123A# show policy-map interface control-plane
Control Plane
service-policy input: copp-system-policy
class-map copp-s-selfIp (match-any)
police pps 500
OutPackets 0
DropPackets 0
class-map copp-s-default (match-any)
police pps 400
OutPackets 0
DropPackets 0
class-map copp-s-l2switched (match-any)
police pps 200
OutPackets 12410119
DropPackets 0
class-map copp-s-ping (match-any)
match access-group name copp-system-acl-ping
police pps 100
OutPackets 0
DropPackets 0
class-map copp-s-l3destmiss (match-any)
police pps 100
OutPackets 0
DropPackets 0
class-map copp-s-glean (match-any)
police pps 500
OutPackets 0
DropPackets 0
class-map copp-s-l3mtufail (match-any)
police pps 100
OutPackets 0
DropPackets 0
class-map copp-s-ttl1 (match-any)
police pps 100
OutPackets 0
DropPackets 0
class-map copp-s-ipmcmiss (match-any)
police pps 400
OutPackets 0
DropPackets 0
class-map copp-s-l3slowpath (match-any)
police pps 100
OutPackets 0
DropPackets 0
class-map copp-s-dhcpreq (match-any)
police pps 300
OutPackets 0
DropPackets 0
class-map copp-s-dhcpresp (match-any)
match access-group name copp-system-dhcp-relay
police pps 300
OutPackets 0
DropPackets 0
class-map copp-s-dai (match-any)
police pps 300
OutPackets 0
DropPackets 0
class-map copp-s-igmp (match-any)
match access-group name copp-system-acl-igmp
police pps 400
OutPackets 268424
DropPackets 0
class-map copp-s-routingProto2 (match-any)
match access-group name copp-system-acl-routingproto2
police pps 1300
OutPackets 4159515
DropPackets 0
class-map copp-s-v6routingProto2 (match-any)
match access-group name copp-system-acl-v6routingProto2
police pps 1300
OutPackets 0
DropPackets 0
class-map copp-s-eigrp (match-any)
match access-group name copp-system-acl-eigrp
match access-group name copp-system-acl-eigrp6
police pps 200
OutPackets 0
DropPackets 0
class-map copp-s-pimreg (match-any)
match access-group name copp-system-acl-pimreg
police pps 200
OutPackets 0
DropPackets 0
class-map copp-s-pimautorp (match-any)
police pps 200
OutPackets 0
DropPackets 0
class-map copp-s-routingProto1 (match-any)
match access-group name copp-system-acl-routingproto1
match access-group name copp-system-acl-v6routingproto1
police pps 1000
OutPackets 19
DropPackets 0
class-map copp-s-arp (match-any)
police pps 200
OutPackets 703489
DropPackets 0
class-map copp-s-ptp (match-any)
police pps 1000
OutPackets 0
DropPackets 0
class-map copp-s-vxlan (match-any)
police pps 1000
OutPackets 0
DropPackets 0
class-map copp-s-bfd (match-any)
police pps 350
OutPackets 0
DropPackets 0
class-map copp-s-bpdu (match-any)
police pps 12000
OutPackets 11940143
DropPackets 0
class-map copp-s-dpss (match-any)
police pps 1000
OutPackets 0
DropPackets 0
class-map copp-icmp (match-any)
match access-group name copp-system-acl-icmp
police pps 200
OutPackets 0
DropPackets 0
class-map copp-telnet (match-any)
match access-group name copp-system-acl-telnet
police pps 500
OutPackets 0
DropPackets 0
class-map copp-ssh (match-any)
match access-group name copp-system-acl-ssh
police pps 500
OutPackets 0
DropPackets 0
class-map copp-snmp (match-any)
match access-group name copp-system-acl-snmp
police pps 500
OutPackets 0
DropPackets 0
class-map copp-ntp (match-any)
match access-group name copp-system-acl-ntp
police pps 100
OutPackets 0
DropPackets 0
class-map copp-tacacsradius (match-any)
match access-group name copp-system-acl-tacacsradius
police pps 400
OutPackets 0
DropPackets 0
class-map copp-stftp (match-any)
match access-group name copp-system-acl-stftp
police pps 400
OutPackets 0
DropPackets 0
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide