06-28-2013 05:06 PM - edited 03-01-2019 07:21 AM
with Cisco Expert Vinayak Sudame
Welcome to the Cisco Support Community Ask the Expert conversation. This is an opportunity to learn and ask questions how to configure and troubleshoot the Cisco Nexus 2000, 5000 and 6000 Series Switches with Cisco subject matter expert Vinayak Sudame. You can ask any question on configuration, troubleshooting, features, design and Fiber Channel over Ethernet (FCoE).
Vinayak Sudame is a Technical Lead in Data Center Switching Support Team within Cisco's Technical Services in RTP, North Carolina. His current responsibilities include but are not limited to Troubleshooting Technical support problems and Escalations in the areas of Nexus 5000, Nexus 2000, FCoE. Vinayak is also involved in developing technical content for Cisco Internal as well as external. eg, Nexus 5000 Troubleshooting Guide (CCO), Nexus 5000 portal (partners), etc. This involves cross team collaboration and working with multiple different teams within Cisco. Vinayak has also contributed to training account teams and partners in CAE (Customer Assurance Engineering) bootcamp dealing with Nexus 5000 technologies. In the past, Vinayak's responsibilities included supporting MDS platform (Fiber Channel Technologies) and work with EMC support on Escalated MDS cases. Vinayak was the Subject Matter Expert for Santap Technologies before moving to Nexus 5000 support. Vinayak holds a Masters in Electrical Engineering with Specialization in Networking from Wichita State University, Kansas. He also holds Cisco Certification CCIE (#20672) in Routing and Switching.
Remember to use the rating system to let Vinayak know if you have received an adequate response.
Vinayak might not be able to answer each question due to the volume expected during this event. Remember that you can continue the conversation on the Data Center sub-community, Other Data Center Topics discussion forum shortly after the event.
This event last through Friday July 12, 2013. Visit the community often to view responses to youe questions of other community members.
07-01-2013 01:42 AM
Hi Vnayak,
We have two Nexus 5548 switches along with a coupe of dual homed FEXes. All vlan, portchannel, interface configurations were done in 'conf sync' mode.
We continued like this and faced no problems. However recently i found out that after a change, i tried to verify and commit, however there is an error saying that the peer is not reachable. But i am able to ping both the Nexus switches from each others console via the appropiate vrfs. I noticed that on switch 1 i am able to see the cfs peers, but on switch 2 on giving the command following is the output:-
5548_02# show cfs peers
CFS Discovery is in Progress ..Please wait
Could not get response. The network topology may be under going change.
Please try after about 30 seconds
5548_02#
{{ Above output takes some time to display}}
I am not able to find much information about this. I disabled CFSoIP and re enabled it. Still same result. I am actually stuck here. The other things to do would be delete the switch profile and add back again or may be reload the switch.
Any help would be appreciated.
The NX OS is:-
BIOS: version 3.6.0
loader: version N/A
kickstart: version 5.2(1)N1(2a)
system: version 5.2(1)N1(2a)
power-seq: Module 1: version v1.0
Module 3: version v2.0
On switch 1 peers are showing up as normal:-
5548_01# sh cfs peer
Physical Fabric
-------------------------------------------------------------------------
Switch WWN IP Address
-------------------------------------------------------------------------
20:00:54:7f:ee:b7:c2:80 172.16.1.53 [Local]
5548_01
20:00:54:7f:ee:b6:99:c0 172.16.1.51
20:00:54:7f:ee:b6:9c:00 172.16.1.52
20:00:54:7f:ee:b6:3f:80 172.16.1.54
Total number of entries = 4
Any feed back would be appreciated.
Thanks in advance!!.
Regards
07-01-2013 07:47 AM
Hello Sonu
I have few initial questions on your query.
- Is switch 2 vpc primary or secondary ?
- Are these switches configured for FC/FCoE ?
- Were these switches upgraded non-disruptively in the past ?
- How are the mgmt ports physically connected ? Are they back-to-back or connecting to other 5k/2k switch ?
- On Switch 2 are you able to perform 'show run', 'config t' and other tasks without any problems ?
Can you send output of
show cfs lock
show system internal csm info trace
show cfs internal ethernet-peer database
- 'show cfs peers' output on Switch 2 was it working correctly before this issue happened? Do you know ?
- I have seen before that sometimes it would take time for that command output to be displayed but in our case Switch 2 could be having issues syncing with Switch1.
Thanks, Vinayak
07-01-2013 08:53 AM
Hi Vinayak,
Thanks for the reply.
As of now i do not have access to the device, but i will answer your queries.
Is switch 2 vpc primary or secondary ?
Vpc secondary
-Are these switches configured for FC/FCoE ?
No
Were these switches upgraded non-disruptively in the past ?
Yes, the upgrade went smoothly
On Switch 2 are you able to perform 'show run', 'config t' and other tasks without any problems?
Yes
How are the mgmt ports physically connected ? Are they back-to-back or connecting to other 5k/2k switch ?
The management ports of both Nexus 5548 switches are connected to a 3750X stack switch in an appropriate vlan
Since i do not have access to switches, i am not in a position to send the outputs but from memory the "show cfs lock" shows no output i.e. no pending locks
'show cfs peers' output on Switch 2 was it working correctly before this issue happened? Do you know ?
Yes "show cfs peers" command was giving proper output before.
Everything was working properly. Ping from one switch to another to peer IP is working properly via appropriate vrf.
I will send the output of the other commands as soon as i get them.
We have about 40 live servers connected to these 5548/FEXes.
Regards.
07-01-2013 11:47 PM
Hi Vinayak,
Output of "show cfs internal ethernet-peer database"
Switch 1
ETH Fabric
-------------------------------------------------------------------------
Switch WWN logical-if_index
-------------------------------------------------------------------------
20:00:54:7f:ee:b7:c2:80 [Local]
20:00:54:7f:ee:b6:3f:80 16000005
Total number of entries = 2
Switch 2
ETH Fabric
-------------------------------------------------------------------------
Switch WWN logical-if_index
-------------------------------------------------------------------------
20:00:54:7f:ee:b6:3f:80 [Local]
20:00:54:7f:ee:b7:c2:80 16000005
Total number of entries = 2
Output of "show system internal csm info trace"
Switch 1 in which "show cfs peers" show proper output
Mon Jul 1 05:46:19.145339 (CSM_T) csm_sp_buf_cmd_tbl_expand_range(8604): No range command in buf_cmd_tbl.
Mon Jul 1 05:46:19.145280 (CSM_T) csm_set_sync_status(6257): Peer RT status PSSed
Mon Jul 1 05:46:19.145188 (CSM_T) csm_sp_handle_local_verify_commit(4291):
Mon Jul 1 05:46:19.145131 csm_continue_verify_ac[597]: peer is not reachable over CFS so continuing with local verify/commit
Mon Jul 1 05:46:19.145071 csm_tl_lock(766): Peer information not found for IP address: '172.16.1.54'
Mon Jul 1 05:46:19.145011 csm_tl_lock(737):
Mon Jul 1 05:46:19.144955 (CSM_EV) csm_sp_build_tl_lock_req_n_send(941): sending lock-request for CONF_SYNC_TL_SESSION_TYPE_VERIFY subtype 0 to Peer ip = (172.16.1.54)
Mon Jul 1 05:46:19.143819 (CSM_T) csm_copy_image_and_internal_versions(788): sw_img_ver: 5.2(1)N1(2a), int_rev: 1
Mon Jul 1 05:46:19.143761 (CSM_T) csm_sp_get_peer_sync_rev(329): found the peer with address=172.16.1.54 and sync_rev=78
Mon Jul 1 05:46:19.143699 (CSM_T) csm_sp_get_peer_sync_rev(315):
Mon Jul 1 05:46:19.143641 (CSM_EV) csm_sp_build_tl_lock_req_n_send(838): Entered fn
Mon Jul 1 05:46:19.143582 (CSM_T) csm_set_sync_status(6257): Peer RT status PSSed
Switch 2 in which "show cfs peers" does not show proper output
Mon Jul 1 06:13:11.885354 (CSM_ERR) csm_pss_cmd_tree_walk_cb(2057): Parent command not found for cmd switchport mode trunk, cmd pseq 77 seq 482
Mon Jul 1 06:13:11.884992 (CSM_ERR) csm_pss_cmd_tree_walk_cb(2057): Parent command not found for cmd channel-group 51 mode active, cmd pseq 357 seq 369
Mon Jul 1 06:13:11.884932 (CSM_ERR) csm_pss_cmd_tree_walk_cb(2057): Parent command not found for cmd switchport trunk allowed vlan 2, 11, cmd pseq 357 seq 368
Mon Jul 1 06:13:11.884872 (CSM_ERR) csm_pss_cmd_tree_walk_cb(2057): Parent command not found for cmd switchport mode trunk, cmd pseq 357 seq 367
Mon Jul 1 06:13:11.884811 (CSM_ERR) csm_pss_cmd_tree_walk_cb(2057): Parent command not found for cmd description process_vpc, cmd pseq 357 seq 366
Mon Jul 1 06:13:11.884750 (CSM_ERR) csm_pss_cmd_tree_walk_cb(2057): Parent command not found for cmd channel-group 51 mode active, cmd pseq 352 seq 365
Mon Jul 1 06:13:11.884690 (CSM_ERR) csm_pss_cmd_tree_walk_cb(2057): Parent command not found for cmd switchport trunk allowed vlan 2, 11, cmd pseq 352 seq 364
Mon Jul 1 06:13:11.884630 (CSM_ERR) csm_pss_cmd_tree_walk_cb(2057): Parent command not found for cmd switchport mode trunk, cmd pseq 352 seq 363
Mon Jul 1 06:13:11.884568 (CSM_ERR) csm_pss_cmd_tree_walk_cb(2057): Parent command not found for cmd description process_vpc, cmd pseq 352 seq 362
Mon Jul 1 06:13:11.884207 (CSM_EV) csm_sp_acfg_gen_handler(3011): Preparing config into /tmp/csm_sp_acfg_1733916569.txt
Mon Jul 1 06:13:11.878695 csm_get_locked_ssn_ctxt[539]: Lock not yet taken.
Mon Jul 1 06:13:11.878638 (CSM_EV) csm_sp_acfg_gen_handler(2937): Recieved sp acfg merge request for type: running cfg
Mon Jul 1 06:12:29.527840 (CSM_T) csm_pss_del_seq_tbl(1989): Freeing seq tbl data
Mon Jul 1 06:12:29.513255 (CSM_T) csm_sp_acfg_gen_handler(3106): Done acfg file write
Mon Jul 1 06:12:29.513179 (CSM_EV) csm_sp_acfg_gen_handler(3011): Preparing config into /tmp/csm_sp_acfg_1733911262.txt
Mon Jul 1 06:12:29.508859 csm_get_locked_ssn_ctxt[539]: Lock not yet taken.
Mon Jul 1 06:12:29.508803 (CSM_EV) csm_sp_acfg_gen_handler(2937): Recieved sp acfg merge request for type: running cfg
Mon Jul 1 05:53:17.651236 Collecting peer info
Mon Jul 1 05:53:17.651181 Failed to get the argumentvalue for 'ip-address'
Mon Jul 1 05:40:59.262736 DB Unlocked Successfully
Mon Jul 1 05:40:59.262654 Unlocking DB, Lock Owner Details:Client:1 ID:1
Mon Jul 1 05:40:59.262570 (CSM_T) csm_sp_del_buf_cmd(1713): Deleting comand with Id = 1
Mon Jul 1 05:40:59.262513 DB Lock Successful by Client:1 ID:1
Mon Jul 1 05:40:59.262435 Recieved lock request by Client:1 ID:1
Mon Jul 1 05:40:41.741224 ssnmgr_ssn_handle_create_get: Session FSM already present, ID:1
Mon Jul 1 05:40:41.741167 ssnmgr_handle_mgmt_request: Create/Get request received for session[process_n5kprof]
show cfs lock gives no output.
Just to further clarify, we have 4 5548UP switches in the same management vlan. 2 switches are in one location lets say location A and they are CFS peers and are working fine.
These two switches which are having problem are in location B. All the switches are in the same vlan. Essentially the all CFS multicast messages will be seen by all 5548 switches as they are in the same vlan. I am assuming that this might not create any problems as we specify the peers in the respective configurations. Or do we have to change the CFSoIPv4 multicast addresses in location B or may be configure a different region.
Regards.
07-02-2013 09:03 AM
Thanks, will analyze this and get back to you. In the meantime can you also paste output of
'show switch-profile status'
after you make the change and send it to me?
Thanks, Vinayak
07-02-2013 09:16 AM
Hi Vinayak,
Switch 1 in whcih show cfs peers are showing
Profile-Revision: 85
Session-type: Commit
Session-subtype: -
Peer-triggered: No
Profile-status: Sync Success
Local information:
----------------
Status: Commit Success
Error(s):
Peer information:
----------------
IP-address: 172.16.1.54
Sync-status: Not yet merged
Merge Flags: pending_merge:1 rcv_merge:1 pend_validate:0
Status: Peer not reachable
Error(s):
Profile-Revision: 75
Session-type: Initial-Exchange
Session-subtype: Init-Exchange-All
Peer-triggered: No
Profile-status: -
Switch 2 in which show cfs peers are not shown
I tried a copy run start in the switch 2 but it got stuck and following output taken after that.
Local information:
----------------
Status: Verify Failure
Error(s): CFS send failure: [Failure: Lock failed]
Peer information:
----------------
IP-address: 172.16.1.53
Sync-status: Not yet merged
Merge Flags: pending_merge:1 rcv_merge:1 pend_validate:0
Status: -
Error(s):
Regards
07-02-2013 10:54 AM
Sonu
1) Do you have a large number of switches on the mgmt network ?
2) Have you used cfs ipv4 mcast-address config on the peer switches
Multicast address configuration will help in making sure only the 2 peer switches are in the CFS domain for config-sync.
http://www.cisco.com/en/US/docs/switches/datacenter/nexus5000/sw/configuration/guide/cli/cfs.html
3) Can you remove "cfs ipv4 distribute" config and re-config on the switch where the peer is not seen and check if this helps to resolve the issue?
Thanks, Vinayak
07-03-2013 01:38 AM
Hi Vinayak,
I am refering to the thread:https://supportforums.cisco.com/message/3980732#3980732.
Question:
I am planning an ISSU upgrade on my nexus 7000. On this chassis there are also Nexus 2000 connected. Since they don't support vpc, all fexes are connected to a single chassis. (but different card).
The chassis will upgrade without interruption, but what about the fexes? They will also need the new NX-OS. Do they reboot after or during the upgrade? Or is it without interruption like the 7000 chassis?
Kindly let me know your opinion on this pleasE/
Thanks,
Fari.
07-03-2013 01:13 PM
Hi Fari
If the 7k chassis support ISSU, then we should be able to do hitless upgrade on the 2k as well. During ISSU it should not reload your fexes.
Thanks, Vinayak
07-04-2013 12:32 AM
Hi Vinayak,
If I have the following configuration on the router at the moment, how do you migrate the configuration to the nexus 5000?
int vlan 10
ip address 172.31.10.3 255.255.255.0
int vlan 20
ip address 172.31.20.3 255.255.255.0
int vlan 30
ip address 10.10.30.3 255.255.255.0
router rip
version 2
passive-interface default
no passive-interface vlan20
network 10.0.0.0
network 172.31.0.0
no auto-summary
redistribute static route-map test
route-map test permit 10
match ip address 10
ip access-list 10
permit 0.0.0.0
deny any
TIA
PF
07-05-2013 06:39 AM
Hi PF
You should definitely go through our Nexus 5000 Unicast Configuration Guide on Cisco.com as it will assist you in understanding how to configure RIP on Nexus switch and how to configure for route redistribution. Also for Route-Maps we have "Configuring Route Policy Manager" guide which can assist you in configuring for Route-Maps.
"Configuring RIP"
http://www.cisco.com/en/US/docs/switches/datacenter/nexus5000/sw/unicast/521_N11/l3_rip.html
"Configuring Route Policy Manager"
http://www.cisco.com/en/US/docs/switches/datacenter/nexus5000/sw/unicast/521_N11/l3_rpm.html
07-08-2013 01:56 AM
Vinayak,
Thanks very much. The link provides you with the pieces of config for the RIP. What I need is an example of the RIP configuration by putting the required commands together for a working RIP. If I have the following configuration below how is that being achieved? An example will gives a better understanding of the RIP configuration on the Nexus and helps to consolidate the information/understanding I have.
TIA
PF
I also have a ethernet link between 2 data centres. For example
Config for DC B
int gi1/1
desc link to DC A
switchport access vlan 10
int vlan 10
ip address 172.31.10.3 255.255.255.0
int vlan 20
ip address 172.31.20.3 255.255.255.0
int vlan 30
ip address 10.10.30.3 255.255.255.0
router rip
version 2
passive-interface default
no passive-interface vlan20
network 10.0.0.0
network 172.31.0.0
no auto-summary
redistribute static route-map test
route-map test permit 10
match ip address 10
ip access-list 10
permit 0.0.0.0
deny any
07-10-2013 01:18 PM
Ok I can assist you with building an RIP example config based on what you already have.
The document I provided before would have helped you gather the below information which would have been useful to build a configuration which I pasted below.
- passive interface command is configured under vlan SVI.
- no support for automatic route summarization.
- no network statements definition under router rip.
- redistribute and route-map commands are similar.
Based on the above, following is the config.
int e1/1
desc link to DC A
switchport access vlan 10
feature rip
feature interface-vlan
int vlan 10
ip address 172.31.10.3 255.255.255.0
ip rip passive-interface
int vlan 20
ip address 172.31.20.3 255.255.255.0
int vlan 30
ip address 10.10.30.3 255.255.255.0
ip rip passive-interface
router rip abc
address-family ipv4 unicast
redistribute static route-map test
route-map test permit 10
match ip address 10
ip access-list 10
permit ip any any
You can configure similarly for other site.
Hope this helps,
Thanks,
Vinayak
07-11-2013 01:55 AM
Vinayak,
Thanks so much for your reply. It is very much appreciated.
2 more questions.
1. Don't you need " ip router ip abc" defined on the interface? For exmaple
int vlan 10
ip address 172.31.10.3 255.255.255.0
ip rip passive-interface
ip router rip abc <-- ???
int vlan 20
...
ip router rip abc
2. If the Nexus 5500 and 2000 are used (no other switches), 5500 as core switches, can I joint the 2 mgmt 0 interfaces on both the 5500 (peer switches) together?
Thanks again.
PF
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide