Ask the Expert: Nexus 5000, 3000 and 2000 Series - Page 4

ciscomoderator · ‎10-07-2011

With Lucien Avramov

Welcome to the Cisco Support Community Ask the Expert conversation. This is an opportunity to learn about design, configuration and troubleshooting of the Cisco Nexus 2000, 3000, 5000 Series with cisco Expert Lucien Avramov. Lucien Avramov is a technical marketing engineer in the Server Access Virtualization Business Unit at Cisco, where he supports the Cisco Nexus 5000, 3000 and 2000 Series. He has several industry certifications including CCIE #19945 in Routing and Switching, CCDP, DCNIS, and VCP #66183

Remember to use the rating system to let Lucien know if you have received an adequate response.

Lucien might not be able to answer each question due to the volume expected during this event. Remember that you can continue the conversation on the Data Center sub-community discussion forum shortly after the event. This event lasts through October 21, 2011. Visit this forum often to view responses to your questions and the questions of other community members.

Lucien Avramov · ‎10-17-2011

Okay, so when your images match, the next step is to clear the temp space.

This would happen with a reload. In any case your upgrade will be disruptive so I will suggest you, when you have your maintenance window, to first reload the switch and then run the install all.

Providing the show system internal flash will help confirm.

smartboy2255 · ‎10-18-2011

HI Lucien ,

pls find the logs

sh system internal flash

Mount-on 1K-blocks Used Available Use-percent Filesystem
/                            204800    131672     73128 65   /dev/root
/proc                             0         0         0   0   proc
/post                          2048         4      2044   1   none
/sys                              0         0         0   0   none
/isan                        409600    309452    100148 76   none
/var/sysmgr                  153600     36032    117568 24   none
/var/sysmgr/ftp              204800        28    204772   1   none
/dev/shm                     262144     72344    189800 28   none
/volatile                     20480         0     20480   0   none
/debug                         2048         0      2048   0   none
/dev/mqueue                       0         0         0   0   none
/mnt/cfg/0                    38327      4292     32056 12   /dev/sda5
/mnt/cfg/1                    37814      4292     31570 12   /dev/sda6
/var/sysmgr/startup-cfg       61440      2568     58872   5   none
/dev/pts                          0         0         0   0   devpts
/mnt/plog                     56320      1632     54688   3   /dev/mtdblock2
/mnt/pss                      38343      4443     31920 13   /dev/sda4
/bootflash                   834608    234832    557380 30   /dev/sda3

sh system internal log install

lock_config_mode: Acquiring config mode lock
store_modules_info: index: 0 is present, mod_type: 4 mod_state: 0, sw_card_id: 60
store_modules_info: checking to see if index 0 is fabric switch upgrade supported
store_modules_info: index: 1 is present, mod_type: 0 mod_state: 0, sw_card_id: 65
store_modules_info: index: 2 is present, mod_type: 0 mod_state: 0, sw_card_id: 65
update_srg_table_active_sup_running: check for support_loader flag
check_plugin_img: is_given_image_plugin_based: 1
tar: ./isan/plugin_img/core_plugin: Wrote only 7680 of 10240 bytes
tar: Error exit delayed from previous errors
extract_all_srgs_from_plugins: "/isan/sbin/isanadd -x /bootflash/n5000-uk9.4.0.1a.N2.1.bin . isan/plugin_img" failed
check_plugin_img: Failed to xtract SRG's from plugins within the image: -1
srv_installer_job_done: procjob operation failed. (0x404F0003)
process_srg_extraction_done: SRG extraction failed (0x404F0003).
unlock_config_mode: Releasing config mode lock

sh sys internal log install details

Starting MTS bind
MTS bind successful
MTS now ready
installer_state_init: State initialized to INSTALLER_STATE_INIT.
srv_installer: calling get_card_module_info to find out sup type
******srv_installer: setting support_loader to FALSE*****
process_install_start: got install request: abort_installation (0), install_operation (2).
notify_global_status_change: Sending global status notification.
process_install_start: status 0x00000000.
installer_state_change: old_state: INSTALLER_STATE_INIT, new_state: INSTALLER_STATE_PRE_VERIFICATION
installer_state_change: State changed from INSTALLER_STATE_INIT to INSTALLER_STATE_PRE_VERIFICATION.
lock_config_mode: Acquiring config mode lock
build_bootvar_table: Starting to build the bootvar table.
build_bootvar_table: Total number of bootvars are 5.

store_modules_info: index: 0 is present, mod_type: 4 mod_state: 0, sw_card_id: 60
store_modules_info: checking to see if index 0 is fabric switch upgrade supported
store_modules_info: entity_index 22, mod_num 1
store_modules_info: index: 1 is present, mod_type: 0 mod_state: 0, sw_card_id: 65
store_modules_info: entity_index 23, mod_num 2
store_modules_info: index: 2 is present, mod_type: 0 mod_state: 0, sw_card_id: 65
store_modules_info: entity_index 24, mod_num 3
store_modules_info: entity_index 22, mod_num 1
store_modules_info: entity_index 22, mod_num 1
store_modules_info: entity_index 22, mod_num 1
store_modules_info: entity_index 22, mod_num 1
store_modules_info: entity_index 22, mod_num 1
store_modules_info: entity_index 22, mod_num 1
store_modules_info: entity_index 22, mod_num 1
store_modules_info: entity_index 22, mod_num 1
store_modules_info: entity_index 22, mod_num 1
store_modules_info: entity_index 22, mod_num 1
store_modules_info: entity_index 22, mod_num 1

store_modules_info: entity_index 22, mod_num 1
store_modules_info: entity_index 22, mod_num 1
mod_num 2 mod_status 1 type 0
mod_num 3 mod_status 1 type 0
mod_num 1 mod_status 0 type 0
mod_num 1 mod_status 0 type 0
mod_num 1 mod_status 0 type 0
mod_num 1 mod_status 0 type 0
mod_num 1 mod_status 0 type 0
mod_num 1 mod_status 0 type 0
mod_num 1 mod_status 0 type 0
mod_num 1 mod_status 0 type 0
mod_num 1 mod_status 0 type 0
mod_num 1 mod_status 0 type 0
mod_num 1 mod_status 0 type 0
mod_num 1 mod_status 0 type 0
mod_num 1 mod_status 0 type 0
store_redundancy_status: admin_mode:SYSMGR_REDUN_MODE_HA operational_mode:SYSMGR_REDUN_MODE_NONE.
installer_state_change: old_state: INSTALLER_STATE_PRE_VERIFICATION, new_state: INSTALLER_STATE_NORMALIZATION
installer_state_change: State changed from INSTALLER_
STATE_PRE_VERIFICATION to INSTALLER_STATE_NORMALIZATION.
clear_lost_found_dir: Executing command - cd /bootflash; rm -rf /bootflash/lost+found; mklost+found;
perform_uri_copy: scheme for src_uri "bootflash:/n5000-uk9-kickstart.4.0.1a.N2.1.bin" is "bootflash"
perform_uri_copy: path for src_uri "bootflash:/n5000-uk9-kickstart.4.0.1a.N2.1.bin" is "n5000-uk9-kickstart.4.0.1a.N2.1.bin".
perform_uri_copy: no copy needed for src_uri "bootflash:/n5000-uk9-kickstart.4.0.1a.N2.1.bin".
perform_uri_copy: scheme for src_uri "bootflash:/n5000-uk9.4.0.1a.N2.1.bin" is "bootflash"
perform_uri_copy: path for src_uri "bootflash:/n5000-uk9.4.0.1a.N2.1.bin" is "n5000-uk9.4.0.1a.N2.1.bin".
perform_uri_copy: no copy needed for src_uri "bootflash:/n5000-uk9.4.0.1a.N2.1.bin".
find_next_add_on_image:number of add-on images for module 2 is 0.
find_next_add_on_image:number of add-on images for module 3 is 0.
installer_state_change: old_state: INSTALLER_STATE_NORMALIZATION, new_state: INSTALLER_STATE_IMAGE_VERIFICATION
installer_state_change: State changed from INSTALLER_
STATE_NORMALIZATION to INSTALLER_STATE_IMAGE_VERIFICATION.
perform_image_verification: verify: procjob_handle 24622 syserr "success"
installer_state_change: old_state: INSTALLER_STATE_IMAGE_VERIFICATION, new_state: INSTALLER_STATE_IMAGE_VERIFY_WAIT
installer_state_change: State changed from INSTALLER_STATE_IMAGE_VERIFICATION to INSTALLER_STATE_IMAGE_VERIFY_WAIT.
procjobcb_job_ready: Handshake done: 24622
image_verify_func: successfully able to open libbootverify.so
image_verify_func: successfully able to dlsym.
Imghdr: platform type from hardware is NUOVA
Imghdr: isan image's platform type is NUOVA
Imghdr: boot image's platform type is NUOVA
42fbe4b24a1f677c859dc42bc4d69cd4
Verify OK!!
image_verify_func: rval for verify fn 0.
procjobcb_job_done: Handle 24622 syserr "success" ret_info_size 0 ret_syserr 0 ret_info b7f70540
installer_state_change: old_state: INSTALLER_STATE_IMAGE_VERIFY_WAIT, new_state: INSTALLER_STATE_IMAGE_VERIFICATION
installer_state_change: State changed from INSTALLER_STATE_IMAGE_VERIFY_WAIT to INSTALLER_STATE_IMAGE_VERIFICATION.

perform_image_verification: verify: procjob_handle 24624 syserr "success"
installer_state_change: old_state: INSTALLER_STATE_IMAGE_VERIFICATION, new_state: INSTALLER_STATE_IMAGE_VERIFY_WAIT
installer_state_change: State changed from INSTALLER_STATE_IMAGE_VERIFICATION to INSTALLER_STATE_IMAGE_VERIFY_WAIT.
procjobcb_job_ready: Handshake done: 24624
image_verify_func: successfully able to open libbootverify.so
image_verify_func: successfully able to dlsym.
Imghdr: platform type from hardware is NUOVA
Imghdr: isan image's platform type is NUOVA
Imghdr: boot image's platform type is NUOVA
Verify OK!!
image_verify_func: rval for verify fn 0.
procjobcb_job_done: Handle 24624 syserr "success" ret_info_size 0 ret_syserr 0 ret_info b7f70540
installer_state_change: old_state: INSTALLER_STATE_IMAGE_VERIFY_WAIT, new_state: INSTALLER_STATE_IMAGE_VERIFICATION
installer_state_change: State changed from INSTALLER_STATE_IMAGE_VERIFY_WAIT to INSTALLER_STATE_IMAGE_VERIFICATION.
installer_state_change: old_state: INSTALLER_STATE_IMAGE_VERIFICATION, new_state: INSTALLER_STATE_SRG_COLLECTION

installer_state_change: State changed from INSTALLER_STATE_IMAGE_VERIFICATION to INSTALLER_STATE_SRG_COLLECTION.
clone_local_srg: SRG Size for /isan/etc/srg.sro is 380 bytes
clone_local_srg: SRG Retrieval succeeded for /isan/etc/srg.sro
update_srg_table_active_sup_running: Got Running system SRG for active sup
clone_local_srg: SRG Size for /etc/srg-ks.sro is 244 bytes
clone_local_srg: SRG Retrieval succeeded for /etc/srg-ks.sro
update_srg_table_active_sup_running: Got Running kickstart SRG for active sup
get_bios_ver: calling get_module_bios_version_ext,mod_num 1,ver_len 7464
get_bios_ver: get_module_bios_version_ext returned, num_entries is 1, pri ver_str is v1.0.0(04/01/08), sw_id is 11
update_srg_table_active_sup_running: Got Running bios version for active sup, primary bios: v1.0.0(04/01/08)
update_srg_table_active_sup_running: check for support_loader flag

update_srg_table_active_sup_running: Total number of running swids for active sup are 3
collect_running_srgs: collecting running srg's from lc(1)
update_srg_table_lc_running: Got 0 srg's for module 2
update_srg_table_lc_running: Total number of running swids for lc_mod_index 1 are 0
collect_running_srgs: collecting running srg's from lc(2)
update_srg_table_lc_running: Got 0 srg's for module 3
update_srg_table_lc_running: Total number of running swids for lc_mod_index 2 are 0
fsm_action_srg_collection: running image plugin based ?: 1
fsm_action_srg_collection: checking if tobe installed image is plugin based
check_plugin_img_async: procjob_handle 24626 syserr "success"
waiting for plugin image check to complete .
leaving_srg_collection_state: entering.. procjob_hand
le is 24626 installer_state_change: old_state: INSTALLER_STATE_SRG_COLLECTION, new_state: INSTALLER_STATE_SRG_EXTRACTION_WAIT
installer_state_change: State changed from INSTALLER_STATE_SRG_COLLECTION to INSTALLER_STATE_SRG_EXTRACTION_WAIT.
procjobcb_job_ready: Handshake done: 24626
#######################################
EXTRACT
#######################################
> extracting files: /isan/plugin_img/img.conf
--> /isan/plugin_img/img.conf
#######################################
EXTRACT
#######################################
> extracting files: isan/plugin_img
--> isan/plugin_img
Error extracting the file
extract_all_srgs_from_plugins: Extracting all srgs within plugin(s)from the image "/bootflash/n5000-uk9.4.0.1a.N2.1.bin"
extract_all_srgs_from_plugins: /tmp/extracted-srgs directory exists
extract_all_srgs_from_plugins: clearing /tmp/extracted-srgs directory
extract_all_srgs_from_plugins: Create working dir with pid 24627
extract_all_srgs_from_plugins: Executing cmd "/isan/sbin/isanadd -x /bootflash/n
5000-uk9.4.0.1a.N2.1.bin . isan/plugin_img"
procjobcb_job_done: Handle 24626 syserr "success" ret_info_size 4 ret_syserr 0 ret_info b7f70540
process_srg_extraction_done: success
process_srg_extraction_done: plugin image check returned: 1
installer_state_change: old_state: INSTALLER_STATE_SRG_EXTRACTION_WAIT, new_state: INSTALLER_STATE_SRG_COLLECTION
installer_state_change: State changed from INSTALLER_STATE_SRG_EXTRACTION_WAIT to INSTALLER_STATE_SRG_COLLECTION.
fsm_action_srg_collection: find_next_srg_to_extract returns 1
extract_srg_async: srg extract: procjob_handle 24651 syserr "success"
Waiting for SRG extraction to complete .
extract_srg_async: leaving...
perform_srg_extraction: handle not 0, leaving
leaving_srg_collection_state: entering.. procjob_handle is 24651 installer_state_change: old_state: INSTALLER_STATE_SRG_COLLECTION, new_state: INSTALLER_STATE_SRG_EXTRACTION_WAIT
installer_state_change: State changed from INSTALLER_STATE_SRG_COLLECTION to INSTALLER_STATE_SRG_EXTRACTION_WAIT.

procjobcb_job_ready: Handshake done: 24651
extract_srg_file_func: extracting srg "/isan/etc/srg.sro" from image "/bootflash/n5000-uk9.4.0.1a.N2.1.bin".
search_extracted_srg_dir: Searching SRG "/isan/etc/srg.sro" in "/tmp/extracted-srgs"
extract_srg_file_func: extraction of srg "/isan/etc/srg.sro" from image "/bootflash/n5000-uk9.4.0.1a.N2.1.bin": rval (0x404F0003).
procjobcb_job_done: Handle 24651 syserr "success" ret_info_size 0 ret_syserr 1078919171 ret_info b7f70540
srv_installer_job_done: procjob operation failed. (0x404F0003)
process_srg_extraction_done: SRG extraction failed (0x404F0003).
installer_state_change: old_state: INSTALLER_STATE_SRG_EXTRACTION_WAIT, new_state: INSTALLER_STATE_ABORTING
installer_state_change: State changed from INSTALLER_STATE_SRG_EXTRACTION_WAIT to INSTALLER_STATE_ABORTING.
notify_global_status_change: Sending global status notification.
fsm_action_srv_installer_aborting: Waiting for 10 secs before exiting..
process_install_start: got install request: abort_ins
tallation (0), install_operation (1).
abort_installation: We are in state "INSTALLER_STATE_ABORTING" and got a request for abort_and_exit, going to state DONE.
installer_state_change: old_state: INSTALLER_STATE_ABORTING, new_state: INSTALLER_STATE_DONE
installer_state_change: State changed from INSTALLER_STATE_ABORTING to INSTALLER_STATE_DONE.
unlock_local_module: Trying to unlock module 1
unlock_local_module: Trying to unlock module 2
unlock_local_module: Trying to unlock module 3
unlock_config_mode: Releasing config mode lock

Lucien Avramov · ‎10-18-2011

Your root partition is 65 % full, this is not enough for the upgrade to complete.

The next I suggest you here is to reload to remove all temporary cache files and then run the upgrade.

Keep us posted.

smartboy2255 · ‎10-18-2011

Can you pls tell me how much disk space is required to complete the installation. Also is there any option to see the temp folder usage.

Lucien Avramov · ‎10-18-2011

There is no way to see the details of the root partition on your CLI, you have to follow the reload procedure and then run the install all.

Lucien Avramov · ‎10-17-2011

N7k is compatible with the FEX 2224,2248 and 2232PP. We have now L3 as well on FEX from 5.2 code, meaning you can configure a FEX port as an L3 port if you desire.

Some FEX limitations:

http://www.cisco.com/en/US/docs/switches/datacenter/sw/5_x/nx-os/release/notes/51_nx-os_release_note.html#wp351847

Also here are the release notes for 5.2:

http://www.cisco.com/en/US/docs/switches/datacenter/sw/5_x/nx-os/release/notes/52_nx-os_release_note.html

paulw · ‎10-17-2011

Hi Lucien,

Per the data sheet on the Cisco Nexus 5548 / 5596, the switches support "ACL logging and statistics". The URL is

http://www.cisco.com/en/US/prod/collateral/switches/ps9441/ps9670/data_sheet_c78-618603.html

I would expect to be able to use the "log" keyword in ACLs, as in "permit ip any 10.0.0.2/32 log".

However, the configuration guides don't mention the "log" option in the command syntax. Further, I found this:

https://supportforums.cisco.com/message/3336094#3336094 which implies that logging is not supported.

Can you tell me if logging is or is not actually supported, and if not, if there is any plan to incorporate this in a future release? This is a very useful capability.

Thanks,

Paul

Lucien Avramov · ‎10-18-2011

Thanks for pointing this out. We will clarify the documentation. You are absolutely right on the feature expectation. Today the hardware supports this, however it has not been implemented yet. You can discuss offline the timelines with your account team or contact me offline for more information regarding future plans on this feature.

ovalledw · ‎10-18-2011

Hi Lucien,

Looking at the Nexus 3064 for ultra low latency in our multicast environment, but limited to 64x10Gigports.

We are thinking of creating a port channel between two 3064 using 2x40Gig interfaces as the interconnect to increase port capacity. The question is how much latency will be added between two host, one on each 3064. I understand that latency varies on

packet size, buffer, etc., but lets say the two devices are sending traffic to each other with fixed packet size with a steady stream and no microbursts.

What other solution do you recommend if my environment consist of 70+ servers (need to be on same broadcast domain) and is in need of a ultra low latency solution? <1us

Eddie O.

Lucien Avramov · ‎10-18-2011

In your Ultra Low Latency environment, you mention multicast. Let's take for example packet sizes between 64 and 512 bytes. In this case, 40 GE is actually the best choice. Using 40 GE as aggregation in the following scheme : Topology 1: 10GE server ---nexus 3k---(40GE)--nexus3k---(40GE)---nexus3k----10GE

Topology 2: 10GE server ---nexus 3k---(10GE)--nexus3k---(10GE)---nexus3k----10GE

Topology 1, even though it is store-forward mode for the end to end latency (since there is a 10->40 GE) will save you between .260 usec up to .580 usec for packet sizes between 64 and 512 bytes.

I'd recommend you to have a 40 GE aggregation, this will give you predictable latency and will let you scale to a high number of servers if you need growth.

vinay_verma80 · ‎10-19-2011

Hi Lucien,

I don't know if it is a corrcet forum to ask this questing in this forum or not but I am having a problem at hand and i need few answere based on LAN (VRRP/HSRP) feature.

Actually we are having ES+ cards (on 7609) and we are using multiple groups(say 350 vrrp groups) running on the router . the routers are connected as

router 1>>> mux(which is working as switches)>>> router2

my questing are

1. does their will be "multicast packets" (for VRRP group) "from backup router to Master router" also when in stable state( ie when Master and backup are already chosen) or the packet from backup to master should be usnicast

I know for sure the packet from master to back is multicast packets

I am not sure but I think from backup to master it should be multicast

2. what is frequency of these packets( from backup to master)

3. As i have multiper group on a single interface ( we are using q-in-q), when the connectivity from router's is broken, then does all the groups will muticast their active roll in the lan sengment "at once" or it will be in a groups say 100 groups at once, and after few ms few 100's and sone ( as is on OSPF or RIP)

we are in between troubleshooting I hope we get the ans

Thanks in advance

Warm regards

Vinay Verma

Lucien Avramov · ‎10-19-2011

Please try posting your question on the LAN forum:

https://supportforums.cisco.com/community/netpro/network-infrastructure/switching

Luciano Vigano' · ‎10-19-2011

Ciao Lucien,

Are you aware about the NX-OS version 6 and when it will be released?

I know that this question is not pertinent with this section, but I'm studing a solution for a customer and it will be a great thing insert in the solution (it's a 7010) the new Fabric2 and N7K-F248XP-25 linecard (which is not clear about the L2/L3 support and other things, as per my request in another post)... but both of them are supported with NX-OS 6 :-(

Another question (this one for the N2K-C2148T-1GE): if I understand correctly this device can connect to two N5K in these ways:

via vPC (dual homed) using at max 1 link to first N5K and another one to the second N5K (so a vPC with 2 x 10Gb) and the connection to a server must be active\standby
via a PO (single homed) to a N5K but the link can be done using at max 4 link (PO with 4 X 10Gb) and the conenction to a server could be active\active via vPC

Is this right?

Tnx in advance if u can help!

Ciao!

Luciano

Lucien Avramov · ‎10-19-2011

Thanks for your question. I can not publish on this forum the release timeframes for 6.0 NX-OS and the best contact point for you to get this information is your Cisco Account Team.

Regarding vPC:

1) You can use with 2148T up to, 2 links going to N5k-1 and 2 other links going to N5K-2 : all the 4 uplinks can therefore be used. All those links can be in a PO which is a ''vPC".

2) Sure that is correct

In short, on any fabric extenders (2148,2200s..) you can use all the uplinks if you desire, the minimum is 1 uplink.

Here is how to configure either case (look at figure 4 and figure 5.):

http://www.cisco.com/en/US/prod/collateral/switches/ps9441/ps9670/configuration_guide_c07-543563.html

Luciano Vigano' · ‎10-19-2011

Ciao Lucien,

tnx for the info but I'm a little confused (maybe it's due to my poor english ), for case 1 (vPC) it's stated:

"The Cisco Nexus 2148T supports two-link PortChannels, with a maximum of one port per fabric extender. The Cisco Nexus 2148T thus supports only vPCs with a total of two ports. Figure 23 illustrates the topologies that the Cisco Nexus 2148T supports"

on page 32 of "Chapter 6 Data Center Access Design with Cisco Nexus 5000 Series Switches and 2000 Series Fabric Extenders and Virtual PortChannels"

Ciao!

Luciano