Nexus 5548 rebooting continually after installing L3 Base License

[randerson]

Here is the situation - we recently purchased L3 daughter cards (N55-D160L3-V2) for our 5548UPs along with the LAN Enterprise Services licenses (

N55-LAN1K9).

I installed the cards and Ent Svcs lic this past weekend and everything went well excep that I couldn't enable the EIGRP feature .. apparently EIGRP requires the LAN Base (N55-BAS1K9-BUN) license even when you have the Enterprise lic installed, so I was able to dig up that license eventually (which comes with the L3 daughter card, incidentally). Another side effect of not installing the Lan Base lic is that the L3 card/module will be in an "offline" state until the Lan Base lic is installed. Either way, everything was working normally in L2 mode at this time.

So, here I was thinking I was ready to proceed with enabling L3 - this morning I installed the LAN Base license on one of the 5548s which apparently triggered the L3 Daughter card module to go active and triggered a switch reboot. Ok, not that big of a deal, knew that was probably coming. I expected the switch to come back up normally .. NOPE. The switch went into a continual reboot cycle where the console prompt would appear, then about 30 seconds later, would reboot again.

I finally ended up re-installing the original L2 card and the switch came right up. I see that LAN Base and LAN Enterprise are both still installed as well.

What have I done wrong here?  Why did the switch go into a continuous reboot cycle once I installed the LAN Base license?

Thanks in advance!

Here are more details :

Software

  BIOS:      version 3.5.0

  loader:    version N/A

  kickstart: version 5.0(3)N1(1c)

  system:    version 5.0(3)N1(1c)

  power-seq: Module 1: version v1.0

  uC:        version v1.2.0.1

  SFP uC:    Module 1: v1.0.0.0

  BIOS compile time:       02/03/2011

  kickstart image file is: bootflash:/n5000-uk9-kickstart.5.0.3.N1.1c.bin

  kickstart compile time:  5/4/2011 3:00:00 [05/04/2011 06:43:31]

  system image file is:    bootflash:/n5000-uk9.5.0.3.N1.1c.bin

  system compile time:     5/4/2011 3:00:00 [05/04/2011 08:35:20]

!

!

Feature                      Ins  Lic   Status Expiry Date Comments

                                 Count

--------------------------------------------------------------------------------

FM_SERVER_PKG                 No    -   Unused             -

ENTERPRISE_PKG                No    -   Unused             -

FC_FEATURES_PKG               No    -   Unused             -

LAN_BASE_SERVICES_PKG         Yes   -   In use Never       -

LAN_ENTERPRISE_SERVICES_PKG   Yes   -   Unused Never       -

--------------------------------------------------------------------------------

!

!

Mod Ports  Module-Type                      Model                  Status

--- -----  -------------------------------- ---------------------- ------------

1    32     O2 32X10GE/Modular Universal Pla N5K-C5548UP-SUP        active *

3    0      O2 Non L3 Daughter Card          N55-DL2                ok     <--- *** had to reinstall the original L2 card in order to get switch to boot correctly***

Mod  Sw              Hw      World-Wide-Name(s) (WWN)

---  --------------  ------  --------------------------------------------------

1    5.0(3)N1(1c)    1.0     --

3    5.0(3)N1(1c)    1.0     --

[Leo Laohoo]

Hi Ross,

Can you plug your console cable back in?

I would like to see the entire bootup process?

I want to know what's causing it.

If you un-install or remove the Layer 3 daughter card, will the chassis still go into a reboot?  If it doesn't, make sure you check the flash to ensure the flash is not filled with crashinfo files/directories.

[randerson]

Thanks for the response.

Unfortunately I can not reinstall the L3 card at the moment as I can not reboot during the day. Plus, the logs get cleared after a reboot so I do not have a copy of the messages - I had a console cable connected while the reboots were happening though and I saw a bunch of ominous messages, which likely would shed light on the problem - I just didn't have a chance to write them down as I was trying to bring the switch up asap.

I do have a maintenance window this weekend and will attempt to recreate the issues. We have (2) 5548s (we'll call them A and B) and one (A) now has the original L2 card (this was the switch that started rebooting continually after installing the LAN Base so I put the original L2 card back in - that allowed the switch to boot again) and the other one (B) has the new L3 card with Enterprise Services license installed - I will be installing the Lan Base license on (B) and see if the exact same continous reboots happen again. I noted that even though the L3 card and Ent Svcs lic were installed, the L3 card was still offline since the Lan Base Svcs lic had not yet been installed - this is why I didn't see any issues until I installed the Lan Base lic, which in turn "activates" the L3 module.

Mainly, I need to know if there is something I am missing here - does our IOS (version 5.0(3)N1(1c)) not work with our L3 daughter card (N55-D160L3-V2)? Do you have to install the licenses in a particular order? Is there a step I am missing?

I thought it would be pretty straightforward .. turn off switch, install L3 card, power up, install Lan Base and Lan Enterprise Svcs lics, enable new features (hsrp, interface_vlan, eigrp), put feet up and drink cold one. Alas, it has not worked out like that.

Thanks again for any help!!

[Leo Laohoo · Accepted solution]

I had a console cable connected while the reboots were happening though and I saw a bunch of ominous messages, which likely would shed light on the problem

I'd be very interested to see this.

Mainly, I need to know if there is something I am missing here - does our IOS (version 5.0(3)N1(1c)) not work with our L3 daughter card (N55-D160L3-V2)?

Hmmm ... good point.  You could try either 5.1(3)N2(1a) or 5.2(1)N1(1).

[randerson]

UPDATE

I tried to enable the LAN Base license on our secondary 5548 and the same problem happened - continuous reboot. However, this time I got a log of what is happening .. (see full log below)

Here is what is happening :

1) L3 daughter card is installed (N55-D160L3-V2) and no licenses are present; L3 card is in "offline" status

2) I install the LAN Base license - L3 module goes into "powered-up" state

3) Switch reboots about 10 seconds later after posting this message :

2012 Aug  5 08:12:07 ATL5548-DC02 %$ VDC-1 %$ %SYSMGR-2-SERVICE_CRASHED: Service "bcm_usd" (PID 4571) hasn't caught signal 11 (core will be saved).

Shutdown Ports.. writing reset reason 16, bcm_usd hap reset

Broadcast me

INIT: Sending processes the TERM signalAug  5 08:12:13 %LIBSYSMGR-3-SIGTERM_FORCE_EXIT Service "sunnyvale" (PID 4548) is forced exit.

Aug  5 08:12:13 %STP-2-PROFILE SIGNAL Handler:: crossed (0 secs), elapsed (0 secs 2 usecs), usr (0 secs  0 usecs), sys (0 secs 0 usecs) : 

Aug  5 08:12:13 %LIBSYSMGR-3-SIGTERM_FORCE_EXIT Service "statsclient" (PID 4590) is forced exit.

4) Once the switch has rebooted it will show the console login prompt for about 10 seconds and then reboots again with the same messages

So I have a predicament -- what am I doing wrong? Does the V2 daughter card require a newer IOS? The only way to get the switch back up is to re-install the original L2 daughter card.

PLEASE HELP!!!

TIA - sorry for the huge log.

[Jerry Ye]

Yes, Leo gave you the answer already.  The N55-D160L3-V2 requires NX-OS 5.1(3)N1(1) or above.

HTH,

jerry

[randerson]

Thanks everyone - I discovered that same thing this morning and have downloaded 5.1. Hopefully I will be able to upgrade sometime this week. I'll post an update after upgrading.

Thanks again!!

[randerson]

Just wanted to close out this discussion. Once NX-OS 5.1 was loaded and the 5548UP rebooted, the switch (and L3 V2 card) started functioning normally.

[Leo Laohoo]

Thanks Ross.

Just want to let you know that NX-OS 5.0 and 5.1 is known as the "short life" NX-OS.  These two will not have long to go.

The "long life" NX-OS is the 5.2 line.

[randerson]

Thanks again leolahoo -- in general I don't like to be on the very latest OS but I've 5.2 ready to go for when we do decide to upgrade, probably in a couple of months I suppose.

[Leo Laohoo]

 in general I don't like to be on the very latest OS

Don't want to pick an argument but NX-OS 5.0 and 5.1 is known, in the old tongue, as the "M" and "T" train respectively.  "M" train is known as the Maintenance Release and the "T" train is known as the Technology Release.

The NX-OS 5.2 long-life is known as the General Release.  This will be the version where all the "un-tamed" 5.0 and 5.1 will be improved and "tamed".

[besnik baftijari]

Hi there,

How did you go on installing the license, where did you get the license and steps please on how you did it? 

thanks 

[neil_titchener]

I've just done the very same thing.  Cisco provided an RMA with a switch running nxos 5.0 I've installed the base and enterprise licenses and now I'm seeing continual reboots exactly as described Randerson.  i'll update once we have a solution.  I'll start by doing what been suggested here.

thanks for the great information in this forum, hopefully it will resolve my issue.

[neil_titchener]

Hi All,  I recovered the system following Randersons guidelines.  Interestingly we are using Opengear console switches and the break sequence we used to holt to reboots was ctl/Shift/r.