6509 stuck in Rommon mode

[pardeep_verma]

   Hi all,

Any one can help , as i have 6509 E.  Actually what happen it last 3-4 times it reload its self and got stuck in rommon mode, i tried to boot it with boot by connecting a console cable on supervisor 720 2b.

Anyone can help me exact what amd where is problem and why is stuck in rommon after reload.

With Regards

Pardeep Kumar
( Rate please if it is helpful )

[mahmoodmkl]

HI

Can u do a dir sup-bootflash and check if u have any valid image on it.

Thanks

[pardeep_verma]

Yes i have a image in it.

s72033-adventerprisek9_wan-vz.122-33.SXI6.bin

along with crash information

2  -rw-    33554432  May 14 2011 15:56:02 +00:00  sea_console.dat

    3  -rw-    16646144  May 15 2011 00:47:46 +00:00  sea_log.dat

    4  -rw-      401517  Nov 22 2012 08:21:50 +00:00  crashinfo_ios-base-20121122-082144

    5  -rw-      324960  Nov 22 2012 08:13:24 +00:00  crashinfo_ios-base-20121122-081319

    6  -rw-      324873  Nov 22 2012 08:19:58 +00:00  crashinfo_ios-base-20121122-081954

    7  -rw-      324252  Nov 22 2012 08:26:34 +00:00  crashinfo_ios-base-20121122-082629

    8  -rw-      381629  Nov 26 2012 11:18:36 +00:00  crashinfo_ios-base-20121126-111831

    9  -rw-      319455  Dec 15 2012 09:23:48 +00:00  crashinfo_ios-base-20121215-092343

   10  -rw-      319007  Dec 15 2012 10:02:00 +00:00  crashinfo_ios-base-20121215-100155

   11  -rw-      323885  Dec 15 2012 10:11:40 +00:00  crashinfo_ios-base-20121215-101134

   12  -rw-      319322  Dec 15 2012 12:58:40 +00:00  crashinfo_ios-base-20121215-125836

   13  -rw-      319346  Dec 15 2012 14:47:48 +00:00  crashinfo_ios-base-20121215-144744

   14  -rw-      319052  Dec 16 2012 08:08:04 +00:00  crashinfo_ios-base-20121216-080759

With Regards
Pardeep Kumar
( Rate please if it is helpful )

[Raju Sekharan]

Upload the crashinfo to find the reason for crash

more sup-bootflash:crashinfo_ios-base-20121216-080759

Thanks

Raju

[Raju Sekharan]

Paste the following outputs

Is there a crashinfo on the router? That will help to find the cause of reload

Paste the following outputs

1. Show bootvar

2. show ver

3. Remote command switch show ver

4. remote command switch show nvlog

5. Show fabric channel-counters

Thanks

Raju

[pardeep_verma]

Thanks raju.........

[pardeep_verma]

one more things i wanna add after by running this command

redundancy force-switchover

Secondary become active but primary goes in rommon mode again

With Regards
Pardeep Kumar
( Rate please if it is helpful )

[Raju Sekharan]

Hi

SUP is going to rommon because the active SUP was crashing continuously

29. 12/16/2012 08:37:43: FC_pf_redun_active_init_crash_handle:Active has crashed 3 times, dropping to rommon for analysis

So if you fix the crash, your both issues will get resolved.

Please upload the crashinfo to analyze the crash

Thanks

Raju

[pardeep_verma]

more bootflash:crashinfo_ios-base-20121126-111831

please find the attached logs

With Regards
Pardeep Kumar
( Rate please if it is helpful )

[Raju Sekharan]

Hi

The crashinfo you have uploaded is if 26th Nov.

I can see the route reproting lot of fabric errors

Nov 26 11:14:50.675: %FABRIC-SP-3-SYNC_FAILED: the module in slot 2 could not synchronize with the standby fabric

--More--         Nov 26 11:14:51.295: %OIR-SP-3-PWRCYCLE: Card in module 6, is being power-cycled 'off (Fabric channel errors)'

Nov 26 11:14:51.295: %OIR-SP-3-PWRCYCLE: Card in module 6, is being power-cycled 'Power cycle'

Nov 26 11:14:52.260: %PFREDUN-SP-6-ACTIVE: Standby processor removed or reloaded, changing to Simplex mode

Nov 26 11:14:54.007: %SNMP-5-MODULETRAP: Module 6 [Down] Trap

Nov 26 11:16:07.358: %SNMP-5-MODULETRAP: Module 2 [Down] Trap

Nov 26 11:16:07.356: %C6KPWR-SP-4-DISABLED: power to module in slot 2 set off (Fabric channel errors)

Please send the following outputs

1, Show fabric channel-counters

2. Show module

[pardeep_verma]

HI.....

#sh module

Mod Ports Card Type                              Model              Serial No.

--- ----- -------------------------------------- ------------------ -----------

  1   48  CEF720 48 port 10/100/1000mb Ethernet  WS-X6748-GE-TX     SAL151194ZK

  2   48  CEF720 48 port 10/100/1000mb Ethernet  WS-X6748-GE-TX     SAL15108FRM

  5    2  Supervisor Engine 720 (Hot)            WS-SUP720-3B       SAL1519D736

  6    2  Supervisor Engine 720 (Active)         WS-SUP720-3B       SAL1514ARJS

Mod MAC addresses                       Hw    Fw           Sw           Status

--- ---------------------------------- ------ ------------ ------------ -------

  1  c89c.1d2f.64e8 to c89c.1d2f.6517   4.1   12.2(18r)S1  12.2(33)SXI6 Ok

  2  503d.e5cd.74b0 to 503d.e5cd.74df   4.1   12.2(18r)S1  12.2(33)SXI6 Ok

  5  588d.098a.57e0 to 588d.098a.57e3   5.12  8.5(4)       12.2(33)SXI6 Ok

  6  588d.098a.3d38 to 588d.098a.3d3b   5.12  8.5(4)       12.2(33)SXI6 Ok

Mod  Sub-Module                  Model              Serial       Hw     Status

---- --------------------------- ------------------ ----------- ------- -------

  1  Distributed Forwarding Card WS-F6700-DFC3B     SAL1518CTZX  4.9    Ok

  2  Distributed Forwarding Card WS-F6700-DFC3B     SAL1518CTZ8  4.9    Ok

  5  Policy Feature Card 3       WS-F6K-PFC3B       SAL1519D6LT  2.6    Ok

  5  MSFC3 Daughterboard         WS-SUP720          SAL1518D4J2  5.1    Ok

  6  Policy Feature Card 3       WS-F6K-PFC3B       SAL1514ART2  2.6    Ok

  6  MSFC3 Daughterboard         WS-SUP720          SAL1514AKTT  5.1    Ok

Mod  Online Diag Status

---- -------------------

  1  Pass

  2  Pass

  5  Pass

  6  Pass

# sh fabric channel-counters

slot channel   rxErrors   txErrors    txDrops  lbusDrops

    1       0          0          0          0          0

    1       1          0          0          0          0

    2       0          0          0          0          0

    2       1          0          0          0          0

    5       0          0          0          0          0

    6       0          0          0          0          0

[Arumugam Muthaiah · Accepted solution]

Hi Pradeep,

I decoded your 26th Nov crashinfo file and it looks very genric and I would like to inform you that that modular IOS which you have currently installed is not developing anymore and no bug fixes.

You could share the other two or three crashinfo file to reconfirm this behavior,

12  -rw-      319322  Dec 15 2012 12:58:40 +00:00  crashinfo_ios-base-20121215-125836

13  -rw-      319346  Dec 15 2012 14:47:48 +00:00  crashinfo_ios-base-20121215-144744

14  -rw-      319052  Dec 16 2012 08:08:04 +00:00  crashinfo_ios-base-20121216-080759

Is there any specific reason to run this ION code on this switch, if possible, you could upgrade the code to aviod this kind of behavior.

Regards,

Aru

*** Please rate if the post is useful ***