4900M WS-X4908-10G-RJ45 Port Startup Delay

shane_mann · ‎03-05-2013

Hi all,

We have recently purchased 2 x 4900M switches with the following configuration:

2 x X2-10GB-SR installed in port te1/1, 1/2 (optical)

1 x 20 x 1GB RJ45 installed in the top left of chasis ge2/1 - 2/20

1 x WS-X4908-10G-RJ45 8 port 1/2 card in the top right of chassis te3/1 - 3/8

We have te1/1 connecting the 2 switches, and various vlan's and connections on the 1GB RJ45's with no problems.

However, we have a number of 10Gbps BaseT connections te3/1 - te3/5 which are causing us issues. On a reload of the switch or removal / re-install of the cable it can take anywhere up to an hour for the port to become active again. So when we reload the switch, we may get a connection come up after say 3 mins, but the port next to it may not come up for 15-60 min. The cabling is all the same (CAT6 and has been tested/verified) and all connections go to Broadcom 10Gb cards in Dell 820 servers running vmware. We have the following set on the ports:

switchport mode access

spanning-tree portfast

No speed or duplex can be set on these ports - these options are not available in the config.

Is this normal? Is there something I am missing in config for these ports? By the way, once a link is finally established, it will hold the link solid with no problems, but this is a problem for us, we can't have a reboot take an undetermined amount of time for the link to come back.

Any help appreciated.

Cheers,

Shane

Reza Sharifi · ‎03-05-2013

Hi,

No that is not normal at all. What if you use ports (te3/1 - te3/5) instead of te1/1 to connect the switches together?

Do you still have the same problem?

I know it is hard to find a laptop or desktop with 10Gig interface, but is there is device other then the servers you can connect to these ports?

HTH

shane_mann · ‎03-05-2013

Thanks for reply,

I connected te3/1 to te3/1 on each switch and the link took 5 min 3 seconds to establish.

I also have some dell 10Gb rj-45 switches and if I connect the server to a dell switch the link is basically instant (as one would expect).

It's really got us scratching our heads...

Cheers,

Shane

Reza Sharifi · ‎03-05-2013

What version of IOS are you running?

Cisco IOS Software Release 12.2(54)SG continues to deliver data center smart top-of-rack services:

• Hardware support for the new Cisco Catalyst 4948E

• Hardware support for the new Cisco Catalyst 4900M 8-Port 10GBASE-T RJ-45 Half Card (WS-X4908-10G-RJ45)

more info:

http://www.cisco.com/en/US/prod/collateral/switches/ps5718/ps6021/product_bulletin_c25_608183.html

HTH

shane_mann · ‎03-05-2013

Seems our output below has ROM: 12.2(44r)SG10, should we be running 12.2(54)SG?

We only purchased these switches a couple of weeks ago, I didn't look to see if they were up to the latest IOS.

Output from show version:

Cisco IOS Software, Catalyst 4500 L3 Switch Software (cat4500e-ENTSERVICESK9-M),

Version 15.0(2)SG3, RELEASE SOFTWARE (fc2)

Technical Support: http://www.cisco.com/techsupport

Compiled Mon 09-Jan-12 01:56 by prod_rel_team

Image text-base: 0x10000000, data-base: 0x12E99770

ROM: 12.2(44r)SG10

Sushi Revision 11, Tatooine Revision 141, Forerunner Revision 1.79

swe11 uptime is 22 hours, 22 minutes

System returned to ROM by power-on

System image file is "bootflash:cat4500e-entservicesk9-mz.150-2.SG3.bin"

This product contains cryptographic features and is subject to United

States and local country laws governing import, export, transfer and

use. Delivery of Cisco cryptographic products does not imply

third-party authority to import, export, distribute or use encryption.

Importers, exporters, distributors and users are responsible for

compliance with U.S. and local country laws. By using this product you

agree to comply with applicable laws and regulations. If you are unable

to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:

http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to

export@cisco.com.

cisco WS-C4900M (MPC8548) processor (revision 2) with 1048576K bytes of memory.

Processor board ID JAE16110AY1

MPC8548 CPU at 1.33GHz, Cisco Catalyst 4900M

Last reset from PowerUp

1 Virtual Ethernet interface

28 Gigabit Ethernet interfaces

16 Ten Gigabit Ethernet interfaces

511K bytes of non-volatile configuration memory.

Configuration register is 0x2101

Cheers,

Shane

Reza Sharifi · ‎03-05-2013

should we be running 12.2(54)SG?

No, that is firm ware for Rommon and usually doesn't need to be ugraded often. Your IOS version is 150-2.SG3, which is pretty new. Also one more thing and this shouldn't make any difference on the issue you have, but your config register is 0x2101. Can you change it to 0x2102, reboot and test again?

HTH

shane_mann · ‎03-05-2013

That particular switch had no compact flash card installed - so 0x2102 sent it into a reboot loop as is couldn't find an image in slot0. I have since added a card with the image and it's rebooting now.

I also contacted cisco and they are making 15.1.2SG available as these switches were purchased within the last 90 days.

I will test this as soon as I receive the new image.

Thanks for your help... getting there...

Cheers,

Shane

shane_mann · ‎03-06-2013

I have now deployed cat4500e-entservicesk9-mz.151-2.SG.bin on both switches, and found a vmware driver update for the broadcom cards they are connected to (BCM-NetXtremeII-7.0-1036215.zip released 2 days ago). Also 1 of the ports is connected directly to a Windows 2k8 server.

It didn't help. A reload of both switches saw the following:

sw2 port 3/4 up at 40 seconds

sw2 port 3/3 up at 1:42 seconds

sw1 port 3/3 up at 6:50 seconds

Then no other ports up to 15 mins, so I went and got a coffee, came back and the last port came on-line at 40 minutes 11 seconds (the others somewhere between 15-40 mins).

I was timing from when the module status light turned green during reload.

Still stumped.

Cheers,

Shane

shane_mann · ‎03-06-2013

A further search on these discussions found that downgrading to 122-54.SG1 may fix the problem. I'll test that and see.

Cheers,

Shane

shane_mann · ‎03-07-2013

Downgrading to 122-54.SG1 has fixed the problem. Each of the ports now starts approximately 5 seconds after the modules status light turns green. If you unplug a link and plug it back in - the port again comes up after approx 5 seconds.

A Cisco support case has been opened, how can they release an IOS and ship it on a device that clearly does not work properly...

Cheers,

Shane

claytongump · ‎04-23-2013

I've got a very similar problem except mine won't come up at all.

Upgraded to 15.1(2) from 122-54

and now a few of the 10GBaseT connections are acting strange.

The ports that were up when I upgraded have stayed up however I can't get a new connection to work. This was from a ESX vmWare server. This connection used to work perfectly fine. We only plug it in when needed.

I even tried using the hw-module command to change the port to 1G but it still will not come up.

So I tried moving an working 10G connection from port Te2/4 to port Te2/3 and now this connection wont come up no matter where I try and connect it.

I've verified it isn't the server becasue they come up fine on another switch at 1G.

I've got a TAC case in. I'll let you know what happens. This sucks becasue the last time I tried moving to 15.0 I had to back down to 12.2 becasue of these stupid RF45 ports.

claytongump · ‎04-24-2013

So the solution from TAC is to downgrade back to 12.2.54,

This is the second time I've tried to upgrade these switches only to spend wasted hours troubleshooting the problem and then having to downgrade anyway.

shane_mann · ‎04-25-2013

Thanks for your update. I am still providing debugging info to my support case (have provided heaps so far). No solution as yet though...

Like you - we are currently running 12.2.54 while we wait for a working update.

Cheers,

Shane

claytongump · ‎04-30-2013

I wanted to give anyone finding this post an update.

We downgraded to 12.2.54 last week and the 10GBaseT ports work perfectly now.

I wanted to let you all know about another problem with the 4900M that cropped up when I went to 15.1 that also has gone away. It has to do with the switched reporting a %SNMP-3-CPUHOG error. I would get this error when a network management device I have did a SNMP discovery of the device. I have all the nasty details documented so I can discuss in more detail if needed.

The short version is the 4900M CPU would spike to over 90% for about 5 minutes blocking any other SNMP requests.

The final resolution from TAC was

"the SNMP Engine seems to be getting in a loop while polling the following OID:lldpXMedLocMediaPolicyEntry"

The TAC solution (basically telling the 4900M to ignore requests for that OID) did resolve the problem however when I downgraded back to 12.2.54 the issue went away.

So the best advice if you have a 4900M is to just stay away from 15.x for now.

Thanks.

-Clay

glen.grant · ‎04-30-2013

My take has always been unless a code upgrade is needed to support new hardware or to fix a specific problem or security issue , leave it alone. If it's not broke don't fix it .