issue with switch

[ADIL ALTAEE]

I have this cisco 6509 switch which belongs to a government organization, they brought me to diagnoze it

it hasn't been operated since 1.5 years, they only opened it a month ago to try to diagnose it. 

anyway, problem is that when i connect a host to any port and ping to it, it says request time out. all the switch supervisor engine LEDs are green, the status LED IS also green. but the ports LEDs are all orange when connecting a host to it accept for one LED. i checked the status of the ports. most of them are disabled, others are err-disabled

by the way, there are no port-security activated in any port

update:

now i deleted previous vlans and set all port access to vlan1, 12 ports are working now

anyway, when issue shut command to any port, the LED stays orange, after no shutdown, the error is: 

%PM-SP-4-ERR_DISABLE: diagnostics error detected on Gi2/24, putting Gi2/24 in err-disable state

even after rebooting the same message for all other ports

this is the boot for the switch

Press RETURN to get started!

*Jan 28 07:57:08.927: RP: Currently running ROMMON from S (Gold) region
*Jan 28 07:57:09.747: %SPANTREE-5-EXTENDED_SYSID: Extended SysId enabled for type vlan. The Bridge IDs of all active STP instances have been updated, which might change the spanning tree topology
*Jan 28 07:57:13.827: %SYS-6-CLOCKUPDATE: System clock has been updated from 07:57:13 UTC Mon Jan 28 2019 to 10:57:13 GMT Mon Jan 28 2019, configured from console by console.
*Jan 28 07:57:14.803: %SYS-5-CONFIG_I: Configured from memory by console
*Jan 28 07:57:18.151: %SYS-5-RESTART: System restarted --
Cisco IOS Software, s72033_rp Software (s72033_rp-IPSERVICESK9_WAN-M), Version 12.2(33)SXI9, RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2012 by Cisco Systems, Inc.
Compiled Fri 24-Feb-12 21:38 by prod_rel_team
*Jan 28 07:55:55.111: %SYS-SP-3-LOGGER_FLUSHED: System was paused for 00:00:00 to ensure console debugging output.
*Jan 28 07:57:04.015: %SPANTREE-SP-5-EXTENDED_SYSID: Extended SysId enabled for type vlan. The Bridge IDs of all active STP instances have been updated, which might change the spanning tree topology
*Jan 28 07:57:04.031: SP: SP: Currently running ROMMON from S (Gold) region
*Jan 28 07:57:17.065: %OIR-SP-6-INSPS: Power supply inserted in slot 1
*Jan 28 07:57:17.097: %C6KPWR-SP-4-PSOK: power supply 1 turned on.
*Jan 28 07:57:17.448: %SYS-SP-5-RESTART: System restarted --
Cisco IOS Software, s72033_sp Software (s72033_sp-IPSERVICESK9_WAN-M), Version 12.2(33)SXI9, RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2012 by Cisco Systems, Inc.
Compiled Fri 24-Feb-12 21:56 by prod_rel_team
*Jan 28 07:57:17.892: %SYS-SP-6-BOOTTIME: Time taken to reboot after reload = 209 seconds
*Jan 28 07:57:26.287: %FABRIC-SP-5-CLEAR_BLOCK: Clear block option is off for the fabric in slot 5.
*Jan 28 07:57:26.287: %FABRIC-SP-5-FABRIC_MODULE_ACTIVE: The Switch Fabric Module in slot 5 became active.
*Jan 28 07:57:28.959: %DIAG-SP-6-RUN_MINIMUM: Module 5: Running Minimal Diagnostics...
*Jan 28 07:57:52.679: %DIAG-SP-6-DIAG_OK: Module 5: Passed Online Diagnostics
*Jan 28 07:57:54.679: %OIR-SP-6-INSCARD: Card inserted in slot 5, interfaces are now online
*Mar 1 00:00:02.299: DaughterBoard (Centralized Forwarding Card)

Firmware compiled 19-Jan-12 11:10 by integ Build [100]
Jan 28 07:58:15.859: %SYS-CFC2-5-RESTART: System restarted --
Cisco IOS Software, c6lc2 Software (c6lc2-SP-M), Version 12.2(33)SXI9, RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2012 by Cisco Systems, Inc.
Compiled Fri 24-Feb-12 21:38 by prod_rel_team
*Jan 28 07:58:19.525: %PFREDUN-SP-4-VERSION_MISMATCH: Defaulting to RPR mode (Different sw versions)
*Jan 28 07:58:20.537: %PFREDUN-SP-6-ACTIVE: Standby initializing for RPR mode
Jan 28 07:58:15.863: CFC2: Currently running ROMMON from S (Gold) region
*Jan 28 07:58:20.834: %SYS-SP-3-LOGGER_FLUSHED: System was paused for 00:00:00 to ensure console debugging output.
*Jan 28 07:58:22.102: %PFINIT-SP-5-CONFIG_SYNC: Sync'ing the startup configuration to the standby Router.
*Jan 28 07:58:24.519: %FABRIC-SP-5-CLEAR_BLOCK: Clear block option is off for the fabric in slot 6.
*Jan 28 07:58:24.519: %FABRIC-SP-5-FABRIC_MODULE_BACKUP: The Switch Fabric Module in slot 6 became standby
*Jan 28 07:58:25.435: %DIAG-SP-6-RUN_MINIMUM: Module 2: Running Minimal Diagnostics...
*Jan 28 07:58:28.063: %DIAG-SP-6-RUN_MINIMUM: Module 6: Running Minimal Diagnostics...
*Jan 28 07:58:29.287: %RF-SP-5-RF_TERMINAL_STATE: Terminal state reached for (RPR)
Core-MOIM>
Core-MOIM>
Core-MOIM>
Core-MOIM>
*Jan 28 08:00:31.159: %DIAG-SP-3-MINOR: Module 2: Online Diagnostics detected a Minor Error. Please use 'show diagnostic result <target>' to see test results.
*Jan 28 08:00:31.163: %CONST_DIAG-SP-3-BOOTUP_TEST_FAIL: Module 2: TestLoopback failed on port(s) 1-24,26,28,30,32,34,36,38,40,42,44,46,48
*Jan 28 08:00:32.427: %PM-SP-4-ERR_DISABLE: diagnostics error detected on Gi2/3, putting Gi2/3 in err-disable state
*Jan 28 08:00:32.603: %PM-SP-4-ERR_DISABLE: diagnostics error detected on Gi2/4, putting Gi2/4 in err-disable state
*Jan 28 08:00:32.759: %PM-SP-4-ERR_DISABLE: diagnostics error detected on Gi2/5, putting Gi2/5 in err-disable state
*Jan 28 08:00:32.911: %PM-SP-4-ERR_DISABLE: diagnostics error detected on Gi2/6, putting Gi2/6 in err-disable state
*Jan 28 08:00:32.931: %PM-SP-4-ERR_DISABLE: diagnostics error detected on Gi2/7, putting Gi2/7 in err-disable state
*Jan 28 08:00:33.079: %PM-SP-4-ERR_DISABLE: diagnostics error detected on Gi2/8, putting Gi2/8 in err-disable state
*Jan 28 08:00:33.099: %PM-SP-4-ERR_DISABLE: diagnostics error detected on Gi2/9, putting Gi2/9 in err-disable state
*Jan 28 08:00:33.115: %PM-SP-4-ERR_DISABLE: diagnostics error detected on Gi2/10, putting Gi2/10 in err-disable state
*Jan 28 08:00:33.135: %PM-SP-4-ERR_DISABLE: diagnostics error detected on Gi2/11, putting Gi2/11 in err-disable state
*Jan 28 08:00:33.279: %PM-SP-4-ERR_DISABLE: diagnostics error detected on Gi2/12, putting Gi2/12 in err-disable state
*Jan 28 08:00:33.431: %PM-SP-4-ERR_DISABLE: diagnostics error detected on Gi2/21, putting Gi2/21 in err-disable state
*Jan 28 08:00:33.627: %PM-SP-4-ERR_DISABLE: diagnostics error detected on Gi2/22, putting Gi2/22 in err-disable state
*Jan 28 08:00:33.775: %PM-SP-4-ERR_DISABLE: diagnostics error detected on Gi2/23, putting Gi2/23 in err-disable state
*Jan 28 08:00:33.791: %PM-SP-4-ERR_DISABLE: diagnostics error detected on Gi2/24, putting Gi2/24 in err-disable state
*Jan 28 08:00:33.951: %PM-SP-4-ERR_DISABLE: diagnostics error detected on Gi2/26, putting Gi2/26 in err-disable state
*Jan 28 08:00:34.111: %PM-SP-4-ERR_DISABLE: diagnostics error detected on Gi2/28, putting Gi2/28 in err-disable state
*Jan 28 08:00:34.147: %PM-SP-4-ERR_DISABLE: diagnostics error detected on Gi2/30, putting Gi2/30 in err-disable state
*Jan 28 08:00:34.179: %PM-SP-4-ERR_DISABLE: diagnostics error detected on Gi2/32, putting Gi2/32 in err-disable state
*Jan 28 08:00:34.211: %PM-SP-4-ERR_DISABLE: diagnostics error detected on Gi2/34, putting Gi2/34 in err-disable state
*Jan 28 08:00:34.243: %PM-SP-4-ERR_DISABLE: diagnostics error detected on Gi2/36, putting Gi2/36 in err-disable state
*Jan 28 08:00:34.395: %PM-SP-4-ERR_DISABLE: diagnostics error detected on Gi2/38, putting Gi2/38 in err-disable state
*Jan 28 08:00:34.427: %PM-SP-4-ERR_DISABLE: diagnostics error detected on Gi2/40, putting Gi2/40 in err-disable state
*Jan 28 08:00:34.463: %PM-SP-4-ERR_DISABLE: diagnostics error detected on Gi2/42, putting Gi2/42 in err-disable state
*Jan 28 08:00:34.619: %PM-SP-4-ERR_DISABLE: diagnostics error detected on Gi2/44, putting Gi2/44 in err-disable state
*Jan 28 08:00:34.783: %PM-SP-4-ERR_DISABLE: diagnostics error detected on Gi2/46, putting Gi2/46 in err-disable state
*Jan 28 08:00:34.951: %PM-SP-4-ERR_DISABLE: diagnostics error detected on Gi2/48, putting Gi2/48 in err-disable state
*Jan 28 08:00:34.963: %OIR-SP-6-INSCARD: Card inserted in slot 2, interfaces are now online
*Jan 28 08:00:35.403: %OIR-SP-6-INSCARD: Card inserted in slot 6, interfaces are now online

[Mark Malone]

Are the modules all online fully , can you post the show module
Is there anything in the logging relating to faults ?

so your pc as example is 10.1.1.1 you set a switchport to no switchport add an ip address and you cant ping it when its directly connected and in an UP/UP state ? even after shut no shut on the port if errdisabled ?

Why are the ports all err-disabled is there any logs showing the cause of the errdisables , shut the ports and retstart the lot , interface rang g1/0/1 - 48 , you can do this for each effected module

can you post the out put of your port your pinging , show interface gx/x/x

[ADIL ALTAEE]

thank for your concern
the error message is
%PM-SP-4-ERR_DISABLE: diagnostics error detected on Gi2/24, putting Gi2/24 in err-disable state
shut and no shut doesn't work too,
even after I issued no errdisable recovery cause all

[Mark Malone]

Hi
thats a diagnostic error it seen when booting up the module and the switch itself shut the port down , could be a hardware fault
can you post the show tech

have you re-seated the modules , to start them back up and see if it makes a change

[ADIL ALTAEE]

I have shutdown and restarted multiple ports, when restarting port, the error shows again
and by the way, when issuing shutdown command, the port LED stays orange and doesn't turn off
thank you again

[Mark Malone]

Yes its a hardware issue , can you pull the show tech and post it and we can run it and see if it shows why its failing
At the moment you have a hardware issue and the switch is failing diagnostics
If you want to know why post the show tech it may contain it

[Mark Malone]

DID you also try re-seat the actual modules back into the switch ?

[ADIL ALTAEE]

if I understood you, you mean to remove the module and put it back in. or put it in another slot??
anyway, I haven't done that before, and I don't want to try with it unless it is a last solution

[Mark Malone]

yes these cards are OIR they can be taken out and reseated while the chassis is on
there hot swappable , same slot

I haven't done that before, and I don't want to try with it unless it is a last solution

Its actually one of the first things you should do when its failing diag test , the hardware failing to come up , reseating it will start the process completely again , if its still the same the modules probably damaged and needs and RMA , replaced

[ADIL ALTAEE]

Hello
i will do that tomorrow when i have access to the switch
before that i want to ask, can this happened to 36 ports while other 12 ports are working
thank you very much for your help

[Mark Malone]

yes there are multiple asics on a blade in sets behind the ports , some of them may have blown completely , so you may have a couple of blocks gone

[ADIL ALTAEE]

how can i stop show tech command, caus it goes all the way down. i am using putty

[Mark Malone]

Yes its everything from the switch, you set the  putty session to log/save to the desktop (logging tab in putty), run the showtech let it finish , stop the log collection then post it

[Peter Marshall]

I don't want to assume anything, but are you able to get access to the network via that 6509?
If yes, troubleshoot from the 6509. If not, there may be other issues, such as a firewall or restrictive ACL.

If you can get on that network, do an "ipconfig /all" from a PowerShell session. See if you can ping the following:
Your IP / self-ping
The Default Gateway
The DNS server
The DHCP server

If you can't ping those, start asking questions.

Log into the switch with Putty, set it to log the session, and enter these commands:

terminal length 0
show spanning-tree root
show spanning-tree vlan
show spanning-tree blockedports

show run
show version

Save and close that session, then open it with Notepad as you proceed - this will allow you to revert to the original configuration. Start a new Putty session.

On your Ethernet connection, run the following:

show run interface GigabitEthernetn/n

If it doesn't look like this, make the change on your connection:
configure terminal
interface GigabitEthernetn/n
shutdown
description My.Test
switchport
switchport access vlan (the access vlan)
switchport host
switchport mode access
switchport nonegotiate
spanning-tree portfast edge
spanning-tree bpduguard enable
spanning-tree guard root
no shutdown

HTH

[ADIL ALTAEE]

I have access to the switch through console cable, no worries for network
i will update the post, i hope you will read it 
very much appreciation for your help