Solved: Re: urgent issue between core and Distribution

coolj_212 · ‎08-12-2018

I configure ospf l3 between core switch and distribution and after 3 days I lose the connectivity between the core, Distribution and access switch after I restart the Distribution the network become normal the configuration that have been done as layer 3 is below and also I observe that one of the Distribution hang Also note that we did The below configuration in different 3 site and its working Find so please anyone have the same issue please advice

This configuration from core side

interface Port-channel44

description connected to DIS- port-channel 44

no switchport

ip address 10.x.x.x 255.255.255.252

no ip redirects

no ip unreachables

no ip proxy-arp

interface TenGigabitEthernet2/3/3

description To | SWITCH |

no switchport

no ip address

no ip redirects

no ip unreachables

no ip proxy-arp

channel-group 44 mode on

!

interface TenGigabitEthernet2/3/4

description To | SWITCH |

no switchport

no ip address

no ip redirects

no ip unreachables

no ip proxy-arp

channel-group 44 mode on

!

router ospf 1

router-id 2.2.2.2

ispf

log-adjacency-changes

auto-cost reference-bandwidth 100000

area 50 nssa

redistribute connected subnets

network 10.x.x.x 0.0.0.0 area 50

This configuration from Distribution side

interface Port-channel44

description connected to port-channel 44

ip address 10.x.x.x 255.255.255.252

no ip redirects

no ip unreachables

no ip proxy-arp

ip ospf 1 area 50

bfd interval 750 min_rx 750 multiplier 3

interface TenGigabitEthernet1/2/3

description To | SWITCH

no switchport

no ip address

channel-group 44 mode on

!
interface TenGigabitEthernet2/2/3

description To | SWITCH |

no switchport

no ip address

channel-group 44 mode on

!
router ospf 1

router-id 5.5.5.5

ispf

auto-cost reference-bandwidth 100000

area 50 nssa

redistribute connected subnets

distribution model: WS-C4507R+E

IOS:cat4500e-universalk9.SPA.03.05.00.E.152-1.E.bin

CORE MODEL: WS-C6506-E

CORE: s2t54-ipservicesk9-mz.SPA.122-50.SY.bin

paul driver · ‎08-13-2018

Hello

Can you post the outputs for the follwoing:

Show switch virtual
Show switch virtual role
Show switch virtual link
Show switch virtual port-channel
show switch module

Please rate and mark as an accepted solution if you have found any of the information provided useful.
This then could assist others on these forums to find a valuable answer and broadens the community’s global network.

Kind Regards
Paul

View solution in original post

omz · ‎08-12-2018

What is the issue? One of distribution switch hangs? Please explain.

coolj_212 · ‎08-12-2018

yes one of the Distribution hang and all the up link on that Distribution Go down and also we lose the neighbor ship from ospf between the core and Distribution so

if you type the command show ip ospf neighbor:

2.2.2.2 1 exstart/BDR 00:00:00 10.x.x.x Port-channel44

omz · ‎08-12-2018

Is OSPF stuck in exstart? That will be due to MTU mismatch.

Is the port-channel stable?

Does anything stands out in logs?

debug ip ospf adj

omz · ‎08-12-2018

debug ip ospf adj

What do you see?

paul driver · ‎08-12-2018

Hello

Prior to enabling opsf what was you using and was everything was okay?

Please rate and mark as an accepted solution if you have found any of the information provided useful.
This then could assist others on these forums to find a valuable answer and broadens the community’s global network.

Kind Regards
Paul

coolj_212 · ‎08-12-2018

before the i confider ospf i was using static route and everything was working fine please note that i have 4 area i enable ospf in the 4 area 3 area was working fine and one area is not working fine i enable when i enable the ospf its work 3 days and after that i lose the ospf the neighbor ship go down so i need to restart the distribution and after that the L3 will work fine

omz · ‎08-12-2018

so after dist restart, it works? is it working now?

coolj_212 · ‎08-12-2018

yes its working and the neighbor ship is up

debug ip ospf ad

*Aug 12 01:26:27.579: OSPF-1 PAK : rcv. v:2 t:1 l:48 rid:2.2.2.2 aid:0.0.0.50 chk:C0FB aut:0 auk: from Port-channel44
*Aug 12 01:26:36.847: OSPF-1 PAK : rcv. v:2 t:1 l:48 rid:2.2.2.2 aid:0.0.0.50 chk:C0FB aut:0 auk: from Port-channel44
*Aug 12 01:26:46.827: OSPF-1 PAK : rcv. v:2 t:1 l:48 rid:2.2.2.2 aid:0.0.0.50 chk:C0FB aut:0 auk: from Port-channel44
*Aug 12 01:26:56.413: OSPF-1 PAK : rcv. v:2 t:1 l:48 rid:2.2.2.2 aid:0.0.0.50 chk:C0FB aut:0 auk: from Port-channel44
*Aug 12 01:27:06.267: OSPF-1 PAK : rcv. v:2 t:1 l:48 rid:2.2.2.2 aid:0.0.0.50 chk:C0FB aut:0 auk: from Port-channel44
*Aug 12 01:27:06.752: OSPF-1 PAK : rcv. v:2 t:4 l:64 rid:2.2.2.2 aid:0.0.0.50 chk:87F4 aut:0 auk: from Port-channel44

omz · ‎08-12-2018

Are you trying to find out the root cause of why it went down after 3 days?

Can you see anything in the logs just before the OSPF neighbour went down?

coolj_212 · ‎08-12-2018

I didn't check the log so can you please advice why the ospf go down or what can i do when the ospf go down which command i have to issue

omz · ‎08-12-2018

Proper logging is very important. Preferably to a Syslog server. You have log-adjacency-changes enabled under OSPF. Logs will tell why the OSPF went down. If the local interface went down/flapped or the OSPF hold time expired which would indicate remote interface went down.

For a quick check to see what happened if OSPF went down -

- sh logg | i OSPF

- sh logg | i OSPF|UPDOWN

These are helpful for detailed troubleshooting -

- show ip ospf nei (neighbors and adjacency state)

- show ip ospf int (“Hello” and “Dead” timer intervals)

- show ip prot (if OSPF is running)

- show ip route (routes learned by OSPF)

- show ip ospf (all OSPF processes running)

- debug ip ospf packet (hello packets being sent and received on the router)
- debug ip ospf hello (detailed info regarding packets being sent and received)
- debug ip ospf adj (shows DR and BDR elections)

This may be helpful in troubleshooting OSPF -

https://www.cisco.com/c/en/us/support/docs/ip/open-shortest-path-first-ospf/12151-trouble-main.html

Please rate helpful posts :)

coolj_212 · ‎08-12-2018

please find the log when the issue happen

show log

Aug 12 01:26:27.579: OSPF-1 PAK : rcv. v:2 t:1 l:48 rid:2.2.2.2 aid:0.0.0.50 chk:C0FB aut:0 auk: from Port-channel44

*Aug 12 01:26:36.847: OSPF-1 PAK : rcv. v:2 t:1 l:48 rid:2.2.2.2 aid:0.0.0.50 chk:C0FB aut:0 auk: from Port-channel44

*Aug 12 01:26:46.827: OSPF-1 PAK : rcv. v:2 t:1 l:48 rid:2.2.2.2 aid:0.0.0.50 chk:C0FB aut:0 auk: from Port-channel44

*Aug 12 01:26:56.413: OSPF-1 PAK : rcv. v:2 t:1 l:48 rid:2.2.2.2 aid:0.0.0.50 chk:C0FB aut:0 auk: from Port-channel44

*Aug 12 01:27:06.267: OSPF-1 PAK : rcv. v:2 t:1 l:48 rid:2.2.2.2 aid:0.0.0.50 chk:C0FB aut:0 auk: from Port-channel44

*Aug 12 01:27:06.752: OSPF-1 PAK : rcv. v:2 t:4 l:64 rid:2.2.2.2 aid:0.0.0.50 chk:87F4 aut:0 auk: from Port-channel44

*Aug 11 21:38:41.810: %C4K_IOSVSLENCR-3-VSLPMKKEYSTOREERROR: Failed to open PMK keystore file.

*Aug 11 21:38:41.832: %C4K_IOSSYS-6-IMAGELEVEL: Supervisor booting in image level 'entservices'

*Aug 11 21:39:45.342: %C4K_IOSINTF-5-LMPHWSESSIONSTATE: Lmp HW session UP on slot 3 port 2.

*Aug 11 21:39:45.474: %C4K_IOSINTF-5-LMPHWSESSIONSTATE: Lmp HW session UP on slot 3 port 1.

*Aug 11 21:40:01.351: %VSLP-5-VSL_UP: Ready for control traffic

*Aug 11 21:40:06.344: %VSLP-5-RRP_ROLE_RESOLVED: Role resolved as ACTIVE by VSLP

*Aug 11 21:40:36.651: %C4K_REDUNDANCY-6-INIT: Initializing as ACTIVE supervisor

*Aug 11 21:40:37.309: %SPANTREE-5-EXTENDED_SYSID: Extended SysId enabled for type vlan

*Aug 11 21:40:39.095: %C4K_REDUNDANCY-6-DUPLEX_MODE: The peer Supervisor has been detected

*Aug 11 21:40:39.129: %C4K_IOSMODPORTMAN-6-MODULEONLINE: Module 1 (WS-X4448-GB-SFP S/N: JAE174905CD Hw: 1.3) is online

*Aug 11 21:40:39.129: %C4K_IOSMODPORTMAN-6-MODULEONLINE: Module 2 (WS-X4606-X2-E S/N: JAE180109L7 Hw: 2.0) is online

*Aug 11 21:40:39.129: %C4K_IOSMODPORTMAN-6-MODULEONLINE: Module 3 (WS-X45-SUP7L-E S/N: CAT1748L7GQ Hw: 3.0) is online

*Aug 11 21:40:42.291: %SPANTREE-6-PORTDEL_ALL_VLANS: Port-channel1 deleted from all Vlans

*Aug 11 21:40:42.344: %SPANTREE-6-PORTDEL_ALL_VLANS: Port-channel2 deleted from all Vlans

*Aug 11 21:40:44.462: %SW_VLAN-6-VTP_DOMAIN_NAME_CHG: VTP domain name changed to.

*Aug 11 21:40:45.854: %SYS-5-CONFIG_I: Configured from memory by console

*Aug 11 21:40:46.940: %SYS-5-RESTART: System restarted --

Cisco IOS Software, IOS-XE Software, Catalyst 4500 L3 Switch Software (cat4500e-UNIVERSALK9-M), Version 03.05.00.E RELEASE SOFTWARE (fc3)

Technical Support: http://www.cisco.com/techsupport

Compiled Tue 27-Aug-13 14:56 by prod_rel_team

*Aug 11 21:40:46.981: %SSH-5-ENABLED: SSH 1.99 has been enabled

*Aug 11 21:40:47.005: %OSPF-4-NSSA_NO_FA: OSPF process 1 lacks forwarding address for type 7 LSA 5.5.5.5 in NSSA 50 - P-bit cleared

*Aug 11 21:40:48.807: %SNMP-5-COLDSTART: SNMP agent on host site is undergoing a cold start

*Aug 11 21:40:48.983: %EC-5-BUNDLE: Interface TenGigabitEthernet1/3/1 joined port-channel Port-channel1

*Aug 11 21:40:49.314: %EC-5-BUNDLE: Interface TenGigabitEthernet1/3/2 joined port-channel Port-channel1

*Aug 11 21:40:52.778: %EC-5-BUNDLE: Interface TenGigabitEthernet1/2/4 joined port-channel Port-channel44

*Aug 11 21:40:54.668: %EC-5-BUNDLE: Interface GigabitEthernet1/1/8 joined port-channel Port-channel11

*Aug 11 21:40:55.132: CDP-EV: RCVD CDP packet on GigabitEthernet1/1/8 with len (1)

*Aug 11 21:40:55.132: CDP Packet Process DONE

*Aug 11 21:40:56.584: %EC-5-BUNDLE: Interface GigabitEthernet1/1/2 joined port-channel Port-channel5

*Aug 11 21:40:56.590: %EC-5-BUNDLE: Interface GigabitEthernet1/1/4 joined port-channel Port-channel7

*Aug 11 21:40:56.623: %EC-5-BUNDLE: Interface GigabitEthernet1/1/7 joined port-channel Port-channel10

*Aug 11 21:40:57.696: %EC-5-BUNDLE: Interface GigabitEthernet1/1/1 joined port-channel Port-channel4

*Aug 11 21:40:59.297: %SFF8472-5-THRESHOLD_VIOLATION: Te1/2/4: Tx power low alarm; Operating value: -40.0 dBm, Threshold value: -12.2 dBm.

*Aug 11 21:40:59.515: %EC-5-BUNDLE: Interface GigabitEthernet1/1/6 joined port-channel Port-channel9

*Aug 11 21:41:03.518: %EC-5-BUNDLE: Interface GigabitEthernet1/1/9 joined port-channel Port-channel12

*Aug 11 21:41:08.861: %C4K_IOSMODPORTMAN-6-MODULEONLINE: Module 13 (WS-X45-SUP7L-E S/N: CAT1748L7DK Hw: 3.0) is online

*Aug 11 21:41:09.412: %C4K_REDUNDANCY-6-MODE: ACTIVE supervisor initializing for sso mode

*Aug 11 21:41:09.703: %C4K_REDUNDANCY-3-COMMUNICATION: Communication with the peer Supervisor has been established

*Aug 11 21:41:10.409: %C4K_REDUNDANCY-6-MODE: ACTIVE supervisor initializing for sso mode

*Aug 11 21:41:28.521: %C4K_IOSINTF-5-TRANSCEIVERINSERTED: Slot=12 Port=3: Transceiver has been inserted

*Aug 11 21:41:28.521: %C4K_IOSINTF-5-TRANSCEIVERINSERTED: Slot=12 Port=4: Transceiver has been inserted

*Aug 11 21:41:29.801: %C4K_REDUNDANCY-5-CONFIGSYNC: The bootvar has been successfully synchronized to the standby supervisor

*Aug 11 21:41:29.801: %C4K_REDUNDANCY-5-CONFIGSYNC: The config-reg has been successfully synchronized to the standby supervisor

*Aug 11 21:41:29.803: %C4K_REDUNDANCY-5-CALENDAR: The calendar has been successfully synchronized to the standby supervisor for the first time

*Aug 11 21:41:29.803: %C4K_REDUNDANCY-5-CONFIGSYNC: The startup-config has been successfully synchronized to the standby supervisor

*Aug 11 21:41:32.430: %C4K_REDUNDANCY-5-CONFIGSYNC: The private-config has been successfully synchronized to the standby supervisor

*Aug 11 21:41:36.282: %OSPF-5-ADJCHG: Process 1, Nbr 2.2.2.2 on Port-channel44 from LOADING to FULL, Loading Done

*Aug 11 21:41:36.896: %C4K_REDUNDANCY-5-CONFIGSYNC_RATELIMIT: The vlan database has been successfully synchronized to the standby supervisor

*Aug 11 21:41:48.238: %C4K_IOSMODPORTMAN-6-MODULEONLINE: Module 11 (WS-X4448-GB-SFP S/N: JAE174904Q2 Hw: 1.3) is online

*Aug 11 21:41:48.238: %C4K_IOSMODPORTMAN-6-MODULEONLINE: Module 12 (WS-X4606-X2-E S/N: JAE18010APJ Hw: 2.0) is online

*Aug 11 21:41:51.514: %C4K_IOSINTF-5-TRANSCEIVERINSERTED: Slot=13 Port=1: Transceiver has been inserted

*Aug 11 21:41:51.514: %C4K_IOSINTF-5-TRANSCEIVERINSERTED: Slot=13 Port=2: Transceiver has been inserted

*Aug 11 21:42:26.513: %C4K_IOSINTF-5-TRANSCEIVERREMOVED: Slot=12 Port=3: Transceiver has been removed

*Aug 11 21:42:26.513: %C4K_IOSINTF-5-TRANSCEIVERREMOVED: Slot=12 Port=4: Transceiver has been removed

*Aug 11 21:42:26.516: %EC-5-BUNDLE: Interface TenGigabitEthernet2/3/1 joined port-channel Port-channel2

*Aug 11 21:42:26.548: %EC-5-BUNDLE: Interface TenGigabitEthernet2/3/2 joined port-channel Port-channel2

*Aug 11 21:42:29.185: %HA_CONFIG_SYNC-6-BULK_CFGSYNC_SUCCEED: Bulk Sync succeeded

*Aug 11 21:42:29.192: %RF-5-RF_TERMINAL_STATE: 3 ha_mgr: Terminal state reached for (SSO)

*Aug 11 21:42:27.570: %C4K_IOSMODPORTMAN-6-MODULEONLINE: STANDBY:Module 1 (WS-X4448-GB-SFP S/N: JAE174905CD Hw: 1.3) is online

*Aug 11 21:42:27.570: %C4K_IOSMODPORTMAN-6-MODULEONLINE: STANDBY:Module 2 (WS-X4606-X2-E S/N: JAE180109L7 Hw: 2.0) is online

*Aug 11 21:42:27.570: %C4K_IOSMODPORTMAN-6-MODULEONLINE: STANDBY:Module 3 (WS-X45-SUP7L-E S/N: CAT1748L7GQ Hw: 3.0) is online

*Aug 11 21:42:27.570: %C4K_IOSMODPORTMAN-6-MODULEONLINE: STANDBY:Module 11 (WS-X4448-GB-SFP S/N: JAE174904Q2 Hw: 1.3) is online

*Aug 11 21:42:27.570: %C4K_IOSMODPORTMAN-6-MODULEONLINE: STANDBY:Module 12 (WS-X4606-X2-E S/N: JAE18010APJ Hw: 2.0) is online

*Aug 11 21:42:27.570: %C4K_IOSMODPORTMAN-6-MODULEONLINE: STANDBY:Module 13 (WS-X45-SUP7L-E S/N: CAT1748L7DK Hw: 3.0) is online

*Aug 11 21:42:31.233: %EC-5-BUNDLE: Interface TenGigabitEthernet2/2/3 joined port-channel Port-channel44

*Aug 11 21:42:31.235: %C4K_IOSINTF-5-TRANSCEIVERINSERTED: Slot=12 Port=3: Transceiver has been inserted

*Aug 11 21:42:31.236: %EC-5-BUNDLE: Interface TenGigabitEthernet2/2/4 joined port-channel Port-channel44

*Aug 11 21:42:31.237: %C4K_IOSINTF-5-TRANSCEIVERINSERTED: Slot=12 Port=4: Transceiver has been inserted

*Aug 11 21:42:29.497: %EC-5-BUNDLE: STANDBY:Interface TenGigabitEthernet2/2/3 joined port-channel Port-channel44

*Aug 11 21:42:29.591: %EC-5-BUNDLE: STANDBY:Interface TenGigabitEthernet2/2/4 joined port-channel Port-channel44

*Aug 11 21:42:34.355: %EC-5-BUNDLE: Interface GigabitEthernet2/1/6 joined port-channel Port-channel9

*Aug 11 21:42:34.365: %EC-5-BUNDLE: Interface GigabitEthernet2/1/7 joined port-channel Port-channel10

*Aug 11 21:42:34.369: %EC-5-BUNDLE: Interface GigabitEthernet2/1/9 joined port-channel Port-channel12

*Aug 11 21:42:34.639: %EC-5-BUNDLE: Interface GigabitEthernet2/1/1 joined port-channel Port-channel4

*Aug 11 21:42:34.644: %EC-5-BUNDLE: Interface GigabitEthernet2/1/2 joined port-channel Port-channel5

*Aug 11 21:42:34.648: %EC-5-BUNDLE: Interface GigabitEthernet2/1/4 joined port-channel Port-channel7

*Aug 11 21:42:32.675: %EC-5-BUNDLE: STANDBY:Interface GigabitEthernet2/1/6 joined port-channel Port-channel9

*Aug 11 21:42:32.677: %EC-5-BUNDLE: STANDBY:Interface GigabitEthernet2/1/7 joined port-channel Port-channel10

*Aug 11 21:42:32.679: %EC-5-BUNDLE: STANDBY:Interface GigabitEthernet2/1/9 joined port-channel Port-channel12

*Aug 11 21:42:32.830: %EC-5-BUNDLE: STANDBY:Interface GigabitEthernet2/1/1 joined port-channel Port-channel4

*Aug 11 21:42:32.832: %EC-5-BUNDLE: STANDBY:Interface GigabitEthernet2/1/2 joined port-channel Port-channel5

*Aug 11 21:42:32.925: %EC-5-BUNDLE: STANDBY:Interface GigabitEthernet2/1/4 joined port-channel Port-channel7

*Aug 11 21:42:41.407: %SFF8472-5-THRESHOLD_VIOLATION: Te2/2/3: Tx power low alarm; Operating value: -40.0 dBm, Threshold value: -12.2 dBm.

*Aug 11 21:42:41.409: %SFF8472-5-THRESHOLD_VIOLATION: Te2/2/4: Tx power low alarm; Operating value: -40.0 dBm, Threshold value: -12.2 dBm.

*Aug 12 01:13:47.063: OSPF-1 HELLO Po44: Send hello to 224.0.0.5 area 50 from 10.x.x.x

*Aug 12 01:13:49.763: OSPF-1 HELLO Po44: Rcv hello from 2.2.2.2 area 50 10.x.x.x

*Aug 12 01:14:18.944: OSPF-1 PAK : rcv. v:2 t:1 l:48 rid:2.2.2.2 aid:0.0.0.50 chk:C0FB aut:0 auk: from Port-channel44

*Aug 12 01:14:22.863: OSPF-1 PAK : rcv. v:2 t:4 l:64 rid:2.2.2.2 aid:0.0.0.50 chk:7C aut:0 auk: from Port-channel44

*Aug 12 01:14:24.160: OSPF-1 PAK : rcv. v:2 t:4 l:64 rid:2.2.2.2 aid:0.0.0.50 chk:81F6 aut:0 auk: from Port-channel44

*Aug 12 01:14:24.200: OSPF-1 PAK : rcv. v:2 t:4 l:64 rid:2.2.2.2 aid:0.0.0.50 chk:85F2 aut:0 auk: from Port-channel44

*Aug 12 01:14:24.232: OSPF-1 PAK : rcv. v:2 t:4 l:996 rid:2.2.2.2 aid:0.0.0.50 chk:FB1E aut:0 auk: from Port-channel44

*Aug 12 01:14:28.579: OSPF-1 PAK : rcv. v:2 t:1 l:48 rid:2.2.2.2 aid:0.0.0.50 chk:C0FB aut:0 auk: from Port-channel44

*Aug 12 01:14:29.214: OSPF-1 PAK : rcv. v:2 t:4 l:64 rid:2.2.2.2 aid:0.0.0.50 chk:2A52 aut:0 auk: from Port-channel44

*Aug 12 01:14:29.247: OSPF-1 PAK : rcv. v:2 t:4 l:672 rid:2.2.2.2 aid:0.0.0.50 chk:8A5C aut:0 auk: from Port-channel44

*Aug 12 01:14:34.388: OSPF-1 PAK : rcv. v:2 t:4 l:112 rid:2.2.2.2 aid:0.0.0.50 chk:202D aut:0 auk: from Port-channel44

*Aug 12 01:15:02.807: %SYS-5-CONFIG_I: Configured from console by cisco on vty0 (10.41.65.46)

*Aug 12 01:15:32.223: OSPF-1 PAK : rcv. v:2 t:4 l:60 rid:2.2.2.2 aid:0.0.0.50 chk:7607 aut:0 auk: from Port-channel44

*Aug 12 01:15:34.872: OSPF-1 PAK : rcv. v:2 t:1 l:48 rid:2.2.2.2 aid:0.0.0.50 chk:C0FB aut:0 auk: from Port-channel44

*Aug 12 01:15:44.364: OSPF-1 PAK : rcv. v:2 t:1 l:48 rid:2.2.2.2 aid:0.0.0.50 chk:C0FB aut:0 auk: from Port-channel44

*Aug 12 01:15:54.316: OSPF-1 PAK : rcv. v:2 t:1 l:48 rid:2.2.2.2 aid:0.0.0.50 chk:C0FB aut:0 auk: from Port-channel44

*Aug 12 01:16:03.781: OSPF-1 PAK : rcv. v:2 t:1 l:48 rid:2.2.2.2 aid:0.0.0.50 chk:C0FB aut:0 auk: from Port-channel44

*Aug 12 01:16:13.493: OSPF-1 PAK : rcv. v:2 t:1 l:48 rid:2.2.2.2 aid:0.0.0.50 chk:C0FB aut:0 auk: from Port-channel44

*Aug 12 01:16:22.553: OSPF-1 PAK : rcv. v:2 t:1 l:48 rid:2.2.2.2 aid:0.0.0.50 chk:C0FB aut:0 auk: from Port-channel44

*Aug 12 01:16:31.909: OSPF-1 PAK : rcv. v:2 t:1 l:48 rid:2.2.2.2 aid:0.0.0.50 chk:C0FB aut:0 auk: from Port-channel44

*Aug 12 01:16:36.179: OSPF-1 PAK : rcv. v:2 t:4 l:64 rid:2.2.2.2 aid:0.0.0.50 chk:4138 aut:0 auk: from Port-channel44

*Aug 12 01:16:36.180: OSPF-1 PAK : rcv. v:2 t:4 l:64 rid:2.2.2.2 aid:0.0.0.50 chk:4B2E aut:0 auk: from Port-channel44

*Aug 12 01:16:36.200: OSPF-1 PAK : rcv. v:2 t:4 l:568 rid:2.2.2.2 aid:0.0.0.50 chk:A189 aut:0 auk: from Port-channel44

*Aug 12 01:16:41.177: OSPF-1 PAK : rcv. v:2 t:1 l:48 rid:2.2.2.2 aid:0.0.0.50 chk:C0FB aut:0 auk: from Port-channel44

*Aug 12 01:16:50.313: OSPF-1 PAK : rcv. v:2 t:1 l:48 rid:2.2.2.2 aid:0.0.0.50 chk:C0FB aut:0 auk: from Port-channel44

*Aug 12 01:19:02.013: %SYS-5-CONFIG_I: Configured from console by cisco on vty0 (10.41.65.46)

*Aug 12 01:19:29.584: OSPF-1 PAK : rcv. v:2 t:1 l:48 rid:2.2.2.2 aid:0.0.0.50 chk:C0FB aut:0 auk: from Port-channel44

*Aug 12 01:19:39.351: OSPF-1 PAK : rcv. v:2 t:1 l:48 rid:2.2.2.2 aid:0.0.0.50 chk:C0FB aut:0 auk: from Port-channel44

*Aug 12 01:19:41.659: OSPF-1 PAK : rcv. v:2 t:4 l:88 rid:2.2.2.2 aid:0.0.0.50 chk:B3AC aut:0 auk: from Port-channel44

*Aug 12 01:19:49.043: OSPF-1 PAK : rcv. v:2 t:1 l:48 rid:2.2.2.2 aid:0.0.0.50 chk:C0FB aut:0 auk: from Port-channel44

*Aug 12 01:19:58.931: OSPF-1 PAK : rcv. v:2 t:1 l:48 rid:2.2.2.2 aid:0.0.0.50 chk:C0FB aut:0 auk: from Port-channel44

*Aug 12 01:20:08.587: OSPF-1 PAK : rcv. v:2 t:1 l:48 rid:2.2.2.2 aid:0.0.0.50 chk:C0FB aut:0 auk: from Port-channel44

*Aug 12 01:20:15.666: OSPF-1 PAK : rcv. v:2 t:4 l:64 rid:2.2.2.2 aid:0.0.0.50 chk:1266 aut:0 auk: from Port-channel44

*Aug 12 01:20:15.701: OSPF-1 PAK : rcv. v:2 t:4 l:64 rid:2.2.2.2 aid:0.0.0.50 chk:E98E aut:0 auk: from Port-channel44

*Aug 12 01:20:15.731: OSPF-1 PAK : rcv. v:2 t:4 l:424 rid:2.2.2.2 aid:0.0.0.50 chk:A127 aut:0 auk: from Port-channel44

*Aug 12 01:20:17.639: OSPF-1 PAK : rcv. v:2 t:1 l:48 rid:2.2.2.2 aid:0.0.0.50 chk:C0FB aut:0 auk: from Port-channel44

*Aug 12 01:20:26.943: OSPF-1 PAK : rcv. v:2 t:1 l:48 rid:2.2.2.2 aid:0.0.0.50 chk:C0FB aut:0 auk: from Port-channel44

*Aug 12 01:20:36.184: OSPF-1 PAK : rcv. v:2 t:1 l:48 rid:2.2.2.2 aid:0.0.0.50 chk:C0FB aut:0 auk: from Port-channel44

*Aug 12 01:20:42.923: OSPF-1 PAK : rcv. v:2 t:4 l:64 rid:2.2.2.2 aid:0.0.0.50 chk:4831 aut:0 auk: from Port-channel44

*Aug 12 01:20:42.962: OSPF-1 PAK : rcv. v:2 t:4 l:64 rid:2.2.2.2 aid:0.0.0.50 chk:2950 aut:0 auk: from Port-channel44

*Aug 12 01:20:42.991: OSPF-1 PAK : rcv. v:2 t:4 l:1468 rid:2.2.2.2 aid:0.0.0.50 chk:FD0F aut:0 auk: from Port-channel44

*Aug 12 01:20:43.032: OSPF-1 PAK : rcv. v:2 t:4 l:388 rid:2.2.2.2 aid:0.0.0.50 chk:F805 aut:0 auk: from Port-channel44

*Aug 12 01:20:46.063: OSPF-1 PAK : rcv. v:2 t:1 l:48 rid:2.2.2.2 aid:0.0.0.50 chk:C0FB aut:0 auk: from Port-channel44

*Aug 12 01:20:52.066: OSPF-1 PAK : rcv. v:2 t:4 l:100 rid:2.2.2.2 aid:0.0.0.50 chk:DE76 aut:0 auk: from Port-channel44

*Aug 12 01:20:55.407: OSPF-1 PAK : rcv. v:2 t:1 l:48 rid:2.2.2.2 aid:0.0.0.50 chk:C0FB aut:0 auk: from Port-channel44

*Aug 12 01:21:03.496: OSPF-1 PAK : rcv. v:2 t:4 l:64 rid:2.2.2.2 aid:0.0.0.50 chk:6B0D aut:0 auk: from Port-channel44

*Aug 12 01:21:03.531: OSPF-1 PAK : rcv. v:2 t:4 l:64 rid:2.2.2.2 aid:0.0.0.50 chk:F087 aut:0 auk: from Port-channel44

*Aug 12 01:21:03.563: OSPF-1 PAK : rcv. v:2 t:4 l:564 rid:2.2.2.2 aid:0.0.0.50 chk:7FA0 aut:0 auk: from Port-channel44

*Aug 12 01:21:04.659: OSPF-1 PAK : rcv. v:2 t:1 l:48 rid:2.2.2.2 aid:0.0.0.50 chk:C0FB aut:0 auk: from Port-channel44

*Aug 12 01:21:13.984: OSPF-1 PAK : rcv. v:2 t:1 l:48 rid:2.2.2.2 aid:0.0.0.50 chk:C0FB aut:0 auk: from Port-channel44

*Aug 12 01:21:23.952: OSPF-1 PAK : rcv. v:2 t:1 l:48 rid:2.2.2.2 aid:0.0.0.50 chk:C0FB aut:0 auk: from Port-channel44

*Aug 12 01:21:33.494: OSPF-1 PAK : rcv. v:2 t:1 l:48 rid:2.2.2.2 aid:0.0.0.50 chk:C0FB aut:0 auk: from Port-channel44

*Aug 12 01:21:43.252: OSPF-1 PAK : rcv. v:2 t:1 l:48 rid:2.2.2.2 aid:0.0.0.50 chk:C0FB aut:0 auk: from Port-channel44

*Aug 12 01:21:44.251: OSPF-1 PAK : rcv. v:2 t:4 l:64 rid:2.2.2.2 aid:0.0.0.50 chk:7309 aut:0 auk: from Port-channel44

*Aug 12 01:21:44.300: OSPF-1 PAK : rcv. v:2 t:4 l:64 rid:2.2.2.2 aid:0.0.0.50 chk:BCBF aut:0 auk: from Port-channel44

*Aug 12 01:21:44.328: OSPF-1 PAK : rcv. v:2 t:4 l:204 rid:2.2.2.2 aid:0.0.0.50 chk:8162 aut:0 auk: from Port-channel44

*Aug 12 01:21:52.580: OSPF-1 PAK : rcv. v:2 t:1 l:48 rid:2.2.2.2 aid:0.0.0.50 chk:C0FB aut:0 auk: from Port-channel44

*Aug 12 01:22:02.044: OSPF-1 PAK : rcv. v:2 t:1 l:48 rid:2.2.2.2 aid:0.0.0.50 chk:C0FB aut:0 auk: from Port-channel44

*Aug 12 01:22:12.032: OSPF-1 PAK : rcv. v:2 t:1 l:48 rid:2.2.2.2 aid:0.0.0.50 chk:C0FB aut:0 auk: from Port-channel44

*Aug 12 01:22:21.651: OSPF-1 PAK : rcv. v:2 t:4 l:268 rid:2.2.2.2 aid:0.0.0.50 chk:6A6 aut:0 auk: from Port-channel44

*Aug 12 01:22:21.691: OSPF-1 PAK : rcv. v:2 t:4 l:60 rid:2.2.2.2 aid:0.0.0.50 chk:235A aut:0 auk: from Port-channel44

*Aug 12 01:22:21.720: OSPF-1 PAK : rcv. v:2 t:4 l:156 rid:2.2.2.2 aid:0.0.0.50 chk:9575 aut:0 auk: from Port-channel44

*Aug 12 01:22:21.752: OSPF-1 PAK : rcv. v:2 t:1 l:48 rid:2.2.2.2 aid:0.0.0.50 chk:C0FB aut:0 auk: from Port-channel44

*Aug 12 01:22:24.595: OSPF-1 PAK : rcv. v:2 t:4 l:64 rid:2.2.2.2 aid:0.0.0.50 chk:3445 aut:0 auk: from Port-channel44

*Aug 12 01:22:24.650: OSPF-1 PAK : rcv. v:2 t:4 l:64 rid:2.2.2.2 aid:0.0.0.50 chk:E98F aut:0 auk: from Port-channel44

*Aug 12 01:22:24.684: OSPF-1 PAK : rcv. v:2 t:4 l:780 rid:2.2.2.2 aid:0.0.0.50 chk:5ED3 aut:0 auk: from Port-channel44

*Aug 12 01:22:30.940: OSPF-1 PAK : rcv. v:2 t:1 l:48 rid:2.2.2.2 aid:0.0.0.50 chk:C0FB aut:0 auk: from Port-channel44

*Aug 12 01:22:40.060: OSPF-1 PAK : rcv. v:2 t:1 l:48 rid:2.2.2.2 aid:0.0.0.50 chk:C0FB aut:0 auk: from Port-channel44

*Aug 12 01:22:49.417: OSPF-1 PAK : rcv. v:2 t:1 l:48 rid:2.2.2.2 aid:0.0.0.50 chk:C0FB aut:0 auk: from Port-channel44

*Aug 12 01:22:59.213: OSPF-1 PAK : rcv. v:2 t:1 l:48 rid:2.2.2.2 aid:0.0.0.50 chk:C0FB aut:0 auk: from Port-channel44

*Aug 12 01:23:08.881: OSPF-1 PAK : rcv. v:2 t:1 l:48 rid:2.2.2.2 aid:0.0.0.50 chk:C0FB aut:0 auk: from Port-channel44

*Aug 12 01:23:18.129: OSPF-1 PAK : rcv. v:2 t:1 l:48 rid:2.2.2.2 aid:0.0.0.50 chk:C0FB aut:0 auk: from Port-channel44

*Aug 12 01:23:27.985: OSPF-1 PAK : rcv. v:2 t:1 l:48 rid:2.2.2.2 aid:0.0.0.50 chk:C0FB aut:0 auk: from Port-channel44

*Aug 12 01:23:37.489: OSPF-1 PAK : rcv. v:2 t:1 l:48 rid:2.2.2.2 aid:0.0.0.50 chk:C0FB aut:0 auk: from Port-channel44

*Aug 12 01:23:47.325: OSPF-1 PAK : rcv. v:2 t:1 l:48 rid:2.2.2.2 aid:0.0.0.50 chk:C0FB aut:0 auk: from Port-channel44

*Aug 12 01:23:56.329: OSPF-1 PAK : rcv. v:2 t:1 l:48 rid:2.2.2.2 aid:0.0.0.50 chk:C0FB aut:0 auk: from Port-channel44

*Aug 12 01:24:05.349: OSPF-1 PAK : rcv. v:2 t:1 l:48 rid:2.2.2.2 aid:0.0.0.50 chk:C0FB aut:0 auk: from Port-channel44

*Aug 12 01:24:15.136: OSPF-1 PAK : rcv. v:2 t:1 l:48 rid:2.2.2.2 aid:0.0.0.50 chk:C0FB aut:0 auk: from Port-channel44

*Aug 12 01:24:24.381: OSPF-1 PAK : rcv. v:2 t:1 l:48 rid:2.2.2.2 aid:0.0.0.50 chk:C0FB aut:0 auk: from Port-channel44

*Aug 12 01:24:32.976: OSPF-1 PAK : rcv. v:2 t:4 l:124 rid:2.2.2.2 aid:0.0.0.50 chk:53EA aut:0 auk: from Port-channel44

*Aug 12 01:24:33.531: OSPF-1 PAK : rcv. v:2 t:1 l:48 rid:2.2.2.2 aid:0.0.0.50 chk:C0FB aut:0 auk: from Port-channel44

*Aug 12 01:24:43.026: OSPF-1 PAK : rcv. v:2 t:1 l:48 rid:2.2.2.2 aid:0.0.0.50 chk:C0FB aut:0 auk: from Port-channel44

*Aug 12 01:24:52.914: OSPF-1 PAK : rcv. v:2 t:1 l:48 rid:2.2.2.2 aid:0.0.0.50 chk:C0FB aut:0 auk: from Port-channel44

*Aug 12 01:25:02.722: OSPF-1 PAK : rcv. v:2 t:1 l:48 rid:2.2.2.2 aid:0.0.0.50 chk:C0FB aut:0 auk: from Port-channel44

*Aug 12 01:25:12.534: OSPF-1 PAK : rcv. v:2 t:1 l:48 rid:2.2.2.2 aid:0.0.0.50 chk:C0FB aut:0 auk: from Port-channel44

*Aug 12 01:25:21.602: OSPF-1 PAK : rcv. v:2 t:1 l:48 rid:2.2.2.2 aid:0.0.0.50 chk:C0FB aut:0 auk: from Port-channel44

*Aug 12 01:25:30.826: OSPF-1 PAK : rcv. v:2 t:1 l:48 rid:2.2.2.2 aid:0.0.0.50 chk:C0FB aut:0 auk: from Port-channel44

*Aug 12 01:25:40.394: OSPF-1 PAK : rcv. v:2 t:1 l:48 rid:2.2.2.2 aid:0.0.0.50 chk:C0FB aut:0 auk: from Port-channel44

*Aug 12 01:25:49.658: OSPF-1 PAK : rcv. v:2 t:1 l:48 rid:2.2.2.2 aid:0.0.0.50 chk:C0FB aut:0 auk: from Port-channel44

*Aug 12 01:25:58.691: OSPF-1 PAK : rcv. v:2 t:1 l:48 rid:2.2.2.2 aid:0.0.0.50 chk:C0FB aut:0 auk: from Port-channel44

*Aug 12 01:26:04.383: OSPF-1 PAK : rcv. v:2 t:4 l:184 rid:2.2.2.2 aid:0.0.0.50 chk:D827 aut:0 auk: from Port-channel44

*Aug 12 01:26:07.871: OSPF-1 PAK : rcv. v:2 t:1 l:48 rid:2.2.2.2 aid:0.0.0.50 chk:C0FB aut:0 auk: from Port-channel44

*Aug 12 01:26:17.635: OSPF-1 PAK : rcv. v:2 t:1 l:48 rid:2.2.2.2 aid:0.0.0.50 chk:C0FB aut:0 auk: from Port-channel44

*Aug 12 01:26:27.579: OSPF-1 PAK : rcv. v:2 t:1 l:48 rid:2.2.2.2 aid:0.0.0.50 chk:C0FB aut:0 auk: from Port-channel44

*Aug 12 01:26:36.847: OSPF-1 PAK : rcv. v:2 t:1 l:48 rid:2.2.2.2 aid:0.0.0.50 chk:C0FB aut:0 auk: from Port-channel44

*Aug 12 01:26:46.827: OSPF-1 PAK : rcv. v:2 t:1 l:48 rid:2.2.2.2 aid:0.0.0.50 chk:C0FB aut:0 auk: from Port-channel44

*Aug 12 01:26:56.413: OSPF-1 PAK : rcv. v:2 t:1 l:48 rid:2.2.2.2 aid:0.0.0.50 chk:C0FB aut:0 auk: from Port-channel44

*Aug 12 01:27:06.267: OSPF-1 PAK : rcv. v:2 t:1 l:48 rid:2.2.2.2 aid:0.0.0.50 chk:C0FB aut:0 auk: from Port-channel44

*Aug 12 01:27:06.752: OSPF-1 PAK : rcv. v:2 t:4 l:64 rid:2.2.2.2 aid:0.0.0.50 chk:87F4 aut:0 auk: from Port-channel44

*Aug 12 01:27:16.011: OSPF-1 PAK : rcv. v:2 t:1 l:48 rid:2.2.2.2 aid:0.0.0.50 chk:C0FB aut:0 auk: from Port-channel44

omz · ‎08-12-2018

*Aug 11 21:42:29.497: %EC-5-BUNDLE: STANDBY:Interface TenGigabitEthernet2/2/3 joined port-channel Port-channel44

*Aug 11 21:42:29.591: %EC-5-BUNDLE: STANDBY:Interface TenGigabitEthernet2/2/4 joined port-channel Port-channel44

.

*Aug 11 21:42:41.407: %SFF8472-5-THRESHOLD_VIOLATION: Te2/2/3: Tx power low alarm; Operating value: -40.0 dBm, Threshold value: -12.2 dBm.

*Aug 11 21:42:41.409: %SFF8472-5-THRESHOLD_VIOLATION: Te2/2/4: Tx power low alarm; Operating value: -40.0 dBm, Threshold value: -12.2 dBm.

*Aug 12 01:13:47.063: OSPF-1 HELLO Po44: Send hello to 224.0.0.5 area 50 from 10.x.x.x

*Aug 12 01:13:49.763: OSPF-1 HELLO Po44: Rcv hello from 2.2.2.2 area 50 10.x.x.x

Clean / Replace fibres and/or transceivers. Start with fibre first.

Use below to check light levels.

show interfaces transceiver threshold violations

show interfaces Te2/2/3 transceiver threshold violations

show interfaces Te2/2/4 transceiver threshold violations

show interfaces Te2/2/3 transceiver detail

show interfaces Te2/2/4 transceiver detail

coolj_212 · ‎08-12-2018

ITU Channel not available (Wavelength not available),
Transceiver is internally calibrated.
mA: milliamperes, dBm: decibels (milliwatts), NA or N/A: not applicable.
++ : high alarm, + : high warning, - : low warning, -- : low alarm.
A2D readouts (if they differ), are reported in parentheses.
The threshold values are calibrated.

High Alarm High Warn Low Warn Low Alarm
Temperature Threshold Threshold Threshold Threshold
Port (Celsius) (Celsius) (Celsius) (Celsius) (Celsius)
--------- ------------------ ---------- --------- --------- ---------
Te1/2/4 25.8 74.0 70.0 0.0 -4.0

High Alarm High Warn Low Warn Low Alarm
Voltage Threshold Threshold Threshold Threshold
Port (Volts) (Volts) (Volts) (Volts) (Volts)
--------- --------------- ---------- --------- --------- ---------
Te1/2/4 0.00 N/A N/A N/A N/A

Optical High Alarm High Warn Low Warn Low Alarm
Transmit Power Threshold Threshold Threshold Threshold
Port (dBm) (dBm) (dBm) (dBm) (dBm)
--------- ----------------- ---------- --------- --------- ---------
Te1/2/4 -2.0 4.4 0.4 -8.2 -12.2

Optical High Alarm High Warn Low Warn Low Alarm
Receive Power Threshold Threshold Threshold Threshold
Port (dBm) (dBm) (dBm) (dBm) (dBm)
------- ----------------- ---------- --------- --------- ---------
Te1/2/4 -2.6 4.4 0.4 -14.4 -18.4

DIS #show interfaces Ten 1/2/4 transceiver detail /2/4 transceiver detail 2/2/4 transceiver detail
ITU Channel not available (Wavelength not available),
Transceiver is internally calibrated.
mA: milliamperes, dBm: decibels (milliwatts), NA or N/A: not applicable.
++ : high alarm, + : high warning, - : low warning, -- : low alarm.
A2D readouts (if they differ), are reported in parentheses.
The threshold values are calibrated.

High Alarm High Warn Low Warn Low Alarm
Temperature Threshold Threshold Threshold Threshold
Port (Celsius) (Celsius) (Celsius) (Celsius) (Celsius)
--------- ------------------ ---------- --------- --------- ---------
Te2/2/4 25.6 74.0 70.0 0.0 -4.0

High Alarm High Warn Low Warn Low Alarm
Voltage Threshold Threshold Threshold Threshold
Port (Volts) (Volts) (Volts) (Volts) (Volts)
--------- --------------- ---------- --------- --------- ---------
Te2/2/4 0.00 N/A N/A N/A N/A

Optical High Alarm High Warn Low Warn Low Alarm
Transmit Power Threshold Threshold Threshold Threshold
Port (dBm) (dBm) (dBm) (dBm) (dBm)
--------- ----------------- ---------- --------- --------- ---------
Te2/2/4 -2.0 4.4 0.4 -8.2 -12.2

Optical High Alarm High Warn Low Warn Low Alarm
Receive Power Threshold Threshold Threshold Threshold
Port (dBm) (dBm) (dBm) (dBm) (dBm)
------- ----------------- ---------- --------- --------- ---------
Te2/2/4 -2.0 4.4 0.4 -14.4 -18.4

DIS #show interfaces Ten 2/2/4 transceiver detail transceiver detail 3 transceiver detail
ITU Channel not available (Wavelength not available),
Transceiver is internally calibrated.
mA: milliamperes, dBm: decibels (milliwatts), NA or N/A: not applicable.
++ : high alarm, + : high warning, - : low warning, -- : low alarm.
A2D readouts (if they differ), are reported in parentheses.
The threshold values are calibrated.

High Alarm High Warn Low Warn Low Alarm
Temperature Threshold Threshold Threshold Threshold
Port (Celsius) (Celsius) (Celsius) (Celsius) (Celsius)
--------- ------------------ ---------- --------- --------- ---------
Te2/2/3 26.8 74.0 70.0 0.0 -4.0

High Alarm High Warn Low Warn Low Alarm
Voltage Threshold Threshold Threshold Threshold
Port (Volts) (Volts) (Volts) (Volts) (Volts)
--------- --------------- ---------- --------- --------- ---------
Te2/2/3 0.00 N/A N/A N/A N/A

Optical High Alarm High Warn Low Warn Low Alarm
Transmit Power Threshold Threshold Threshold Threshold
Port (dBm) (dBm) (dBm) (dBm) (dBm)
--------- ----------------- ---------- --------- --------- ---------
Te2/2/3 -2.0 4.4 0.4 -8.2 -12.2

Optical High Alarm High Warn Low Warn Low Alarm
Receive Power Threshold Threshold Threshold Threshold
Port (dBm) (dBm) (dBm) (dBm) (dBm)
------- ----------------- ---------- --------- --------- ---------
Te2/2/3 -2.6 4.4 0.4 -14.4 -18.4