Catalyst 9K Unknown Error Message In Switch Logs

[zachartl]

Hello,

We have a Catalyst 9400 that appeared to simply cease operating the other day. We had to reload it to restore it to service. We managed to get this message out of it in the logs - Nov 23 17:17:49.532: %FED_L3M_ERRMSG-3-RSRC_ERR: F0/0: fed: Failed to allocate hardware resource for Adjacency - rc:FED_L3_MCAST_OUT_OF_RESOURCES

We've tried looking it up and can not find it in any of the System Error Message Log Manuals from either IOS-XE Versions 16.9 and 17.8.1

We upgraded the Switch to 17.9.1 thinking this an IOS-XE bug but the message has returned. We thought it might have something to do with udld and disabled it on both Uplink interfaces but that did not have any affect. The message persists in the Switch Logs. I've cut and paste the Log message from today but it was the same message in IOS-XE version 16.9

Thank you,

Terry

[balaji.bandi]

It looks to me like some resource issue, how is your network?

check the  resources using :

show platform hardware fed

https://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst9500/software/release/16-6/command_reference/b_166_9500_cr/b_166_9500_cr_chapter_01011.html#wp2243595564

[MHM Cisco World]

check the TCAM utilize, 

Switch#show platform hardware fed active fwd-asic resource tcam utilization

L3 Multicast           EM           I        16384        0    0.00%        0        0       0        0
L3 Multicast           TCAM         I         1024        9    0.88%        3        6       0        0

check the value for L3 multicast 

[zachartl]

Hello,

I'm not seeing anything too compelling in the output. I've included an attachment encompassing the output in it's entirety. Would the tcam usage of the control plane, currently at 54% be deemed normal?

Thank you, you're input is most appreciated.

Terry

[MHM Cisco World]

Switch#show platform hardware fed active fwd-asic resource features ip-adjacency utilization

can also share the output of above command ? 

[zachartl]

Yes, of course, please see enclosed.

Thank you again!

[MHM Cisco World]

are you run any BGP ?
can you
show ip bgp 
show ip route 

[zachartl]

No BGP, just IGP - OSPFv2

mh10-9410a#sh ip route summ
IP routing table name is default (0x0)
IP routing table maximum-paths is 32
Route Source Networks Subnets Replicates Overhead Memory (bytes)
static 0 0 0 0 0
connected 0 27 0 3024 8424
ospf 10 2 1531 0 228928 484428
Intra-area: 162 Inter-area: 1371 External-1: 0 External-2: 0
NSSA External-1: 0 NSSA External-2: 0
internal 4 125208
Total 6 1558 0 231952 618060

[zachartl]

However, there has been a development. My switch logs just generated this message and when the suggested command is evoked, there is a plethora of information provided.

Nov 25 12:02:31.836: %FMFP_CCK-4-INCONSISTENCY_FOUND: F1/0: fman_fp_image: Consistency Checker(CCK) detected inconsistency for l3m_entry. Check 'show consistency-checker run-id 45 detail'.
Nov 25 12:02:31.836: %FMRP_CCK-4-INCONSISTENCY_FOUND: R0/0: fman_rp: Consistency Checker(CCK) detected inconsistency for l3m_entry. Check 'show consistency-checker run-id 45 detail'.
Nov 25 12:02:31.836: %FMFP_CCK-4-INCONSISTENCY_FOUND: F0/0: fman_fp_image: Consistency Checker(CCK) detected inconsistency for l3m_entry. Check 'show consistency-checker run-id 45 detail'.

Very interesting. I hope it is to you as well.

Again, Thank you!

[MHM Cisco World]

ospf 10         2           1531        0           228928      484428

WoW 1531 subnet in OSPF 10, that so huge that what eat your TCAM I think.