C9500-16X - %FMFP-3-OBJ_DWNLD_TO_DP_STUCK: Switch 1 R0/0: fman_fp_image: AOM download to Data Plane is stuck for more than 1800 seconds for obj[2253953] type[21] pending-issue Req-delete Issued-none 'uRPF-list(hdl=0x500000d6)'

[Daniel Marques]

Hello Guys,

Since last week my catalyst 9500 start to show the following logs:

001204: Oct 4 00:10:08.930 BST: %FMFP-6-OBJ_DWNLD_TO_DP_RESUME: Switch 1 R0/0: fman_fp_image: AOM download of objects to Data Plane is back to normal
001206: Oct 4 11:40:08.855 BST: %FMFP-3-OBJ_DWNLD_TO_DP_STUCK: Switch 1 R0/0: fman_fp_image: AOM download to Data Plane is stuck for more than 1800 seconds for obj[274354] type[21] pending-issue Req-delete Issued-none 'uRPF-list(hdl=0x5000021b)'
001207: Oct 4 17:10:08.820 BST: %FMFP-6-OBJ_DWNLD_TO_DP_RESUME: Switch 1 R0/0: fman_fp_image: AOM download of objects to Data Plane is back to normal
001208: Oct 4 18:10:08.814 BST: %FMFP-3-OBJ_DWNLD_TO_DP_STUCK: Switch 1 R0/0: fman_fp_image: AOM download to Data Plane is stuck for more than 1800 seconds for obj[274354] type[21] pending-issue Req-delete Issued-none 'uRPF-list(hdl=0x5000021b)'
001210: Oct 4 21:40:08.795 BST: %FMFP-6-OBJ_DWNLD_TO_DP_RESUME: Switch 1 R0/0: fman_fp_image: AOM download of objects to Data Plane is back to normal
001221: Oct 7 23:40:08.310 BST: %FMFP-3-OBJ_DWNLD_TO_DP_STUCK: Switch 1 R0/0: fman_fp_image: AOM download to Data Plane is stuck for more than 1800 seconds for obj[1939605] type[21] pending-issue Req-delete Issued-none 'uRPF-list(hdl=0x50000578)'
001222: Oct 8 03:40:08.281 BST: %FMFP-6-OBJ_DWNLD_TO_DP_RESUME: Switch 1 R0/0: fman_fp_image: AOM download of objects to Data Plane is back to normal
001223: Oct 8 10:40:08.234 BST: %FMFP-3-OBJ_DWNLD_TO_DP_STUCK: Switch 1 R0/0: fman_fp_image: AOM download to Data Plane is stuck for more than 1800 seconds for obj[2253953] type[21] pending-issue Req-delete Issued-none 'uRPF-list(hdl=0x500000d6)'
001224: Oct 8 11:10:08.232 BST: %FMFP-6-OBJ_DWNLD_TO_DP_RESUME: Switch 1 R0/0: fman_fp_image: AOM download of objects to Data Plane is back to normal

Checking the following Cisco doc i found this system error:

https://www.cisco.com/c/en/us/td/docs/ios-xml/ios/16_xe/smg/xe-16-10/b-sem-16-10-1/b-sem-16-10-1_chapter_010.html

%FMFP-3-OBJ_DWNLD_TO_DP_STUCK : AOM download of [chars] to Data Plane is stuck for more than [int] seconds

Explanation	An object download from FMAN-FP to lower layer has taken long time. It can be caused by incomplete configuration or software defects
Recommended Action	Run show platform software object-manager fp [active|standby] [pending-issue-update|pending-issue-batch] sorted <min_pending_time> to see the sorted list of update/batch/ command in pending state for more than the min_pending_time. For incomplete configuration, use show platform platform software object fp [active|standby] resolve to see if there is any resolve object

 %FMFP-6-OBJ_DWNLD_TO_DP_RESUME : AOM download of objects to Data Plane is back to normal

Explanation	An object download from FMAN-FP to lower layer has finished after a temporary stuck. It can be caused by incomplete configuration or software defects
Recommended Action	check if there is any functional impact after recovery

Using the commands in "Recommended Action" i don't get any conclusive idea if this cloud lead to a future problem or not.

<catalyst 9500># show platform software object-manager switch 1 R0 object-type-info | i RPF

Object type Name Count Del Pend Layer
------------------------------------------------------------------------------
RPF_CFG_IPV4 rpf_cfg_ipv4 0 0 24
RPF_CFG_IPV6 rpf_cfg_ipv6 0 0 24
RPF_LIST rpf_list 533 0 24
RPF_LOOSE_SDROP_IPV4 rpf_loose_sdrop_ipv4 0 0 24
RPF_LOOSE_SDROP_IPV6 rpf_loose_sdrop_ipv6 0 0 24

Note: The counter sometimes increment and other times decrement.

After search on the running-config for any feature related to RPF i found this four lines:

<catalyst 9500>#sh run all | i rpf

platform urpf loose counter ipv4 supress asymmetric_only
platform urpf loose counter ipv6 supress asymmetric_only
ipv6 multicast rpf use-bgp
ipv6 multicast vrf Mgmt-vrf rpf use-bgp

The software on the switch is:

Cisco IOS Software [Gibraltar], Catalyst L3 Switch Software (CAT9K_IOSXE), Version 16.11.1, RELEASE SOFTWARE (fc3)

BOOTLDR: System Bootstrap, Version 16.10.1r[FC1], RELEASE SOFTWARE (P)

Any one have a clue of what this is?

[Deepak Kumar]

Hi,

Have you checked this bug CSCvn08705

https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvn08705/?rfs=iqvred

[Daniel Marques]

Yes i checked the bug that you refer, is related with QOS and the issue is related with uRPF-list.

The switch is running an 16.11.1 and the detailed notes of the bug show this release as a "Known Fixed Releases" so this bug is fixed on my code.

Any other ideia?

[Leo Laohoo]

Upgrade to the latest 16.12.X rebuild and see if the problem goes away.

[Daniel Marques]

Hello Leo,

Reading the release notes of this version (16.12.1) i don't find any resolved Caveats related with this issue.

https://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst9500/software/release/16-12/release_notes/ol-16-12-9500.html

Do you have any evidence that an upgrade to the code will solve this issue?

[Leo Laohoo]

Raise a TAC Case. 

Based on my experience Bug IDs are seldom and/or rarely updated (or accurate).

16.11.X is not a "stable" code. 

[EdionDiasSage]

Hello!

I have the same failure. Have you opened a TAC and did they solve the failure? What have they done to resolve?

Thanks!

[Leo Laohoo]

What firmware are you running on?

[EdionDiasSage]

I'm on the most recent recommended version: Catalyst L3 Switch Software (CAT9K_IOSXE), Version 16.9.4.

[EdionDiasSage]

The issue is auto resolving itself. But I need to know this is something I should be worried about:

SW-SP-CORE_NEW#sh log | i FMFP
Oct 22 18:05:55.936: %FMFP-3-OBJ_DWNLD_TO_DP_STUCK: Switch 1 R0/0: fman_fp_image: AOM download of obj[53053] type[21] pending-issue Req-delete Issued-noneuRPF-list(hdl=0x000004cf) to Data Plane is stuck for more than 1800 seconds
Oct 22 19:05:55.935: %FMFP-6-OBJ_DWNLD_TO_DP_RESUME: Switch 1 R0/0: fman_fp_image: AOM download of objects to Data Plane is back to normal
Oct 22 19:35:55.935: %FMFP-3-OBJ_DWNLD_TO_DP_STUCK: Switch 1 R0/0: fman_fp_image: AOM download of obj[53053] type[21] pending-issue Req-delete Issued-noneuRPF-list(hdl=0x000004cf) to Data Plane is stuck for more than 1800 seconds
Oct 22 20:35:55.937: %FMFP-6-OBJ_DWNLD_TO_DP_RESUME: Switch 1 R0/0: fman_fp_image: AOM download of objects to Data Plane is back to normal
Oct 22 21:05:55.936: %FMFP-3-OBJ_DWNLD_TO_DP_STUCK: Switch 1 R0/0: fman_fp_image: AOM download of obj[53053] type[21] pending-issue Req-delete Issued-noneuRPF-list(hdl=0x000004cf) to Data Plane is stuck for more than 1800 seconds
Oct 23 07:35:55.942: %FMFP-6-OBJ_DWNLD_TO_DP_RESUME: Switch 1 R0/0: fman_fp_image: AOM download of objects to Data Plane is back to normal
SW-SP-CORE_NEW#

[Leo Laohoo]

Raise a TAC Case. I suspect this is CSCvn08705.

[Emmanuel USCLADE]

Hello!

I have the same issue using the same release on my C9500-40X. Have you raised a case and did they solve the issue ? 

Thanks!

[Kiley]

Hi there, 

I know this post is a few years old; however, I have a 9407 running 17.0.4 with firmware 17.8.1r and I got the same error message as above this morning.  Interestingly enough, the error message indicated a problem with a mac address in our quarantine vlan.  In my case, I was able to find the mac, bounce the port and clear port-security, which cleared the mac address from the CAM table.  The port still showed connected but the mac address did not register with stick mac and was still showing up as a "pending-issue-update".  I then cleared the cam table for vlan 99 and bounced the port again.  At that point, I got a message that issue was cleared and the "Data Plane was back to normal".  

Checking for pending issue updates is not something that is part of my normal checks after a reboot but it will be now.