CSCvk11367 - 3850 Failed to Ping New host when hardware resource exhausted

[polarbrian]

#show platform hard fed switch active fwd-asic resource tcam utilization
CAM Utilization for ASIC# 0
 Table                                              Max Values        Used Values
 --------------------------------------------------------------------------------
 Unicast MAC addresses                              32768/512          62/23
 Directly or indirectly connected routes            16384/7168       16196/7167
 L2 Multicast groups                                4096/512           0/9
 L3 Multicast groups                                4096/512           0/7
 QoS Access Control Entries                         2560               154
 Security Access Control Entries                    3072               236
 Netflow ACEs                                        768                15
 Input Microflow policer ACEs                        256                 7
 Output Microflow policer ACEs                       256                 7
 Flow SPAN ACEs                                      256                 5
 Output Flow SPAN ACEs                               256                 8
 Control Plane Entries                               512               285
 Policy Based Routing ACEs                          1024                 9
 Tunnels                                             256                17
 Lisp Instance Mapping Entries                       256                 2
 Input Security Associations                         256                 4
 SPD                                                 256                 2
 Output Security Associations and Policies           256                 9
 SGT_DGT                                            3840/512           0/0
 CLIENT_LE                                          4096/256           0/0
 INPUT_GROUP_LE                                     6144                 0
 OUTPUT_GROUP_LE                                    6144                 0
CAM Utilization for ASIC# 1
 Table                                              Max Values        Used Values
 --------------------------------------------------------------------------------
 Unicast MAC addresses                              32768/512          62/23
 Directly or indirectly connected routes            16384/7168       16196/7167
 L2 Multicast groups                                4096/512           0/9
 L3 Multicast groups                                4096/512           0/7
 QoS Access Control Entries                         2560               154
 Security Access Control Entries                    3072               236
 Netflow ACEs                                        768                15
 Input Microflow policer ACEs                        256                 7
 Output Microflow policer ACEs                       256                 7
 Flow SPAN ACEs                                      256                 5
 Output Flow SPAN ACEs                               256                 8
 Control Plane Entries                               512               285
 Policy Based Routing ACEs                          1024                 9
 Tunnels                                             256                17
 Lisp Instance Mapping Entries                       256                 2
 Input Security Associations                         256                 3
 SPD                                                 256                 2
 Output Security Associations and Policies           256                 9
 SGT_DGT                                            3840/512           0/0
 CLIENT_LE                                          4096/256           0/0
 INPUT_GROUP_LE                                     6144                 0
 OUTPUT_GROUP_LE                                    6144                 0

We are unable to have new hosts ping even the local vlan gateway on this switch. We get IP addresses in the correct scope, but then cannot ping or route out on that address. How do I fix this?

[wingwf]

We just had this exact problem, our core switch-stack was up for over 3+ years.

What we did is run the command to check both the active and standby switches in the stack.

The standby did not have the same issue as the primary (direct/indirect routes exhausted).

We forced a switchover from the Active to the standby:  redundancy force-switchover

This allowed the standby to takeover all functions, and rebooted the former Active switch.

Everything started working after that.