Hi folks,
I'm currently investigating an issue for one of our customers where one of their 3750 Core Switch Stacks crash / becomes unresponsive during a NESSUS Scan.
They've diabled DoS testing and have ensured that safe scanning is enabled. For the test they are port scanning all of their VLANs (around 600 internal addresses).
The network consists of 2x 3750 Switch Stacks connected via fiber, edge switches connect into these cores. Both cores are running HSRP, for VLAN gateway redundancy.
Issue Being faced is as follows:
During the scan, Core 1 becomes unreachable from Core 2. We can telnet to Core 2 and administer as necessary. However we cannot telnet to Core1, a console connection also fails - the switch stack is unresponsive, but does respond to pings.
On Core 2 I've performed a show proc cpu sorted and can see the IP Input process is running at around 60% and the CPU is highly utilised.
Once Core 1 becomes unreachable the network gradually grinds to a halt, almost mimicking some sort of broadcast storm or Spanning Tree loop.
Interestingly Core 1 HSRP is still active, so the hello packets are still being sent.
The only resolution to the issue is to perform a hard reset of the Core to restore service.
Logs from core 1 show the CPU becomes fully utilised. There is also an error logged indiciating:
%FIB-2-FIBDOWN: CEF has been disabled due to a low memory condition. It can be re-enabled by configuring "ip cef [distributed]"
Both cores are running IOS 12.2.(52) SE IPBASE. I've attempted to reproduce the issue in the office here and although a NESSUS scan does increase switch CPU utilisation I couldn't reproduce the failure scenario.
Has anyone else experienced similar issues or have any suggestions as to what may be causing the 1st core to become unresponsive? I've found some articles with regard to a 6500 switch rebooting during a NESSUS scan, and also some HP switches exhibiting similar behaviour but nothing that matches the exact scenario I'm investigating.
Any help or suggestions would be greatly appreciated.
Many thanks
Ryan
