Lost Carrier issues on PIX 515e

[ucmattscott]

I have a Cisco PIX 515e device that sits behind my ISP's router.  The device has been in operation for many years with no config changes for the entire 2 years that I have been with the company.  A couple of days ago we started experiencing very random network related issues and I think I have tracked it down to the PIX.  Symptoms include:  Very slow performance on web server (but very fast when working correctly), very high ping times to servers in DMZ, etc.  Since I am not a network guy (software dev wearing many hats ), I am unsure what some of the items I am seeing really mean.  I have done a ton of investigation today, and I need some help from those that know more than I do.

Below is what I am seeing when I run a "show interface" command on the PIX (public facing IP on outside interface has been changed for security reasons).  There are 2 items that concern me.  First, is the lost carrier errors on the DMZ interface.  Second is the overrun errors on the outside interface.  I have replaced the cable on the DMZ side (many posts pointed to cabling being the culprit), but the issues I have persist.  At this point, I am thinking that there is a faulty component (cable, interface port, etc.), but I don't know how to "prove" it.

Thanks in advance for any assistance that is provided.

interface ethernet0 "outside" is up, line protocol is up

  Hardware is i82559 ethernet, address is 000e.8492.65c8

  IP address xxx.xxx.xxx.xxx, subnet mask 255.255.255.192

  MTU 1500 bytes, BW 100000 Kbit full duplex

69004224 packets input, 3364593099 bytes, 0 no buffer

Received 35197 broadcasts, 0 runts, 0 giants

1 input errors, 0 CRC, 0 frame, 1 overrun, 0 ignored, 0 abort

107167736 packets output, 3996617543 bytes, 0 underruns

0 output errors, 0 collisions, 0 interface resets

0 babbles, 0 late collisions, 0 deferred

0 lost carrier, 0 no carrier

input queue (curr/max blocks): hardware (128/128) software (1/128)

output queue (curr/max blocks): hardware (5/128) software (0/1)

interface ethernet1 "inside" is up, line protocol is up

  Hardware is i82559 ethernet, address is 000e.8492.65c9

  IP address 10.1.1.254, subnet mask 255.255.255.0

  MTU 1500 bytes, BW 100000 Kbit full duplex

237783930 packets input, 967866251 bytes, 0 no buffer

Received 276294 broadcasts, 0 runts, 0 giants

0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort

257605616 packets output, 2035478712 bytes, 0 underruns

0 output errors, 0 collisions, 0 interface resets

0 babbles, 0 late collisions, 0 deferred

30 lost carrier, 0 no carrier

input queue (curr/max blocks): hardware (128/128) software (0/57)

output queue (curr/max blocks): hardware (5/115) software (0/1)

interface ethernet2 "DMZ" is up, line protocol is up

  Hardware is i82559 ethernet, address is 0002.b3e8.2f9f

  IP address 10.2.1.254, subnet mask 255.255.255.0

  MTU 1500 bytes, BW 100000 Kbit full duplex

359324942 packets input, 2751360655 bytes, 0 no buffer

Received 8940 broadcasts, 0 runts, 0 giants

6 input errors, 0 CRC, 0 frame, 6 overrun, 0 ignored, 0 abort

300290612 packets output, 994574232 bytes, 0 underruns

0 output errors, 0 collisions, 0 interface resets

0 babbles, 0 late collisions, 0 deferred

24394 lost carrier, 0 no carrier

input queue (curr/max blocks): hardware (128/128) software (1/128)

output queue (curr/max blocks): hardware (0/125) software (0/1)

[Jouni Forss]

Hi,

Are the "inside" and "DMZ" interfaces perhaps connected to same device on the other end? Could there be a switch that is faulty? I ask this because I dont see any similiar output for the "outside" physical interface. So it would lead me to believe that if the problem isnt with cabling it might be with the opposite device.

Could you check the device where "inside" and "DMZ" are connected to for anything out of the ordinary. Is there perhaps some even older device behind the PIX on the LAN? Is it a device that could be replaced easily with some spare device to be able to monitor if the situation/problem persists?

And the PIX515E isnt exactly a fresh model anymore   Then again I've lately witnessed old PIX firewalls which have been happily up for 7 years straight! (Not certainly an indication thats everythings been handled well )

- Jouni

[ucmattscott]

Thanks for your reply Jouni...

Yes, both the inside and DMZ are connected to the same device.  However, I replaced that device last night to ensure that it wasn't the problem (replaced with a brand new Cisco SG300-28 switch).  Unfortunately, the issue started again this morning once the web traffic started to increase.

There are a couple of older 10/100 Linksys switches that are behind the PIX (off the replaced switch) that provide networking to all my end-users, but I unplugged those all yesterday during the day to see if something on that end was the culprit.  This also didn't help my issues.

The other odd behavior that I noticed today while monitoring the PIX was the traffic (inbound and outbound at different time) would gradually climb on the DMZ interface and would suddenly fall to 0 and then start it's climb back up.

I hear you on the age of the device...I can't wait to replace it!    Have a new firewall and prepping it for production now that this has happened, but a little apprehensive as our production system supports about a million unique visitors a month.

Thanks again for your time.

Matt

[ucmattscott]

Additional info from current state:

From what I can tell, the overruns mean that the traffic on the interface has exceeeded the interface's max capacity and that the firewall needs to be upgraded.

interface ethernet2 "DMZ" is up, line protocol is up

  Hardware is i82559 ethernet, address is 0002.b3e8.2f9f

  IP address 10.2.1.254, subnet mask 255.255.255.0

  MTU 1500 bytes, BW 100000 Kbit full duplex

363287686 packets input, 455639047 bytes, 0 no buffer

Received 9584 broadcasts, 0 runts, 0 giants

144 input errors, 0 CRC, 0 frame, 144 overrun, 0 ignored, 0 abort

305107926 packets output, 4056247068 bytes, 0 underruns

0 output errors, 0 collisions, 0 interface resets

0 babbles, 0 late collisions, 0 deferred

24394 lost carrier, 0 no carrier

input queue (curr/max blocks): hardware (128/128) software (0/135)

output queue (curr/max blocks): hardware (0/128) software (0/8)