10-28-2004
07:49 AM
- last edited on
02-21-2020
11:14 PM
by
cc_security_adm
Hi All,
I see these logs from the PIX firewalls on 2 sites. It keeps happening quite frequently and then stops for a while and then starts again. I know that there are no problems with the switch ports where the PIX is connected to, but could it be hardware related to the PIX itself.
Oct 26 03:01:10 10.20.2.7 Oct 25 2004 22:43:31: %PIX-1-104004: (Secondary) Switching to OK.
Oct 26 03:01:10 10.20.2.7 Oct 25 2004 22:43:31: %PIX-1-105003: (Secondary) Monitoring on interface 2 waiting
Oct 26 03:01:10 10.20.2.7 Oct 25 2004 22:43:31: %PIX-1-105003: (Secondary) Monitoring on interface 1 waiting
Oct 26 03:01:25 10.20.2.7 Oct 25 2004 22:43:46: %PIX-1-104004: (Secondary) Switching to OK.
Oct 26 03:01:40 10.20.2.7 Oct 25 2004 22:44:01: %PIX-1-105004: (Secondary) Monitoring on interface 2 normal
Oct 26 03:01:40 10.20.2.7 Oct 25 2004 22:44:01: %PIX-1-105004: (Secondary) Monitoring on interface 1 normal
Oct 26 03:16:46 10.20.2.6 Oct 26 2004 12:00:36: %PIX-1-103003: (Primary) Other firewall network interface 0 failed.
Oct 26 03:16:55 10.20.2.7 Oct 25 2004 22:59:16: %PIX-1-104004: (Secondary) Switching to OK.
Oct 26 03:16:55 10.20.2.7 Oct 25 2004 22:59:16: %PIX-1-105003: (Secondary) Monitoring on interface 2 waiting
Oct 26 03:16:55 10.20.2.7 Oct 25 2004 22:59:16: %PIX-1-105003: (Secondary) Monitoring on interface 1 waiting
Oct 26 03:17:10 10.20.2.7 Oct 25 2004 22:59:31: %PIX-1-104004: (Secondary) Switching to OK.
Oct 26 03:17:25 10.20.2.7 Oct 25 2004 22:59:46: %PIX-1-105004: (Secondary) Monitoring on interface 2 normal
Oct 26 03:17:25 10.20.2.7 Oct 25 2004 22:59:46: %PIX-1-105004: (Secondary) Monitoring on interface 1 normal
Oct 26 03:17:39 10.20.2.6 Oct 26 2004 12:01:29: %PIX-1-103003: (Primary) Other firewall network interface 0 failed.
Oct 26 03:17:40 10.20.2.7 Oct 25 2004 23:00:01: %PIX-1-104004: (Secondary) Switching to OK.
Oct 26 03:17:40 10.20.2.7 Oct 25 2004 23:00:01: %PIX-1-105003: (Secondary) Monitoring on interface 2 waiting
Any help would be appreciated.
10-28-2004 06:51 PM
I doubt this is a hardware related issue. My guess is that the interface 0, 1, and 2 are overloaded which is causing the hello packets between the 2 PIX's to fail. Any chance you can provide a 'sh int' from the active PIX in this failover pair?
Scott
10-29-2004 05:49 AM
Here is the "show int" output for e0 to e2 from the active PIX
interface ethernet0 "outside" is up, line protocol is up
Hardware is i82559 ethernet, address is 000f.f77a.b163
IP address x.x.x.x, subnet mask 255.255.255.128
MTU 1500 bytes, BW 100000 Kbit full duplex
3241298791 packets input, 2930622504 bytes, 0 no buffer
Received 476343 broadcasts, 0 runts, 0 giants
50 input errors, 0 CRC, 0 frame, 50 overrun, 0 ignored, 0 abort
2842147749 packets output, 257956032 bytes, 26 underruns
0 output errors, 0 collisions, 0 interface resets
0 babbles, 0 late collisions, 0 deferred
0 lost carrier, 0 no carrier
input queue (curr/max blocks): hardware (128/128) software (0/128)
output queue (curr/max blocks): hardware (0/128) software (0/20)
interface ethernet1 "inside" is up, line protocol is up
Hardware is i82559 ethernet, address is 000f.f77a.b164
IP address x.x.x.x, subnet mask 255.255.255.0
MTU 1500 bytes, BW 100000 Kbit full duplex
3307607836 packets input, 2439814712 bytes, 0 no buffer
Received 337501 broadcasts, 0 runts, 0 giants
262 input errors, 0 CRC, 0 frame, 262 overrun, 0 ignored, 0 abort
4205769341 packets output, 3658178793 bytes, 18 underruns
0 output errors, 0 collisions, 0 interface resets
0 babbles, 0 late collisions, 0 deferred
0 lost carrier, 0 no carrier
input queue (curr/max blocks): hardware (128/128) software (0/128)
output queue (curr/max blocks): hardware (2/128) software (0/3969)
interface ethernet2 "nada" is administratively down, line protocol is down
Hardware is i82558 ethernet, address is 00e0.b604.2af5
IP address x.x.x.x, subnet mask 255.255.255.0
MTU 1500 bytes, BW 10000 Kbit half duplex
0 packets input, 0 bytes, 0 no buffer
Received 0 broadcasts, 0 runts, 0 giants
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
0 packets output, 0 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 babbles, 0 late collisions, 0 deferred
0 lost carrier, 0 no carrier
input queue (curr/max blocks): hardware (128/128) software (0/0)
output queue (curr/max blocks): hardware (0/0) software (0/0)
10-29-2004 05:56 AM
Here is the "show int" command for e3 to e5 on the active PIX. Is there a way where I can turn on the
debug command just to see the hello packets? Any help would be appreciated.
interface ethernet3 "edmz" is up, line protocol is up
Hardware is i82558 ethernet, address is 00e0.b604.2af4
IP address x.x.x.x, subnet mask 255.255.255.0
MTU 1500 bytes, BW 100000 Kbit full duplex
53210714 packets input, 3031228629 bytes, 0 no buffer
Received 399756 broadcasts, 0 runts, 0 giants
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
54928993 packets output, 2279011716 bytes, 21 underruns
0 output errors, 0 collisions, 0 interface resets
0 babbles, 0 late collisions, 0 deferred
0 lost carrier, 0 no carrier
input queue (curr/max blocks): hardware (128/128) software (0/15)
output queue (curr/max blocks): hardware (0/99) software (0/1)
interface ethernet4 "radmz" is up, line protocol is up
Hardware is i82558 ethernet, address is 00e0.b604.2af3
IP address 10.20.2.1, subnet mask 255.255.255.0
MTU 1500 bytes, BW 100000 Kbit full duplex
623731113 packets input, 3683438095 bytes, 0 no buffer
Received 7716838 broadcasts, 0 runts, 0 giants
4 input errors, 4 CRC, 0 frame, 0 overrun, 4 ignored, 0 abort
741891222 packets output, 733382163 bytes, 18 underruns
0 output errors, 0 collisions, 0 interface resets
0 babbles, 0 late collisions, 0 deferred
0 lost carrier, 0 no carrier
input queue (curr/max blocks): hardware (128/128) software (0/77)
output queue (curr/max blocks): hardware (0/98) software (0/1)
interface ethernet5 "failover" is up, line protocol is up
Hardware is i82558 ethernet, address is 00e0.b604.2af2
IP address x.x.x.x, subnet mask 255.255.255.0
MTU 1500 bytes, BW 100000 Kbit full duplex
1774202 packets input, 184935558 bytes, 0 no buffer
Received 46 broadcasts, 0 runts, 0 giants
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
57391215 packets output, 3331498958 bytes, 10 underruns
0 output errors, 0 collisions, 0 interface resets
0 babbles, 0 late collisions, 0 deferred
0 lost carrier, 0 no carrier
input queue (curr/max blocks): hardware (128/128) software (0/11)
output queue (curr/max blocks): hardware (0/9) software (0/1)
10-29-2004 06:27 AM
I have the same problem, can't figure it out. My pixs are not overloaded, it happens at night sometimes.
Anyone with that has expierenced this problem please respond..
Thanks.
10-29-2004 08:21 AM
I have the same problem with a pix pair. 515's with 6.2. I even had cisco exchange equipment after changeing all cards, switchs and cables. I have been fighting this problem for over 6 months. Does not seem to affect users but I get paged when it happens. Utilization is not the issue. We use netview and the graphs look normal plus when it happens everyone is out of the office.
10-30-2004 05:58 AM
My problems seems to be with the outside interface..just like the original post, the interface is stuck in waiting. They don't affect users cause mine keep state but i need to fix this. When it flops to the secondary everthing works fine..
Anyone ever expierence this..Please help.
code level 6.3.
11-05-2004 06:35 AM
Anyone have an answer this yet or some suggestions. The PIXS use the same mac-addresses, when it flip-flops it causes some problems. The switches get confused with the mac-addresses. I also get mac-address flipping between switches error on the two switches the firewalls are on.
I have one interface on the secondary that is always in waiting.
11-05-2004 07:41 AM
Hi:
Can you post the output of "show failover" on the primary PIX, as well as the relevant failover statements to the group so we can take a look at it.
I assume that you are using LAN-based failover. Here's a sample config of LAN-based failover from CCO document just for your reference:
PRIMARY UNIT:
interface ethernet0 100full
interface ethernet1 100full
interface ethernet2 100full
interface ethernet3 100full
nameif ethernet0 outside security0
nameif ethernet1 inside security100
nameif ethernet2 failover security10
nameif ethernet3 state security20
enable password xxxx
password xxxx
telnet 192.168.2.45 255.255.255.255
hostname pixfirewall
ip address outside 209.x.x.x.255.255.224
ip address inside 192.168.2.1 255.255.255.0
ip address failover 192.168.254.1 255.255.255.0
ip address state 192.168.253.1 255.255.255.252
failover ip address outside 209.165.201.2
failover ip address inside 192.168.2.2
failover ip address failover 192.168.254.2
failover ip address state 192.168.253.2
failover link state
failover lan unit primary
failover lan interface failover
failover lan key 12345678
failover lan enable
failover
SECONDARY UNIT:
interface ethernet2 100full
nameif ethernet2 failover security10
ip address failover 192.168.254.1 255.255.255.0
failover ip address failover 192.168.254.2
failover lan unit secondary
failover lan interface failover
failover lan key 12345678
failover lan enable
failover
A few more questions:
1. Are you hard-coding the failover MAC address by using "failover mac address" statement?
2. Are all interfaces hard-coded to 100Full/1000Full?
Thanks,
Binh
11-05-2004 08:21 AM
Here is the "Show failover" output on the primary PIX
Failover On
Cable status: Normal
Reconnect timeout 0:00:00
Poll frequency 15 seconds
Last Failover at: 06:38:11 UTC Sun Jul 18 2004
This host: Primary - Active
Active time: 9537300 (sec)
Interface outside (x.x.x.x): Normal
Interface inside (10.x.x.x): Normal
Interface na (10.x.x.x): Link Down (Shutdown)
Interface edmz (10.x.x.x): Normal
Interface radmz (10.x.x.x): Normal
Interface failover (10.x.x.x): Normal
Other host: Secondary - Standby
Active time: 2070 (sec)
Interface outside (x.x.x.x): Normal
Interface inside (10.x.x.x): Normal
Interface da (10.x.x.x): Link Down (Shutdown)
Interface edmz (10.x.x.x): Normal
Interface rdmz (10.x.x.x): Normal
Interface failover (10.x.x.x): Normal
Stateful Failover Logical Update Statistics
Link : failover
Stateful Obj xmit xerr rcv rerr
General 308836146 0 1262649 0
sys cmd 1260364 0 1260364 0
up time 2 0 2 0
xlate 5701738 0 0 0
tcp conn 301690867 0 2277 0
udp conn 164964 0 6 0
ARP tbl 0 0 0 0
RIP Tbl 0 0 0 0
Logical Update Queue Information
Cur Max Total
Recv Q: 0 1 1260991
Xmit Q: 1 1 60134738
Here is the "Show Failover" on the failover PIX
Failover On
Cable status: Normal
Reconnect timeout 0:00:00
Poll frequency 15 seconds
Last Failover at: 06:38:16 UTC Sun Jul 18 2004
This host: Secondary - Standby
Active time: 2070 (sec)
Interface outside (x.x.x.x): Normal
Interface inside (10.x.x.x): Normal
Interface da (10.x.x.x): Link Down (Shutdown)
Interface edmz (10.x.x.x): Normal
Interface rdmz (10.x.x.x): Normal
Interface failover (10.x.x.x): Normal
Other host: Primary - Active
Active time: 9537705 (sec)
Interface outside (x.x.x.x): Normal
Interface inside (10.x.x.x): Normal
Interface da (10.x.x.x): Link Down (Shutdown)
Interface edmz (10.x.x.x): Normal
Interface rdmz (10.x.x.x): Normal
Interface failover (10.x.x.x): Normal
Stateful Failover Logical Update Statistics
Link : failover
Stateful Obj xmit xerr rcv rerr
General 1260353 0 308846788 0
sys cmd 1260353 0 1260351 0
up time 0 0 2 0
xlate 0 0 5702116 0
tcp conn 0 0 301701144 0
udp conn 0 0 164964 0
ARP tbl 0 0 0 0
RIP Tbl 0 0 0 0
Logical Update Queue Information
Cur Max Total
Recv Q: 0 18 60135691
Xmit Q: 0 1 1260353
No "failover mac address" is used and all interfcaes are hard-coded to 100-full.
11-05-2004 08:28 AM
Here is the "Show failover" output on the primary PIX
Failover On
Cable status: Normal
Reconnect timeout 0:00:00
Poll frequency 15 seconds
Last Failover at: 06:38:11 UTC Sun Jul 18 2004
This host: Primary - Active
Active time: 9537300 (sec)
Interface outside (x.x.x.x): Normal
Interface inside (10.x.x.x): Normal
Interface na (10.x.x.x): Link Down (Shutdown)
Interface edmz (10.x.x.x): Normal
Interface radmz (10.x.x.x): Normal
Interface failover (10.x.x.x): Normal
Other host: Secondary - Standby
Active time: 2070 (sec)
Interface outside (x.x.x.x): Normal
Interface inside (10.x.x.x): Normal
Interface da (10.x.x.x): Link Down (Shutdown)
Interface edmz (10.x.x.x): Normal
Interface rdmz (10.x.x.x): Normal
Interface failover (10.x.x.x): Normal
Stateful Failover Logical Update Statistics
Link : failover
Stateful Obj xmit xerr rcv rerr
General 308836146 0 1262649 0
sys cmd 1260364 0 1260364 0
up time 2 0 2 0
xlate 5701738 0 0 0
tcp conn 301690867 0 2277 0
udp conn 164964 0 6 0
ARP tbl 0 0 0 0
RIP Tbl 0 0 0 0
Logical Update Queue Information
Cur Max Total
Recv Q: 0 1 1260991
Xmit Q: 1 1 60134738
Here is the "Show Failover" on the failover PIX
Failover On
Cable status: Normal
Reconnect timeout 0:00:00
Poll frequency 15 seconds
Last Failover at: 06:38:16 UTC Sun Jul 18 2004
This host: Secondary - Standby
Active time: 2070 (sec)
Interface outside (x.x.x.x): Normal
Interface inside (10.x.x.x): Normal
Interface da (10.x.x.x): Link Down (Shutdown)
Interface edmz (10.x.x.x): Normal
Interface rdmz (10.x.x.x): Normal
Interface failover (10.x.x.x): Normal
Other host: Primary - Active
Active time: 9537705 (sec)
Interface outside (x.x.x.x): Normal
Interface inside (10.x.x.x): Normal
Interface da (10.x.x.x): Link Down (Shutdown)
Interface edmz (10.x.x.x): Normal
Interface rdmz (10.x.x.x): Normal
Interface failover (10.x.x.x): Normal
Stateful Failover Logical Update Statistics
Link : failover
Stateful Obj xmit xerr rcv rerr
General 1260353 0 308846788 0
sys cmd 1260353 0 1260351 0
up time 0 0 2 0
xlate 0 0 5702116 0
tcp conn 0 0 301701144 0
udp conn 0 0 164964 0
ARP tbl 0 0 0 0
RIP Tbl 0 0 0 0
Logical Update Queue Information
Cur Max Total
Recv Q: 0 18 60135691
Xmit Q: 0 1 1260353
No "failover mac address" is used and all interfcaes are hard-coded to 100-full.
11-05-2004 09:19 AM
no, I am not using lan based failover just the cable with a state. Duplex is clean, i will provide the sh int and sh fail. The outside interface seems to have the problem.
when it does failover it work fine, but it flip-flops
the collisions you see are old..I fixed the duplex problem already, i have supplied the switch interface counters also and a sh log for the secondary.
here is the primary conf.
Failover On
Cable status: Normal
Reconnect timeout 0:00:00
Poll frequency 15 seconds
Last Failover at: 08:24:12 EST Fri Nov 5 2004
This host: Primary - Active
Active time: 1196865 (sec)
Interface outside (): Normal
Interface inside (10.1.1.4): Normal
Interface PDMZ (192.168.100.1): Normal
Interface WIRELESS (192.168.2.1): Normal
Interface stateful (172.17.17.1): Normal
Interface ISOLATED (192.168.102.1): Normal
Other host: Secondary - Standby
Active time: 0 (sec)
Interface outside (): Normal (Waiting)
Interface inside (10.1.1.3): Normal
Interface PDMZ (192.168.100.2): Normal
Interface WIRELESS (192.168.2.2): Normal
Interface stateful (172.17.17.2): Normal
Interface ISOLATED (192.168.102.2): Normal
Stateful Failover Logical Update Statistics
Link : stateful
Stateful Obj xmit xerr rcv rerr
General 22124378 1 5750602 0
sys cmd 202988 0 202978 0
up time 10 0 0 0
xlate 2157484 0 564464 0
tcp conn 19763910 0 4983160 3
udp conn 0 0 0 0
ARP tbl 0 0 0 0
RIP Tbl 0 0 0 0
Logical Update Queue Information
Cur Max Total
Recv Q: 0 10 1829642
Xmit Q: 1 1 6157180
___________________________________________________
sh int
___________________________________________________
interface ethernet0 "outside" is up, line protocol is up
Hardware is i82559 ethernet, address is 00b4.0080.d29c
IP address subnet mask 255.255.255.128
MTU 1500 bytes, BW 100000 Kbit full duplex
354412584 packets input, 2721098732 bytes, 0 no buffer
Received 525278 broadcasts, 0 runts, 0 giants
1 input errors, 0 CRC, 0 frame, 1 overrun, 0 ignored, 0 abort
394810630 packets output, 3316815700 bytes, 0 underruns
0 output errors, 1277018 collisions, 0 interface resets
0 babbles, 826467 late collisions, 1370038 deferred
166 lost carrier, 0 no carrier
input queue (curr/max blocks): hardware (128/128) software (0/128)
output queue (curr/max blocks): hardware (0/100) software (0/1)
_____________________________________________________
11-05-2004 09:22 AM
continued..
___________________________________________________
switch interface counters on the switch for primary
____________________________________________________
FastEthernet0/10 is up, line protocol is up
Hardware is Fast Ethernet, address is 0050.8070.e80a (bia 0050.8070.e80a)
Description: NYPIX1-Outside
MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,
reliability 255/255, txload 5/255, rxload 4/255
Encapsulation ARPA, loopback not set
Keepalive not set
Auto-duplex (Full), Auto Speed (100), 100BaseTX/FX
ARP type: ARPA, ARP Timeout 04:00:00
Last input never, output 00:00:00, output hang never
Last clearing of "show interface" counters 2w1d
Queueing strategy: fifo
Output queue 0/40, 0 drops; input queue 0/75, 0 drops
5 minute input rate 1584000 bits/sec, 366 packets/sec
5 minute output rate 2164000 bits/sec, 373 packets/sec
335456451 packets input, 2591643272 bytes
Received 64406 broadcasts, 0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 782 ignored
0 watchdog, 0 multicast
0 input packets with dribble condition detected
299274798 packets output, 1334532211 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier
0 output buffer failures, 0 output buffers
_____________________________________________________
11-05-2004 09:22 AM
Secondary PIX
____________________________________________________
Failover On
Cable status: Normal
Reconnect timeout 0:00:00
Poll frequency 15 seconds
Last Failover at: 08:25:04 EST Fri Nov 5 2004
This host: Secondary - Standby
Active time: 0 (sec)
Interface outside (): Normal Waiting)
Interface inside (10.1.1.3): Normal
Interface PDMZ (192.168.100.2): Normal
Interface WIRELESS (192.168.2.2): Normal
Interface stateful (172.17.17.2): Normal
Interface ISOLATED (192.168.102.2): Normal
Other host: Primary - Active
Active time: 1197255 (sec)
Interface outside (): Normal
Interface inside (10.1.1.4): Normal
Interface PDMZ (192.168.100.1): Normal
Interface WIRELESS (192.168.2.1): Normal
Interface stateful (172.17.17.1): Normal
Interface ISOLATED (192.168.102.1): Normal
Stateful Failover Logical Update Statistics
Link : stateful
Stateful Obj xmit xerr rcv rerr
General 1803 0 400954 0
sys cmd 1805 0 1804 0
up time 0 0 2 0
xlate 0 0 46221 0
tcp conn 0 0 352961 0
udp conn 0 0 0 0
ARP tbl 0 0 0 0
RIP Tbl 0 0 0 0
Logical Update Queue Information
Cur Max Total
Recv Q: 0 3 83033
Xmit Q: 0 1 1805
_____________________________________________________
11-05-2004 09:24 AM
sh int outside
_____________________________________________________
interface ethernet0 "outside" is up, line protocol is up
Hardware is i82559 ethernet, address is 00b4.0080.d29c
IP address , subnet mask 255.255.255.128
MTU 1500 bytes, BW 100000 Kbit full duplex
14255 packets input, 5466275 bytes, 0 no buffer
Received 3779 broadcasts, 0 runts, 0 giants
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
3543 packets output, 213870 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 babbles, 0 late collisions, 0 deferred
0 lost carrier, 0 no carrier
input queue (curr/max blocks): hardware (128/128) software (0/3)
output queue (curr/max blocks): hardware (0/1) software (0/1)
_____________________________________________________
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide