Re: Flip-flops on the PIX failover monitoring

mchockalingam · ‎10-28-2004

Hi All,

I see these logs from the PIX firewalls on 2 sites. It keeps happening quite frequently and then stops for a while and then starts again. I know that there are no problems with the switch ports where the PIX is connected to, but could it be hardware related to the PIX itself.

Oct 26 03:01:10 10.20.2.7 Oct 25 2004 22:43:31: %PIX-1-104004: (Secondary) Switching to OK.

Oct 26 03:01:10 10.20.2.7 Oct 25 2004 22:43:31: %PIX-1-105003: (Secondary) Monitoring on interface 2 waiting

Oct 26 03:01:10 10.20.2.7 Oct 25 2004 22:43:31: %PIX-1-105003: (Secondary) Monitoring on interface 1 waiting

Oct 26 03:01:25 10.20.2.7 Oct 25 2004 22:43:46: %PIX-1-104004: (Secondary) Switching to OK.

Oct 26 03:01:40 10.20.2.7 Oct 25 2004 22:44:01: %PIX-1-105004: (Secondary) Monitoring on interface 2 normal

Oct 26 03:01:40 10.20.2.7 Oct 25 2004 22:44:01: %PIX-1-105004: (Secondary) Monitoring on interface 1 normal

Oct 26 03:16:46 10.20.2.6 Oct 26 2004 12:00:36: %PIX-1-103003: (Primary) Other firewall network interface 0 failed.

Oct 26 03:16:55 10.20.2.7 Oct 25 2004 22:59:16: %PIX-1-104004: (Secondary) Switching to OK.

Oct 26 03:16:55 10.20.2.7 Oct 25 2004 22:59:16: %PIX-1-105003: (Secondary) Monitoring on interface 2 waiting

Oct 26 03:16:55 10.20.2.7 Oct 25 2004 22:59:16: %PIX-1-105003: (Secondary) Monitoring on interface 1 waiting

Oct 26 03:17:10 10.20.2.7 Oct 25 2004 22:59:31: %PIX-1-104004: (Secondary) Switching to OK.

Oct 26 03:17:25 10.20.2.7 Oct 25 2004 22:59:46: %PIX-1-105004: (Secondary) Monitoring on interface 2 normal

Oct 26 03:17:25 10.20.2.7 Oct 25 2004 22:59:46: %PIX-1-105004: (Secondary) Monitoring on interface 1 normal

Oct 26 03:17:39 10.20.2.6 Oct 26 2004 12:01:29: %PIX-1-103003: (Primary) Other firewall network interface 0 failed.

Oct 26 03:17:40 10.20.2.7 Oct 25 2004 23:00:01: %PIX-1-104004: (Secondary) Switching to OK.

Oct 26 03:17:40 10.20.2.7 Oct 25 2004 23:00:01: %PIX-1-105003: (Secondary) Monitoring on interface 2 waiting

Any help would be appreciated.

scoclayton · ‎10-28-2004

I doubt this is a hardware related issue. My guess is that the interface 0, 1, and 2 are overloaded which is causing the hello packets between the 2 PIX's to fail. Any chance you can provide a 'sh int' from the active PIX in this failover pair?

Scott

mchockalingam · ‎10-29-2004

Here is the "show int" output for e0 to e2 from the active PIX

interface ethernet0 "outside" is up, line protocol is up

Hardware is i82559 ethernet, address is 000f.f77a.b163

IP address x.x.x.x, subnet mask 255.255.255.128

MTU 1500 bytes, BW 100000 Kbit full duplex

3241298791 packets input, 2930622504 bytes, 0 no buffer

Received 476343 broadcasts, 0 runts, 0 giants

50 input errors, 0 CRC, 0 frame, 50 overrun, 0 ignored, 0 abort

2842147749 packets output, 257956032 bytes, 26 underruns

0 output errors, 0 collisions, 0 interface resets

0 babbles, 0 late collisions, 0 deferred

0 lost carrier, 0 no carrier

input queue (curr/max blocks): hardware (128/128) software (0/128)

output queue (curr/max blocks): hardware (0/128) software (0/20)

interface ethernet1 "inside" is up, line protocol is up

Hardware is i82559 ethernet, address is 000f.f77a.b164

IP address x.x.x.x, subnet mask 255.255.255.0

MTU 1500 bytes, BW 100000 Kbit full duplex

3307607836 packets input, 2439814712 bytes, 0 no buffer

Received 337501 broadcasts, 0 runts, 0 giants

262 input errors, 0 CRC, 0 frame, 262 overrun, 0 ignored, 0 abort

4205769341 packets output, 3658178793 bytes, 18 underruns

0 output errors, 0 collisions, 0 interface resets

0 babbles, 0 late collisions, 0 deferred

0 lost carrier, 0 no carrier

input queue (curr/max blocks): hardware (128/128) software (0/128)

output queue (curr/max blocks): hardware (2/128) software (0/3969)

interface ethernet2 "nada" is administratively down, line protocol is down

Hardware is i82558 ethernet, address is 00e0.b604.2af5

IP address x.x.x.x, subnet mask 255.255.255.0

MTU 1500 bytes, BW 10000 Kbit half duplex

0 packets input, 0 bytes, 0 no buffer

Received 0 broadcasts, 0 runts, 0 giants

0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort

0 packets output, 0 bytes, 0 underruns

0 output errors, 0 collisions, 0 interface resets

0 babbles, 0 late collisions, 0 deferred

0 lost carrier, 0 no carrier

input queue (curr/max blocks): hardware (128/128) software (0/0)

output queue (curr/max blocks): hardware (0/0) software (0/0)

mchockalingam · ‎10-29-2004

Here is the "show int" command for e3 to e5 on the active PIX. Is there a way where I can turn on the

debug command just to see the hello packets? Any help would be appreciated.

interface ethernet3 "edmz" is up, line protocol is up

Hardware is i82558 ethernet, address is 00e0.b604.2af4

IP address x.x.x.x, subnet mask 255.255.255.0

MTU 1500 bytes, BW 100000 Kbit full duplex

53210714 packets input, 3031228629 bytes, 0 no buffer

Received 399756 broadcasts, 0 runts, 0 giants

0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort

54928993 packets output, 2279011716 bytes, 21 underruns

0 output errors, 0 collisions, 0 interface resets

0 babbles, 0 late collisions, 0 deferred

0 lost carrier, 0 no carrier

input queue (curr/max blocks): hardware (128/128) software (0/15)

output queue (curr/max blocks): hardware (0/99) software (0/1)

interface ethernet4 "radmz" is up, line protocol is up

Hardware is i82558 ethernet, address is 00e0.b604.2af3

IP address 10.20.2.1, subnet mask 255.255.255.0

MTU 1500 bytes, BW 100000 Kbit full duplex

623731113 packets input, 3683438095 bytes, 0 no buffer

Received 7716838 broadcasts, 0 runts, 0 giants

4 input errors, 4 CRC, 0 frame, 0 overrun, 4 ignored, 0 abort

741891222 packets output, 733382163 bytes, 18 underruns

0 output errors, 0 collisions, 0 interface resets

0 babbles, 0 late collisions, 0 deferred

0 lost carrier, 0 no carrier

input queue (curr/max blocks): hardware (128/128) software (0/77)

output queue (curr/max blocks): hardware (0/98) software (0/1)

interface ethernet5 "failover" is up, line protocol is up

Hardware is i82558 ethernet, address is 00e0.b604.2af2

IP address x.x.x.x, subnet mask 255.255.255.0

MTU 1500 bytes, BW 100000 Kbit full duplex

1774202 packets input, 184935558 bytes, 0 no buffer

Received 46 broadcasts, 0 runts, 0 giants

0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort

57391215 packets output, 3331498958 bytes, 10 underruns

0 output errors, 0 collisions, 0 interface resets

0 babbles, 0 late collisions, 0 deferred

0 lost carrier, 0 no carrier

input queue (curr/max blocks): hardware (128/128) software (0/11)

output queue (curr/max blocks): hardware (0/9) software (0/1)

rosario.garufi · ‎10-29-2004

I have the same problem, can't figure it out. My pixs are not overloaded, it happens at night sometimes.

Anyone with that has expierenced this problem please respond..

Thanks.

charlie.ford · ‎10-29-2004

I have the same problem with a pix pair. 515's with 6.2. I even had cisco exchange equipment after changeing all cards, switchs and cables. I have been fighting this problem for over 6 months. Does not seem to affect users but I get paged when it happens. Utilization is not the issue. We use netview and the graphs look normal plus when it happens everyone is out of the office.

rosario.garufi · ‎10-30-2004

My problems seems to be with the outside interface..just like the original post, the interface is stuck in waiting. They don't affect users cause mine keep state but i need to fix this. When it flops to the secondary everthing works fine..

Anyone ever expierence this..Please help.

code level 6.3.

rosario.garufi · ‎11-05-2004

Anyone have an answer this yet or some suggestions. The PIXS use the same mac-addresses, when it flip-flops it causes some problems. The switches get confused with the mac-addresses. I also get mac-address flipping between switches error on the two switches the firewalls are on.

I have one interface on the secondary that is always in waiting.

hoangbp · ‎11-05-2004

Hi:

Can you post the output of "show failover" on the primary PIX, as well as the relevant failover statements to the group so we can take a look at it.

I assume that you are using LAN-based failover. Here's a sample config of LAN-based failover from CCO document just for your reference:

PRIMARY UNIT:

interface ethernet0 100full

interface ethernet1 100full

interface ethernet2 100full

interface ethernet3 100full

nameif ethernet0 outside security0

nameif ethernet1 inside security100

nameif ethernet2 failover security10

nameif ethernet3 state security20

enable password xxxx

password xxxx

telnet 192.168.2.45 255.255.255.255

hostname pixfirewall

ip address outside 209.x.x.x.255.255.224

ip address inside 192.168.2.1 255.255.255.0

ip address failover 192.168.254.1 255.255.255.0

ip address state 192.168.253.1 255.255.255.252

failover ip address outside 209.165.201.2

failover ip address inside 192.168.2.2

failover ip address failover 192.168.254.2

failover ip address state 192.168.253.2

failover link state

failover lan unit primary

failover lan interface failover

failover lan key 12345678

failover lan enable

failover

SECONDARY UNIT:

interface ethernet2 100full

nameif ethernet2 failover security10

ip address failover 192.168.254.1 255.255.255.0

failover ip address failover 192.168.254.2

failover lan unit secondary

failover lan interface failover

failover lan key 12345678

failover lan enable

failover

A few more questions:

1. Are you hard-coding the failover MAC address by using "failover mac address" statement?

2. Are all interfaces hard-coded to 100Full/1000Full?

Thanks,

Binh

mchockalingam · ‎11-05-2004

Here is the "Show failover" output on the primary PIX

Failover On

Cable status: Normal

Reconnect timeout 0:00:00

Poll frequency 15 seconds

Last Failover at: 06:38:11 UTC Sun Jul 18 2004

This host: Primary - Active

Active time: 9537300 (sec)

Interface outside (x.x.x.x): Normal

Interface inside (10.x.x.x): Normal

Interface na (10.x.x.x): Link Down (Shutdown)

Interface edmz (10.x.x.x): Normal

Interface radmz (10.x.x.x): Normal

Interface failover (10.x.x.x): Normal

Other host: Secondary - Standby

Active time: 2070 (sec)

Interface outside (x.x.x.x): Normal

Interface inside (10.x.x.x): Normal

Interface da (10.x.x.x): Link Down (Shutdown)

Interface edmz (10.x.x.x): Normal

Interface rdmz (10.x.x.x): Normal

Interface failover (10.x.x.x): Normal

Stateful Failover Logical Update Statistics

Link : failover

Stateful Obj xmit xerr rcv rerr

General 308836146 0 1262649 0

sys cmd 1260364 0 1260364 0

up time 2 0 2 0

xlate 5701738 0 0 0

tcp conn 301690867 0 2277 0

udp conn 164964 0 6 0

ARP tbl 0 0 0 0

RIP Tbl 0 0 0 0

Logical Update Queue Information

Cur Max Total

Recv Q: 0 1 1260991

Xmit Q: 1 1 60134738

Here is the "Show Failover" on the failover PIX

Failover On

Cable status: Normal

Reconnect timeout 0:00:00

Poll frequency 15 seconds

Last Failover at: 06:38:16 UTC Sun Jul 18 2004

This host: Secondary - Standby

Active time: 2070 (sec)

Interface outside (x.x.x.x): Normal

Interface inside (10.x.x.x): Normal

Interface da (10.x.x.x): Link Down (Shutdown)

Interface edmz (10.x.x.x): Normal

Interface rdmz (10.x.x.x): Normal

Interface failover (10.x.x.x): Normal

Other host: Primary - Active

Active time: 9537705 (sec)

Interface outside (x.x.x.x): Normal

Interface inside (10.x.x.x): Normal

Interface da (10.x.x.x): Link Down (Shutdown)

Interface edmz (10.x.x.x): Normal

Interface rdmz (10.x.x.x): Normal

Interface failover (10.x.x.x): Normal

Stateful Failover Logical Update Statistics

Link : failover

Stateful Obj xmit xerr rcv rerr

General 1260353 0 308846788 0

sys cmd 1260353 0 1260351 0

up time 0 0 2 0

xlate 0 0 5702116 0

tcp conn 0 0 301701144 0

udp conn 0 0 164964 0

ARP tbl 0 0 0 0

RIP Tbl 0 0 0 0

Logical Update Queue Information

Cur Max Total

Recv Q: 0 18 60135691

Xmit Q: 0 1 1260353

No "failover mac address" is used and all interfcaes are hard-coded to 100-full.

mchockalingam · ‎11-05-2004

Here is the "Show failover" output on the primary PIX

Failover On

Cable status: Normal

Reconnect timeout 0:00:00

Poll frequency 15 seconds

Last Failover at: 06:38:11 UTC Sun Jul 18 2004

This host: Primary - Active

Active time: 9537300 (sec)

Interface outside (x.x.x.x): Normal

Interface inside (10.x.x.x): Normal

Interface na (10.x.x.x): Link Down (Shutdown)

Interface edmz (10.x.x.x): Normal

Interface radmz (10.x.x.x): Normal

Interface failover (10.x.x.x): Normal

Other host: Secondary - Standby

Active time: 2070 (sec)

Interface outside (x.x.x.x): Normal

Interface inside (10.x.x.x): Normal

Interface da (10.x.x.x): Link Down (Shutdown)

Interface edmz (10.x.x.x): Normal

Interface rdmz (10.x.x.x): Normal

Interface failover (10.x.x.x): Normal

Stateful Failover Logical Update Statistics

Link : failover

Stateful Obj xmit xerr rcv rerr

General 308836146 0 1262649 0

sys cmd 1260364 0 1260364 0

up time 2 0 2 0

xlate 5701738 0 0 0

tcp conn 301690867 0 2277 0

udp conn 164964 0 6 0

ARP tbl 0 0 0 0

RIP Tbl 0 0 0 0

Logical Update Queue Information

Cur Max Total

Recv Q: 0 1 1260991

Xmit Q: 1 1 60134738

Here is the "Show Failover" on the failover PIX

Failover On

Cable status: Normal

Reconnect timeout 0:00:00

Poll frequency 15 seconds

Last Failover at: 06:38:16 UTC Sun Jul 18 2004

This host: Secondary - Standby

Active time: 2070 (sec)

Interface outside (x.x.x.x): Normal

Interface inside (10.x.x.x): Normal

Interface da (10.x.x.x): Link Down (Shutdown)

Interface edmz (10.x.x.x): Normal

Interface rdmz (10.x.x.x): Normal

Interface failover (10.x.x.x): Normal

Other host: Primary - Active

Active time: 9537705 (sec)

Interface outside (x.x.x.x): Normal

Interface inside (10.x.x.x): Normal

Interface da (10.x.x.x): Link Down (Shutdown)

Interface edmz (10.x.x.x): Normal

Interface rdmz (10.x.x.x): Normal

Interface failover (10.x.x.x): Normal

Stateful Failover Logical Update Statistics

Link : failover

Stateful Obj xmit xerr rcv rerr

General 1260353 0 308846788 0

sys cmd 1260353 0 1260351 0

up time 0 0 2 0

xlate 0 0 5702116 0

tcp conn 0 0 301701144 0

udp conn 0 0 164964 0

ARP tbl 0 0 0 0

RIP Tbl 0 0 0 0

Logical Update Queue Information

Cur Max Total

Recv Q: 0 18 60135691

Xmit Q: 0 1 1260353

No "failover mac address" is used and all interfcaes are hard-coded to 100-full.

rosario.garufi · ‎11-05-2004

no, I am not using lan based failover just the cable with a state. Duplex is clean, i will provide the sh int and sh fail. The outside interface seems to have the problem.

when it does failover it work fine, but it flip-flops

the collisions you see are old..I fixed the duplex problem already, i have supplied the switch interface counters also and a sh log for the secondary.

here is the primary conf.

Failover On

Cable status: Normal

Reconnect timeout 0:00:00

Poll frequency 15 seconds

Last Failover at: 08:24:12 EST Fri Nov 5 2004

This host: Primary - Active

Active time: 1196865 (sec)

Interface outside (): Normal

Interface inside (10.1.1.4): Normal

Interface PDMZ (192.168.100.1): Normal

Interface WIRELESS (192.168.2.1): Normal

Interface stateful (172.17.17.1): Normal

Interface ISOLATED (192.168.102.1): Normal

Other host: Secondary - Standby

Active time: 0 (sec)

Interface outside (): Normal (Waiting)

Interface inside (10.1.1.3): Normal

Interface PDMZ (192.168.100.2): Normal

Interface WIRELESS (192.168.2.2): Normal

Interface stateful (172.17.17.2): Normal

Interface ISOLATED (192.168.102.2): Normal

Stateful Failover Logical Update Statistics

Link : stateful

Stateful Obj xmit xerr rcv rerr

General 22124378 1 5750602 0

sys cmd 202988 0 202978 0

up time 10 0 0 0

xlate 2157484 0 564464 0

tcp conn 19763910 0 4983160 3

udp conn 0 0 0 0

ARP tbl 0 0 0 0

RIP Tbl 0 0 0 0

Logical Update Queue Information

Cur Max Total

Recv Q: 0 10 1829642

Xmit Q: 1 1 6157180

___________________________________________________

sh int

___________________________________________________

interface ethernet0 "outside" is up, line protocol is up

Hardware is i82559 ethernet, address is 00b4.0080.d29c

IP address subnet mask 255.255.255.128

MTU 1500 bytes, BW 100000 Kbit full duplex

354412584 packets input, 2721098732 bytes, 0 no buffer

Received 525278 broadcasts, 0 runts, 0 giants

1 input errors, 0 CRC, 0 frame, 1 overrun, 0 ignored, 0 abort

394810630 packets output, 3316815700 bytes, 0 underruns

0 output errors, 1277018 collisions, 0 interface resets

0 babbles, 826467 late collisions, 1370038 deferred

166 lost carrier, 0 no carrier

input queue (curr/max blocks): hardware (128/128) software (0/128)

output queue (curr/max blocks): hardware (0/100) software (0/1)

_____________________________________________________

rosario.garufi · ‎11-05-2004

continued..

___________________________________________________

switch interface counters on the switch for primary

____________________________________________________

FastEthernet0/10 is up, line protocol is up

Hardware is Fast Ethernet, address is 0050.8070.e80a (bia 0050.8070.e80a)

Description: NYPIX1-Outside

MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,

reliability 255/255, txload 5/255, rxload 4/255

Encapsulation ARPA, loopback not set

Keepalive not set

Auto-duplex (Full), Auto Speed (100), 100BaseTX/FX

ARP type: ARPA, ARP Timeout 04:00:00

Last input never, output 00:00:00, output hang never

Last clearing of "show interface" counters 2w1d

Queueing strategy: fifo

Output queue 0/40, 0 drops; input queue 0/75, 0 drops

5 minute input rate 1584000 bits/sec, 366 packets/sec

5 minute output rate 2164000 bits/sec, 373 packets/sec

335456451 packets input, 2591643272 bytes

Received 64406 broadcasts, 0 runts, 0 giants, 0 throttles

0 input errors, 0 CRC, 0 frame, 0 overrun, 782 ignored

0 watchdog, 0 multicast

0 input packets with dribble condition detected

299274798 packets output, 1334532211 bytes, 0 underruns

0 output errors, 0 collisions, 0 interface resets

0 babbles, 0 late collision, 0 deferred

0 lost carrier, 0 no carrier

0 output buffer failures, 0 output buffers

_____________________________________________________

rosario.garufi · ‎11-05-2004

Secondary PIX

____________________________________________________

Failover On

Cable status: Normal

Reconnect timeout 0:00:00

Poll frequency 15 seconds

Last Failover at: 08:25:04 EST Fri Nov 5 2004

This host: Secondary - Standby

Active time: 0 (sec)

Interface outside (): Normal Waiting)

Interface inside (10.1.1.3): Normal

Interface PDMZ (192.168.100.2): Normal

Interface WIRELESS (192.168.2.2): Normal

Interface stateful (172.17.17.2): Normal

Interface ISOLATED (192.168.102.2): Normal

Other host: Primary - Active

Active time: 1197255 (sec)

Interface outside (): Normal

Interface inside (10.1.1.4): Normal

Interface PDMZ (192.168.100.1): Normal

Interface WIRELESS (192.168.2.1): Normal

Interface stateful (172.17.17.1): Normal

Interface ISOLATED (192.168.102.1): Normal

Stateful Failover Logical Update Statistics

Link : stateful

Stateful Obj xmit xerr rcv rerr

General 1803 0 400954 0

sys cmd 1805 0 1804 0

up time 0 0 2 0

xlate 0 0 46221 0

tcp conn 0 0 352961 0

udp conn 0 0 0 0

ARP tbl 0 0 0 0

RIP Tbl 0 0 0 0

Logical Update Queue Information

Cur Max Total

Recv Q: 0 3 83033

Xmit Q: 0 1 1805

_____________________________________________________

rosario.garufi · ‎11-05-2004

sh int outside

_____________________________________________________

interface ethernet0 "outside" is up, line protocol is up

Hardware is i82559 ethernet, address is 00b4.0080.d29c

IP address , subnet mask 255.255.255.128

MTU 1500 bytes, BW 100000 Kbit full duplex

14255 packets input, 5466275 bytes, 0 no buffer

Received 3779 broadcasts, 0 runts, 0 giants

0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort

3543 packets output, 213870 bytes, 0 underruns

0 output errors, 0 collisions, 0 interface resets

0 babbles, 0 late collisions, 0 deferred

0 lost carrier, 0 no carrier

input queue (curr/max blocks): hardware (128/128) software (0/3)

output queue (curr/max blocks): hardware (0/1) software (0/1)

_____________________________________________________