We recently experienced an issue with our WAE-7341 where the fail-to-wire mechanism didn't perform as expected, which has generated a couple of questions. I already have a TAC case open, but our engineer has left on vacation and we probably won't see a follow-up until next week. In the meantime, I thought I would ask here as well to see if anyone had real-world experience with this particular scenario.
The WAE's data partition needed to be rebuilt due to a software mismatch (the partition schema was from 4.0.17 and the OS was 4.4.3b), so the "disk delete-data-partition" and "reload" command were issued. At this point, the WAE started to prepare for its reload. One of the first tasks that it performed was to drop the link on its inline NIC ports. Almost five seconds later, the inline groups went into bypass mode. It did the same in reverse once the reboot was complete and the groups went back to active mode.
Questions:
- The watchdog timer, configured under the InlineGroup interface, is set to the Cisco default of 1 second. Why did it take five seconds to fail to bypass?
- Is there a way to verify the failover timer? I couldn't find any sort of show command that displayed this information.
- Why did the WAE physically drop the links on the inline NIC?
- I was previously under the assumption that the fail-to-wire mechanism was nearly hitless. Am I wrong?
Here is the actual InlineGroup configuration from the CLI:
!
interface InlineGroup 1/1
inline vlan all
exit
!
And here is the output from a "show interface" command on the same Group:
Interface is in intercept operating mode.
Standard NIC mode is off.
Disable bypass mode is off.
VLAN IDs configured for inline interception: All
Watchdog timer is enabled.
Timer frequency: 1600 ms.
Autoreset frequency 500 ms.
The watchdog timer will expire in 1140 ms.
Inline Port Statistics Of The Group:
------------------------------------
WAN Port :
===========
Device name : eth2. Bypass master interface.
Packets Received : 1532302642
Packets Intercepted: 1278405667
Packets Bridged : 253896993
Packets Forwarded : 1348015134
Packets Dropped : 0
Packets Received on native : 0
Active flows for this interface : 0
Ethernet Driver Status
-------------------------
Type:Ethernet
Ethernet address:00:E0:ED:12:8A:00
Maximum Transfer Unit Size:1500
Metric:1
Packets Received: 1532295055
Input Errors: 0
Input Packets Dropped: 0
Input Packets Overruns: 0
Input Packets Frames: 0
Packet Sent: 1510461071
Output Errors: 0
Output Packets Dropped: 0
Output Packets Overruns: 0
Output Packets Carrier: 0
Output Queue Length:1000
Collisions: 0
Base address:0x6000
Flags:UP BROADCAST RUNNING PROMISC ALLMULTI MULTICAST
Link State: Interface is up,line protocol up
Mode: autoselect, full-duplex, 1000baseTX
LAN Port :
===========
Device name : eth3. Bypass slave interface.
Packets Received : 1539048782
Packets Intercepted: 1376598851
Packets Bridged : 162449947
Packets Forwarded : 1329125070
Packets Dropped : 0
Packets Received on native : 0
Active flows for this interface : 0
Ethernet Driver Status
-------------------------
Type:Ethernet
Ethernet address:00:E0:ED:12:8A:01
Maximum Transfer Unit Size:1500
Metric:1
Packets Received: 1539040555
Input Errors: 0
Input Packets Dropped: 0
Input Packets Overruns: 0
Input Packets Frames: 0
Packet Sent: 1583018046
Output Errors: 0
Output Packets Dropped: 0
Output Packets Overruns: 0
Output Packets Carrier: 0
Output Queue Length:1000
Collisions: 0
Base address:0x6040
Flags:UP BROADCAST RUNNING PROMISC ALLMULTI MULTICAST
Link State: Interface is up,line protocol up
Mode: autoselect, full-duplex, 1000baseTX
And finally, a snippet from our log files showing the NICs going down and the inline groups entering bypass mode.
2013 Apr 10 03: kernel: %WAAS-SYS-3-900000: e1000bp: eth3: e1000bp_watchdog: NIC Link is Down
2013 Apr 10 03: kernel: %WAAS-SYS-3-900000: e1000bp: eth2: e1000bp_watchdog: NIC Link is Down
2013 Apr 10 03: kernel: %WAAS-SYS-6-900000: Inline: InlinePort wan (eth2) lost link.
2013 Apr 10 03: kernel: %WAAS-SYS-6-900000: Inline: Waiting for link on InlinePort wan (eth2).
2013 Apr 10 03: kernel: %WAAS-SYS-6-900000: Inline: InlinePort lan (eth3) shutting down.
...
2013 Apr 9 20: Sysmon: %WAAS-SYSMON-3-445007: Fault detected: The interface InlinePort 1/0/wan is in bypass mode.
2013 Apr 9 20: Sysmon: %WAAS-SYSMON-3-445007: Fault detected: The interface InlinePort 1/0/lan is in bypass mode.
2013 Apr 9 20: Sysmon: %WAAS-SYSMON-3-445007: Fault detected: The interface InlinePort 1/1/wan is in bypass mode.
2013 Apr 9 20: Sysmon: %WAAS-SYSMON-3-445007: Fault detected: The interface InlinePort 1/1/lan is in bypass mode.
Any insight would be appreciated!
