Solved: Re: Nexus Fail over

galaga · ‎09-19-2018

New to Nexus, we have a Nexus 56128P, we experienced Cisco Bug: CSCus68610
https://quickview.cloudapps.cisco.com/quickview/bug/CSCus68610

During this our fail over did not work since the device was hung. Is there a way to set up the fail over/vpc to fail over if an IP is not reachable or if there is an internal failure ?

sbhadrav@cisco.com · ‎09-25-2018

The fix for bug CSCus68610 was incomplete and does not address all conditions for this issue.)

A Nexus 5672 or Nexus 56128 may hang or reset with unknown reason

From the "show logging onboard internal reset-reason",
Reset Reason (HW): uC reset code: 0x4800 <<<<
OR
Reset Reason (HW): uC reset code: 0x400b

Prior to reset/hang, syslog messages such as following can be displayed (if the code version is above 6.0(2)N2(5) or 7.0(3)N1(1))
Conditions:
Correctable errors seen on the internal PCIE bus.

Workaround:
Refer to the following field notice to obtain fix for this issue
http://www.cisco.com/c/en/us/support/docs/field-notices/641/fn64110.html

View solution in original post

ADP_89 · ‎09-19-2018

Hello Galaga,

There is no way to failover the entire VPC peer as VPC is an active-active technology. If all of your interfaces are VPC port-channel you could use lacp fast timers in order to have the connected devices noticing the problem faster. Generally these are corner case that Cisco fixes with software, like in your case.

HTH,

ADP

galaga · ‎09-20-2018

Could you elaborate on lacp fast timers ?

We have a WAN router from a vendor for internet access, is there a way to monitor that IP, if that IP is not reachable, then fail over ?

When this happened we lost access to everything behind the MPLS (our LAN)

How to tie this into VCP ? or use timers ? The vendor WAN router only has (1) physical interface, I believe......

nazimkha · ‎09-20-2018

You may want to enable vPC object tracking. You can go through the following cisco live session for details on vPC
https://www.ciscolive.com/global/on-demand-library/?search=vpc#/session/14479207929320017eHp

galaga · ‎09-20-2018

With object tracking, I could add a non Nexus device (Cisco 3900 Router) via IP address, and if that IP is not reachable, SW1 goes to SW2 ? See attached PDF

Thanks for your input

ADP_89 · ‎09-20-2018

Normally LACP ports will drop out from a Port-Channel once they do not receive LACP packets for 90 seconds (basically 3 miss LACPDU as they are sent every 30seconds). Fast rate LACP are sent instead every 1 second, and eventually the port will be deactivated after 3 seconds. So they will help you only if the devices connected to the Nexus are in a Port-Channel. If your router is connected to you LAN only via a single interface you will not achieve any improvements.

I am not too sure on what failover you would like to achieve, can you post a simple topology of your network and make a better statement on what you would like to get in a specific condition?

Cheers,

ADP

galaga · ‎09-20-2018

galaga · ‎09-20-2018

It appears out management interface was down that may be why vPC did not kick in.

Is there a design improvement to check for VPC/Management interface connectivity ?

Add auto recovery and or object tracking ?

galaga · ‎09-20-2018

[image: image.png]

nazimkha · ‎09-20-2018

Hi,

For optimal operation of vPC it is recommended to dual-home to all external devices. In your case it would be ideal to dual-home to the vendor switch.

If you cannot dual home to the vendor switch, I would suggest that you connect the link to the ISP using the switch which is vPC primary. Infact all single homed devices should be connected to vPC primary. As vPC secondary switch suspends all its connection if the vPC peer-link between the switches fail.

Also are you running IP routing protocol between the nexus and the vendor switch, if yes, you may have to run the NXOS version 7.3(0)N1(1)

Dynamic routing over vPC or L3 vPC is only supported from this release.

https://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus5500/sw/release/notes/7x/Nexus5500_Release_Notes_7x.html#pgfId-622079

galaga · ‎09-20-2018

In our case turns out management interface was down, we experienced Cisco Bug: CSCus68610 on 5K1, which hung. NK2 did not kick in, Rebooting 5K1 resolved the issue.

Everything (our LAN) behind the Vendor WAN router Cisco 3900 was not reachable.

Sorry still learning, homing this is the solution ? Need to add a physical interface on the vendor WAN router to do this ?

I am trying to figure out if we encountered the perfect storm, or we have a design flaw. We should update the IOS, but in the mean time I would like to fix the design flaw.

IOS 7.0(7)

vpc domain 1
role priority 7096
peer-keepalive destination 158.100.1.2 source 158.100.1.1
delay restore 150

interface mgmt0
vrf member management
ip address 158.100.1.1/24

interface port-channel100
description ndcsw0102 (vpc-peer link)
switchport mode trunk
spanning-tree port type network
speed 40000
vpc peer-link

ADP_89 · ‎09-20-2018

Your WAN router needs to suppor LACP as well so you can team the two interfaces into a Port-Channel. The 2 interfaces will be connected one per Nexus and the configuration will look like the following:

SW1:

interface Ethernet1/5

description TO_MPLS_ROUTER

channel-group 5 mode active

interface Po5

switchport

switchport mode access

switchport access VLAN X

vpc 5

SW2:

interface Ethernet1/5

description TO_MPLS_ROUTER

channel-group 5 mode active

interface Po5

switchport

switchport mode access

switchport access VLAN X

vpc 5

But ideally all your devices should be dual homed to both Nexus.

One more thing, you are saying that NK2 did not kick in, that is not clear. VPC switches are always active active, can you post a "show vpc" from both devices so we can check the status?

Thanks,

ADP

galaga · ‎09-20-2018

By homing, you mean configuring the WAN as you stated ?

NK2 did not kick in, by say that, I assume if it was working correctly, we would not have had an outage.

ADP_89 · ‎09-20-2018

By dual homing I mean connecting the device (in your case the 3900) to both Nexus, and configuring the interfaces as I told you earlier.

galaga · ‎09-20-2018

NK1

Legend:
(*) - local vPC is down, forwarding via vPC peer-link

vPC domain id : 1
Peer status : peer adjacency formed ok
vPC keep-alive status : peer is alive
Configuration consistency status : success
Per-vlan consistency status : success
Type-2 consistency status : success
vPC role : primary, operational secondary
Number of vPCs configured : 592
Peer Gateway : Disabled
Dual-active excluded VLANs : -
Graceful Consistency Check : Enabled
Auto-recovery status : Enabled (timeout = 240 seconds)

vPC Peer-link status
---------------------------------------------------------------------
id Port Status Active vlans
-- ---- ------ --------------------------------------------------
1 Po100 up 1,10,20,22,50,100-104,150-154,200-204,206-208,210-
214,300-305,310-313,350-354,400-404,450-454,500-50
4,671,681,690,700-702,707-708,710-711,730-731,800,
997-999,1100

vPC status
----------------------------------------------------------------------------
id Port Status Consistency Reason Active vlans
------ ----------- ------ ----------- -------------------------- -----------
101 Po101 up success success -
102 Po102 up success success -
103 Po103 up success success -
104 Po104 up success success -
105 Po105 up success success -
106 Po106 up success success -
107 Po107 up success success -
108 Po108 up success success -
109 Po109 up success success -
110 Po110 up success success -
111 Po111 up success success -
112 Po112 up success success -
102400 Eth101/1/1 up success success 102

102401 Eth101/1/2 up success success 103

102402 Eth101/1/3 down* Not Consistency Check Not -
Applicable Performed
102403 Eth101/1/4 down* Not Consistency Check Not -
Applicable Performed
102404 Eth101/1/5 down* Not Consistency Check Not -
Applicable Performed
102405 Eth101/1/6 down* Not Consistency Check Not -
Applicable Performed
102406 Eth101/1/7 down* Not Consistency Check Not -
Applicable Performed
102407 Eth101/1/8 down* Not Consistency Check Not -
Applicable Performed
102408 Eth101/1/9 down* Not Consistency Check Not -
Applicable Performed
102409 Eth101/1/10 down* Not Consistency Check Not -
Applicable Performed
102410 Eth101/1/11 down* Not Consistency Check Not -
Applicable Performed
102411 Eth101/1/12 up success success 102

102412 Eth101/1/13 down* Not Consistency Check Not -
Applicable Performed
102413 Eth101/1/14 down* Not Consistency Check Not -
Applicable Performed
102414 Eth101/1/15 down* Not Consistency Check Not -
Applicable Performed
102415 Eth101/1/16 down* Not Consistency Check Not -
Applicable Performed
102416 Eth101/1/17 down* Not Consistency Check Not -
Applicable Performed
102417 Eth101/1/18 down* Not Consistency Check Not -
Applicable Performed
102418 Eth101/1/19 down* Not Consistency Check Not -
Applicable Performed
102419 Eth101/1/20 down* Not Consistency Check Not -
Applicable Performed
102420 Eth101/1/21 down* Not Consistency Check Not -
Applicable Performed
102421 Eth101/1/22 down* Not Consistency Check Not -
Applicable Performed
102422 Eth101/1/23 up success success 210

102423 Eth101/1/24 up success success 208

102424 Eth101/1/25 down* Not Consistency Check Not -
Applicable Performed
102425 Eth101/1/26 down* Not Consistency Check Not -
Applicable Performed
102426 Eth101/1/27 up success success 100

102427 Eth101/1/28 down* Not Consistency Check Not -
Applicable Performed
102428 Eth101/1/29 down* Not Consistency Check Not -
Applicable Performed
102429 Eth101/1/30 down* Not Consistency Check Not -
Applicable Performed
102430 Eth101/1/31 down* Not Consistency Check Not -
Applicable Performed
102431 Eth101/1/32 down* Not Consistency Check Not -
Applicable Performed
102432 Eth101/1/33 down* Not Consistency Check Not -
Applicable Performed
102433 Eth101/1/34 down* Not Consistency Check Not -
Applicable Performed
102434 Eth101/1/35 down* Not Consistency Check Not -
Applicable Performed
102435 Eth101/1/36 up success success 504

102436 Eth101/1/37 down* Not Consistency Check Not -
Applicable Performed
102437 Eth101/1/38 down* Not Consistency Check Not -
Applicable Performed
102438 Eth101/1/39 down* Not Consistency Check Not -
Applicable Performed
102439 Eth101/1/40 down* Not Consistency Check Not -
Applicable Performed
102441 Eth101/1/42 up success success 102

102444 Eth101/1/45 up success success 310

102445 Eth101/1/46 down* Not Consistency Check Not -
Applicable Performed
102446 Eth101/1/47 down* Not Consistency Check Not -
Applicable Performed
102447 Eth101/1/48 down* Not Consistency Check Not -
Applicable Performed
103424 Eth102/1/1 up success success 102

103425 Eth102/1/2 up success success 103

103426 Eth102/1/3 down* Not Consistency Check Not -
Applicable Performed
103427 Eth102/1/4 down* Not Consistency Check Not -
Applicable Performed
103428 Eth102/1/5 down* Not Consistency Check Not -
Applicable Performed
103429 Eth102/1/6 down* Not Consistency Check Not -
Applicable Performed
103430 Eth102/1/7 up success success 711

103431 Eth102/1/8 down* Not Consistency Check Not -
Applicable Performed
103432 Eth102/1/9 up success success 800

103433 Eth102/1/10 down* Not Consistency Check Not -
Applicable Performed
103434 Eth102/1/11 down* Not Consistency Check Not -
Applicable Performed
103435 Eth102/1/12 up success success 102

103436 Eth102/1/13 down* Not Consistency Check Not -
Applicable Performed
103437 Eth102/1/14 down* Not Consistency Check Not -
Applicable Performed
103438 Eth102/1/15 down* Not Consistency Check Not -
Applicable Performed
103439 Eth102/1/16 down* Not Consistency Check Not -
Applicable Performed
103440 Eth102/1/17 down* Not Consistency Check Not -
Applicable Performed
103441 Eth102/1/18 down* Not Consistency Check Not -
Applicable Performed
103442 Eth102/1/19 down* Not Consistency Check Not -
Applicable Performed
103443 Eth102/1/20 down* Not Consistency Check Not -
Applicable Performed
103444 Eth102/1/21 down* Not Consistency Check Not -
Applicable Performed
103445 Eth102/1/22 down* Not Consistency Check Not -
Applicable Performed
103446 Eth102/1/23 up success success 210

103447 Eth102/1/24 up success success 208

103448 Eth102/1/25 down* Not Consistency Check Not -
Applicable Performed
103449 Eth102/1/26 down* Not Consistency Check Not -
Applicable Performed
103450 Eth102/1/27 down* Not Consistency Check Not -
Applicable Performed
103451 Eth102/1/28 down* Not Consistency Check Not -
Applicable Performed
103452 Eth102/1/29 down* Not Consistency Check Not -
Applicable Performed
103453 Eth102/1/30 down* Not Consistency Check Not -
Applicable Performed
103454 Eth102/1/31 down* Not Consistency Check Not -
Applicable Performed
103455 Eth102/1/32 down* Not Consistency Check Not -
Applicable Performed
103456 Eth102/1/33 down* Not Consistency Check Not -
Applicable Performed
103457 Eth102/1/34 down* Not Consistency Check Not -
Applicable Performed
103458 Eth102/1/35 down* Not Consistency Check Not -
Applicable Performed
103459 Eth102/1/36 up success success 504

103460 Eth102/1/37 down* Not Consistency Check Not -
Applicable Performed
103461 Eth102/1/38 down* Not Consistency Check Not -
Applicable Performed
103462 Eth102/1/39 down* Not Consistency Check Not -
Applicable Performed
103463 Eth102/1/40 down* Not Consistency Check Not -
Applicable Performed
1034 Eth102/1/41 down* Not Consistency Check Not -
64 Applicable Performed
103465 Eth102/1/42 down* Not Consistency Check Not -
Applicable Performed
103468 Eth102/1/45 up success success 310

103469 Eth102/1/46 down* Not Consistency Check Not -
Applicable Performed
103470 Eth102/1/47 down* Not Consistency Check Not -
Applicable Performed
103471 Eth102/1/48 down* Not Consistency Check Not -
Applicable Performed
104448 Eth103/1/1 down* Not Consistency Check Not -
Applicable Performed
104449 Eth103/1/2 down* Not Consistency Check Not -
Applicable Performed
104450 Eth103/1/3 down* Not Consistency Check Not -
Applicable Performed
104451 Eth103/1/4 down* Not Consistency Check Not -
Applicable Performed
104452 Eth103/1/5 down* Not Consistency Check Not -
Applicable Performed
104453 Eth103/1/6 down* Not Consistency Check Not -
Applicable Performed
104454 Eth103/1/7 down* Not Consistency Check Not -
Applicable Performed
104455 Eth103/1/8 down* Not Consistency Check Not -
Applicable Performed
104456 Eth103/1/9 down* Not Consistency Check Not -
Applicable Performed
104457 Eth103/1/10 down* Not Consistency Check Not -
Applicable Performed
104458 Eth103/1/11 down* Not Consistency Check Not -
Applicable Performed
104459 Eth103/1/12 up success success 102

104460 Eth103/1/13 down* Not Consistency Check Not -
Applicable Performed
104461 Eth103/1/14 up success success 102

104462 Eth103/1/15 down* Not Consistency Check Not -
Applicable Performed
104463 Eth103/1/16 down* Not Consistency Check Not -
Applicable Performed
104464 Eth103/1/17 down* Not Consistency Check Not -
Applicable Performed
104465 Eth103/1/18 down* Not Consistency Check Not -
Applicable Performed
104466 Eth103/1/19 down* Not Consistency Check Not -
Applicable Performed
104467 Eth103/1/20 down* Not Consistency Check Not -
Applicable Performed
104468 Eth103/1/21 up success success 102

104469 Eth103/1/22 down* Not Consistency Check Not -
Applicable Performed
104470 Eth103/1/23 down* Not Consistency Check Not -
Applicable Performed
104471 Eth103/1/24 down* Not Consistency Check Not -
Applicable Performed
104472 Eth103/1/25 up success success 102

104473 Eth103/1/26 down* Not Consistency Check Not -
Applicable Performed
104474 Eth103/1/27 down* Not Consistency Check Not -
Applicable Performed
104475 Eth103/1/28 down* Not Consistency Check Not -
Applicable Performed
104476 Eth103/1/29 down* Not Consistency Check Not -
Applicable Performed
104477 Eth103/1/30 down* Not Consistency Check Not -
Applicable Performed
104478 Eth103/1/31 down* Not Consistency Check Not -
Applicable Performed
104479 Eth103/1/32 down* Not Consistency Check Not -
Applicable Performed
104480 Eth103/1/33 down* Not Consistency Check Not -
Applicable Performed
104481 Eth103/1/34 down* Not Consistency Check Not -
Applicable Performed
104482 Eth103/1/35 down* Not Consistency Check Not -
Applicable Performed
104483 Eth103/1/36 down* Not Consistency Check Not -
Applicable Performed
104484 Eth103/1/37 down* Not Consistency Check Not -
Applicable Performed
104485 Eth103/1/38 down* Not Consistency Check Not -
Applicable Performed
104486 Eth103/1/39 down* Not Consistency Check Not -
Applicable Performed
104487 Eth103/1/40 down* Not Consistency Check Not -
Applicable Performed
104488 Eth103/1/41 down* Not Consistency Check Not -
Applicable Performed
104489 Eth103/1/42 down* Not Consistency Check Not -
Applicable Performed
104490 Eth103/1/43 down* Not Consistency Check Not -
Applicable Performed
104491 Eth103/1/44 down* Not Consistency Check Not -
Applicable Performed
104492 Eth103/1/45 down* Not Consistency Check Not -
Applicable Performed
104493 Eth103/1/46 down* Not Consistency Check Not -
Applicable Performed
104494 Eth103/1/47 down* Not Consistency Check Not -
Applicable Performed
104495 Eth103/1/48 down* Not Consistency Check Not -
Applicable Performed
105472 Eth104/1/1 down* Not Consistency Check Not -
Applicable Performed
105473 Eth104/1/2 down* Not Consistency Check Not -
Applicable Performed
105474 Eth104/1/3 down* Not Consistency Check Not -
Applicable Performed
105475 Eth104/1/4 up success success 200

105476 Eth104/1/5 down* Not Consistency Check Not -
Applicable Performed
105477 Eth104/1/6 down* Not Consistency Check Not -
Applicable Performed
105478 Eth104/1/7 down* Not Consistency Check Not -
Applicable Performed
105479 Eth104/1/8 down* Not Consistency Check Not -
Applicable Performed
105480 Eth104/1/9 down* Not Consistency Check Not -
Applicable Performed
105481 Eth104/1/10 down* Not Consistency Check Not -
Applicable Performed
105482 Eth104/1/11 up success success 1

105483 Eth104/1/12 up success success 102

105484 Eth104/1/13 down* Not Consistency Check Not -
Applicable Performed
105485 Eth104/1/14 down* Not Consistency Check Not -
Applicable Performed
105486 Eth104/1/15 down* Not Consistency Check Not -
Applicable Performed
105487 Eth104/1/16 down* Not Consistency Check Not -
Applicable Performed
105488 Eth104/1/17 down* Not Consistency Check Not -
Applicable Performed
105489 Eth104/1/18 down* Not Consistency Check Not -
Applicable Performed
105490 Eth104/1/19 down* Not Consistency Check Not -
Applicable Performed
105491 Eth104/1/20 down* Not Consistency Check Not -
Applicable Performed
105492 Eth104/1/21 down* Not Consistency Check Not -
Applicable Performed
105493 Eth104/1/22 down* Not Consistency Check Not -
Applicable Performed
105494 Eth104/1/23 down* Not Consistency Check Not -
Applicable Performed
105495 Eth104/1/24 down* Not Consistency Check Not -
Applicable Performed
105496 Eth104/1/25 up success success 102

105497 Eth104/1/26 down* Not Consistency Check Not -
Applicable Performed
105498 Eth104/1/27 down* Not Consistency Check Not -
Applicable Performed
105499 Eth104/1/28 down* Not Consistency Check Not -
Applicable Performed
105500 Eth104/1/29 down* Not Consistency Check Not -
Applicable Performed
105501 Eth104/1/30 down* Not Consistency Check Not -
Applicable Performed
105502 Eth104/1/31 down* Not Consistency Check Not -
Applicable Performed
105503 Eth104/1/32 down* Not Consistency Check Not -
Applicable Performed
105504 Eth104/1/33 down* Not Consistency Check Not -
Applicable Performed
105505 Eth104/1/34 down* Not Consistency Check Not -
Applicable Performed
105506 Eth104/1/35 down* Not Consistency Check Not -
Applicable Performed
105507 Eth104/1/36 down* Not Consistency Check Not -
Applicable Performed
105508 Eth104/1/37 down* Not Consistency Check Not -
Applicable Performed
105509 Eth104/1/38 down* Not Consistency Check Not -
Applicable Performed
105510 Eth104/1/39 down* Not Consistency Check Not -
Applicable Performed
105511 Eth104/1/40 down* Not Consistency Check Not -
Applicable Performed
105512 Eth104/1/41 down* Not Consistency Check Not -
Applicable Performed
105513 Eth104/1/42 down* Not Consistency Check Not -
Applicable Performed
105514 Eth104/1/43 down* Not Consistency Check Not -
Applicable Performed
105515 Eth104/1/44 down* Not Consistency Check Not -
Applicable Performed
1055 Eth104/1/45 down* Not Consistency Check Not -
16 Applicable Performed
105517 Eth104/1/46 down* Not Consistency Check Not -
Applicable Performed
105518 Eth104/1/47 down* Not Consistency Check Not -
Applicable Performed
105519 Eth104/1/48 down* Not Consistency Check Not -
Applicable Performed
106500 Eth105/1/5 up success success 100,102

106501 Eth105/1/6 down* Not Consistency Check Not -
Applicable Performed
106502 Eth105/1/7 down* Not Consistency Check Not -
Applicable Performed
106503 Eth105/1/8 down* Not Consistency Check Not -
Applicable Performed
106504 Eth105/1/9 down* Not Consistency Check Not -
Applicable Performed
106505 Eth105/1/10 down* Not Consistency Check Not -
Applicable Performed
106506 Eth105/1/11 down* Not Consistency Check Not -
Applicable Performed
106507 Eth105/1/12 down* Not Consistency Check Not -
Applicable Performed
106508 Eth105/1/13 up success success 100,102

106509 Eth105/1/14 down* Not Consistency Check Not -
Applicable Performed
106510 Eth105/1/15 up success success 102

106511 Eth105/1/16 down* Not Consistency Check Not -
Applicable Performed
106512 Eth105/1/17 down* Not Consistency Check Not -
Applicable Performed
106513 Eth105/1/18 down* Not Consistency Check Not -
Applicable Performed
106514 Eth105/1/19 down* Not Consistency Check Not -
Applicable Performed
106515 Eth105/1/20 down* Not Consistency Check Not -
Applicable Performed
106516 Eth105/1/21 down* Not Consistency Check Not -
Applicable Performed
106517 Eth105/1/22 down* Not Consistency Check Not -
Applicable Performed
106518 Eth105/1/23 down* Not Consistency Check Not -
Applicable Performed
106519 Eth105/1/24 down* Not Consistency Check Not -
Applicable Performed
106520 Eth105/1/25 down* Not Consistency Check Not -
Applicable Performed
106521 Eth105/1/26 down* Not Consistency Check Not -
Applicable Performed
106522 Eth105/1/27 down* Not Consistency Check Not -
Applicable Performed
106523 Eth105/1/28 down* Not Consistency Check Not -
Applicable Performed
106524 Eth105/1/29 down* Not Consistency Check Not -
Applicable Performed
106525 Eth105/1/30 down* Not Consistency Check Not -
Applicable Performed
106526 Eth105/1/31 down* Not Consistency Check Not -
Applicable Performed
106527 Eth105/1/32 down* Not Consistency Check Not -
Applicable Performed
106528 Eth105/1/33 down* Not Consistency Check Not -
Applicable Performed
106529 Eth105/1/34 down* Not Consistency Check Not -
Applicable Performed
106530 Eth105/1/35 down* Not Consistency Check Not -
Applicable Performed
106531 Eth105/1/36 down* Not Consistency Check Not -
Applicable Performed
106532 Eth105/1/37 down* Not Consistency Check Not -
Applicable Performed
106533 Eth105/1/38 down* Not Consistency Check Not -
Applicable Performed
106534 Eth105/1/39 down* Not Consistency Check Not -
Applicable Performed
106535 Eth105/1/40 down* Not Consistency Check Not -
Applicable Performed
106536 Eth105/1/41 down* Not Consistency Check Not -
Applicable Performed
106537 Eth105/1/42 down* Not Consistency Check Not -
Applicable Performed
106538 Eth105/1/43 down* Not Consistency Check Not -
Applicable Performed
106539 Eth105/1/44 down* Not Consistency Check Not -
Applicable Performed
106540 Eth105/1/45 up success success 102

106541 Eth105/1/46 down* Not Consistency Check Not -
Applicable Performed
106542 Eth105/1/47 down* Not Consistency Check Not -
Applicable Performed
106543 Eth105/1/48 down* Not Consistency Check Not -
Applicable Performed
107524 Eth106/1/5 up success success 100,102

107525 Eth106/1/6 down* Not Consistency Check Not -
Applicable Performed
107526 Eth106/1/7 down* Not Consistency Check Not -
Applicable Performed
107527 Eth106/1/8 down* Not Consistency Check Not -
Applicable Performed
107528 Eth106/1/9 down* Not Consistency Check Not -
Applicable Performed
107529 Eth106/1/10 down* Not Consistency Check Not -
Applicable Performed
107530 Eth106/1/11 down* Not Consistency Check Not -
Applicable Performed
107531 Eth106/1/12 down* Not Consistency Check Not -
Applicable Performed
107532 Eth106/1/13 down* Not Consistency Check Not -
Applicable Performed
107533 Eth106/1/14 down* Not Consistency Check Not -
Applicable Performed
107534 Eth106/1/15 down* Not Consistency Check Not -
Applicable Performed
107535 Eth106/1/16 up success success 100,102

107536 Eth106/1/17 down* Not Consistency Check Not -
Applicable Performed
107537 Eth106/1/18 down* Not Consistency Check Not -
Applicable Performed
107538 Eth106/1/19 down* Not Consistency Check Not -
Applicable Performed
107539 Eth106/1/20 down* Not Consistency Check Not -
Applicable Performed
107540 Eth106/1/21 down* Not Consistency Check Not -
Applicable Performed
107541 Eth106/1/22 down* Not Consistency Check Not -
Applicable Performed
107542 Eth106/1/23 down* Not Consistency Check Not -
Applicable Performed
107543 Eth106/1/24 down* Not Consistency Check Not -
Applicable Performed
107544 Eth106/1/25 down* Not Consistency Check Not -
Applicable Performed
107545 Eth106/1/26 down* Not Consistency Check Not -
Applicable Performed
107546 Eth106/1/27 down* Not Consistency Check Not -
Applicable Performed
107547 Eth106/1/28 down* Not Consistency Check Not -
Applicable Performed
107548 Eth106/1/29 down* Not Consistency Check Not -
Applicable Performed
107549 Eth106/1/30 down* Not Consistency Check Not -
Applicable Performed
107550 Eth106/1/31 down* Not Consistency Check Not -
Applicable Performed
107551 Eth106/1/32 down* Not Consistency Check Not -
Applicable Performed
107552 Eth106/1/33 down* Not Consistency Check Not -
Applicable Performed
107553 Eth106/1/34 down* Not Consistency Check Not -
Applicable Performed
107554 Eth106/1/35 down* Not Consistency Check Not -
Applicable Performed
107555 Eth106/1/36 down* Not Consistency Check Not -
Applicable Performed
107556 Eth106/1/37 down* Not Consistency Check Not -
Applicable Performed
107557 Eth106/1/38 down* Not Consistency Check Not -
Applicable Performed
107558 Eth106/1/39 down* Not Consistency Check Not -
Applicable Performed
107559 Eth106/1/40 down* Not Consistency Check Not -
Applicable Performed
107560 Eth106/1/41 down* Not Consistency Check Not -
Applicable Performed
107561 Eth106/1/42 down* Not Consistency Check Not -
Applicable Performed
107562 Eth106/1/43 down* Not Consistency Check Not -
Applicable Performed
107563 Eth106/1/44 down* Not Consistency Check Not -
Applicable Performed
107564 Eth106/1/45 up success success 102

107565 Eth106/1/46 down* Not Consistency Check Not -
Applicable Performed
107566 Eth106/1/47 down* Not Consistency Check Not -
Applicable Performed
107567 Eth106/1/48 down* Not Consistency Check Not -
Applicable Performed
1085 Eth107/1/1 down* Not Consistency Check Not -
44 Applicable Performed
108545 Eth107/1/2 down* Not Consistency Check Not -
Applicable Performed
108546 Eth107/1/3 up success success 10

108547 Eth107/1/4 down* Not Consistency Check Not -
Applicable Performed
108548 Eth107/1/5 down* Not Consistency Check Not -
Applicable Performed
108549 Eth107/1/6 up success success 202

108550 Eth107/1/7 up success success 202

108551 Eth107/1/8 up success success 100

108552 Eth107/1/9 up success success 202

108553 Eth107/1/10 down* Not Consistency Check Not -
Applicable Performed
108554 Eth107/1/11 down* Not Consistency Check Not -
Applicable Performed
108555 Eth107/1/12 up success success 102

108556 Eth107/1/13 up success success 100

108557 Eth107/1/14 up success success 210

108558 Eth107/1/15 up success success 200-203,212
,501-503
108559 Eth107/1/16 up success success 102

108560 Eth107/1/17 down* Not Consistency Check Not -
Applicable Performed
108561 Eth107/1/18 down* Not Consistency Check Not -
Applicable Performed
108562 Eth107/1/19 down* Not Consistency Check Not -
Applicable Performed
108563 Eth107/1/20 down* Not Consistency Check Not -
Applicable Performed
108564 Eth107/1/21 down* Not Consistency Check Not -
Applicable Performed
108565 Eth107/1/22 down* Not Consistency Check Not -
Applicable Performed
108566 Eth107/1/23 up success success 200

108567 Eth107/1/24 up success success 100

108568 Eth107/1/25 up success success 210

108569 Eth107/1/26 up success success 200-201,212
,501
108570 Eth107/1/27 up success success 100

108571 Eth107/1/28 down* Not Consistency Check Not -
Applicable Performed
108572 Eth107/1/29 down* Not Consistency Check Not -
Applicable Performed
108573 Eth107/1/30 down* Not Consistency Check Not -
Applicable Performed
108574 Eth107/1/31 down* Not Consistency Check Not -
Applicable Performed
108575 Eth107/1/32 down* Not Consistency Check Not -
Applicable Performed
108576 Eth107/1/33 down* Not Consistency Check Not -
Applicable Performed
108577 Eth107/1/34 down* Not Consistency Check Not -
Applicable Performed
108578 Eth107/1/35 down* Not Consistency Check Not -
Applicable Performed
108579 Eth107/1/36 down* Not Consistency Check Not -
Applicable Performed
108580 Eth107/1/37 down* Not Consistency Check Not -
Applicable Performed
108581 Eth107/1/38 down* Not Consistency Check Not -
Applicable Performed
108582 Eth107/1/39 down* Not Consistency Check Not -
Applicable Performed
108583 Eth107/1/40 down* Not Consistency Check Not -
Applicable Performed
108584 Eth107/1/41 down* Not Consistency Check Not -
Applicable Performed
108585 Eth107/1/42 down* Not Consistency Check Not -
Applicable Performed
108586 Eth107/1/43 down* Not Consistency Check Not -
Applicable Performed
108587 Eth107/1/44 down* Not Consistency Check Not -
Applicable Performed
108588 Eth107/1/45 up success success 102

108589 Eth107/1/46 down* Not Consistency Check Not -
Applicable Performed
108590 Eth107/1/47 down* Not Consistency Check Not -
Applicable Performed
108591 Eth107/1/48 down* Not Consistency Check Not -
Applicable Performed
109568 Eth108/1/1 down* Not Consistency Check Not -
Applicable Performed
109569 Eth108/1/2 down* Not Consistency Check Not -
Applicable Performed
109570 Eth108/1/3 up success success 10

109571 Eth108/1/4 up success success 100

109572 Eth108/1/5 down* Not Consistency Check Not -
Applicable Performed
109573 Eth108/1/6 up success success 202

109574 Eth108/1/7 up success success 202

109575 Eth108/1/8 up success success 102

109576 Eth108/1/9 up success success 202

109577 Eth108/1/10 down* Not Consistency Check Not -
Applicable Performed
109578 Eth108/1/11 down* Not Consistency Check Not -
Applicable Performed
109579 Eth108/1/12 down* Not Consistency Check Not -
Applicable Performed
109580 Eth108/1/13 down* Not Consistency Check Not -
Applicable Performed
109581 Eth108/1/14 up success success 210

109582 Eth108/1/15 up success success 200-203,212
,501-503
109583 Eth108/1/16 down* Not Consistency Check Not -
Applicable Performed
109584 Eth108/1/17 down* Not Consistency Check Not -
Applicable Performed
109585 Eth108/1/18 down* Not Consistency Check Not -
Applicable Performed
109586 Eth108/1/19 down* Not Consistency Check Not -
Applicable Performed
109587 Eth108/1/20 down* Not Consistency Check Not -
Applicable Performed
109588 Eth108/1/21 down* Not Consistency Check Not -
Applicable Performed
109589 Eth108/1/22 down* Not Consistency Check Not -
Applicable Performed
109590 Eth108/1/23 up success success 200

109591 Eth108/1/24 up success success 102

109592 Eth108/1/25 up success success 210

109593 Eth108/1/26 up success success 200-201,212
,501
109594 Eth108/1/27 down* Not Consistency Check Not -
Applicable Performed
109595 Eth108/1/28 down* Not Consistency Check Not -
Applicable Performed
109596 Eth108/1/29 down* Not Consistency Check Not -
Applicable Performed
109597 Eth108/1/30 down* Not Consistency Check Not -
Applicable Performed
109598 Eth108/1/31 down* Not Consistency Check Not -
Applicable Performed
109599 Eth108/1/32 down* Not Consistency Check Not -
Applicable Performed
109600 Eth108/1/33 down* Not Consistency Check Not -
Applicable Performed
109601 Eth108/1/34 down* Not Consistency Check Not -
Applicable Performed
109602 Eth108/1/35 down* Not Consistency Check Not -
Applicable Performed
109603 Eth108/1/36 down* Not Consistency Check Not -
Applicable Performed
109604 Eth108/1/37 down* Not Consistency Check Not -
Applicable Performed
109605 Eth108/1/38 down* Not Consistency Check Not -
Applicable Performed
109606 Eth108/1/39 down* Not Consistency Check Not -
Applicable Performed
109607 Eth108/1/40 down* Not Consistency Check Not -
Applicable Performed
109608 Eth108/1/41 down* Not Consistency Check Not -
Applicable Performed
109609 Eth108/1/42 down* Not Consistency Check Not -
Applicable Performed
109610 Eth108/1/43 down* Not Consistency Check Not -
Applicable Performed
109611 Eth108/1/44 down* Not Consistency Check Not -
Applicable Performed
109612 Eth108/1/45 down* Not Consistency Check Not -
Applicable Performed
109613 Eth108/1/46 down* Not Consistency Check Not -
Applicable Performed
109614 Eth108/1/47 down* Not Consistency Check Not -
Applicable Performed
109615 Eth108/1/48 down* Not Consistency Check Not -
Applicable Performed
110592 Eth109/1/1 up success success 203

110593 Eth109/1/2 down* Not Consistency Check Not -
Applicable Performed
110594 Eth109/1/3 down* Not Consistency Check Not -
Applicable Performed
110595 Eth109/1/4 down* Not Consistency Check Not -
Applicable Performed
1105 Eth109/1/5 down* Not Consistency Check Not -
96 Applicable Performed
110597 Eth109/1/6 down* Not Consistency Check Not -
Applicable Performed
110598 Eth109/1/7 up success success 102

110599 Eth109/1/8 up success success 102

110600 Eth109/1/9 down* Not Consistency Check Not -
Applicable Performed
110601 Eth109/1/10 down* Not Consistency Check Not -
Applicable Performed
110602 Eth109/1/11 up success success 102

110603 Eth109/1/12 down* Not Consistency Check Not -
Applicable Performed
110604 Eth109/1/13 down* Not Consistency Check Not -
Applicable Performed
110605 Eth109/1/14 down* Not Consistency Check Not -
Applicable Performed
110606 Eth109/1/15 down* Not Consistency Check Not -
Applicable Performed
110607 Eth109/1/16 down* Not Consistency Check Not -
Applicable Performed
110608 Eth109/1/17 down* Not Consistency Check Not -
Applicable Performed
110609 Eth109/1/18 up success success 102

110610 Eth109/1/19 down* Not Consistency Check Not -
Applicable Performed
110611 Eth109/1/20 up success success 102

110612 Eth109/1/21 down* Not Consistency Check Not -
Applicable Performed
110613 Eth109/1/22 down* Not Consistency Check Not -
Applicable Performed
110614 Eth109/1/23 down* Not Consistency Check Not -
Applicable Performed
110615 Eth109/1/24 down* Not Consistency Check Not -
Applicable Performed
110616 Eth109/1/25 up success success 203

110617 Eth109/1/26 down* Not Consistency Check Not -
Applicable Performed
110618 Eth109/1/27 down* Not Consistency Check Not -
Applicable Performed
110619 Eth109/1/28 down* Not Consistency Check Not -
Applicable Performed
110620 Eth109/1/29 down* Not Consistency Check Not -
Applicable Performed
110621 Eth109/1/30 up success success 102

110622 Eth109/1/31 up success success 102

110623 Eth109/1/32 up success success 102

110624 Eth109/1/33 down* Not Consistency Check Not -
Applicable Performed
110625 Eth109/1/34 down* Not Consistency Check Not -
Applicable Performed
110626 Eth109/1/35 up success success 102

110627 Eth109/1/36 up success success 102

110628 Eth109/1/37 down* Not Consistency Check Not -
Applicable Performed
110629 Eth109/1/38 down* Not Consistency Check Not -
Applicable Performed
110630 Eth109/1/39 down* Not Consistency Check Not -
Applicable Performed
110631 Eth109/1/40 down* Not Consistency Check Not -
Applicable Performed
110632 Eth109/1/41 down* Not Consistency Check Not -
Applicable Performed
110633 Eth109/1/42 down* Not Consistency Check Not -
Applicable Performed
110634 Eth109/1/43 down* Not Consistency Check Not -
Applicable Performed
110635 Eth109/1/44 down* Not Consistency Check Not -
Applicable Performed
110636 Eth109/1/45 up success success 102

110637 Eth109/1/46 up success success 102

110638 Eth109/1/47 down* Not Consistency Check Not -
Applicable Performed
110639 Eth109/1/48 up success success 102

111616 Eth110/1/1 down* Not Consistency Check Not -
Applicable Performed
111617 Eth110/1/2 down* Not Consistency Check Not -
Applicable Performed
111618 Eth110/1/3 down* Not Consistency Check Not -
Applicable Performed
111619 Eth110/1/4 down* Not Consistency Check Not -
Applicable Performed
111620 Eth110/1/5 down* Not Consistency Check Not -
Applicable Performed
111621 Eth110/1/6 down* Not Consistency Check Not -
Applicable Performed
111622 Eth110/1/7 down* Not Consistency Check Not -
Applicable Performed
111623 Eth110/1/8 down* Not Consistency Check Not -
Applicable Performed
111624 Eth110/1/9 down* Not Consistency Check Not -
Applicable Performed
111625 Eth110/1/10 down* Not Consistency Check Not -
Applicable Performed
111626 Eth110/1/11 down* Not Consistency Check Not -
Applicable Performed
111627 Eth110/1/12 down* Not Consistency Check Not -
Applicable Performed
111628 Eth110/1/13 down* Not Consistency Check Not -
Applicable Performed
111629 Eth110/1/14 down* Not Consistency Check Not -
Applicable Performed
111630 Eth110/1/15 down* Not Consistency Check Not -
Applicable Performed
111631 Eth110/1/16 down* Not Consistency Check Not -
Applicable Performed
111632 Eth110/1/17 down* Not Consistency Check Not -
Applicable Performed
111633 Eth110/1/18 down* Not Consistency Check Not -
Applicable Performed
111634 Eth110/1/19 down* Not Consistency Check Not -
Applicable Performed
111635 Eth110/1/20 down* Not Consistency Check Not -
Applicable Performed
111636 Eth110/1/21 down* Not Consistency Check Not -
Applicable Performed
111637 Eth110/1/22 down* Not Consistency Check Not -
Applicable Performed
111638 Eth110/1/23 down* Not Consistency Check Not -
Applicable Performed
111639 Eth110/1/24 down* Not Consistency Check Not -
Applicable Performed
111640 Eth110/1/25 down* Not Consistency Check Not -
Applicable Performed
111641 Eth110/1/26 down* Not Consistency Check Not -
Applicable Performed
111642 Eth110/1/27 down* Not Consistency Check Not -
Applicable Performed
111643 Eth110/1/28 down* Not Consistency Check Not -
Applicable Performed
111644 Eth110/1/29 down* Not Consistency Check Not -
Applicable Performed
111645 Eth110/1/30 down* Not Consistency Check Not -
Applicable Performed
111646 Eth110/1/31 down* Not Consistency Check Not -
Applicable Performed
111647 Eth110/1/32 down* Not Consistency Check Not -
Applicable Performed
111648 Eth110/1/33 down* Not Consistency Check Not -
Applicable Performed
111649 Eth110/1/34 down* Not Consistency Check Not -
Applicable Performed
111650 Eth110/1/35 down* Not Consistency Check Not -
Applicable Performed
111651 Eth110/1/36 down* Not Consistency Check Not -
Applicable Performed
111652 Eth110/1/37 down* Not Consistency Check Not -
Applicable Performed
111653 Eth110/1/38 down* Not Consistency Check Not -
Applicable Performed
111654 Eth110/1/39 down* Not Consistency Check Not -
Applicable Performed
111655 Eth110/1/40 down* Not Consistency Check Not -
Applicable Performed
111656 Eth110/1/41 down* Not Consistency Check Not -
Applicable Performed
111657 Eth110/1/42 down* Not Consistency Check Not -
Applicable Performed
111658 Eth110/1/43 down* Not Consistency Check Not -
Applicable Performed
111659 Eth110/1/44 down* Not Consistency Check Not -
Applicable Performed
111660 Eth110/1/45 down* Not Consistency Check Not -
Applicable Performed
111661 Eth110/1/46 down* Not Consistency Check Not -
Applicable Performed
111662 Eth110/1/47 down* Not Consistency Check Not -
Applicable Performed
111663 Eth110/1/48 down* Not Consistency Check Not -
Applicable Performed
112640 Eth111/1/1 down* Not Consistency Check Not -
Applicable Performed
112641 Eth111/1/2 up success success 208

112642 Eth111/1/3 up success success 711

112643 Eth111/1/4 down* Not Consistency Check Not -
Applicable Performed
112644 Eth111/1/5 down* Not Consistency Check Not -
Applicable Performed
112645 Eth111/1/6 up success success 700-702,710
-711,730-73
1,800
112646 Eth111/1/7 down* Not Consistency Check Not -
Applicable Performed
112647 Eth111/1/8 down* Not Consistency Check Not -
Applicable Performed
1126 Eth111/1/9 up success success 102
48
112649 Eth111/1/10 down* Not Consistency Check Not -
Applicable Performed
112650 Eth111/1/11 up success success 690

112651 Eth111/1/12 down* Not Consistency Check Not -
Applicable Performed
112652 Eth111/1/13 down* Not Consistency Check Not -
Applicable Performed
112653 Eth111/1/14 down* Not Consistency Check Not -
Applicable Performed
112654 Eth111/1/15 down* Not Consistency Check Not -
Applicable Performed
112655 Eth111/1/16 down* Not Consistency Check Not -
Applicable Performed
112656 Eth111/1/17 down* Not Consistency Check Not -
Applicable Performed
112657 Eth111/1/18 down* Not Consistency Check Not -
Applicable Performed
112658 Eth111/1/19 down* Not Consistency Check Not -
Applicable Performed
112659 Eth111/1/20 down* Not Consistency Check Not -
Applicable Performed
112660 Eth111/1/21 down* Not Consistency Check Not -
Applicable Performed
112661 Eth111/1/22 up success success 102

112662 Eth111/1/23 up success success 210

112663 Eth111/1/24 up success success 208

112664 Eth111/1/25 up success success 100

112665 Eth111/1/26 up success success 210

112666 Eth111/1/27 up success success 208

112667 Eth111/1/28 up success success 208

112668 Eth111/1/29 up success success 999

112669 Eth111/1/30 up success success 711

112670 Eth111/1/31 up success success 700

112671 Eth111/1/32 down* Not Consistency Check Not -
Applicable Performed
112672 Eth111/1/33 down* Not Consistency Check Not -
Applicable Performed
112673 Eth111/1/34 down* Not Consistency Check Not -
Applicable Performed
112674 Eth111/1/35 down* Not Consistency Check Not -
Applicable Performed
112675 Eth111/1/36 up success success 504

112676 Eth111/1/37 down* Not Consistency Check Not -
Applicable Performed
112677 Eth111/1/38 down* Not Consistency Check Not -
Applicable Performed
112678 Eth111/1/39 down* Not Consistency Check Not -
Applicable Performed
112679 Eth111/1/40 down* Not Consistency Check Not -
Applicable Performed
112681 Eth111/1/42 up success success 102

112684 Eth111/1/45 down* Not Consistency Check Not -
Applicable Performed
112685 Eth111/1/46 down* Not Consistency Check Not -
Applicable Performed
112686 Eth111/1/47 down* Not Consistency Check Not -
Applicable Performed
112687 Eth111/1/48 down* Not Consistency Check Not -
Applicable Performed
113664 Eth112/1/1 down* Not Consistency Check Not -
Applicable Performed
113665 Eth112/1/2 down* Not Consistency Check Not -
Applicable Performed
113666 Eth112/1/3 up success success 711

113667 Eth112/1/4 down* Not Consistency Check Not -
Applicable Performed
113668 Eth112/1/5 down* Not Consistency Check Not -
Applicable Performed
113669 Eth112/1/6 up success success 700-702,710
-711,730-73
1,800
113670 Eth112/1/7 up success success 711

113671 Eth112/1/8 down* Not Consistency Check Not -
Applicable Performed
113672 Eth112/1/9 up success success 800

113673 Eth112/1/10 down* Not Consistency Check Not -
Applicable Performed
113674 Eth112/1/11 down* Not Consistency Check Not -
Applicable Performed
113675 Eth112/1/12 up success success 690

113676 Eth112/1/13 up success success 690

113677 Eth112/1/14 down* Not Consistency Check Not -
Applicable Performed
113678 Eth112/1/15 down* Not Consistency Check Not -
Applicable Performed
113679 Eth112/1/16 down* Not Consistency Check Not -
Applicable Performed
113680 Eth112/1/17 up success success 100

113681 Eth112/1/18 down* Not Consistency Check Not -
Applicable Performed
113682 Eth112/1/19 up success success 1

113683 Eth112/1/20 down* Not Consistency Check Not -
Applicable Performed
113684 Eth112/1/21 up success success 102

113685 Eth112/1/22 down* Not Consistency Check Not -
Applicable Performed
113686 Eth112/1/23 up success success 210

113687 Eth112/1/24 up success success 208

113688 Eth112/1/25 up success success 100

113689 Eth112/1/26 up success success 210

113690 Eth112/1/27 up success success 208

113691 Eth112/1/28 up success success 208

113692 Eth112/1/29 up success success 999

113693 Eth112/1/30 down* Not Consistency Check Not -
Applicable Performed
113694 Eth112/1/31 down* Not Consistency Check Not -
Applicable Performed
113695 Eth112/1/32 down* Not Consistency Check Not -
Applicable Performed
113696 Eth112/1/33 down* Not Consistency Check Not -
Applicable Performed
113697 Eth112/1/34 down* Not Consistency Check Not -
Applicable Performed
113698 Eth112/1/35 down* Not Consistency Check Not -
Applicable Performed
113699 Eth112/1/36 up success success 504

113700 Eth112/1/37 down* Not Consistency Check Not -
Applicable Performed
113701 Eth112/1/38 down* Not Consistency Check Not -
Applicable Performed
113702 Eth112/1/39 down* Not Consistency Check Not -
Applicable Performed
113703 Eth112/1/40 down* Not Consistency Check Not -
Applicable Performed
113704 Eth112/1/41 down* Not Consistency Check Not -
Applicable Performed
113705 Eth112/1/42 down* Not Consistency Check Not -
Applicable Performed
113706 Eth112/1/43 down* Not Consistency Check Not -
Applicable Performed
113707 Eth112/1/44 down* Not Consistency Check Not -
Applicable Performed
113708 Eth112/1/45 down* Not Consistency Check Not -
Applicable Performed
113709 Eth112/1/46 down* Not Consistency Check Not -
Applicable Performed
113710 Eth112/1/47 up success success 102

113711 Eth112/1/48 up success success 102

262153 Po10 up success success 10,102-104,
200-204,206
-208,210-21
2,214,310,5
01-503,1100
262154 Po11 up success success 10,102-104,
200-204,206
-208,210-21
2,214,310,5
01-503,1100
262163 Po20 up success success 103

262164 Po21 up success success 103