08-04-2015 11:20 AM - edited 03-05-2019 02:00 AM
Hey all,
got a strange one here. I am using ME 3600 and 3800 on a point to point fiber link. I have done my config as below, however, the IS-IS adjacency isn't forming. When i do a "show clns neighbor" i can see the neighbor but the protocol is stuck on ES-IS. Checking the logs i am getting the below error message; please advise. Most documentation is pointing to a mismatched MTU but i have checked and i don't have any mismatched MTU.
router isis as47888
net 49.0001.aaaa.bbbb.ccfg.00
is-type level-2-only
ispf level-2 60
metric-style wide
fast-flood 10
ignore-attached-bit
set-overload-bit on-startup wait-for-bgp
max-lsp-lifetime 65535
lsp-refresh-interval 65000
spf-interval 5 1 20
prc-interval 5 1 20
lsp-gen-interval 5 1 20
no hello padding
log-adjacency-changes
metric 100000
fast-reroute per-prefix level-2 all
fast-reroute remote-lfa level-2 mpls-ldp
passive-interface Loopback0
bfd all-interfaces
!
!
interface GigabitEthernet0/11
no switchport
mtu 9000
ip address 10.0.0.245 255.255.255.254
ip router isis as47888
bfd interval 50 min_rx 50 multiplier 3
no bfd echo
isis network point-to-point
isis metric 2 level-2
isis csnp-interval 10
end
####################
router isis as47888
net 49.0001.aaaa.bbbb.ccde.00
is-type level-2-only
ispf level-2 60
metric-style wide
fast-flood 10
ignore-attached-bit
set-overload-bit on-startup wait-for-bgp
max-lsp-lifetime 65535
lsp-refresh-interval 65000
spf-interval 5 1 20
prc-interval 5 1 20
lsp-gen-interval 5 1 20
no hello padding
log-adjacency-changes
metric 100000
fast-reroute per-prefix level-2 all
fast-reroute remote-lfa level-2 mpls-ldp
passive-interface Loopback0
bfd all-interfaces
!
!
interface GigabitEthernet0/24
no switchport
mtu 9000
ip address 10.0.0.244 255.255.255.254
ip router isis as47888
bfd interval 50 min_rx 50 multiplier 3
no bfd echo
isis network point-to-point
isis metric 2 level-2
isis csnp-interval 10
end
sh logg
056533: *Aug 4 20:04:05.080 CAT: ISIS-Adj: Sending serial IIH on GigabitEthernet0/24, 3way state:DOWN, length 8996
056534: *Aug 4 20:04:14.268 CAT: ISIS-Adj: Sending serial IIH on GigabitEthernet0/24, 3way state:DOWN, length 8996
#sh clns neighbors
Tag as47888:
System Id Interface SNPA State Holdtime Type Protocol
2883.3004.1180 Gi0/24 362.882a.fdcf Up 254 IS ES-IS
08-04-2015 12:55 PM
Amos,
Can you please post the unabridged output of the following commands from both switches?
show clns
show clns interface gi0/X !!! Replace X with the proper interface
show isis neighbor
Best regards,
Peter
08-04-2015 10:52 PM
#sh clns
Global CLNS Information:
1 Interfaces Enabled for CLNS
Configuration Timer: 60, Default Holding Timer: 300, Packet Lifetime 64
ERPDU's requested on locally generated packets
Running IS-IS in IP-only mode (CLNS forwarding not allowed)
NET: 49.0001.aaaa.bbbb.ccde.00
#sh clns int gig 0/24
GigabitEthernet0/24 is up, line protocol is up
Checksums enabled, MTU 8997, Encapsulation SAP
ERPDUs enabled, min. interval 10 msec.
CLNS fast switching disabled
CLNS SSE switching disabled
DEC compatibility mode OFF for this interface
Next ESH/ISH in 25 seconds
Routing Protocol: IS-IS
Circuit Type: level-1-2
Interface number 0x0, local circuit ID 0x100
Level-2 Metric: 2, Priority: 64, Circuit ID: NNI-OPK-UPE-01.00
Level-2 IPv6 Metric: 10
Number of active level-2 adjacencies: 0
Next IS-IS Hello in 3 seconds
No hello padding
if state DOWN
BFD enabled: (MTID:0, ipv4)
#sh isis nei
Tag as47888:
System Id Type Interface IP Address State Holdtime Circuit Id
#############################################
#sh clns
Global CLNS Information:
5 Interfaces Enabled for CLNS
Configuration Timer: 60, Default Holding Timer: 300, Packet Lifetime 64
ERPDU's requested on locally generated packets
Running IS-IS in IP/IPv6-only mode (CLNS forwarding not allowed)
NET: 49.0001.aaaa.bbbb.ccfg.00
1#sh clns int g0/11
GigabitEthernet0/11 is up, line protocol is up
Checksums enabled, MTU 8997, Encapsulation SAP
ERPDUs enabled, min. interval 10 msec.
CLNS fast switching disabled
CLNS SSE switching disabled
DEC compatibility mode OFF for this interface
Next ESH/ISH in 26 seconds
Routing Protocol: IS-IS
Circuit Type: level-1-2
Interface number 0x2, local circuit ID 0x102
Level-2 Metric: 2, Priority: 64, Circuit ID: NNI-UNZ-UPE-01.02
Level-2 IPv6 Metric: 10
Number of active level-2 adjacencies: 0
Next IS-IS Hello in 0 seconds
No hello padding
if state DOWN
BFD enabled: (MTID:0, ipv4)
08-04-2015 11:11 PM
Hello Amos,
It's just an observation and it very well might be just a coincidence, but on the Gi0/11 interface on the second switch, notice that the show clns int gi0/11 says "Next IS-IS Hello in 0 seconds". Can you please issue that command multiple times and confirm that the timer is actually moving and that it isn't stuck?
If this is determined to be a non-issue, do you believe you would be able to run debug isis adj on both devices and post the result? To limit down the amount of output, you could use debug condition interface Gi0/X beforehand (if you do, do not forget to use undebug condition 1 afterwards as undebug all does not remove debug conditions). According to what you have posted so far, your NNI-OPK-UPE-01 is sending IIH out Gi0/24 but is not receiving any over it. You did not post the same output for NNI-UNZ-UPE-01.
One more question: Is it possible that the configurations of Gi0/24 and Gi0/11 posted here are not complete, and that one of those interfaces has IPv6 enabled while the other has not?
Best regards,
Peter
08-04-2015 11:50 PM
Hi Peter,
i have done the show clns int gi0/11 multiple times and can confirm that the timer is stuck on 0! it is not moving. I can also confirm that there is no IPv6 enabled on both interfaces. below is the debug output. I have some adjacency formed on the main NNI-UNZ-UPE-01 which are currently active and are showing in the debug, however, i cannot see anything on the interface of interest. see below;
NNI-OPK-UPE-01# sh logg
063276: *Aug 5 08:22:00.777 CAT: ISIS-Adj: Sending serial IIH on GigabitEthernet0/24, 3way state:DOWN, length 8996
063277: *Aug 5 08:22:10.777 CAT: ISIS-Adj: Sending serial IIH on GigabitEthernet0/24, 3way state:DOWN, length 8996
063278: *Aug 5 08:22:16.281 CAT: ES-IS: ISH sent to All ESs (GigabitEthernet0/24): NET 49.0001.aaaa.bbbb.ccde.00, HT 300, HLEN 23
063279: *Aug 5 08:22:18.601 CAT: ISIS-Adj: Sending serial IIH on GigabitEthernet0/24, 3way state:DOWN, length 8996
063280: *Aug 5 08:22:27.777 CAT: ISIS-Adj: Sending serial IIH on GigabitEthernet0/24, 3way state:DOWN, length 8996
063281: *Aug 5 08:22:35.577 CAT: ES-IS: ISH from 3462.882a.fdcf (GigabitEthernet0/24), HT 300
063282: *Aug 5 08:22:35.953 CAT: ISIS-Adj: Sending serial IIH on GigabitEthernet0/24, 3way state:DOWN, length 8996
063283: *Aug 5 08:22:45.457 CAT: ISIS-Adj: Sending serial IIH on GigabitEthernet0/24, 3way state:DOWN, length 8996
063284: *Aug 5 08:22:54.437 CAT: ISIS-Adj: Sending serial IIH on GigabitEthernet0/24, 3way state:DOWN, length 8996
063285: *Aug 5 08:23:02.813 CAT: ISIS-Adj: Sending serial IIH on GigabitEthernet0/24, 3way state:DOWN, length 8996
063286: *Aug 5 08:23:03.221 CAT: ES-IS: ISH sent to All ESs (GigabitEthernet0/24): NET 49.0001.1972.2003.1172.00, HT 300, HLEN 23
063287: *Aug 5 08:23:10.401 CAT: ISIS-Adj: Sending serial IIH on GigabitEthernet0/24, 3way state:DOWN, length 8996
063288: *Aug 5 08:23:19.001 CAT: ISIS-Adj: Sending serial IIH on GigabitEthernet0/24, 3way state:DOWN, length 8996
063289: *Aug 5 08:23:23.593 CAT: ES-IS: ISH from 3462.882a.fdcf (GigabitEthernet0/24), HT 300
063290: *Aug 5 08:23:28.585 CAT: ISIS-Adj: Sending serial IIH on GigabitEthernet0/24, 3way state:DOWN, length 8996
063291: *Aug 5 08:23:38.561 CAT: ISIS-Adj: Sending serial IIH on GigabitEthernet0/24, 3way state:DOWN, length 8996
NNI-UNZ-UPE-01#
Log Buffer (4096 bytes):
j: Action = ACCEPT
097865: *Aug 5 08:20:12.565 CAT: ISIS-Adj: ACTION_ACCEPT:
097866: *Aug 5 08:20:15.073 CAT: ISIS-Adj: Sending serial IIH on TenGigabitEthernet0/2, 3way state:UP, length 48
097867: *Aug 5 08:20:15.281 CAT: ISIS-Adj: Sending serial IIH on GigabitEthernet0/12, 3way state:UP, length 48
097868: *Aug 5 08:20:15.953 CAT: ISIS-Adj: Rec serial IIH from f41f.c273.19da (TenGigabitEthernet0/2), cir type L2, cir id 00, length 48
097869: *Aug 5 08:20:15.953 CAT: ISIS-Adj: Local mode (IP), remote mode (IP)
097870: *Aug 5 08:20:15.953 CAT: ISIS-Adj: rcvd state UP, old state UP, new state UP, nbr usable TRUE
097871: *Aug 5 08:20:15.953 CAT: ISIS-Adj: newstate:0, state_changed:0, going_up:0, going_down:0
097872: *Aug 5 08:20:15.953 CAT: ISIS-Adj: Action = ACCEPT
097873: *Aug 5 08:20:15.953 CAT: ISIS-Adj: ACTION_ACCEPT:
097874: *Aug 5 08:20:17.333 CAT: ES-IS: ISH sent to All ESs (GigabitEthernet0/12): NET 49.0001.aaaa.bbbb.ccfg.00, HT 300, HLEN 24
097875: *Aug 5 08:20:18.661 CAT: ISIS-Adj: Rec serial IIH from 3462.882a.fbd7 (GigabitEthernet0/12), cir type L2, cir id 00, length 48
097876: *Aug 5 08:20:18.661 CAT: ISIS-Adj: Local mode (IP), remote mode (IP)
097877: *Aug 5 08:20:18.661 CAT: ISIS-Adj: rcvd state UP, old state UP, new state UP, nbr usable TRUE
097878: *Aug 5 08:20:18.661 CAT: ISIS-Adj: newstate:0, state_changed:0, going_up:0, going_down:0
097879: *Aug 5 08:20:18.661 CAT: ISIS-Adj: Action = ACCEPT
097880: *Aug 5 08:20:18.661 CAT: ISIS-Adj: ACTION_ACCEPT:
097881: *Aug 5 08:20:19.469 CAT: ISIS-Adj: Sending serial IIH on GigabitEthernet0/13, 3way state:UP, length 48
097882: *Aug 5 08:20:20.017 CAT: ISIS-Adj: Sending serial IIH on TenGigabitEthernet0/1, 3way state:UP, length 48
097883: *Aug 5 08:20:20.821 CAT: ISIS-Adj: Rec serial IIH from 3462.882b.0d58 (GigabitEthernet0/13), cir type L2, cir id 00, length 48
097884: *Aug 5 08:20:20.821 CAT: ISIS-Adj: Local mode (IP), remote mode (IP)
097885: *Aug 5 08:20:20.821 CAT: ISIS-Adj: rcvd state UP, old state UP, new state UP, nbr usable TRUE
097886: *Aug 5 08:20:20.821 CAT: ISIS-Adj: newstate:0, state_changed:0, going_up:0, going_down:0
097887: *Aug 5 08:20:20.821 CAT: ISIS-Adj: Action = ACCEPT
097888: *Aug 5 08:20:20.821 CAT: ISIS-Adj: ACTION_ACCEPT:
097889: *Aug 5 08:20:22.469 CAT: ISIS-Adj: Rec serial IIH from 1cde.a79a.17da (TenGigabitEthernet0/1), cir type L2, cir id 01, length 48
097890: *Aug 5 08:20:22.469 CAT: ISIS-Adj: Local mode (IP), remote mode (IP)
097891: *Aug 5 08:20:22.469 CAT: ISIS-Adj: rcvd state UP, old state UP, new state UP, nbr usable TRUE
097892: *Aug 5 08:20:22.469 CAT: ISIS-Adj: newstate:0, state_changed:0, going_up:0, going_down:0
097893: *Aug 5 08:20:22.469 CAT: ISIS-Adj: Action = ACCEPT
097894: *Aug 5 08:20:22.469 CAT: ISIS-Adj: ACTION_ACCEPT:
097895: *Aug 5 08:20:23.789 CAT: ISIS-Adj: Sending serial IIH on TenGigabitEthernet0/2, 3way state:UP, length 48
097896: *Aug 5 08:20:24.253 CAT: ISIS-Adj: Sending serial IIH on GigabitEthernet0/12, 3way state:UP, length 48
097897: *Aug 5 08:20:24.713 CAT: ISIS-Adj: Rec serial IIH from f41f.c273.19da (TenGigabitEthernet0/2), cir type L2, cir id 00, length 48
097898: *Aug 5 08:20:24.713 CAT: ISIS-Adj: Local mode (IP), remote mode (IP)
097899: *Aug 5 08:20:24.713 CAT: ISIS-Adj: rcvd state UP, old state UP, new state UP, nbr usable TRUE
097900: *Aug 5 08:20:24.713 CAT: ISIS-Adj: newstate:0, state_changed:0, going_up:0, going_down:0
097901: *Aug 5 08:20:24.713 CAT: ISIS-Adj: Action = ACCEPT
097902: *Aug 5 08:20:24.713 CAT: ISIS-Adj: ACTION_ACCEPT:
097903: *Aug 5 08:20:26.629 CAT: ISIS-Adj: Rec serial IIH from 3462.882a.fbd7 (GigabitEthernet0/12), cir type L2, cir id 00, length 48
097904: *Aug 5 08:20:26.629 CAT: ISIS-Adj: Local mode (IP), remote mode (IP)
097905: *Aug 5 08:20:26.629 CAT: ISIS-Adj: rcvd state UP, old state UP, new state UP, nbr usable TRUE
097906: *Aug 5 08:20:26.629 CAT: ISIS-Adj: newstate:0, state_changed:0, going_up:0, going_down:0
097907: *Aug 5 08:20:26.629 CAT: ISIS-Adj: Action = ACCEPT
097908: *Aug 5 08:20:26.629 CAT: ISIS-Adj: ACTION_ACCEPT:
08-05-2015 12:05 AM
Amos,
It appears as if on NNI-UNZ-UPE-01, the IIH timer on Gi0/11 got stuck - that would be an IOS bug. If the timer is stuck, no IIH are sent out which would explain why the IS-IS adjacency is not coming up. The question is what can we do to get it "unstuck". Would you mind trying the following suggestions in the order listed here, proceeding to the next option if the previous did not cause the timer to start running again?
Best regards,
Peter
08-05-2015 12:27 AM
Peter,
i have tried all 4 options and the timer is still stuck! I will try the following things during the change window period;
I will update once i have tried this.
Also to note, the problem started when the device NNI-OPK-UPE-01 detected that there was a duplicate NET ID on the network. When I reconfigured the NET ID and tried to restart the process it led to this.
08-05-2015 12:43 AM
Amos,
I see. I am afraid we don't have many choices left. If I come across an idea of what yet else to try without causing too much disruption, I will let you know. Also, please, keep me informed about any findings you'll have.
Best regards,
Peter
08-07-2015 02:08 AM
Hi Peter,
so I restarted the ISIS process and rebooted the ME too but the hello's are still not being sent, timer still stuck on zero. Am stuck for options at the moment.
08-07-2015 03:36 AM
Amos,
Truth be told, I am also staring at the entire issue in utter disbelief.
Just wondering: On NNI-UNZ-UPE-01, if you enter the following command:
show isis as47888 database NNI-UNZ-UPE-01.00-00 detail
does the output of this LSP also show the network on the Gi0/11 interface, that is, 10.0.0.244/31? I am trying to find out whether IS-IS registers this interface at all, at least the IP subnet there. I also suppose you do not have that LSP so large that it is fragmented (if it is, check also other fragments with -01, -02, etc.)
I suppose you do not run multiple IS-IS processes on NNI-UNZ-UPE-01, do you?
Best regards,
Peter
08-07-2015 05:40 AM
Peter,
below is the output of the command. I am/ was running 5 ISIS instances until this one went down. So currently only 4 are active.
IS-IS Level-2 LSP NNI-UNZ-UPE-01.00-00
LSPID LSP Seq Num LSP Checksum LSP Holdtime ATT/P/OL
LSK-UNZ-UPE-01.00-00* 0x00000629 0xB4CE 65268 0/0/0
Area Address: 49.0001
NLPID: 0xCC 0x8E
Topology: IPv4 (0x0)
IPv6 (0x2)
Hostname: LSK-UNZ-UPE-01
Metric: 2 IS-Extended XXXX.2003.1181.00
Metric: 20 IS-Extended NNI-COM-UPE-01.00
Metric: 20 IS-Extended NNI-WLD-UPE-01.00
IP Address: 197.220.31.180
Metric: 20 IP 10.0.0.202/31
Metric: 20 IP 10.0.0.244/31
Metric: 20 IP 10.0.0.250/31
Metric: 2 IP 10.0.0.254/31
Metric: 0 IP XXX.XXX.31.180/32
08-07-2015 06:42 AM
Amos,
Just curious, why are you running 5 different ISIS instances? Do you need to keep the topologies separate? Are you having them in separate VRFs?
I am wondering if we are perhaps hitting some limitation of your platform on the number of running IS-IS instances. If your IS-IS instances all run in the global routing table (no VRFs) then what you are doing is effectively an IS-IS multi-area configuration. The docs about running multiple IS-IS areas insist that only one process can perform Level2 routing, all other processes will be forced to Level1 routing. Check this document for more details:
Can you describe in more detail, then, how do you actually deploy the multiple IS-IS processes?
Best regards,
Peter
08-09-2015 08:20 AM
Peter,
my set up is that; I have one aggregation point to which my 5 remote sites are linking to, this is the NNI-UNZ-UPE where I have 5 ISIS neighbors connecting. They are all running in the global routing table (no VRF's). I am using the same process for all devices and all 4 are running okay. Is there a limit to how many ISIS neighbors a single device may have?
I have tried to directly connect another device, an ASR9k to the ME and its also not forming.
08-09-2015 08:46 AM
Peter,
this is my complete ISIS config. Only Link_C is not forming an adjacency the rest are forming just okay. In addition, like i mentioned before, i have tried to direclty connect an ASR9K to this ME and try to form an adjacency but it will not form. hello packets stuck on zero!
router isis as47888
net 49.0001.aaaa.bbbb.ccde.00
is-type level-2-only
ispf level-2 60
metric-style wide
fast-flood 10
ignore-attached-bit
set-overload-bit on-startup wait-for-bgp
max-lsp-lifetime 65535
lsp-refresh-interval 65000
spf-interval 5 1 20
prc-interval 5 1 20
lsp-gen-interval 5 1 20
no hello padding
log-adjacency-changes
metric 100000
fast-reroute per-prefix level-2 all
fast-reroute remote-lfa level-2 mpls-ldp
passive-interface Loopback0
bfd all-interfaces
!
interface GigabitEthernet0/12
description LINK_A
no switchport
mtu 9000
ip address 10.0.0.203 255.255.255.254
ip router isis as47888
bfd interval 50 min_rx 50 multiplier 3
no bfd echo
isis network point-to-point
isis metric 20 level-2
isis csnp-interval 10
!
interface GigabitEthernet0/13
description Link_B
no switchport
mtu 9000
ip address 10.0.0.239 255.255.255.254
ip router isis as47888
bfd interval 50 min_rx 50 multiplier 3
no bfd echo
isis network point-to-point
isis metric 20 level-2
isis csnp-interval 10
!
interface GigabitEthernet0/11
description Link_C
no switchport
mtu 9000
ip address 10.0.0.245 255.255.255.254
ip router isis as47888
bfd interval 50 min_rx 50 multiplier 3
no bfd echo
isis network point-to-point
isis metric 20 level-2
isis csnp-interval 10
!
interface TenGigabitEthernet0/1
description Link_D
no switchport
mtu 9000
ip address 10.0.0.254 255.255.255.254
ip router isis as47888
bfd interval 50 min_rx 50 multiplier 3
no bfd echo
isis network point-to-point
isis metric 2 level-2
isis csnp-interval 10
08-09-2015 09:02 AM
Adding to Peter's point, If you want to segregate multiple topologies, you can also use MTR feature.
Regarding the down isis nei, can you try to configure a lower clns mtu to see if that makes any difference.
Could you please share both the IOS and XR version running between the two problematic nodes.
Thanks,
Vinit
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide