IS-IS not forming Adjacency - stuck on ES-IS

[Amos Kafwembe]

Hey all,

got a strange one here. I am using ME 3600 and 3800 on a point to point fiber link. I have done my config as below, however, the IS-IS adjacency isn't forming. When i do a "show clns neighbor" i can see the neighbor but the protocol is stuck on ES-IS. Checking the logs i am getting the below error message; please advise. Most documentation is pointing to a mismatched MTU but i have checked and i don't have any mismatched MTU.

router isis as47888
 net 49.0001.aaaa.bbbb.ccfg.00
 is-type level-2-only
 ispf level-2 60
 metric-style wide
 fast-flood 10
 ignore-attached-bit
 set-overload-bit on-startup wait-for-bgp
 max-lsp-lifetime 65535
 lsp-refresh-interval 65000
 spf-interval 5 1 20
 prc-interval 5 1 20
 lsp-gen-interval 5 1 20
 no hello padding
 log-adjacency-changes
 metric 100000
 fast-reroute per-prefix level-2 all
 fast-reroute remote-lfa level-2 mpls-ldp
 passive-interface Loopback0
 bfd all-interfaces
!
!
interface GigabitEthernet0/11
 no switchport
 mtu 9000
 ip address 10.0.0.245 255.255.255.254
 ip router isis as47888
 bfd interval 50 min_rx 50 multiplier 3
 no bfd echo
 isis network point-to-point
 isis metric 2 level-2
 isis csnp-interval 10
end

####################

router isis as47888
 net 49.0001.aaaa.bbbb.ccde.00
 is-type level-2-only
 ispf level-2 60
 metric-style wide
 fast-flood 10
 ignore-attached-bit
 set-overload-bit on-startup wait-for-bgp
 max-lsp-lifetime 65535
 lsp-refresh-interval 65000
 spf-interval 5 1 20
 prc-interval 5 1 20
 lsp-gen-interval 5 1 20
 no hello padding
 log-adjacency-changes
 metric 100000
 fast-reroute per-prefix level-2 all
 fast-reroute remote-lfa level-2 mpls-ldp
 passive-interface Loopback0
 bfd all-interfaces
!
!
interface GigabitEthernet0/24
 no switchport
 mtu 9000
 ip address 10.0.0.244 255.255.255.254
 ip router isis as47888
 bfd interval 50 min_rx 50 multiplier 3
 no bfd echo
 isis network point-to-point
 isis metric 2 level-2
 isis csnp-interval 10
end

sh logg

056533: *Aug  4 20:04:05.080 CAT: ISIS-Adj: Sending serial IIH on GigabitEthernet0/24, 3way state:DOWN, length 8996
056534: *Aug  4 20:04:14.268 CAT: ISIS-Adj: Sending serial IIH on GigabitEthernet0/24, 3way state:DOWN, length 8996

#sh clns neighbors
Tag as47888:
System Id      Interface   SNPA                State  Holdtime  Type Protocol
2883.3004.1180 Gi0/24      362.882a.fdcf      Up     254       IS   ES-IS

[Peter Paluch]

Amos,

Can you please post the unabridged output of the following commands from both switches?

show clns
show clns interface gi0/X !!! Replace X with the proper interface
show isis neighbor

Best regards,
Peter

[Amos Kafwembe]

#sh clns
Global CLNS Information:
  1 Interfaces Enabled for CLNS
  Configuration Timer: 60, Default Holding Timer: 300, Packet Lifetime 64
  ERPDU's requested on locally generated packets
  Running IS-IS in IP-only mode (CLNS forwarding not allowed)
  NET: 49.0001.aaaa.bbbb.ccde.00

#sh clns int gig 0/24
GigabitEthernet0/24 is up, line protocol is up
  Checksums enabled, MTU 8997, Encapsulation SAP
  ERPDUs enabled, min. interval 10 msec.
  CLNS fast switching disabled
  CLNS SSE switching disabled
  DEC compatibility mode OFF for this interface
  Next ESH/ISH in 25 seconds
  Routing Protocol: IS-IS
    Circuit Type: level-1-2
    Interface number 0x0, local circuit ID 0x100
    Level-2 Metric: 2, Priority: 64, Circuit ID: NNI-OPK-UPE-01.00
    Level-2 IPv6 Metric: 10
    Number of active level-2 adjacencies: 0
    Next IS-IS Hello in 3 seconds
    No hello padding
    if state DOWN
  BFD enabled: (MTID:0, ipv4)

#sh isis nei

Tag as47888:
System Id      Type Interface   IP Address      State Holdtime Circuit Id

#############################################

#sh clns
Global CLNS Information:
  5 Interfaces Enabled for CLNS
  Configuration Timer: 60, Default Holding Timer: 300, Packet Lifetime 64
  ERPDU's requested on locally generated packets
  Running IS-IS in IP/IPv6-only mode (CLNS forwarding not allowed)
  NET: 49.0001.aaaa.bbbb.ccfg.00

1#sh clns int g0/11
GigabitEthernet0/11 is up, line protocol is up
  Checksums enabled, MTU 8997, Encapsulation SAP
  ERPDUs enabled, min. interval 10 msec.
  CLNS fast switching disabled
  CLNS SSE switching disabled
  DEC compatibility mode OFF for this interface
  Next ESH/ISH in 26 seconds
  Routing Protocol: IS-IS
    Circuit Type: level-1-2
    Interface number 0x2, local circuit ID 0x102
    Level-2 Metric: 2, Priority: 64, Circuit ID: NNI-UNZ-UPE-01.02
    Level-2 IPv6 Metric: 10
    Number of active level-2 adjacencies: 0
    Next IS-IS Hello in 0 seconds
    No hello padding
    if state DOWN
  BFD enabled: (MTID:0, ipv4)

[Peter Paluch]

Hello Amos,

It's just an observation and it very well might be just a coincidence, but on the Gi0/11 interface on the second switch, notice that the show clns int gi0/11 says "Next IS-IS Hello in 0 seconds". Can you please issue that command multiple times and confirm that the timer is actually moving and that it isn't stuck?

If this is determined to be a non-issue, do you believe you would be able to run debug isis adj on both devices and post the result? To limit down the amount of output, you could use debug condition interface Gi0/X beforehand (if you do, do not forget to use undebug condition 1 afterwards as undebug all does not remove debug conditions). According to what you have posted so far, your NNI-OPK-UPE-01 is sending IIH out Gi0/24 but is not receiving any over it. You did not post the same output for NNI-UNZ-UPE-01.

One more question: Is it possible that the configurations of Gi0/24 and Gi0/11 posted here are not complete, and that one of those interfaces has IPv6 enabled while the other has not?

Best regards,
Peter

[Amos Kafwembe]

Hi Peter,

i have done the show clns int gi0/11 multiple times and can confirm that the timer is stuck on 0! it is not moving. I can also confirm that there is no IPv6 enabled on both interfaces. below is the debug output. I have some adjacency formed on the main NNI-UNZ-UPE-01 which are currently active and are showing in the debug, however, i cannot see anything on the interface of interest. see below;

NNI-OPK-UPE-01# sh logg
063276: *Aug  5 08:22:00.777 CAT: ISIS-Adj: Sending serial IIH on GigabitEthernet0/24, 3way state:DOWN, length 8996
063277: *Aug  5 08:22:10.777 CAT: ISIS-Adj: Sending serial IIH on GigabitEthernet0/24, 3way state:DOWN, length 8996
063278: *Aug  5 08:22:16.281 CAT: ES-IS: ISH sent to All ESs (GigabitEthernet0/24): NET 49.0001.aaaa.bbbb.ccde.00, HT 300, HLEN 23
063279: *Aug  5 08:22:18.601 CAT: ISIS-Adj: Sending serial IIH on GigabitEthernet0/24, 3way state:DOWN, length 8996
063280: *Aug  5 08:22:27.777 CAT: ISIS-Adj: Sending serial IIH on GigabitEthernet0/24, 3way state:DOWN, length 8996
063281: *Aug  5 08:22:35.577 CAT: ES-IS: ISH from 3462.882a.fdcf (GigabitEthernet0/24), HT 300
063282: *Aug  5 08:22:35.953 CAT: ISIS-Adj: Sending serial IIH on GigabitEthernet0/24, 3way state:DOWN, length 8996
063283: *Aug  5 08:22:45.457 CAT: ISIS-Adj: Sending serial IIH on GigabitEthernet0/24, 3way state:DOWN, length 8996
063284: *Aug  5 08:22:54.437 CAT: ISIS-Adj: Sending serial IIH on GigabitEthernet0/24, 3way state:DOWN, length 8996
063285: *Aug  5 08:23:02.813 CAT: ISIS-Adj: Sending serial IIH on GigabitEthernet0/24, 3way state:DOWN, length 8996
063286: *Aug  5 08:23:03.221 CAT: ES-IS: ISH sent to All ESs (GigabitEthernet0/24): NET 49.0001.1972.2003.1172.00, HT 300, HLEN 23
063287: *Aug  5 08:23:10.401 CAT: ISIS-Adj: Sending serial IIH on GigabitEthernet0/24, 3way state:DOWN, length 8996
063288: *Aug  5 08:23:19.001 CAT: ISIS-Adj: Sending serial IIH on GigabitEthernet0/24, 3way state:DOWN, length 8996
063289: *Aug  5 08:23:23.593 CAT: ES-IS: ISH from 3462.882a.fdcf (GigabitEthernet0/24), HT 300
063290: *Aug  5 08:23:28.585 CAT: ISIS-Adj: Sending serial IIH on GigabitEthernet0/24, 3way state:DOWN, length 8996
063291: *Aug  5 08:23:38.561 CAT: ISIS-Adj: Sending serial IIH on GigabitEthernet0/24, 3way state:DOWN, length 8996

NNI-UNZ-UPE-01#
Log Buffer (4096 bytes):
j: Action = ACCEPT
097865: *Aug  5 08:20:12.565 CAT: ISIS-Adj: ACTION_ACCEPT:
097866: *Aug  5 08:20:15.073 CAT: ISIS-Adj: Sending serial IIH on TenGigabitEthernet0/2, 3way state:UP, length 48
097867: *Aug  5 08:20:15.281 CAT: ISIS-Adj: Sending serial IIH on GigabitEthernet0/12, 3way state:UP, length 48
097868: *Aug  5 08:20:15.953 CAT: ISIS-Adj: Rec serial IIH from f41f.c273.19da (TenGigabitEthernet0/2), cir type L2, cir id 00, length 48
097869: *Aug  5 08:20:15.953 CAT: ISIS-Adj: Local mode (IP), remote mode (IP)
097870: *Aug  5 08:20:15.953 CAT: ISIS-Adj: rcvd state UP, old state UP, new state UP, nbr usable TRUE
097871: *Aug  5 08:20:15.953 CAT: ISIS-Adj: newstate:0, state_changed:0, going_up:0, going_down:0
097872: *Aug  5 08:20:15.953 CAT: ISIS-Adj: Action = ACCEPT
097873: *Aug  5 08:20:15.953 CAT: ISIS-Adj: ACTION_ACCEPT:
097874: *Aug  5 08:20:17.333 CAT: ES-IS: ISH sent to All ESs (GigabitEthernet0/12): NET 49.0001.aaaa.bbbb.ccfg.00, HT 300, HLEN 24
097875: *Aug  5 08:20:18.661 CAT: ISIS-Adj: Rec serial IIH from 3462.882a.fbd7 (GigabitEthernet0/12), cir type L2, cir id 00, length 48
097876: *Aug  5 08:20:18.661 CAT: ISIS-Adj: Local mode (IP), remote mode (IP)
097877: *Aug  5 08:20:18.661 CAT: ISIS-Adj: rcvd state UP, old state UP, new state UP, nbr usable TRUE
097878: *Aug  5 08:20:18.661 CAT: ISIS-Adj: newstate:0, state_changed:0, going_up:0, going_down:0
097879: *Aug  5 08:20:18.661 CAT: ISIS-Adj: Action = ACCEPT
097880: *Aug  5 08:20:18.661 CAT: ISIS-Adj: ACTION_ACCEPT:
097881: *Aug  5 08:20:19.469 CAT: ISIS-Adj: Sending serial IIH on GigabitEthernet0/13, 3way state:UP, length 48
097882: *Aug  5 08:20:20.017 CAT: ISIS-Adj: Sending serial IIH on TenGigabitEthernet0/1, 3way state:UP, length 48
097883: *Aug  5 08:20:20.821 CAT: ISIS-Adj: Rec serial IIH from 3462.882b.0d58 (GigabitEthernet0/13), cir type L2, cir id 00, length 48
097884: *Aug  5 08:20:20.821 CAT: ISIS-Adj: Local mode (IP), remote mode (IP)
097885: *Aug  5 08:20:20.821 CAT: ISIS-Adj: rcvd state UP, old state UP, new state UP, nbr usable TRUE
097886: *Aug  5 08:20:20.821 CAT: ISIS-Adj: newstate:0, state_changed:0, going_up:0, going_down:0
097887: *Aug  5 08:20:20.821 CAT: ISIS-Adj: Action = ACCEPT
097888: *Aug  5 08:20:20.821 CAT: ISIS-Adj: ACTION_ACCEPT:
097889: *Aug  5 08:20:22.469 CAT: ISIS-Adj: Rec serial IIH from 1cde.a79a.17da (TenGigabitEthernet0/1), cir type L2, cir id 01, length 48
097890: *Aug  5 08:20:22.469 CAT: ISIS-Adj: Local mode (IP), remote mode (IP)
097891: *Aug  5 08:20:22.469 CAT: ISIS-Adj: rcvd state UP, old state UP, new state UP, nbr usable TRUE
097892: *Aug  5 08:20:22.469 CAT: ISIS-Adj: newstate:0, state_changed:0, going_up:0, going_down:0
097893: *Aug  5 08:20:22.469 CAT: ISIS-Adj: Action = ACCEPT
097894: *Aug  5 08:20:22.469 CAT: ISIS-Adj: ACTION_ACCEPT:
097895: *Aug  5 08:20:23.789 CAT: ISIS-Adj: Sending serial IIH on TenGigabitEthernet0/2, 3way state:UP, length 48
097896: *Aug  5 08:20:24.253 CAT: ISIS-Adj: Sending serial IIH on GigabitEthernet0/12, 3way state:UP, length 48
097897: *Aug  5 08:20:24.713 CAT: ISIS-Adj: Rec serial IIH from f41f.c273.19da (TenGigabitEthernet0/2), cir type L2, cir id 00, length 48
097898: *Aug  5 08:20:24.713 CAT: ISIS-Adj: Local mode (IP), remote mode (IP)
097899: *Aug  5 08:20:24.713 CAT: ISIS-Adj: rcvd state UP, old state UP, new state UP, nbr usable TRUE
097900: *Aug  5 08:20:24.713 CAT: ISIS-Adj: newstate:0, state_changed:0, going_up:0, going_down:0
097901: *Aug  5 08:20:24.713 CAT: ISIS-Adj: Action = ACCEPT
097902: *Aug  5 08:20:24.713 CAT: ISIS-Adj: ACTION_ACCEPT:
097903: *Aug  5 08:20:26.629 CAT: ISIS-Adj: Rec serial IIH from 3462.882a.fbd7 (GigabitEthernet0/12), cir type L2, cir id 00, length 48
097904: *Aug  5 08:20:26.629 CAT: ISIS-Adj: Local mode (IP), remote mode (IP)
097905: *Aug  5 08:20:26.629 CAT: ISIS-Adj: rcvd state UP, old state UP, new state UP, nbr usable TRUE
097906: *Aug  5 08:20:26.629 CAT: ISIS-Adj: newstate:0, state_changed:0, going_up:0, going_down:0
097907: *Aug  5 08:20:26.629 CAT: ISIS-Adj: Action = ACCEPT
097908: *Aug  5 08:20:26.629 CAT: ISIS-Adj: ACTION_ACCEPT:

[Peter Paluch]

Amos,

It appears as if on NNI-UNZ-UPE-01, the IIH timer on Gi0/11 got stuck - that would be an IOS bug. If the timer is stuck, no IIH are sent out which would explain why the IS-IS adjacency is not coming up. The question is what can we do to get it "unstuck". Would you mind trying the following suggestions in the order listed here, proceeding to the next option if the previous did not cause the timer to start running again?

1. Remove and reapply the ip router isis as47888 command on Gi0/11
2. Declare the Gi0/11 as an passive interface, then reapply the ip router isis as47888 command to make it active again
3. Change the IIH timer on Gi0/11 using the isis hello-interval to, say, 5 seconds, then put it back to the default of 10 seconds.
4. Shut the Gi0/11 down if possible, and after 20-30 seconds, turn it on again
5. Reload the box if possible (I know, this is almost out of the question, I am just including this one for completeness)

Best regards,
Peter

[Amos Kafwembe]

Peter,

i have tried all 4 options and the timer is still stuck! I will try the following things during the change window period;

1. change the port
2. restart the isis process (clear isis *)
3. reload the device

I will update once i have tried this.

Also to note, the problem started when the device NNI-OPK-UPE-01 detected that there was a duplicate NET ID on the network. When I reconfigured the NET ID and tried to restart the process it led to this.

[Peter Paluch]

Amos,

I see. I am afraid we don't have many choices left. If I come across an idea of what yet else to try without causing too much disruption, I will let you know. Also, please, keep me informed about any findings you'll have.

Best regards,
Peter

[Amos Kafwembe]

Hi Peter,

so I restarted the ISIS process and rebooted the ME too but the hello's are still not being sent, timer still stuck on zero. Am stuck for options at the moment.

[Peter Paluch]

Amos,

Truth be told, I am also staring at the entire issue in utter disbelief.

Just wondering: On NNI-UNZ-UPE-01, if you enter the following command:

show isis as47888 database NNI-UNZ-UPE-01.00-00 detail

does the output of this LSP also show the network on the Gi0/11 interface, that is, 10.0.0.244/31? I am trying to find out whether IS-IS registers this interface at all, at least the IP subnet there. I also suppose you do not have that LSP so large that it is fragmented (if it is, check also other fragments with -01, -02, etc.)

I suppose you do not run multiple IS-IS processes on NNI-UNZ-UPE-01, do you?

Best regards,
Peter

[Amos Kafwembe]

Peter,

below is the output of the command. I am/ was running 5 ISIS instances until this one went down. So currently only 4 are active.

IS-IS Level-2 LSP NNI-UNZ-UPE-01.00-00
LSPID                 LSP Seq Num  LSP Checksum  LSP Holdtime      ATT/P/OL
LSK-UNZ-UPE-01.00-00* 0x00000629   0xB4CE        65268             0/0/0
  Area Address: 49.0001
  NLPID:        0xCC 0x8E
  Topology:     IPv4 (0x0)
                IPv6 (0x2)
  Hostname: LSK-UNZ-UPE-01
  Metric: 2          IS-Extended XXXX.2003.1181.00
  Metric: 20         IS-Extended NNI-COM-UPE-01.00
  Metric: 20         IS-Extended NNI-WLD-UPE-01.00
  IP Address:   197.220.31.180
  Metric: 20         IP 10.0.0.202/31
  Metric: 20         IP 10.0.0.244/31
  Metric: 20         IP 10.0.0.250/31
  Metric: 2          IP 10.0.0.254/31
  Metric: 0          IP XXX.XXX.31.180/32

[Peter Paluch]

Amos,

Just curious, why are you running 5 different ISIS instances? Do you need to keep the topologies separate? Are you having them in separate VRFs?

I am wondering if we are perhaps hitting some limitation of your platform on the number of running IS-IS instances. If your IS-IS instances all run in the global routing table (no VRFs) then what you are doing is effectively an IS-IS multi-area configuration. The docs about running multiple IS-IS areas insist that only one process can perform Level2 routing, all other processes will be forced to Level1 routing. Check this document for more details:

http://www.cisco.com/c/en/us/td/docs/ios-xml/ios/iproute_isis/configuration/xe-3se/3650/irs-xe-3se-3650-book/irs-ma-sprt.pdf

Can you describe in more detail, then, how do you actually deploy the multiple IS-IS processes?

Best regards,
Peter

[Amos Kafwembe]

Peter,

my set up is that; I have one aggregation point to which my 5 remote sites are linking to, this is the NNI-UNZ-UPE where I have 5 ISIS neighbors connecting. They are all running in the global routing table (no VRF's). I am using the same process for all devices and all 4 are running okay. Is there a limit to how many ISIS neighbors a single device may have?

I have tried to directly connect another device, an ASR9k to the ME and its also not forming.

[Amos Kafwembe]

Peter,

this is my complete ISIS config. Only Link_C is not forming an adjacency the rest are forming just okay. In addition, like i mentioned before, i have tried to direclty connect an ASR9K to this ME and try to form an adjacency but it will not form. hello packets stuck on zero!

router isis as47888
 net 49.0001.aaaa.bbbb.ccde.00
 is-type level-2-only
 ispf level-2 60
 metric-style wide
 fast-flood 10
 ignore-attached-bit
 set-overload-bit on-startup wait-for-bgp
 max-lsp-lifetime 65535
 lsp-refresh-interval 65000
 spf-interval 5 1 20
 prc-interval 5 1 20
 lsp-gen-interval 5 1 20
 no hello padding
 log-adjacency-changes
 metric 100000
 fast-reroute per-prefix level-2 all
 fast-reroute remote-lfa level-2 mpls-ldp
 passive-interface Loopback0
 bfd all-interfaces
 !

interface GigabitEthernet0/12
 description LINK_A
 no switchport
 mtu 9000
 ip address 10.0.0.203 255.255.255.254
 ip router isis as47888
 bfd interval 50 min_rx 50 multiplier 3
 no bfd echo
 isis network point-to-point
 isis metric 20 level-2
 isis csnp-interval 10
!
interface GigabitEthernet0/13
 description Link_B
 no switchport
 mtu 9000
 ip address 10.0.0.239 255.255.255.254
 ip router isis as47888
 bfd interval 50 min_rx 50 multiplier 3
 no bfd echo
 isis network point-to-point
 isis metric 20 level-2
 isis csnp-interval 10
!
interface GigabitEthernet0/11
 description Link_C
 no switchport
 mtu 9000
 ip address 10.0.0.245 255.255.255.254
 ip router isis as47888
 bfd interval 50 min_rx 50 multiplier 3
 no bfd echo
 isis network point-to-point
 isis metric 20 level-2
 isis csnp-interval 10
!
interface TenGigabitEthernet0/1
 description Link_D
 no switchport
 mtu 9000
 ip address 10.0.0.254 255.255.255.254
 ip router isis as47888
 bfd interval 50 min_rx 50 multiplier 3
 no bfd echo
 isis network point-to-point
 isis metric 2 level-2
 isis csnp-interval 10

[Vinit Jain]

Adding to Peter's point, If you want to segregate multiple topologies, you can also use MTR feature. 

Regarding the down isis nei, can you try to configure a lower clns mtu to see if that makes any difference.

Could you please share both the IOS and XR version running between the two problematic nodes.

Thanks,

Vinit