ADSL2+ NIM-VAB-A Troubleshooting

[emuman100]

I'm having an issue where my ADSL dialer interface is not coming up despite the VDSL controller is trained using Verizon ADSL in the Eastern US. Here is Verizon's pertinent configurartion:

VPI:0 VCI:35 Encapsulation: LLC PPPoE username/password: X.X.X (not account specific)

This configuration worked with Verizon in the past on IOS, but now I'm using an ISR 4321 with NIM-VAB-A  and IOS-XE and it seems like there is an issue with the virtual circuit and there is no traffic being passed to initiate the PPPoE link and PPP session. Here is the pertinent configuration:

controller VDSL 0/1/0
 operating mode adsl2+ 
 firmware phy filename bootflash:nim_vab_phy_fw_A39t_B39g1_Bond39t.pkg

interface ATM0/1/0
 no ip address
 atm oversubscribe factor 2
 no atm enable-ilmi-trap
!
interface ATM0/1/0.1 point-to-point
 no atm enable-ilmi-trap
 pvc 0/35 
  encapsulation aal5snap
  pppoe-client dial-pool-number 1
 !
!
interface Dialer1
 ip ddns update hostname <snip>
 ip ddns update <snip>
 ip address negotiated
 ip mtu 1452
 ip nat outside
 encapsulation ppp
 ip tcp adjust-mss 1424
 dialer pool 1
 dialer idle-timeout 0
 dialer persistent
 dialer-group 1
 no cdp enable
 ppp authentication chap callin
 ppp chap hostname newdsl
 ppp chap password 0 X.X.X
 ppp pap sent-username newdsl password 0 X.X.X
 ppp ipcp dns request
 ppp ipcp address accept
!

The VSDL controller is trained with the DSLAM. You can see that the modem status is in showtime and in ADSL2+ mode with good levels:

Controller VDSL 0/1/0 is UP

Daemon Status:		UP 

			XTU-R (DS)		XTU-C (US)
Chip Vendor ID:		'BDCM'			 'IFTN'
Chip Vendor Specific:   0x0000			 0x71C5
Chip Vendor Country:    0xB500			 0xB500
Modem Vendor ID:	'CSCO'			 '    '
Modem Vendor Specific:  0x4602			 0x0000
Modem Vendor Country:   0xB500			 0x0000
Serial Number Near:     X.X.X 
Serial Number Far:      
Modem Version Near:     16.9.3
Modem Version Far:      0x71c5

Modem Status:		TC Sync (Showtime!) 
DSL Config Mode:	ADSL2+ 
Trained Mode:		G.992.5 (ADSL2+) Annex A 

TC Mode:		ATM 
Selftest Result:	0x00 
DELT configuration:	disabled 
DELT state:		not running 

Failed full inits:	0
Short inits:		0
Failed short inits:	0

Modem FW  Version:	4.14L.04
Modem PHY Version:	A2pv6F039t.d26d
Modem PHY Source:	bootflash:
Modem PHY FW Pkg:	nim_vab_phy_fw_A39t_B39g1_Bond39t.pkg

Line 0:

			XTU-R (DS)		XTU-C (US)
Trellis:		 ON			 ON
SRA: 			 enabled		 enabled
SRA count: 		 0			 0
Bit swap: 		 enabled		 enabled
Bit swap count:		 49			 0
Line Attenuation:	 31.0 dB		 14.8 dB
Signal Attenuation:	 29.3 dB		 15.5 dB
Noise Margin:		 14.2 dB		10.4 dB
Attainable Rate:	22736 kbits/s		 1184 kbits/s
Actual Power:		 0.0 dBm		12.7 dBm
Total FECC:		2962			 26
Total ES:		0			 1
Total SES:		0			 0
Total LOSS:		0			 0
Total UAS:		811445			 811445
Total LPRS:		0			 0
Total LOFS:		0			 0
Total LOLS:		0			 0


 		  DS Channel1	  DS Channel0	US Channel1	  US Channel0
Speed (kbps):		  NA	       11768		NA	        1182
SRA Previous Speed:	  NA	           0		NA	           0
Previous Speed:		  NA	           0		NA	           0
Total Cells:		  NA	  2294485443		NA	   230274271
User Cells:		  NA	       10784		NA	           0
Reed-Solomon EC:	  NA	        2962		NA	          26
CRC Errors:		  NA	           0		NA	           1
Header Errors:		  NA	           0		NA	           0
Interleave (ms):	  NA	        8.29		NA	        2.95
Actual INP:		  NA	        4.14		NA	        0.58

 Next, I wanted to make sure the virtual circuit is up:

sh atm vc
Codes: DN - DOWN, IN - INACTIVE

           VCD /                                            Peak Av/Min Burst
Interface  Name         VPI   VCI Type    Encaps     SC     Kbps   Kbps Cells St
0/1/0.1    1              0    35 PVC     SNAP       UBR    1182              UP
                                              (C)    UBR       0

I'm not sure if the above means that the virtual circuit is actually up or if it's talking about the status of the subinterface, which is always up.

Despite this, there is no PPP activity in the debug, and no IP address on the dialer interface. The VPI/VCI could be incorrect and so could the encapsulation, but how can I debug to know for sure? What other debugging can I do to troubleshoot this further? I know that ADSL circuits are still used today, but very limited. As you could have guessed it, Verizon is absolutely no help.

Thanks.

[emuman100]

I believe this issue might stem from going from an IOS to IOS-XE configuration, but I'm still not sure why the PPP session won't come up. Here is the PPPoE session list:

sh pppoe session
     1 client session

Uniq ID  PPPoE  RemMAC          Port                    VT  VA         State
           SID  LocMAC                                      VA-st      Type
    N/A      0  0000.0000.0000  ATM0/1/0.1              Di1 N/A        PADISNT
                0000.0000.0000  VC:  0/35

With PPP debugging on, this came up which seems to be common with some users:

Di1 DDR: Dialer cannot nail-up the profile - dialer string not configured

Here is the output of the pvc:

sh atm pvc 0/35
Description: N/A
ATM0/1/0.1: VCD: 1, VPI: 0, VCI: 35
 Bridge-dot1q-encap:65535
UBR, PeakRate: 1168 (2755 cps)
AAL5-LLC/SNAP, etype:0x0, Flags: 0x1840, VCmode: 0x0, Encapsize: 12
OAM frequency: 0 second(s), OAM retry frequency: 1 second(s)
OAM up retry count: 3, OAM down retry count: 5
OAM Loopback status: OAM Disabled
OAM VC Status: Not Managed
OAM Loop detection: Disabled
ILMI VC status: Not Managed
InARP frequency: 15 minutes(s)
Transmit priority 0
InPkts: 0, OutPkts: 0, InBytes: 0, OutBytes: 0
InPRoc: 0, OutPRoc: 3790, Broadcasts: 0
InFast: 0, OutFast: 0, InAS: 0, OutAS: 0
InPktDrops: 0, OutPktDrops: 0
CrcErrors: 0, SarTimeOuts: 0, OverSizedSDUs: 0, LengthViolation: 0, CPIErrors: 0
Out CLP=1 Pkts: 0
OAM cells received: 0

 I'm not sure if that indicated the pvc is active and running or not. Hopefully this information would lead to some clues.

Thanks.

[paul driver]

Hello
Just noticed I dont see no dailer-list for the relating interface dailer group?.

try this:
conf t
dialer-list 1 protocol ip permit

[Giuseppe Larosa]

Hello @emuman100 ,

>>

Di1 DDR: Dialer cannot nail-up the profile - dialer string not configured

The router IOS XE expects the dialer interface to be used to make a call and it complains it is missing a dialer string that is a phone number to call.

This makes me think of a possible IOS SW bug. As a workaround you could try to configure a dialer string 11111 under the interface dialer configuration just to see if this makes the router to go on.

Hope to help

Giuseppe

[emuman100]

Hi Giuseppe,

I think you might be right, it might be a bug. I did try adding a dialer string and the message went away, however, the interface did not establish a PPP session, but I found the reason for that as described in a post below. I'll still test this once I get to the point where the PPP session is working. Nice catch on this one too!

Thanks.

Jonathan

[paul driver]

Hello
Do you have all interfaces up then?
sh ip int brief
sh arp


Also it could be down to a firmware interoperability issue
sh version

Could try the following, and give the router a reboot.

controller VDSL 0/1/0
operating mode auto

int atm0/1/0
dsl operating-mode auto

int atm0/1/0.1 
encapsulation aal5snap ppp dialer

[emuman100]

Hi Paul,

All interfaces and controllers are up. The IOS version I'm using is 16.9.3 as you can see in my sh controller output above and there are no IOS interoperability issues that I am aware of. Do you know of any that I might have missed?

The atm interface does not accept "dsl-operating mode" since the HWIC-1ADSL and is configured under the controller interface. The pvc sub interface does not accept any commands after "aal5snap".

I still think this is a configuration issue. I was able to try the router I used in 2017 running IOS 15.5 and the ADSL EHWIC. It had the same configuration that worked then and after connecting the interface and powering it up, it works as it did. The dialer interface gets an IP address and I can pass traffic through the interface. Here is the pertinent information from it along with the currently working config. Please review it and see if anything jumps out at you.

*Oct  1 18:00:38.227: %LINK-3-UPDOWN: Interface ATM0/0/0, changed state to up
*Oct  1 18:00:38.899: %SYS-5-CONFIG_I: Configured from console by console
*Oct  1 18:00:39.227: %LINEPROTO-5-UPDOWN: Line protocol on Interface ATM0/0/0, changed state to up
*Oct  1 18:00:48.195: %DIALER-6-BIND: Interface Vi1 bound to profile Di1
*Oct  1 18:00:48.199: %LINK-3-UPDOWN: Interface Virtual-Access1, changed state to up
*Oct  1 18:00:48.691: %LINEPROTO-5-UPDOWN: Line protocol on Interface Virtual-Access1, changed state to up

sh cont vdsl 0/0/0
Controller VDSL 0/0/0 is UP

Daemon Status:           Up

                        XTU-R (DS)              XTU-C (US)
Chip Vendor ID:         'BDCM'                   'IFTN'
Chip Vendor Specific:   0x0000                   0x71C5
Chip Vendor Country:    0xB500                   0xB500
Modem Vendor ID:        'CSCO'                   '    '
Modem Vendor Specific:  0x4602                   0x0000
Modem Vendor Country:   0xB500                   0x0000
Serial Number Near:    FOC16051PAS 1941/K9 15.5(2)T
Serial Number Far:
Modem Version Near:    15.5(2)T
Modem Version Far:     0x71c5

Modem Status:            TC Sync (Showtime!)

DSL Config Mode:         ADSL2+
Trained Mode:   G.992.5 (ADSL2+) Annex A
TC Mode:                 ATM
Selftest Result:         0x00
DELT configuration:      disabled
DELT state:              not running

Full inits:             1
Failed full inits:      1
Short inits:            1
Failed short inits:     0

Firmware        Source          File Name
--------        ------          ----------
VDSL            user config     flash:VA_A_39t_B_35j_24m

Modem FW  Version:      160331_0352-4.02L.03.A2pv6C039t.d24m
Modem PHY Version:      A2pv6C039t.d24m
Trellis:                 ON                       ON
SRA:                     disabled                disabled
 SRA count:              0                       0
Bit swap:                enabled                 enabled
 Bit swap count:         87                      1
Line Attenuation:        31.0 dB                 13.7 dB
Signal Attenuation:      31.3 dB                 13.8 dB
Noise Margin:            14.0 dB                 11.0 dB
Attainable Rate:        22296 kbits/s            1192 kbits/s
Actual Power:             0.0 dBm                11.3 dBm
Total FECC:             13300                    15
Total ES:               8                        0
Total SES:              0                        0
Total LOSS:             0                        0
Total UAS:              64                       64
Total LPRS:             0                        0
Total LOFS:             0                        0
Total LOLS:             0                        0


                  DS Channel1     DS Channel0   US Channel1       US Channel0
Speed (kbps):             0            11768             0              1179
SRA Previous Speed:       0                0             0                 0
Previous Speed:           0                0             0                 0
Total Cells:              0        373754400             0          37373316
User Cells:               0            10147             0              9200
Reed-Solomon EC:          0            13300             0                15
CRC Errors:               0                8             0                 0
Header Errors:            0               13             0                 0
Interleave (ms):       0.00             8.29          0.00              2.96
Actual INP:            0.00             4.14          0.00              0.58

Training Log :  Stopped
Training Log Filename : flash:vdsllog.bin

sh atm pvc 0/35
Description: N/A
ATM0/0/0.1: VCD: 1, VPI: 0, VCI: 35
UBR, PeakRate: 1179 (2781 cps)
AAL5-LLC/SNAP, etype:0x0, Flags: 0x1840, VCmode: 0x0, Encapsize: 12
OAM frequency: 0 second(s), OAM retry frequency: 1 second(s)
OAM up retry count: 3, OAM down retry count: 5
OAM END CC Activate retry count: 3, OAM END CC Deactivate retry count: 3
OAM END CC retry frequency: 30 second(s),
OAM SEGMENT CC Activate retry count: 3, OAM SEGMENT CC Deactivate retry count: 3
OAM SEGMENT CC retry frequency: 30 second(s),
OAM Loopback status: OAM Disabled
OAM VC Status: Not Managed
OAM Loop detection: Disabled
ILMI VC status: Not Managed
InARP frequency: 15 minutes(s)
High Watermark: 0, Low Watermark: 0
InPkts: 3861, OutPkts: 5194, InBytes: 297823, OutBytes: 361290
InPRoc: 2533, OutPRoc: 2, Broadcasts: 0
InFast: 1328, OutFast: 1686, InAS: 0, OutAS: 0
InPktDrops: 0, OutPktDrops: 0/0/0 (holdq/outputq/total)
CrcErrors: 0, SarTimeOuts: 0, OverSizedSDUs: 0, LengthViolation: 0, CPIErrors: 0
Out CLP=1 Pkts: 0
OAM cells received: 1686
F5 InEndloop: 0, F5 InSegloop: 0,
F5 InEndcc: 0, F5 InSegcc: 0, F5 InAIS: 1686, F5 InRDI: 0
OAM cells sent: 1686
F5 OutEndloop: 0, F5 OutSegloop: 0,
F5 OutEndcc: 0, F5 OutSegcc: 0, F5 OutAIS: 0, F5 OutRDI: 1686
OAM cell drops: 0
Status: UP
PPPOE enabled. Current number of pppoe sessions: 1

sh atm vc
Codes: DN - DOWN, IN - INACTIVE

           VCD /                                            Peak Av/Min Burst
Interface  Name         VPI   VCI Type    Encaps     SC     Kbps   Kbps Cells St
0/0/0.1    1              0    35 PVC     SNAP       UBR    1179              UP

sh pppoe session
     1 client session

Uniq ID  PPPoE  RemMAC          Port                    VT  VA         State
           SID  LocMAC                                      VA-st      Type
    N/A   5545  0090.1aa0.a83c  ATM0/0/0.1              Di1 Vi1        UP
                30e4.dbbd.5ca8  VC:  0/35                   UP

controller VDSL 0/0/0
 operating mode adsl2+
 firmware filename flash:VA_A_39t_B_35j_24m
no cdp run

interface ATM0/0/0
 no ip address
 no atm ilmi-keepalive
 hold-queue 224 in
!
interface ATM0/0/0.1 point-to-point
 pvc 0/35
  pppoe-client dial-pool-number 1
 !
!

interface Dialer1
 ip ddns update hostname <snip>
 ip ddns update <snip>
 ip address negotiated
 ip mtu 1452
 ip nat outside
 ip virtual-reassembly in
 encapsulation ppp
 ip tcp adjust-mss 1424
 dialer pool 1
 dialer idle-timeout 0
 dialer persistent
 dialer-group 1
 ppp authentication chap callin
 ppp chap hostname <snip>
 ppp chap password 0 <snip>
 ppp pap sent-username <snip> password 0 <snip>
 ppp ipcp dns request
 no cdp enable

Thanks.

[emuman100]

Hi Paul,

Good catch, but I have that statement in my config already. I just did not post it. It's in the config of both this router and the old router.

Thanks.

Jonathan

[paul driver]

Hello
Have you verified with the ISP regards VPI/VCI values?

Also the difference in the working one are:

interface ATM0/1/0
no ip address
no atm oversubscribe factor 2

!
interface ATM0/1/0.1 point-to-point
no atm enable-ilmi-trap
pvc 0/35
no encapsulation aal5snap
 

[emuman100]

Hi Paul,

The VPI/VCI values are verified in the working config of the old router which was connected and powered on for testing. Everything worked as it should and I was able to pass traffic on the interface. See my post above showing both the controller and pvc up and passing traffic.

ATM oversubscription is set to 2 by default, so the no command will not negate it. I tried the config with or without specifying encapsulation with the same results, so it must be an auto function as it was not required for the old router. You can see the old router shows the circuit with aal5snap encapsulation. See my previous post. 

It has to be something specific to IOS-XE. Many configuration statements changed, and it can be a challenge getting old configs to work.

Thanks.

Jonathan

[emuman100]

Hi All,

I made some progress and found what I was looking for. This is the DSL PPPoE troubleshooting guide which troubleshoots each sublayer of the layer 2 link. The issue that there is no response from the ISP when it tries to establish a PPPoE session, which made perfect sense to be as it seemed like something was not getting communication to bring up the PPP session. The first clue was that it looked like there was something wrong with the PPPoE session:

sh pppoe session
     1 client session

Uniq ID  PPPoE  RemMAC          Port                    VT  VA         State
           SID  LocMAC                                      VA-st      Type
    N/A      0  0000.0000.0000  ATM0/1/0.1              Di1 N/A        PADISNT
                0000.0000.0000  VC:  0/35

The reason for this is because the far end is not seeing my PPPoE PADI requests:

Oct  4 21:13:39.113: Sending PADI: vc=0/35
Oct  4 21:14:11.397:  padi timer expired
Oct  4 21:14:11.397: Sending PADI: vc=0/35
Oct  4 21:14:43.658:  padi timer expired
Oct  4 21:14:43.658: Sending PADI: vc=0/35
Oct  4 21:15:15.926:  padi timer expired
Oct  4 21:15:15.926: Sending PADI: vc=0/35
Oct  4 21:15:48.210:  padi timer expired
Oct  4 21:15:48.210: Sending PADI: vc=0/35
Oct  4 21:16:20.474:  padi timer expired

With the older router and ADSL EHWIC with the working configuration connected to the circuit, the PPPoE session establishes with remote side MAC:

sh pppoe session
     1 client session

Uniq ID  PPPoE  RemMAC          Port                    VT  VA         State
           SID  LocMAC                                      VA-st      Type
    N/A   5545  0090.1aa0.a83c  ATM0/0/0.1              Di1 Vi1        UP
                30e4.dbbd.5ca8  VC:  0/35                   UP

Here is the debug output showing how the session comes up after the received PADO packet:

*Oct  4 23:19:37.296: %LINK-3-UPDOWN: Interface ATM0/0/0, changed state to up
*Oct  4 23:19:37.584:  padi timer expired
*Oct  4 23:19:38.296: %LINEPROTO-5-UPDOWN: Line protocol on Interface ATM0/0/0, changed state to up
*Oct  4 23:19:39.632:  padi timer expired
*Oct  4 23:19:41.680:  padi timer expired
*Oct  4 23:19:43.332: PPPoE:ATM0/0/0.1 0/35, Got ATM VC Event[4]
*Oct  4 23:19:43.540: PPPoE:ATM0/0/0.1 0/35, Got ATM VC Event[2]
*Oct  4 23:19:43.728:  padi timer expired
*Oct  4 23:19:43.728: Sending PADI: vc=0/35
*Oct  4 23:19:43.748: PPPoE 0: I PADO  R:0090.1aa0.a83c L:30e4.dbbd.5ca8 0/35  ATM0/0/0.1
*Oct  4 23:19:45.776:  PPPOE: we've got our pado and the pado timer went off
*Oct  4 23:19:45.776: OUT PADR from PPPoE Session
*Oct  4 23:19:45.916: PPPoE 5925: I PADS  R:0090.1aa0.a83c L:30e4.dbbd.5ca8 0/35  ATM0/0/0.1
*Oct  4 23:19:45.916: IN PADS from PPPoE Session
*Oct  4 23:19:45.916: %DIALER-6-BIND: Interface Vi1 bound to profile Di1
*Oct  4 23:19:45.916: PPPoE: Virtual Access interface obtained.
*Oct  4 23:19:45.916: PPPoE : encap string prepared
*Oct  4 23:19:45.916: [0]PPPoE 5925: data path set to PPPoE Client
*Oct  4 23:19:45.920: %LINK-3-UPDOWN: Interface Virtual-Access1, changed state to up
*Oct  4 23:19:46.516: %LINEPROTO-5-UPDOWN: Line protocol on Interface Virtual-Access1, changed state to up

I also noticed that "vpdn enable" was in the config of the old router, which was normally required for PPPoE configurations, at least with an ethernet interface. I added it to the IOS-XE router and that did not bring up the PPPoE session.

The next step, according to the linked troubleshooting guide, would be to troubleshoot the next upper sub layer, which would be the pvc. Any more tips would be appreciated.

Thanks

Jonathan

[emuman100]

I've made some progress in understanding this issue. So, I tried a different NIM-VAB-A and the PPPoE session came right up and I was able to pass traffic! PPPoE PADI packets make it through to the far end and the session establishes. The configuration I had posted works without issue. Here, I had thought the issue was solved.

In the past few days, there were lots of errors on the downstream channel and my speed was reduced due to an issue with the loop and weather. After a while, the errors increased to where it started retraining and the ATM interface started going up and down. Eventually, it settled on a state where the modem was trained, but PADI packets would not go through, just like before with the other NIM-VAB-A. I was thinking the line was still down but I wanted to verify that, so I tried the old router running IOS 15 and the ADSL EHWIC. The PPPoE session came right up, so the ADSL circuit was operational again. 

I decided that I would try rebooting the router to reboot the NIM-VAB-A. After I rebooted it and the NIM-VAB-A's embedded system booted up, the PPPoE session established again and traffic was passing on the circuit without issue.

In conclusion, it looks like a process or something gets hung up on the NIM-VAB-A's embedded Linux system after the modem is untrained for a long period or poor signal conditions causing retrains and flapping. If it happens again, instead of rebooting the router and taking down network traffic momentarily, I'll try the following:

hw-module subslot 0/1 reload

This should simply reboot the embedded Linux system and the PPPoE session should reestablish. I'll update this thread when I try it.

This seems to be the point where TAC should be contacted so they can investigate this bug and hopefully patch it. Does anyone here actually use this NIM and have experienced the same hang up where it would have to be restarted to pass ATM traffic?

Jonathan

[emuman100]

I wanted to provide an update. I've had a reliable interface since this last post. As of late, I noticed dropped and delayed pings, but did not test any other traffic. At first I thought it was the hosts that I was using, but it turns out it was the interface.

sh controller VDSL 0/1/0 wasn't showing anything that would indicate high pings in either CRC errors or others, which I thought was odd. Using:

hw-module subslot 0/1 reload

rebooted the NIM and retrained the modem and ping times were what they should be. Is this normal for this NIM? The NIM-LTEA-EA has never required reloads, but it does seem to do it automatically on occasion.

Thanks.