Fuji-16.09.05 Cisco ISR 1100 C1111-xP observations, bugs.

[Quentin Kelly]

The below observations may require some peer review.

All results observed on two C1111-8P and one C1111-4P with IOS-XE Fuji 16.09.05. All devices were factory reset and wiped before testing. All devices tested were connected to the internet via two different retail/consumer fibre connections, via ONT connected to Gi0/0/0 using dot1q, dialer using ppp pap authentication and using nat. One connection has a static ip, the other dynamic obtained ip. The dialer interface has (when they are working) no ip unreachables, no ip redirects, and an access-list blocking ping echo etc, and VTY lines blocked from login attempts, trying to be invisible to the internet.

Looking forward to feedback.

Problematic Dialer interface counters.
Priority : low / non impacting
Impacting both C1111-8P and C1111-4P using IOS-XE Fuji-16.09.05
Dialer interface within a few hours of activity will start throwing some wild numbers. Appears to have been introduced with 16.09.05, had not seen this previously. I did note that initially it was only the input rate that was incredible, but soon after the output rate did the same.

5 minute input rate 16029834000 bits/sec, 189 packets/sec
5 minute output rate 10304254000 bits/sec, 100 packets/sec
and
5 minute input rate 1110025000 bits/sec, 315 packets/sec
5 minute output rate 81000 bits/sec, 185 packets/sec
and
5 minute input rate 1106905000 bits/sec, 14 packets/sec
5 minute output rate 25000 bits/sec, 21 packets/sec

VTY Line destination IP(port) is always some random IP address?
Priority : low / non impacting
Impacting both C1111-8P and C1111-4P using any IOS-XE Fuji-16.09.x
Is this supposed to be this way? Has always left me questioning as with every other Cisco I have seen, the destination IP is always 0.0.0.0(port). But on a Cisco ISR1100/IOS-XE it always shows some bogus IP address which changes whenever you restart, regardless of dynamic or static IP addresses.

Using any other Cisco router, ever, I expect to see the following when trying to hit up the box directly.
007284: Feb 19 2020 07:08:50: %SEC-6-IPACCESSLOGP: list SECURE-VTY denied tcp 188.26.219.47(56811) -> 0.0.0.0(22), 1 packet

Yet on a C1111-8P we see the following examples (same box post restarts)
001046: Feb 19 2020 17:56:03: %SEC-6-IPACCESSLOGP: list SECURE-VTY denied tcp 222.254.23.12(53783) -> 57.198.56.160(22), 2 packets
000125: Feb 9 2020 09:47:09: %SEC-6-IPACCESSLOGP: list SECURE-VTY denied tcp 167.86.110.219(36892) -> 90.48.232.160(22), 1 packet
003182: Feb 8 2020 22:16:41: %SEC-6-IPACCESSLOGP: list SECURE-VTY denied tcp 222.186.15.236(9090) -> 70.236.56.160(22), 1 packet

'ip ddns update' fails.
Priority : irritating, as restoring management requires discovering the new dynamic ip address and manually update ddns.
Impacting C1111-8P and C1111-4P. I do not think this has ever worked with IOS-XE Fuji-16.09.x
While 'debug ip ddns' followed with a 'clear pppoe all' shows a successful result, there is also an HTML response displayed showing it returning a 404 URL not found failure, and the ip ddns update is never updated. Appears the configured URL may be truncated somehow before being sent? Yet, the URL in the configuration can be copy/pasted into a browser and it works fine. Note that when using the same configuration on C800 and C1900 series routers works fine.
Documentation for no-ip.com can be found here https://www.noip.com/support/knowledgebase/using-your-cisco-router-with-no-ip-dynamic-dns-services/ which works on every other Cisco IOS, but not on a Cisco ISR1100 with IOS-XE.

Hostname.RTR#clear pppoe all
Hostname.RTR#
000835: Feb 14 11:49:58.162: PPPoE VLAN CoS (pppoe_send_padt)=>GigabitEthernet0/0/0.10 :: cos=0x0
000836: Feb 14 11:49:58.162: [0]PPPoE 1: O PADT R:0023.3e92.2f9b L:b08b.cf8e.9900 Gi0/0/0.10
000837: Feb 15 2020 00:49:58: %DIALER-6-UNBIND: Interface Vi2 unbound from profile Di0
000838: Feb 14 11:49:58.170: [0]PPPoE 1: Destroying R:0023.3e92.2f9b L:b08b.cf8e.9900 10 Gi0/0/0.10
000839: Feb 14 11:49:58.170: PPPoE: Returning Vaccess Virtual-Access2
000840: .Feb 14 11:49:58.174: Sending PADI: Interface = GigabitEthernet0/0/0.10
000841: .Feb 14 11:49:58.174: PPPoE VLAN CoS (pppoe_client_send_padi)=>GigabitEthernet0/0/0.10 :: cos=0x0
000843: .Feb 14 11:49:58.175: DYNUPD: SWIF goingdown 'Virtual-Access2'
000845: .Feb 14 11:49:58.175: DYNDNSUPD: Removing DNS mapping for hostname.ddns.net <=> 1.2.3.4
000846: .Feb 14 11:49:58.176: HTTPDNS: Update remove called for hostname.ddns.net <=> 1.2.3.4
000848: .Feb 14 11:50:02.199: padi timer expired
000849: .Feb 14 11:50:02.199: Sending PADI: Interface = GigabitEthernet0/0/0.10
000850: .Feb 14 11:50:02.199: PPPoE VLAN CoS (pppoe_client_send_padi)=>GigabitEthernet0/0/0.10 :: cos=0x0
000851: .Feb 14 11:50:02.208: PPPoE 0: I PADO R:0023.3e92.2f9b L:b08b.cf8e.9900 10 Gi0/0/0.10
000852: .Feb 14 11:50:04.252: PPPOE: we've got our pado and the pado timer went off
000853: .Feb 14 11:50:04.252: OUT PADR from PPPoE Session
000854: .Feb 14 11:50:04.252: PPPoE VLAN CoS (pppoe_client_send_padr)=>GigabitEthernet0/0/0.10 :: cos=0x0
000855: .Feb 14 11:50:04.260: PPPoE 1: I PADS R:0023.3e92.2f9b L:b08b.cf8e.9900 10 Gi0/0/0.10
000856: .Feb 14 11:50:04.260: IN PADS from PPPoE Session
000858: .Feb 14 11:50:04.263: PPPoE: Virtual Access interface obtained.
000859: .Feb 14 11:50:04.263: [0]PPPoE 1: data path set to PPPoE Client
000862: .Feb 14 11:50:04.316: DYNUPD: SWIF comingup 'Virtual-Access2'
000863: .Feb 14 11:50:04.339: PPPoE : ipfib_encapstr prepared
000864: .Feb 14 11:50:04.340: PPPoE : ipfib_encapstr prepared
000865: .Feb 14 11:50:07.418: DYNDNSUPD: Adding DNS mapping for hostname.ddns.net <=> 1.2.3.4
000866: .Feb 14 11:50:07.418: HTTPDNS: Update add called for hostname.ddns.net <=> 1.2.3.4
000867: .Feb 14 11:50:07.418: HTTPDNSUPD: Session ID = 0xD
000868: .Feb 14 11:50:07.419: HTTPDNSUPD: URL = 'http://username:password@dynupdate.no-ip.com/nic/update?hostname=hostname.ddns.net&myip=1.2.3.4'
000869: .Feb 14 11:50:07.419: HTTPDNSUPD: Sending request
000870: .Feb 14 11:50:07.640: HTTPDNSUPD: Response for update hostname.ddns.net <=> 1.2.3.4
000871: .Feb 14 11:50:07.640: HTTPDNSUPD: DATA START

<!DOCTYPE html>
<html lang=en>
<meta charset=utf-8>
<meta name=viewport content="initial-scale=1, minimum-scale=1, width=device-width">
<title>Error 404 (Not Found)!!1</title>
<style>
*{margin:0;padding:0}html,code{font:15px/22px arial,sans-serif}html{background:#fff;color:#222;padding:15px}body{margin:7% auto 0;max-width:390px;min-height:180px;padding:30px 0 15px}* > body{background:url(//www.google.com/images/errors/robot.png) 100% 5px no-repeat;padding-right:205px}p{margin:11px 0 22px;overflow:hidden}ins{color:#777;text-decoration:none}a img{border:0}@media screen and (max-width:772px){body{background:none;margin-top:0;max-width:none;padding-right:0}}#logo{background:url(//www.google.com/images/branding/googlelogo/1x/googlelogo_color_150x54dp.png) no-repeat;margin-left:-5px}@media only screen and (min-resolution:192dpi){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat 0% 0%/100% 100%;-moz-border-image:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) 0}}@media only screen and (-webkit-min-device-pixel-ratio:2){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat;-webkit-background-size:100% 100%}}#logo{display:inline-block;height:54px;width:150px}
</style>
<a href=//www.google.com/><span id=logo aria-label=Google></span></a>
<p><b>404.</b> <ins>Thatbs an error.</ins>
<p>The requested URL <code>/nic/update?hostname=hostname.ddns.net&amp;myip=1.2.3.4</code> was not found on this server. <ins>Thatbs all we know.</ins>

000872: .Feb 14 11:50:07.651: HTTPDNSUPD: Response for update hostname.ddns.net <=> 1.2.3.4
000873: .Feb 14 11:50:07.651: HTTPDNSUPD: DATA START
000874: .Feb 14 11:50:07.654: HTTPDNSUPD: DATA END, Status is Response data recieved, successfully
000875: .Feb 14 11:50:07.654: HTTPDNSUPD: Call returned SUCCESS, update of hostname.ddns.net <=> 1.2.3.4 succeeded
000876: .Feb 14 11:50:07.654: DYNDNSUPD: Another update completed (outstanding=0, total=0)
000877: .Feb 14 11:50:07.655: HTTPDNSUPD: Clearing all session 13 info

Log sequence line 874 is false and cannot know that. I tried this on a Cisco C800 series that ip ddns update works on, but used some bogus details forcing it to fail and it still returned this message. I think error checking of the returned message is not being checked or understood if/when it fails.
log sequence line 875 is also false as no-ip.com never receive the message to update the ddns record with the details in log sequence line 867, which you can literally copy the URL marked in red into a browser and it works perfectly. This should can be easily replicated.

Dialer interface configuration doesnt work.
Priority : high risk
Impacting C1111-8P and C1111-4P. I do not think this has ever worked with any C1100 IOS-XE.

You can configure and write the following which is visible in the startup-config and running-config.
no ip unreachables
no ip redirects
ip access-group <acl name>
However, the above configurations do not work after power-on nor after a pppoe resync. There may be more configuration that drop off, will need to check.
Work around post restart or pppoe resync you must add ip redirects and ip unreachables and then 'no' them out again, and 'no' the access group and re-configure it to the dialer interface for it to work again. A real problem.

Random upstream traffic generated from Dialer when bound to Virtual-Access2
Priority : critical. but needs peer review.
Impacting C1111-4P only, all version of IOS-XE 16.x.x
Because I have only 1x C1111-4P I cannot replicate this and need a peer review. Note that I have never seen this happen on a C1111-8P on same firmware and configuration.
The C1111-4P box is configured exactly the same as both C1111-8P boxes. I have wiped/factory reset the 4P as I would any Cisco. I have never seen this from any other Cisco configured in this manner.

Observed after replacing a C800 series that was in production for a few years with a new C1111-4P, and a few weeks later the site complained of high outbound traffic usage. Upon investigation I observed higher traffic counters were appearing on the Dialer and Virtual-Access2. Traffic counters were not matching traffic from the LAN interfaces or VLANs. I isolated/shut/admin down all LAN facing interfaces leaving only the Gi0/0/0 (uplink to internet) and Dialer for authentication. Traffic was swiftly rising and easily hit 10mbit/s upstream to the internet.
Troubleshooting steps taken were to restart the box in case of a bug, which traffic went back to normal. However, around 2-3 weeks later the same thing. After restarting, added netflow to dialer and vlans to observe traffic trends and addresses when it started again. Sure enough, 2-3 weeks later. Traffic from Dialer and Virtual-Access2 started incrementing and exceeding 5mbit/s. Netflow showed high amounts of traffic sourced from the Dialer interface (bound to Vi2) ip address was being sent all over the internet, and no correlations. When left for a few days while recording destination addresses, traffic exceeded 15mbit/s upstream. If left longer can assume traffic would increase further.

I stress again this happens with all LAN interfaces shut/admin down leaving only the Dialer interface bound to Virtual-Access2 which show this traffic, which naturally leaves the Gi0/0/0 interface uplink to the internet. And the traffic is real and being counted by the ISP. Performed a 'clear pppoe all' which drops the Virtual-Access2 and obtains new dynamic ip address, traffic drops to zero, and follows this trend after another 2-3 weeks.

This looks very bad. I can only assume a Dialer interface software bug which is oddly affecting only the C1111-4P, as I have the same configuration and IOS-XE being used on two C1111-8P. I can assume this is not some kind of attack as there is little to no traffic inbound, and cannot see a trigger or inbound traffic when it starts. I have not done a packet capture to see what is actually being transmitted. Any ideas on troubleshooting further?
This can be replicated on every IOS-XE 16.x.x. Obviously the new Dialer interface traffic counter bug introduced in 16.09.05 does not help when observing traffic for this problem.

[Quentin Kelly]

Update > the last issue shown above where the traffic from the Virtual-Access2 bound to the Dialer is generating traffic by itself, its happening right now. If anyone has any tips or ideas on what to debug or troubleshoot this please get in touch as I would love to be able to provide more information on this.

LAN interface and VLAN are doing virtually no traffic, like 20k each way, while the Virtual-Access2 and Dialer (if it wasnt for that pesky bug) are showing 1.9mbit upstream, which is being reflected on the Gi0/0/0 uplink interface. I can admin down/shut the LAN/VLAN and this will go on until a reboot or a pppoe resync is forced.

Hostname.RTR#show int di0
Dialer0 is up, line protocol is up (spoofing)
Hardware is Unknown
Description: PPP Dialer
Internet address is <hidden>/32
MTU 1492 bytes, BW 56 Kbit/sec, DLY 20000 usec,
reliability 255/255, txload 255/255, rxload 255/255
Encapsulation PPP, LCP Closed, loopback not set
Keepalive set (10 sec)
DTR is pulsed for 1 seconds on reset
Interface is bound to Vi2
Last input 00:00:00, output 00:00:00, output hang never
Last clearing of "show interface" counters 2w4d
Input queue: 0/375/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: weighted fair
Output queue: 0/1000/64/0 (size/max total/threshold/drops)
Conversations 0/0/16 (active/max active/max total)
Reserved Conversations 0/0 (allocated/max allocated)
Available Bandwidth 42 kilobits/sec
5 minute input rate 59535047000 bits/sec, 102 packets/sec
5 minute output rate 74420472000 bits/sec, 251 packets/sec
268994385 packets input, 5571777922509426 bytes
297594394 packets output, 7158824386719498 bytes
Bound to:
Virtual-Access2 is up, line protocol is up
Hardware is Virtual Access interface
Description: PPP Dialer
Internet address will be negotiated using IPCP
MTU 1492 bytes, BW 56 Kbit/sec, DLY 20000 usec,
reliability 255/255, txload 255/255, rxload 255/255
Encapsulation PPP, LCP Open
Stopped: CDPCP
Open: IPCP
PPPoE vaccess, cloned from Dialer0
Vaccess status 0x44, loopback not set
Keepalive set (10 sec)
Interface is bound to Di0 (Encapsulation PPP)
Last input 00:00:06, output never, output hang never
Last clearing of "show interface" counters 1w5d
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 204000 bits/sec, 107 packets/sec
5 minute output rate 1901000 bits/sec, 254 packets/sec  <<<< to internet, not sourced from LAN
198945990 packets input, 171904255527 bytes, 0 no buffer
Received 0 broadcasts (0 IP multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
204960105 packets output, 180152239420 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 unknown protocol drops
0 output buffer failures, 0 output buffers swapped out
0 carrier transitions