Lindsay,

Linz-cisco · ‎10-28-2016

Hi all,

After much searching I can't seem to find others with the same issues I am seeing. Hopefully you can spare some time to suggest what could be the issue.

We have a Australian NBN connection (FTTP) and a 2921 router.

According to the isp the Uni-D port and the NTD check out fine. The NTD is connected to our Gi0/2 port.

The symptoms are in both directions, outgoing to the internet and incoming from the internet.

The problems show up like a time out but from the limited debugging skills I have in IOS I can see the timeouts related to entries that are not in the CEF (FIB) table. Entries join the table about 90 seconds after the "initial" request. Once in the table they connect fine.

ipv6 cef is disabled (only using ipv4).

The table appears to get up to about 16,000 entries.

I also tried this without CEF and saw the same symptom at least from an outgoing perspective.

DNS resolution is fine.

sh version
Cisco IOS Software, C2900 Software (C2900-UNIVERSALK9-M), Version 15.1(2)T4, RELEASE SOFTWARE (fc1)

interface GigabitEthernet0/2
description NBN link to iiNet 100/40mbps
ip address dhcp
ip access-group OUTGOING-ONLY in
ip access-group INTERNET-ZONE out
ip mtu 1452
ip nat outside
ip virtual-reassembly in
duplex auto
speed auto
no cdp enable

Is there any way to tell the router to use the default gateway by default when it cant resolve via its internal FIB table?

Should I be using a completely different routing method?

Could this be a hardware problem?

Could this be an NBN/isp issue that they just haven't worked out yet?

I do use a small amount of static NAT rules and the router does have phone system components which are currently unused.

Thanks all

Lindsay

Georg Pauwen · ‎10-29-2016

Lindsay,

the default memory for the 2921 is 512MB, which should be plenty for 16000 FIB entries. However, it might be worth looking at CPU utilization when these timeouts occur. The command 'sh proc cpu' will show you the utilization and what processes are causing it.

You might want to add 'ip tcp adjust-mss 1420' to your GigabitEthernet0/2.

You have just one static route I assume ?

ip route 0.0.0.0 0.0.0.0 GigabitEthernet0/2 ?

Linz-cisco · ‎10-29-2016

Thanks for the response.

I'll look into the adjust-mss option.

There are a few static routes (5 + the default) that point back to the internal network, I have a few subnets and until recently had two WAN links, the second WAN link was hooked into an ADSL router. The issues were present with both WAN links (so the cuase is not a change based on that).

The default route is indeed "ip route 0.0.0.0 0.0.0.0 Gi0/2"

There are hosts on the same subnet as the router and an entry for this subnet routed via Gi0/2, not sure if this is needed however but doubt it would cause an issue? (ip route 10.1.1.0 255.255.255.0 Gi0/2).

I'll monitor the processor usage also.

Thanks again.

Georg Pauwen · ‎10-29-2016

Hello,

if possible, can you post the full config of your router ? Maybe there is something else in there that could be adjusted...

Linz-cisco · ‎10-30-2016

Thanks for that, I'll post a sanitized config a bit later.

To continue on the debugging theme, I assume ARP is resolved before CEF/FIB tables?

When debugging, I can see that it is taking about 90 seconds+ to create an ARP entry for new entries not already in the ARP table.

While this is not there, no traffic parses into the debug data (debug all was used to show this).

ping and traceroute are the typical test commands.

Once the ARP entry is created, ping and traceroute and associated debug messages work as expected.

I've tried this with cef enabled and disabled.

From a CPU perspective, the router is not doing all that much and I have not yet seen any CPU spikes (most processes under 0.5% from what I've seen to date).

Is there a way to tell if ARP delays are to do with my router or the ISP end?

I also have a Cisco 3750g switch connected to this 2921 which CDP appears to correctly show.

I found the tcp adjust-mss 1412 was more stable than 1420. With 1420 I appeared to get more regular drop outs (remote connection from another location) but it hasn't been all that long yet so still monitoring this piece also. (other research appears to show MTU - 40 is an appropriate MSS setting).

Cheers again.

Georg Pauwen · ‎10-31-2016

Hello,

I wonder if you create a static ARP entry for the next hop, which is the NBN device ?

arp "ip address" "mac-address" arpa

Linz-cisco · ‎11-07-2016

Hi again,

I'm still struggling to find the most appropriate configuration to make this work properly.

To update, we've been able to flatten the network considerably over the last few days (have been planning to for a while, this forced our hand).

We now have a single static route in the 2921.

We have a 3750 providing DHCP to clients, currently with the 2921 using an ip on the same subnet as the dhcp range.

I saw other details that say to use "ip route 0.0.0.0 0.0.0.0 dhcp" instead of point to an interface.

For us this didn't work effectively.

I did seem to help but clearly some traffic was not being routed effectively (eg nothing getting the destination NATs in the router from the internet side.

I suspect DHCP being directly connected (the 3750) was faster to respond than the ISP dhcp...

So at the moment we are still using Gi0/2 as the ip route but it is typically taking 90 seconds to resolve an ARP entry and create it in the 2921 arp table.

I also tried using "ip route 0.0.0.0 0.0.0.0 gi0/2 dhcp", but this did not seem to be any different.

Here is the current 2921 config;

version 15.1

no service pad

service tcp-keepalives-in

service tcp-keepalives-out

service timestamps debug datetime msec localtime

service timestamps log datetime msec localtime show-timezone

service password-encryption

!

hostname <snip>

!

boot-start-marker

boot-end-marker

!

security authentication failure rate 2 log

security passwords min-length 6

logging buffered 32768

no logging console

<snip>

!

aaa new-model

!

aaa authentication login default local

aaa authorization exec default local

!

aaa session-id common

!

clock <snip>

!

no ipv6 cef

no ip source-route

no ip gratuitous-arps

ip cef

!

no ip dhcp conflict logging

!

no ip bootp server

ip domain name <snip>

ip name-server 8.8.4.4

ip name-server 8.8.8.8

!

multilink bundle-name authenticated

!

crypto pki token default removal timeout 0

!

voice-card 0

dsp services dspfarm

!

license <snip>

hw-module ism 0

!

hw-module pvdm 0/0

!

object-group network local_subnets

any

!

username <snip>

!

redundancy

!

ip tcp synwait-time 10

ip ssh authentication-retries 2

ip ssh version 2

!

interface GigabitEthernet0/0

description --- Data ---

ip address 1.1.1.1 255.255.255.0

no ip redirects

no ip proxy-arp

ip nat inside

ip virtual-reassembly in

ip tcp adjust-mss 1412

duplex auto

speed auto

!

interface ISM0/0

ip address 2.2.2.2.2 255.255.255.252

shutdown

service-module ip address 2.2.2.1 255.255.255.252

!Application: CUE Running on ISM

service-module ip default-gateway 2.2.2.2

!

interface ISM0/1

no ip address

shutdown

!

interface GigabitEthernet0/1

ip address 3.3.3.3 255.255.255.0

no ip redirects

no ip proxy-arp

ip flow ingress

ip flow egress

shutdown

duplex auto

speed auto

!

interface GigabitEthernet0/2

description NBN link

ip address dhcp

ip access-group INTERNET-OUT in

ip access-group INTERNET-ZONE out

no ip redirects

ip mtu 1452

ip nat outside

ip virtual-reassembly in

duplex auto

speed auto

no cdp enable

!

no ip forward-protocol nd

!

ip http server

ip http authentication local

no ip http secure-server

ip http timeout-policy idle 180 life 180 requests 86400

ip http path flash:/cme-gui

!

ip nat inside source static tcp <snip snip>

ip nat inside source list INTERNET-OUT interface GigabitEthernet0/2 overload

ip route 0.0.0.0 0.0.0.0 GigabitEthernet0/2

!

ip access-list standard LOGIN-ACL

<snip>

ip access-list standard SNMP-MONITOR-ACL

<snip>

!

ip access-list extended INTERNET-OUT

permit ip object-group local_subnets any

ip access-list extended INTERNET-ZONE

permit tcp <snip snip>

!

logging <snip>

!

snmp-server <snip>

!

control-plane

!

dspfarm profile 1 transcode

codec g711ulaw

codec g711alaw

codec g729ar8

codec g729abr8

maximum sessions 12

associate application SCCP

!

gatekeeper

shutdown

!

banner login ^C

---------------------------------------------------------------------------

This is a restricted access system. Unauthorised access is prohibited.

---------------------------------------------------------------------------

^C

!

line con 0

!

scheduler allocate 20000 1000

ntp server <snip> iburst

end

Would appreciate your feedback on how I could possibly make the ip route dhcp preferred configuration work or other options.

Thanks kindly

Linz

Georg Pauwen · ‎11-07-2016

Hello,

I have a question:

You wrote:

"I did seem to help but clearly some traffic was not being routed effectively (eg nothing getting the destination NATs in the router from the internet side."

Are you talking about traffic coming FROM the Internet TO your local network, or traffic ORIGINATING on your local network going out TO the Internet ?

Linz-cisco · ‎11-07-2016

What I believe I saw - and this was only brief as this is in a production context hence minimising outage time - the traffic coming FROM the internet to NAT configured on the router pointing to service inside our network - stopped working completely - but due to the short time I could give this, I guess I can't be 100% certain it wouldn't eventually find its way to our router.

Traffic going out improved by way of something like 1 in 2 packets getting to the correct destination - BUT - only when I had both of these in place;

ip route 0.0.0.0 0.0.0.0 gi0/2

ip route 0.0.0.0 0.0.0.0 dhcp

surprisingly the router let me implement two routes for 0.0.0.0

So from what I can deduce, the dhcp based route did actually work for some of the time (perhaps competing with the local dhcp in the 3750...? ) but it still needed the gi0/2 route to work at all.

The ARP issue appeared to reduce (not stop) when both routes were in as new sites resolved for the most part (not 100% however) - perhaps the router was doing some sort of round robin process..

Thanks again, if we can resolve this, I'll owe you a beer or three!

Georg Pauwen · ‎11-07-2016

Hello,

I would remove the static default route out of gi0/2 altogether and leave the dhcp one in there. There are multiple problems with static default routes pointing to local Ethernet interfaces, one of them being that the local ARP table gets very large because for every packet sent out, the interface must ARP for the destination.

The reason you see an improvement with the dhcp default route added is that the router does load sharing using both default routes (since CEF is enabled).

So, try and remove ip route 0.0.0.0 0.0.0.0 gi0/2, and check if that makes a difference.

Dropped/forgotten "routes" - 2921 + iiNet NBN