high retransmission and duplicate ack in core router interface(ASR 1001)

[AbdullahiGuled5724]

Hello

we have network consists of NOC and more 15 branch office.branch offices are interconnect with 2960 switch in ring topology using fiber line.

At NOC we have 2 2960 switch terminating the fiber ring(both switch are interconnect etherchannel to complete the ring topology ) and 2 asr1001 routers and of the switch is connected to router with gig port

For past month client were complain internet speed.after doing investigations i have figured out that router port connecting to switch have high total output drop and high packet dropped from output queue.(i have cleared errors to make sure it wasn't old error counters but it increase immediately ) pictures included

I had SPAN switch port  connecting router and capture packets using Wireshark and after doing Bad TCP i have found found more than 10% and sometimes 30% bad tcp(retransmission, duplicate ack and TCP out of order )

Router interface configuration

interface GigabitEthernet0/0/1
description "All LAN Interface" - -Core-Sw1 - Gi1/0/48
no ip address
no ip redirects
no ip unreachables
no ip proxy-arp
ip nat inside
ip nbar protocol-discovery
ip tcp adjust-mss 1452
speed 1000
no negotiation auto
cdp enable
service-policy output Parent-LAN-Shaper
!

Switch interface configuration

interface GigabitEthernet1/0/48
description ** Connection to MOPT-CE-Router - Gi0/0/1
switchport trunk native vlan 299
switchport mode trunk
switchport nonegotiate
logging event trunk-status
speed 1000
duplex full
srr-queue bandwidth share 1 70 25 5
priority-queue out
mls qos trust dscp
spanning-tree portfast
ip dhcp snooping trust
end

After reading some forums i have suspect that service policy was causing the issue and i have delete the service policy to see if it has any effect and after remove service policy from the router interface the output drop and packet dropped from output queue stopped.but i did another wireshark capture and we are still experience high number of retransmission and duplicate ack

we have more than 20 vlan and subinterface in router port

Any support is highly appreciated.

[Giuseppe Larosa]

Hello Abdullah,

the shaper should be applied on the router that connects to the upstream provider where the Ethernet over SDH should be placed. The following is my understanding:

>>Core Switch >>>>>>NAT router>>>>>>>Router connect to Upstream Provider >>> Eth over SDH STM1 SDH>>>>Upstream provider router

also the ISP upstream router should have a SHAPER towards  your router.

Note: shaping can be applied only outbound on Cisco routers.

warning: verify in which section of your network the STM1 Ethernet over SDH is present and configure the shaper on it on both sides.

Hope to help

Giuseppe

[Giuseppe Larosa]

Hello Abdullah,

we cannot say 100% there isn't an ICMP policer, however the error rate is the same with packet size 1000 bytes.

And this leads to think there is an issue on the link between your router and the upstream ISP router.

At this point I would open a ticket with the service provider. (after having tested a shaper again)

You have removed the shaper in the outbound direction that is the upstream direction.

For standard web access the downstream traffic is greater then upstream traffic,

The slow internet experience is related to all the TCP resending events.

Edit:

Having seen that you have GE interface if there is an ethernet over SDH transport over STM1  between your router and the ISP you need a shaper on your side. See my other post.

Hope to help

Giuseppe

[AbdullahiGuled5724]

Hello Giuseppe.

As planned i wanted to implement the shaper but i want to check how the network is before i change any so i can have baseline i can compare it once i implement the sharper.then something weird(i can't explain) happened.

We use Cacti to monitor the network and provide graphs for our self and to our clients.

router input and output interface value  are very different from cacti one.

Which one is correct router interfaces or cacti??

Also ping to upstream router and internet have improved significantly 

i think the ISP had an issue that they didn't disclose to us and now they have fix now.

But we still have the wireshark bad tcp around 30% 

should i still implement the sharper ???

[Giuseppe Larosa]

Hello Abdullah,

the good news is that connectivity to the ISP router has improved.

About your questions:

CACTI uses SNMP counters and plot differences between last two readings. So it is an average between two SNMP polls.

Cisco interface counters are averaged over last 20 minutes using an exponential weighting factor. For sure CACTI should poll every 5 minutes.

On cisco devices you can get a quicker behaviour if you use

load-interval 30

(seconds) in interface mode. You have the default load interval of 5 minutes. So the average is over 4 * 5 = 20 minutes.

You should look at CACTI graphs as more accurate.

If there is SDH on the path to the provider you still need a shaper as the router can send a burst of packets at line speed 1000 Mbps and some of them can be lost in the SDH equipment.

The shaper avoids the router to send packets in bursts and allows SDH equipment to process all packets.

To be noted also your ISP should implement a similar shaper on his side towards your router, because they have the same problem.

You still have 30% of bad TCP so the network still needs tuning. The shaper can help.

Hope to help

Giuseppe