BGP routing issue

Ultra_Ag3NT01 · ‎10-26-2014

I have a particular vendor whose ISP is dropping ~50% of my network traffic to them. The problem is 7 hops into AT&T's network. This vendor claims the issue is my ISP. It looks more likely that AT&T is influencing the return routing in an attempt to load balance and dropping half of my packets into the bit bucket. My ISP agrees this is the case. My vendor is very problematic and is still pointing the finger at me and my ISP.

Here are the MTRs:

Fri Oct 17 10:55:01 PDT 2014
HOST: mw-02                       Loss%   Snt   Last   Avg Best Wrst StDev
1. 162.222.65.1                  0.0%   100    1.2   0.7   0.4   4.0   0.5
2. v135.core1.lax2.he.net        1.0%   100   10.9   3.8   0.8 35.7   6.7
3. xe-2-1-2.lax21.ip4.gtt.net    0.0%   100    1.0   1.4   0.8   8.9   1.1
4. as7018.lax21.ip4.gtt.net      0.0%   100    1.8   2.6   1.3   5.5   1.3
5. att-ic-153025-las-bb1.c.teli 1.0%   100    5.1 33.1   1.9 63.9 28.7
6. cr1.la2ca.ip.att.net          1.0%   100   55.2 56.8 53.5 147.4   9.3
7. cr2.dvmco.ip.att.net          1.0%   100   54.8 59.1 53.6 220.3 18.7
8. cr2.dvmco.ip.att.net          1.0%   100   60.5 59.5 53.9 69.7   3.6
9. cr81.ipsin.ip.att.net         0.0%   100   59.7 61.7 56.8 126.5   9.0
10. cr81.ipsin.ip.att.net         2.0%   100   58.8 60.8 56.9 123.7   7.8
11. 12.122.152.169                3.0%   100   59.7 63.3 58.3 99.2   9.4
12. 12.89.6.146                  52.0%   100   60.3 59.8 58.2 63.7   1.4
13. ???                          100.0   100    0.0   0.0   0.0   0.0   0.0
Fri Oct 17 11:00:01 PDT 2014
HOST: mw-02                       Loss%   Snt   Last   Avg Best Wrst StDev
1. 162.222.65.1                  0.0%   100    0.8   0.7   0.4   2.1   0.3
2. v135.core1.lax2.he.net        0.0%   100    1.0   4.8   0.8 47.4   9.3
3. xe-2-1-2.lax21.ip4.gtt.net    0.0%   100    1.2   1.7   1.0 13.3   1.4
4. las-b21-link.telia.net        1.0%   100    3.8   4.0   1.3 69.0   6.8
5. cr1.la2ca.ip.att.net          3.0%   100   60.6 34.4   1.8 61.7 26.8
6. cr1.slkut.ip.att.net          1.0%   100   57.2 56.5 53.3 62.4   1.9
7. cr2.dvmco.ip.att.net          1.0%   100   57.5 58.5 53.6 179.2 12.9
8. cr1.cgcil.ip.att.net          0.0%   100   56.4 56.2 53.5 82.1   3.4
9. cr1.cgcil.ip.att.net          0.0%   100   54.9 57.0 53.4 65.2   2.9
10. cr81.ipsin.ip.att.net         0.0%   100   60.2 58.6 53.0 156.3 13.9
11. 12.89.6.146                   0.0%   100   75.4 60.1 53.0 132.5 12.8
12. 12.89.6.146                  51.0%   100   59.1 59.4 58.3 66.1   1.5
13. ???                          100.0   100    0.0   0.0   0.0   0.0   0.0

I am not having this issue on any other links I can find so I really do believe the issue is AT&Ts. How to I prove this?

Thank you

Steve

johnlloyd_13 · ‎10-27-2014

hi,

you can do extended pings to the hops having packet drops and provide a show interface x to your upstream ISP if there's any errors/CRC.

you could ask them to do a path re-route if possible on their network. you need to push and chase your ISP sometimes.

Ultra_Ag3NT01 · ‎10-28-2014

I would totally agree if the problem was with my ISP.

I did do this originally to prove that my ISP was not the root cause of the traffic loss. Every step in the path is good until the 7th hop into the AT&T network (not my ISP). So it appears to me that on that particular device the return path is trying to load balance on two links and one of the return paths has a bad list of ASNs. Resulting in the approx 50% loss of traffic.

The vendor that resides on the AT&T network is not very technical and I need to provide some type of proof that the issue is at their ISP and not mine. The question I have is how to I provide that data so that they can involve their ISP to correct this issue?

Thank you,

Steve