RV082 Dropping Packets? (captures included)

[evanevery]

It appears we might have an issue with our RV082 (v4.2.1.02) dropping packets during the teardown of many TCP conversations.  I have attached two packet captures of what I believe is the same conversation.  One is from outside the router (Wireshark using an Ethernet Tap) and the other is from the client inside the router (SLES11SP2 running TCPDump).  These are both very small captures 9 packets and 18 packets and I'm hoping it will help identify the problem.

It appears that the RV082 is prematurely closing the natted port used to communicate with the host outside the network.  The host sends a FIN, ACK packet, to which the client responds with an ACK,  However, when the client then sends his FIN,ACK sequence, it never makes it outside the router.  The client sends a total of 9 FIN,ACKs trying to contact the outside server, but none of those appear to make it through the router.

Am I missing somethinge here or is the router slamming the door prematurely?  (I've been fighting with this problem for 3 weeks now!)

Inside Capture:

----------------------

No.     Time                       Source                Destination           Protocol Length Info

      1 2013-02-13 19:32:37.827942 192.168.1.45          38.113.116.214        TCP      76     35975 > http [SYN] Seq=0 Win=14600 Len=0 MSS=1460 SACK_PERM=1 TSval=635644783 TSecr=0 WS=128

      2 2013-02-13 19:32:37.895107 38.113.116.214        192.168.1.45          TCP      76     http > 35975 [SYN, ACK] Seq=0 Ack=1 Win=5792 Len=0 MSS=1460 SACK_PERM=1 TSval=1318450816 TSecr=635644783 WS=128

      3 2013-02-13 19:32:37.895137 192.168.1.45          38.113.116.214        TCP      68     35975 > http [ACK] Seq=1 Ack=1 Win=14720 Len=0 TSval=635644800 TSecr=1318450816

      4 2013-02-13 19:32:37.895210 192.168.1.45          38.113.116.214        HTTP     1007   POST /SpamResolverNG/SpamResolverNG.dll?DoNewRequest HTTP/1.0

      5 2013-02-13 19:32:37.965325 38.113.116.214        192.168.1.45          TCP      68     http > 35975 [ACK] Seq=1 Ack=940 Win=7680 Len=0 TSval=1318450887 TSecr=635644800

      6 2013-02-13 19:32:37.966121 38.113.116.214        192.168.1.45          HTTP     377    HTTP/1.1 200 OK  (text/html)

      7 2013-02-13 19:32:37.966142 192.168.1.45          38.113.116.214        TCP      68     35975 > http [ACK] Seq=940 Ack=310 Win=15744 Len=0 TSval=635644818 TSecr=1318450887

      8 2013-02-13 19:32:43.020843 38.113.116.214        192.168.1.45          TCP      68     http > 35975 [FIN, ACK] Seq=310 Ack=940 Win=7680 Len=0 TSval=1318455942 TSecr=635644818

      9 2013-02-13 19:32:43.060371 192.168.1.45          38.113.116.214        TCP      68     35975 > http [ACK] Seq=940 Ack=311 Win=15744 Len=0 TSval=635646092 TSecr=1318455942

     10 2013-02-13 19:33:22.451561 192.168.1.45          38.113.116.214        TCP      68     35975 > http [FIN, ACK] Seq=940 Ack=311 Win=15744 Len=0 TSval=635655939 TSecr=1318455942

     11 2013-02-13 19:33:22.752382 192.168.1.45          38.113.116.214        TCP      68     35975 > http [FIN, ACK] Seq=940 Ack=311 Win=15744 Len=0 TSval=635656015 TSecr=1318455942

     12 2013-02-13 19:33:23.364378 192.168.1.45          38.113.116.214        TCP      68     35975 > http [FIN, ACK] Seq=940 Ack=311 Win=15744 Len=0 TSval=635656167 TSecr=1318455942

     13 2013-02-13 19:33:24.580357 192.168.1.45          38.113.116.214        TCP      68     35975 > http [FIN, ACK] Seq=940 Ack=311 Win=15744 Len=0 TSval=635656472 TSecr=1318455942

     14 2013-02-13 19:33:27.020379 192.168.1.45          38.113.116.214        TCP      68     35975 > http [FIN, ACK] Seq=940 Ack=311 Win=15744 Len=0 TSval=635657082 TSecr=1318455942

     15 2013-02-13 19:33:31.892381 192.168.1.45          38.113.116.214        TCP      68     35975 > http [FIN, ACK] Seq=940 Ack=311 Win=15744 Len=0 TSval=635658300 TSecr=1318455942

     16 2013-02-13 19:33:41.636381 192.168.1.45          38.113.116.214        TCP      68     35975 > http [FIN, ACK] Seq=940 Ack=311 Win=15744 Len=0 TSval=635660736 TSecr=1318455942

     17 2013-02-13 19:34:01.156385 192.168.1.45          38.113.116.214        TCP      68     35975 > http [FIN, ACK] Seq=940 Ack=311 Win=15744 Len=0 TSval=635665616 TSecr=1318455942

     18 2013-02-13 19:34:40.132389 192.168.1.45          38.113.116.214        TCP      68     35975 > http [FIN, ACK] Seq=940 Ack=311 Win=15744 Len=0 TSval=635675360 TSecr=1318455942

Outside Capture:

-------------------------

No.     Time                       Source                Destination           Protocol Length Info

      1 2013-02-13 19:30:03.969765 67.52.35.90           38.113.116.214        TCP      74     64437 > http [SYN] Seq=0 Win=14600 Len=0 MSS=1460 SACK_PERM=1 TSval=637422263 TSecr=0 WS=128

      2 2013-02-13 19:30:04.073302 38.113.116.214        67.52.35.90           TCP      74     http > 64437 [SYN, ACK] Seq=0 Ack=1 Win=5792 Len=0 MSS=1460 SACK_PERM=1 TSval=1325560664 TSecr=637422263 WS=128

      3 2013-02-13 19:30:04.073965 67.52.35.90           38.113.116.214        TCP      66     64437 > http [ACK] Seq=1 Ack=1 Win=14720 Len=0 TSval=637422289 TSecr=1325560664

      4 2013-02-13 19:30:04.074467 67.52.35.90           38.113.116.214        HTTP     1006   POST /SpamResolverNG/SpamResolverNG.dll?DoNewRequest HTTP/1.0

      5 2013-02-13 19:30:04.186232 38.113.116.214        67.52.35.90           TCP      66     http > 64437 [ACK] Seq=1 Ack=941 Win=7680 Len=0 TSval=1325560775 TSecr=637422289

      6 2013-02-13 19:30:04.187020 38.113.116.214        67.52.35.90           HTTP     382    HTTP/1.1 200 OK  (text/html)

      7 2013-02-13 19:30:04.187641 67.52.35.90           38.113.116.214        TCP      66     64437 > http [ACK] Seq=941 Ack=317 Win=15744 Len=0 TSval=637422318 TSecr=1325560776

      8 2013-02-13 19:30:09.242601 38.113.116.214        67.52.35.90           TCP      66     http > 64437 [FIN, ACK] Seq=317 Ack=941 Win=7680 Len=0 TSval=1325565832 TSecr=637422318

      9 2013-02-13 19:30:09.280536 67.52.35.90           38.113.116.214        TCP      66     64437 > http [ACK] Seq=941 Ack=318 Win=15744 Len=0 TSval=637423591 TSecr=1325565832

[Tom Watts]

Hi Edward, I believe this is a software issue that is related to the client trying to used TCP sessions that were closed. It is why those log entries show up.

If that is the case, it is a known issue.

If you want to make a higher point, please open a service request with the small business support center.

-Tom
Please mark answered for helpful posts

[oliversl1]

Hi Edward,

my 2 last suggestions(in this order) should be:

1- try to reproduce the problem with an internal setup

  For example: a PC -> LAN(192.168.1.0) -> RV082 -> WAN1(192.168.2.0) -> an Internal WebServer

  This will definitelly isolate the error to the RV082 and not your ISP Equipment(ADSL) or routers(there should be

  many router between the RV082 and an Internet WebServer)

2- if you want a response from Cisco(or a company), the way to do this is by paying for a support ticket.

I'm very interested in the results of point #1, because I use RV082 every day and want to know if there is a problem with them.

Regards,

Oliver

[evanevery]

After shopping my packet analysis around I got a pretty concise answer on ask.wireshark.org.  It really does appear that the RV082 is dropping packets.  Once the server requests the session to be closed (fin,ack), the router slams the door before the client can send out the required (fin,ack) handhsake from his side.  My application is sensitive to this, and considers the whole conversation to have failed.

The "Policy Violation" log entries that I posted above are not the cause of the problem, they are the effect.  The log entries are posted AFTER the router has slammed the door as those packets from the client have no where to go (the router has already torn down the NAT for that session).

If anyone wants to see how many connections that their RV082 router is improperly terminating, just enable logging for "Deny Policies".  Anything coming from your LAN and going to the WAN, which is logged as a policy violation, is likely due to this problem (unless you have rules other than the default set up).  That is why you also saw them when you turned this on...

My path is already clear.  I have two new routers coming in tomorrow.  (Here's a hint - they're NOT from Cisco!)  The RV082's are being removed from service.  I'm not paying Cisco for a support call in hopes they might get around to fixing this problem someday.  I already spent a good deal of time running this down and if they think I'm going to pay for them to look any further they are greatly mistaken.  Too bad they didn't see the value in this thread...

Good luck and thank you to all the users who provided valuable feedback!  I'm done with Cisco and I'm done here as well...

[jonatrod]

Hi Edward, I apologize for the issues you are having and I want to tell you we are working in your case. I´ve read this full post twice and I saw this is an unusual problem. You can Browse to Firewall-> Attack Checks and uncheck everything. If that helps start checking things and see if one of the options causes the problem. Again I encourage you to do an MTU test: reducing MTU to 1452. I also recommend you please call the Small Business Support Center to confirm if this is a bug. Please go here to find the phone number in your country:

https://www.cisco.com/en/US/support/tsd_cisco_small_business_support_center_contacts.html

Greetings and again I apologize for the issues you are having.

Johnnatan Rodriguez Miranda.

Cisco Network Support Engineer.

[SamirD]

I've been here as well.  Cisco mis-diagnosed my rv016 as faulty and a $400 replacement rv016 did the same thing.  I sent them weeks of syslog data (as well as my ISP since I believe their constant packet storm was the cause) and nothing.

But if you can live with these boxes doing their box as shipped from the factory, they're a great bang for the buck.  Speaking of which, if you're looking to get rid of your rv082s, I'd be interested.  I've got a gateway-to-gateway vpn project that will work great with these.

[dkochnev383]

Hi,

I am having RV082 for a while and started experience the same problems, which was mentioned in this discussion, only 2 weeks ago. I can confirm that it is a router issue itself. I think it might be because of overheated or something but defenetely it is router issue. I tested it on local network and was able to reproduce the same slow ping and packet losses.

I am buying the new one.

[SamirD]

I'm curious.  Did your replacement rv082 solve the issue?  Or did you go with another brand?

Huntsville's Premiere Car and Bike e-magazine: www.huntsvillecarscene.com

[NMITGuy1024]

I realize this is a very old thread to be reviving, but I am curious whether this problem was ever resolved.  I am seeing the same instance on my network with a recently installed RV082, but I haven't seen or heard of any connection problems.  Our setup if very simple, everything is set to default for rules, yet I am seeing exclusively nothing but these connection refused messages in the log and it is very concerning.  I know the firmware I have on this device out-of-the box is a version behind, so I will be upgrading to the latest firmware the next chance I have.

[evanevery]

Never was resolved.  Decomissioned the RV082s and RV042s and replaced ALL of our routers with non-Cisco/Linksys devices.  Cisco doesn't give a crap about these devices.  You can do all the captures you want and point their nose right at the facts but they will not resolve these problems.  Their tech support folks are designed just to keep you busy and wear you down...

We replaced everything with Netgear UTM devices.  All has been well since...

[NMITGuy1024]

Just wonderful. This complete lack of customer service is the major reason I hate being in IT these days.  It seems to be a growing problem in the industry.  Not at all what I was hoping to hear. Now I guess I'm stuck having to convince my vendor to find something that isn't flawed out-of-the-box. Seems there is a large number of people having this problem. I wonder if a class action might get Cisco's attention?

[SamirD]

The whole smb market is the armpit of the IT world. It's not just Cisco that has issues and support like this--it's any brand in the smb market including Netgear, Dlink, Linksys, etc. Why? Because of the price. These devices are pretty powerful and can do so many things for the price. Maybe not so robust, but that's what entry level enterprise products are for (at 2-3x the cost). This definitely is a growing problem in the industry (if it hasn't already been for years), but it's really our fault. We've told these companies (with our wallets) that we'd rather have something cheap and not so perfect than something rock solid at a higher price.

[evanevery]

[QUOTE]  "The whole smb market is the armpit of the IT world. It's not just Cisco that has issues and support like this--it's any brand in the smb market including Netgear, Dlink, Linksys, etc."

Well, that's simply NOT true.  We replaced the RV082 devices with new UTM devices from Netgear and my experience with their tech support is quite the opposite.  They seem genuinely interested in any issues I find and appear to work towards incorporating related improvements to better the product line...

...and the RV082/42 routers actually got BETTER support when they were handled by LinkSys!

Cisco is big, fat, and complacent.  They don't give a crap.  I'm done with them...

[NMITGuy1024]

DISCLAIMER: Sorry, I don't mean to hijack the thread here, I just couldn't help but vent. You can add HP to that list too.  I had given up on their lower-end printing line years ago after problems with PDF's sent to the laserjet 1020 would completely crashing the print spooler service and the only way to be able to restart the service properly was to drill down in an obscure system folder and delete some files before restarting the service.

I switched to Lexmark with relatively minor issues in comparison.  This year I decided to give HP a second chance and was let down once again when their Laserjet Pro refused to print multiple copies, after underwhelming attempts by their support to solve the problem, I replaced it with a Lexmark with no problem. I have another Laserjet Pro that sporadically will not print ANYTHING, but the job leave the queue without error and no error is displayed on the printer, it simply decides not to print. So disappointing and infuriating..

[SamirD]

What did you end up doing with the Cisco devices? If you have any left, I would be interested in them as they would work fine for what I need to put together.

[evanevery]

Since the RV082s and RV042s are obviously not suitable for business use, we simply gave them away to employees...