It appears we might have an issue with our RV082 (v4.2.1.02) dropping packets during the teardown of many TCP conversations. I have attached two packet captures of what I believe is the same conversation. One is from outside the router (Wireshark using an Ethernet Tap) and the other is from the client inside the router (SLES11SP2 running TCPDump). These are both very small captures 9 packets and 18 packets and I'm hoping it will help identify the problem.
It appears that the RV082 is prematurely closing the natted port used to communicate with the host outside the network. The host sends a FIN, ACK packet, to which the client responds with an ACK, However, when the client then sends his FIN,ACK sequence, it never makes it outside the router. The client sends a total of 9 FIN,ACKs trying to contact the outside server, but none of those appear to make it through the router.
Am I missing somethinge here or is the router slamming the door prematurely? (I've been fighting with this problem for 3 weeks now!)
No. Time Source Destination Protocol Length Info
1 2013-02-13 19:32:37.827942 192.168.1.45 184.108.40.206 TCP 76 35975 > http [SYN] Seq=0 Win=14600 Len=0 MSS=1460 SACK_PERM=1 TSval=635644783 TSecr=0 WS=128
2 2013-02-13 19:32:37.895107 220.127.116.11 192.168.1.45 TCP 76 http > 35975 [SYN, ACK] Seq=0 Ack=1 Win=5792 Len=0 MSS=1460 SACK_PERM=1 TSval=1318450816 TSecr=635644783 WS=128
3 2013-02-13 19:32:37.895137 192.168.1.45 18.104.22.168 TCP 68 35975 > http [ACK] Seq=1 Ack=1 Win=14720 Len=0 TSval=635644800 TSecr=1318450816
4 2013-02-13 19:32:37.895210 192.168.1.45 22.214.171.124 HTTP 1007 POST /SpamResolverNG/SpamResolverNG.dll?DoNewRequest HTTP/1.0
5 2013-02-13 19:32:37.965325 126.96.36.199 192.168.1.45 TCP 68 http > 35975 [ACK] Seq=1 Ack=940 Win=7680 Len=0 TSval=1318450887 TSecr=635644800
6 2013-02-13 19:32:37.966121 188.8.131.52 192.168.1.45 HTTP 377 HTTP/1.1 200 OK (text/html)
7 2013-02-13 19:32:37.966142 192.168.1.45 184.108.40.206 TCP 68 35975 > http [ACK] Seq=940 Ack=310 Win=15744 Len=0 TSval=635644818 TSecr=1318450887
8 2013-02-13 19:32:43.020843 220.127.116.11 192.168.1.45 TCP 68 http > 35975 [FIN, ACK] Seq=310 Ack=940 Win=7680 Len=0 TSval=1318455942 TSecr=635644818
9 2013-02-13 19:32:43.060371 192.168.1.45 18.104.22.168 TCP 68 35975 > http [ACK] Seq=940 Ack=311 Win=15744 Len=0 TSval=635646092 TSecr=1318455942
10 2013-02-13 19:33:22.451561 192.168.1.45 22.214.171.124 TCP 68 35975 > http [FIN, ACK] Seq=940 Ack=311 Win=15744 Len=0 TSval=635655939 TSecr=1318455942
11 2013-02-13 19:33:22.752382 192.168.1.45 126.96.36.199 TCP 68 35975 > http [FIN, ACK] Seq=940 Ack=311 Win=15744 Len=0 TSval=635656015 TSecr=1318455942
12 2013-02-13 19:33:23.364378 192.168.1.45 188.8.131.52 TCP 68 35975 > http [FIN, ACK] Seq=940 Ack=311 Win=15744 Len=0 TSval=635656167 TSecr=1318455942
13 2013-02-13 19:33:24.580357 192.168.1.45 184.108.40.206 TCP 68 35975 > http [FIN, ACK] Seq=940 Ack=311 Win=15744 Len=0 TSval=635656472 TSecr=1318455942
14 2013-02-13 19:33:27.020379 192.168.1.45 220.127.116.11 TCP 68 35975 > http [FIN, ACK] Seq=940 Ack=311 Win=15744 Len=0 TSval=635657082 TSecr=1318455942
15 2013-02-13 19:33:31.892381 192.168.1.45 18.104.22.168 TCP 68 35975 > http [FIN, ACK] Seq=940 Ack=311 Win=15744 Len=0 TSval=635658300 TSecr=1318455942
16 2013-02-13 19:33:41.636381 192.168.1.45 22.214.171.124 TCP 68 35975 > http [FIN, ACK] Seq=940 Ack=311 Win=15744 Len=0 TSval=635660736 TSecr=1318455942
17 2013-02-13 19:34:01.156385 192.168.1.45 126.96.36.199 TCP 68 35975 > http [FIN, ACK] Seq=940 Ack=311 Win=15744 Len=0 TSval=635665616 TSecr=1318455942
18 2013-02-13 19:34:40.132389 192.168.1.45 188.8.131.52 TCP 68 35975 > http [FIN, ACK] Seq=940 Ack=311 Win=15744 Len=0 TSval=635675360 TSecr=1318455942
No. Time Source Destination Protocol Length Info
1 2013-02-13 19:30:03.969765 184.108.40.206 220.127.116.11 TCP 74 64437 > http [SYN] Seq=0 Win=14600 Len=0 MSS=1460 SACK_PERM=1 TSval=637422263 TSecr=0 WS=128
2 2013-02-13 19:30:04.073302 18.104.22.168 22.214.171.124 TCP 74 http > 64437 [SYN, ACK] Seq=0 Ack=1 Win=5792 Len=0 MSS=1460 SACK_PERM=1 TSval=1325560664 TSecr=637422263 WS=128
3 2013-02-13 19:30:04.073965 126.96.36.199 188.8.131.52 TCP 66 64437 > http [ACK] Seq=1 Ack=1 Win=14720 Len=0 TSval=637422289 TSecr=1325560664
4 2013-02-13 19:30:04.074467 184.108.40.206 220.127.116.11 HTTP 1006 POST /SpamResolverNG/SpamResolverNG.dll?DoNewRequest HTTP/1.0
5 2013-02-13 19:30:04.186232 18.104.22.168 22.214.171.124 TCP 66 http > 64437 [ACK] Seq=1 Ack=941 Win=7680 Len=0 TSval=1325560775 TSecr=637422289
6 2013-02-13 19:30:04.187020 126.96.36.199 188.8.131.52 HTTP 382 HTTP/1.1 200 OK (text/html)
7 2013-02-13 19:30:04.187641 184.108.40.206 220.127.116.11 TCP 66 64437 > http [ACK] Seq=941 Ack=317 Win=15744 Len=0 TSval=637422318 TSecr=1325560776
8 2013-02-13 19:30:09.242601 18.104.22.168 22.214.171.124 TCP 66 http > 64437 [FIN, ACK] Seq=317 Ack=941 Win=7680 Len=0 TSval=1325565832 TSecr=637422318
9 2013-02-13 19:30:09.280536 126.96.36.199 188.8.131.52 TCP 66 64437 > http [ACK] Seq=941 Ack=318 Win=15744 Len=0 TSval=637423591 TSecr=1325565832
Hi Edward, thank you for using our forum, my name is Johnnatan I am part of the Small business Support community.
I apologize for this issue, I am glad to assist you, I advise you to verify if both sides are in full duplex, to ensure both connections are sending and received packages at the same time. There are several reasons for a slow connection behind the router. Before you troubleshoot the RV wired router, attempt the next basic steps to see if the problem is solved.
Step 1. Contact your ISP (Internet Service Provider) to verify the bandwidth capabilities of your connection.
Step 2. Check if the LAN cards of the devices attached to the network are working properly.
Step 3. Perform connectivity tests such as DNS lookup and Ping. See Connectivity Diagnostic Test On RV016 RV042 RV042G RV082 VPN Routers.
Step 3. Run antivirus programs to detect any spyware/malware or viruses that might slow the speed of your network.
Step 4. Once you determine that all the components in the network are working properly, but the slow connection persists, you have to manually setup the MTU (Maximum Transmission Unit) size on the RV wired router.
The MTU function is to set the parameters for maximum data transmission. This value must be set accordingly to the network topology, in order to take full advantage of the bandwidth available. To Enable MTU manually, first contact your ISP to find out the proper MTU size. The next steps will guide through the implementation of MTU on the RV Wired Routers Series.
*Please mark the question as Answered or rate it so other users can benefit from it"
Johnnatan Rodriguez Miranda.
Cisco Network Support Engineer.
Thank you for your reply. Yes all those things were checked. I could go into great detail about exactly what I did over the last three weeks, but suffice it to say I checked all those items and a bunch more. Checking those are certainly a lot easier than setting up an ethernet tap outside our router (which is where I'm at now).
1. Our ISP has been monitoring our bandwidth for about 2 weeks now at 15 minute intervals. They will continue for another two weeks. Our bandwidth is spot on.
2. We had 4 NIC's (loadbalanced) on the box inside our router. Those had been reduced to a nice simple, single nic solution to ease testing and diagnostics. There are no problems with the connectivity.
3. AntiVirus, Really? This is NOT a virus problem, this is a router problem. There is no problem with the available bandwidth on my network or on the box in question (Linux).
4. MTU? I don't think so. We are running at the default setting. While adjustment of the MTU might improve performance, the default setting should certainly not be a cause of failure of tcp sessions to be properly closed.
I have worked very hard to provide both an inside/outside capture of the problem while it is ocurring. This was no small task. While I thank you for your response, it does not really contain anything insightful as to this particular problem just a list of generic troubleshooting steps.
The RV082 is in its default config wrt firewall config:
1. Default Access Rules: LAN=ALL, WAN1=NONE, WAN2=NONE
2. Default Content filter: No Forbidden Domains, No Blocked Keywords
Nothing in the current logs in default config. I will crank them up...
I will also try disabling DoS checking...
It shouldn't take long to determine if any of this helps. We have many of these improperly terminated connections each day. (In the last 24 hours, the associated server process logged 289 improperly closed connections).
I'll let you know what happens in a couple of hours...
Do you have any of the following config setup?
Firewall -> Access Rules
Firewal -> Content Filter
Also, try disabling
Firewall -> General -> DoS
And try checking the RV082 traffic log, or enabling all the log options of the RV082.
- Access Rules are at the Default setting: LAN=all, WAN1=none, WAN2=none
- Content filter is at the Default setting: No Blocked Domains or Keywords
I will try disabling DoS and cranking up the log detail.
The associated server process reported 189 connection termination problems in the last 24 hrs so I should know the results of these changes in short order...
DoS detection is turned off and I'm logging everything but "Allow Policies". The problem persists. However, I now see a slew of these entries in the router log:
|Feb 20 04:35:46 2013||Connection Refused - Policy violation||TCP 192.168.1.223:1457->184.108.40.206:80 on eth1|
|Feb 20 04:35:46 2013||Connection Refused - Policy violation||TCP 192.168.1.45:9508->220.127.116.11:80 on eth1|
|Feb 20 04:35:56 2013||Connection Refused - Policy violation||TCP 192.168.1.223:1457->18.104.22.168:80 on eth1|
|Feb 20 04:35:59 2013||Connection Refused - Policy violation||TCP 192.168.1.45:21278->22.214.171.124:80 on eth1|
|Feb 20 04:36:39 2013||Connection Refused - Policy violation||TCP 192.168.1.183:65402->126.96.36.199:443 on eth1|
|Feb 20 04:37:00 2013||Connection Refused - Policy violation||TCP 192.168.1.45:46512->188.8.131.52:80 on eth1|
|Feb 20 04:37:00 2013||Connection Refused - Policy violation||TCP 192.168.1.45:63993->184.108.40.206:80 on eth1|
|Feb 20 04:37:03 2013||Connection Refused - Policy violation||TCP 192.168.1.183:65402->220.127.116.11:443 on eth1|
|Feb 20 04:37:03 2013||Connection Refused - Policy violation||TCP 192.168.1.45:21341->18.104.22.168:80 on eth1|
|Feb 20 04:37:06 2013||Connection Refused - Policy violation||TCP 192.168.1.183:65402->22.214.171.124:443 on eth1|
|Feb 20 04:37:08 2013||Connection Refused - Policy violation||TCP 192.168.1.45:63993->126.96.36.199:80 on eth1|
|Feb 20 04:37:08 2013||Connection Refused - Policy violation||TCP 192.168.1.45:46512->188.8.131.52:80 on eth1|
|Feb 20 04:37:10 2013||Connection Refused - Policy violation||TCP 192.168.1.183:65402->184.108.40.206:443 on eth1|
|Feb 20 04:37:10 2013||Connection Refused - Policy violation||TCP 192.168.1.45:21341->220.127.116.11:80 on eth1|
|Feb 20 04:40:20 2013||Connection Refused - Policy violation||TCP 192.168.1.217:45401->18.104.22.168:443 on eth1|
|Feb 20 04:40:22 2013||Connection Refused - Policy violation||TCP 192.168.1.45:16566->22.214.171.124:80 on eth1|
|Feb 20 04:40:25 2013||Connection Refused - Policy violation||TCP 192.168.1.217:37106->126.96.36.199:443 on eth1|
|Feb 20 04:40:26 2013||Connection Refused - Policy violation||TCP 192.168.1.45:16566->188.8.131.52:80 on eth1|
|Feb 20 04:40:28 2013||Connection Refused - Policy violation||TCP 192.168.1.217:37106->184.108.40.206:443 on eth1|
|Feb 20 04:40:31 2013||Connection Refused - Policy violation||TCP 192.168.1.45:16566->220.127.116.11:80 on eth1|
|Feb 20 04:40:33 2013||Connection Refused - Policy violation||TCP 192.168.1.217:37106->18.104.22.168:443 on eth1|
|Feb 20 04:40:36 2013||Connection Refused - Policy violation||TCP 192.168.1.45:21575->22.214.171.124:80 on eth1|
The router is running with just the default access rules. Why do we see connections being refused from the internal LAN (192.168.1.0/24) to the outside (WAN1) if "allow all LAN" is enabled as the first rule? What "policy" could possibly be violated here?
I just enabled Deny Policies Log and I too see those Policy Violations entries. It seems that they are normal, since I don't have packet drops.
In Firewall -> General you have SPI enabled?
In my case, when I have packet loss is when my traffic of packets/s is too high. Maybe you can monitor that rate with a traffic grapher, like PRTG Free edition. Also, a ping -c 100 126.96.36.199 will help.
"Normal" vs "Typical"
It seems to me that there should not be ANY connection refusals for conversations originating in the inside in the default configuration. I have both SPI and Dos now disabled and this has not improved anything.
I would like to hear from CISCO as to why these connections are being dropped and/or why these piolicy violations (?) are being logged!
Also, this is not a traffic volume issue as it seems the various TCP conversations are being terminated inappropriately in EXACTLY the same way. We're not dropping packets randomely here, this is all a very obvious pattern. I have it all captured in 13+ hours of packet captures running simultaneously inside and outside my network. Its all pretty blatant when you isolate a particular conversation both from an internal and an external perspective and then compare the results.
Why is Cisco so quick to suggest: *Please mark the question as Answered or rate it so other users can benefit from it" as they did in the opriginal reply. Why didn't they actually LOOK at the packet traces I provided rather than simply dumping a bunch of boilerplate recommendations and then suggesting I close out this problem? Why hasn't CISCO contributed any further to this thread?
I think I have pretty good positive proof of a possible router firmware problem. I have worked very hard to narrow it down this far. I have provided detailed packet captures and asked some very specific questions. Why is CISCO simply asking me to close out this discussion as "answered" when no real analysis or help has been performed?
I understand your frustation but can not speak for Cisco.
You can wait until someone with a user with a Cisco Avatar responds.
My sugestion will be to reset the settings to default value and reconfigure the router. Another options are:
- test the router in another location with another ISP.
- focus the packet capture to 1 IP and 1 port, for example gmail imap server or a specific small website.
The packet capture has already been focused between two IP's! My mail server and the spam signature server that it uses. I have all traffic between these boxes, both from the inside, and simultaneously from the outside, of my router for over 13 hours. Furthermore, I have matched several conversations accross the router and it appears the router is abnormally terminating many of these conversations without waiting for the full TCP handshake to complete. The initiator sends the FIN/ACK, the responder ACKS the packet, but the router slams the door before the responder can send the other FIN/ACK which is required from the other side. This causes lots of problems with a particular application which simply throws away the whole conversation since the port/conversation "wasn't closed properly"...
I would sure like to hear from someone from CISCO who would actually take the time to consider the issue with all the detail I have provided! I'm getting ready to simply trash several RV082's and replace them with something from a completely different vendor if I can not get any resolution (or education) on this...
Hi Edward, the MTU size is important because if you have a larger MTU size, also means processing of fewer packets for the same amount of data. In some systems, per-packet-processing can be a critical performance limitation. That's why it is advisable to reduce the size from 1492-1500. Also I was wondering, if do you have any VNP configuration?
Hi, just to chime in, you may try to adjust time out periods on the router. May or may not help but can't hurt.
Please mark answered for helpful posts
The TCP timeout is set to 1800 seconds, the UDP timeout is set to 30 seconds. These are the defaults.
When you look at the packet traces I provided you can see that there is less than a second between the last packet (in the conversation) which makes which makes it through the router and the next packet which doesn't.
Look at the inside capture...
- The Outside box sends an ACK/FIN packet to begin closing the conversation (packet 8).
- The Inside Box sends an ACK back to acknowledge the fact (about 40 ms later)
- But when the Inside box sends the ACK/FIN for his side of the conversation (less than a second later) it never gets passed through the router!
- In fact, the router drops all traffic after the inside box sends the ACK to the outside box's ACK/FIN! The inside box tries to resend his ACK/FIN a total of 10 times over the next 2 minutes and none of this traffic makes it through the router!
Since this entire conversation (includeing the dropped packets and the clients retries) ocurr in the space of about two minutes, it appears that the TCP timeout of 1800 seconds (30 minutes) is more than sufficient. Otherwise, what would you suggest I change these value to?
...and, although I don't expect this to be MTU related, I can't seem to find that setting anywhere in the current firmware. I can also try changing that as well, as long as I can find it!
Yes - we do have VPN's configured. We have only a couple tunnels defined with a single persistent tunnel (point-point) set up to another RV082 (same HW and FW revs).
I also believe a specific answer to the question regarding the "Connection Refused - Policy violation" log entries I posted several messages back might be very pertinent. About half of those "Connection Refusals" correspond exactly to the source/dest IP's that we are having trouble with! In fact, they may correspond exactly to the failed conversations I have provided in the packet traces. So, I ask once again, with the default config of "allow all LAN" traffic, exactly what policy is being violated by those log entries? Those connections originate from inside my LAN and are completely legitimate attempts to contact resources on the Internet. Why are they being blocked/refused?
I'm at the end of my rope here. I think I've provided some very detailed observations and posed some very specific questions. I'll try whatever is suggested (even if I think its unlikely), However, If I can't make some headway on this by the end of the day, I've already done enough research to identify the vendor and replacement equipment I will need to purchase to replace all our RV082's (4).
It should be noted that the silence here is deafening!
CISCO either doesn't care about this problem or is simply incapable of processing the detailed information that was provided.
Anyone who is considering purchasing a CISCO SMB product would be wise to review this thread and note the quality and level of support that was provided...
Please take a look at the detailed information that was originally provided (packet caoptures on both sides of the router), consider the work that we did to isolate and identify the problem, note the specific questions that were posed, and then step back and take a look at the type and the content of responses that were provided by the CISCO representatives.