I have a VIP that terminates ssl connections, and
the real servers are http on port 8080. There are three separate apps
involved, example is site/inspector, site/reporter, and site/admin.
I do not have logins for the separate apps, but all three are reachable
and apparently acted as they should, and all was going well.
Today I was informed that the reporter app does not work as it should.
Users can log into the app, and can generate reports. The problem is
that if they choose the export option (the app will allow one to export
the reports via csv file), the export fails. I did a client side sniff just
to see if the app was doing something on other than port 8080, and it
does not appear to. Obviously with the encryption you can't see the payload,
but once again I was just looking for connections on other than 8080,
as that would not have been subject to the url rewrite. I do plan on doing
a server side sniff (behind the VIP) tomorrow.
The symptom when the export fails is that when you hit the ok button,
the "save file" box opens, but is immediately followed by a stop box that
says something like "cannot find the file specified". I also had a user
do a direct connect to one of the reals and do the same thing, and everything
worked fine. The sniffer capture once again did not show anything other than
http on port 8080. But the "save file" box popped up, the user selected the
path and was able to save the report.
This is set up on ACE modules in a 6509 chassis. I can certainly provide
more detail, but was kind of hoping that someone has seen this symptom
and might be able to point me in a direction. Other than that, everything else
with this VIP/these apps seems to be working fine.
Thanks for any advice - chris
These types of situations can be a challenge to troubleshoot. However, there are ways to see the client side of the connection decrypted. Here are some of them:
The best way to go is to SPAN the ACE's tengig port on the chassis so you can get the client side and server side capture in a single capture file. Also, run one of the above browser tools, along with Wireshark, on the client PC. It is also a good idea to get the output of show tech-support from the context your using before and after your test so that you can see if any specific error counters increment during a failed connection.
With this data, you should be able to see if the client is launching a new connection while trying to export the data. Perhaps the button is sending the client to a different port, or maybe there is an HREF within the HTML that the browser is following. The ACE can only rewrite the Location header of a redirect, not an embedded HREF.
Hope this helps,
sorry for not responding sooner. I can't quite
seem to get the certificate imported into wireshark at this point.
I do still need to run the capture to see the server side traffic (behind the
ace). I also appreciate the heads up about http watch.
This situation is not a showstopper at this point. I did verify with the users
that this is the only situation that they have seen a problem with. All other
apps and functions within the apps are working fine.
thanks again, I appreciate it. If I get this figured out I will respond with what is
happening, might be useful for others.
Peter, yes the service is sticky.
Everything else within the three apps appears to be working
fine, I did verify with the users. It's only when they try and
download the csv within the one app that they are having problems
with. I want to run a sniffer capture and see if I can nail this one
myself. If I do get to the point where I'm at a dead end, I will post the
configs. But I will learn more if I dig on this myself first.
I do appreciate the response, and if I get this sorted out, I will post
what I find as it might be useful for others.
Thanks again, chris
I want to run a sniffer capture and see if I can nail this one myself. If I do get to the point where I'm at a dead end, I will post the configs. But I will learn more if I dig on this myself first.
Quite commendable! ;- )
I have taken capture with the span set up to watch the outside and inside (VIP vlan and server vlan). I also have a cleartext capture done while
connecting directly to one of the webservers in the pool. I am not seeing anything in the content that looks out of order. I do see some things related
to the tcp connection like sack being cleared, etc. but nothing really looks much different within the content. In both captures, I can see the GET
of the csv, and also in both captures I can see the 200 OK response. The only difference is that with a direct connection to the server, the csv does
download, and via the VIP I get the stop dialog that says "internet explorer cannot download xyz from sitename. internet explorer was not able to open
the site, it's either unavailable or cannot be found. Please try again later".
So I haven't made much headway with this, although since the proxied capture looks like the direct connect capture, it almost looks like something
with the ssl proxy service. But that's just a guess. Anyway, I have attached both captures as well as a cleaned config of the ace context. The version
Other than the csv download, everything else seems to be working fine. Thanks for any advice - chris
the ace-in-out capture is the span of the ace module
the clear-2 capture is the direct connect
From the capture, I don't see the ACE doing anything wrong that would cause this. I have included a screenshot of the capture where the request for the .csv file is requested. You'll notice that for each packet on the encrypted side, it is exactly 21 bytes larger than it is on the decrypted side. This is for the SSL data. If you start from the packet I have highlighted, which is the GET for the .csv file coming in encrypted to the VIP, you'll notice every packet that is sent by the client or the server is forwarded accordingly by the ACE. The only packet that is not 21 bytes larger on the encrypted side is packet 384. This is because the ACE also had to change the HTTP to HTTPS as configured, which added an extra byte making it 22 bytes larger on the encrypted side.
At this point, I think you're going to need to use HTTPWatch on your client browser both when going through the ACE and when going direct to the server to see exactly what is different from the client's perspective. From the ACE's perspective, things appear fine to me.
One thing you could try is removing the SSL termination from the load balancing and see if it works in clear text through the ACE.
Hope this helps in moving this forward.
thanks for the help.
I did grab httpwatch basic, which does capture the headers, but looks like I need the pro edition to actually see the header content. I'll see what I can
do regarding that.
As far as trying a VIP without the ssl termination, I'll see what I can do because the context this vip is in has multiple production vips (no worry there)
but this particular one is rather important to the organization. I think I might be able to just set up another vip to work on port 8080 just for testing. I'll
look into that.
But I do appreciate the help and it eases my mind a little that you believe the ace is working as expected. Thanks again - chris
thanks again for the help.
A co-worker suggested fiddler to watch the headers. I did install it and watched the headers. I did see in the session
that the result when selecting the csv and hit the submit button, the 200 ok did reach the browser, but the same
"can't find /download the file" happened.
So just for the heck of it, I tried firefox......voila! it worked.
Now I am truly befuddled, but I think I have "plausible denialbitility" now. I don't think that there is a problem
with the setup. Maybe domain policy or something within IE, but I think the ACE/network are exonerated.
just wanted to post this in case anyone was following this. I don't know what the IE deal is, but I think firefox working
proves the ace is not the problem. - chris
did a little more digging, and it looks like this truly is particular to IE.
I found two microsoft kb articles regarding this:
http://support.microsoft.com/kb/323308" (although this one deals with registry hacks on client machines).
So I can probably try header deletion as a test and see if that works.
I hope this might be helpful for others who run into this situation. - chris
Excellent work Chris!
While I'm sure you already know the location, I wanted to let everyone know where they can find out how to remove an HTTP header from a server response using the ACE.
I look forward to seeing whether or not this solves your issue.
well, I wanted to try rewriting the cache-control header to private, but after reading rfc2616, and knowing that
a lot of the apps used in my environment keep us bound to ie6, I decided to just delete the pragma (http1.0)
and cache control (http1.1) headers altogether. If I'm understanding 2616 correctly, the user should not run into
cached content problems courtesy of the expires header. We'll see. I'm not overly worried at this point.
But yes, deleting the headers did the trick and now the csv report does download.
Thanks again for the help, I appreciate it - chris