Certain external websites failing for clients going through our WSAs, but not actually blocked

[Stafford Rau]

We have a pair of Ironport S370s for web content filtering and user activity monitoring. Client sessions are transparently redirected through them via WCCP from our core routers, and we're using Active Directory transparent authentication.

We have experienced several occasions where an external website will fail to load, or will not load completely when the client requesting the site is being redirected through the WSAs. When I exempt that client from going through the proxies (by way of the ACL controlling WCCP redirection), the sites load without trouble.

Nothing in the WSA logs show up as blocked. No config changes on the WSAs have shown to correct this issue - not even adding a custom URL category, adding these sites to the proxy bypass list, etc.

Typically when this happens, I can simply put in an ACL entry exempting traffic going to that website from WCCP redirection. However, the recent trouble spot has been a site that is using the Akamai CDN, and the component that seem to be the main culprit refers to a different service that appears to be hosted by a different CDN. The ACL has to use ip addresses, and as you know addresses on CDNs can change all the time.

Also note that I have an open TAC case on this specific issue, which I opened seven weeks ago. It's still not resolved, nor does it seem like any progress is being made, even after going up one escalation level. My experience to date with Ironport TAC support is that after an initial bit of activity, I simply stop hearing back from the TAC engineer when it turns out to be a non-trivial issue.

Does this sort of issue ring any bells for anyone out there? The particular site causing problems this time is a GIS software vendor, www.esri.com. The component that seems to be the hangup is something from fast.fonts.com.

If any of you Cisco folks would like to take a look at the case, it's 633663619.

[Ken Stieers · Accepted solution]

So first off, instead of using the ACL, you may want to look at Web Security Manager/Bypass Settings. It acts much the same way as the WCCP ACL, but you can use dns names.

Its not as pretty because you can't label stuff like you can in an ACL, but it does the job....

What version of code are you running?

Is this stuff HTTPS? or http?

[Stafford Rau]

Thanks for the reply, Ken.

Working with the TAC engineers previously, we had tried adding items on the individual S370s' proxy bypass list, which didn't seem to work.

However, for whatever reason, I just now added .esri.com and .fonts.com to the bypass settings on our M720 manager/console, and when I published those changes to S370s, that site started working.

Coincidentally, I had a trouble ticket about people not being able to load cnn.com. When I took a look, the browser sessions were hanging on "waiting for fast.fonts.net". So I added .fonts.net to the bypass settings, and that started to work as well.

I'd like to suggest that the Ironport developers might want to have a chat with the fonts.com/fonts.net guys and figure out what's not playing nicely.

And for your questions, we're running version 8.5.1-019 on the WSAs, and while the parent sites involved  are mostly http, I think the fonts.{com|net} components are SSL.

[Ken Stieers]

OK and do you have "Enable Safe Search" and "Enable Site Content rating" turned on?

There's a bug in there that hangs up some SSL sites. CSCur52985

We're just not decrypting those sites until this bug is fixed.  Bypass means they don't get AV scanned, etc...

[Tao Yang]

By design, WSA bypass settings DOES NOT allow partial hostname such as .example.com, but example.com would match www.example.com if T1 interface is monitoring DNS queries.

 IP address or CIDR address such as 10.1.1.0/24 is always preferred.

[spm]

I'm having this exact same issue with fast.fonts.com.  It's been driving me nuts!  It seems like it has something to do with a cascading style sheet the websites try to pull from fast.fonts.com.

[David Byers]

I was having the same issue once I upgraded to 8.5.3.  I started having users calling unable to access www.allscripts.com.  It would hang and never load.  I was not able to see the fast.fonts.com URL trying to load on IE but when I tried the same site on Mozilla I was able to see where the issue was.  I added font.com to the bypass but no luck.  I added fast.fonts.com to the bypass and it worked.  

Thanks for the great Forum.  Been fighting this for a week.

David

[Clint W]

I too had this issue immediately after upgrading to 8.5.1-021 for Web.  Opening a TAC they explained it has to do with caching bug.

NOTE:  I prefer to set a webcache exclusion instead of bypass as you lose statistics on bypass.

THE FIX:

wsaxxxxxx> webcache

Choose the operation you want to perform:
- EVICT - Remove URL from the cache
- DESCRIBE - Describe URL cache status
- IGNORE - Configure domains and URLs never to be cached
[]> ignore

Choose the operation you want to perform:
- DOMAINS - Manage domains
- URLS - Manage urls
[]> domains

Manage domain entries:

Choose the operation you want to perform:
- DELETE - Delete entries
- ADD - Add new entries
- LIST - List entries
[]> add

Enter new domain values; one on each line; an empty line to finish
[]> fast.fonts.com

Enter new domain values; one on each line; an empty line to finish
[]>

Manage domain entries:

Choose the operation you want to perform:
- DELETE - Delete entries
- ADD - Add new entries
- LIST - List entries
[]>

Choose the operation you want to perform:
- DOMAINS - Manage domains
- URLS - Manage urls
[]>

Choose the operation you want to perform:
- EVICT - Remove URL from the cache
- DESCRIBE - Describe URL cache status
- IGNORE - Configure domains and URLs never to be cached
[]>

wsaXXXXXXXX> commit

Please enter some comments describing your changes:
[]> webcache ignore domain add fast.fonts.com

[Ken Stieers]

Nice!  Thanks for this!

Ken

[spm]

That did the trick.  Thanks!

[Brian Vaughn]

Thank you for the update I got the same response from TAC this morning as well and can confirm the workaround allowed the hanging sites to load. Also the bug ID is CSCus56296 if you want to follow it to see when a fix is released.