Large Ironport-PHdr header causing 5.6.0 errors

[Rossco4434]

We are getting occasional errors on the cloud hosted ESA (v13) where the Ironport stamps the email with extremely long Ironport-PHdr headers. This does not happen on every email, i'd say 1/100. We have verified this is the issue.

We see the error when when we increase logging on the ESA, the error occurs between DATA and the error is generated when the final dot is sent.

CSCuw09558 says this has been fixed. I think it's the same issue.

Anybody else with the similar issue?

[msepic]

We are probably seeing the same problem. Our Cisco ESA 13.5.1-277 receives an e-mail with long list of recipients in To:/Cc: headers, and adds an really long IronPort-PHdr header. Next-hop Postfix based device successfully receives e-mail from ESA, but than Exchange 2016 rejects it with 5.6.0 Invalid message content. We are still troubleshooting it, so I'll update this thread when I can confirm or disprove this.

[Rossco4434]

Exchange will fail it because it doesn't meet the RFC. We must be in upside land when Exchange is dealing responsibly with RFC and the Ironport's and other services are not.

[msepic]

Actually, I don't think that single message header size limits are defined in RFC? There ought to exist some limit, and Microsoft decided that for Exchange Server 2016 that limit will be 32KB per single header. We are seeing Ironport-PHdr headers that are larger than 32KB, and because we are using Outbreak filters that can add them again, it seems like there is no point stripping them with Content filters. Cisco engineering teams are aware of this problem with unusually large Ironport-PHdr header, and we were informed by Cisco Support that root cause is known, and that the problem will be solved soon (whatever that means for them).

Exchange Server supports changing single message header size limits, but only by direct editing of transport configuration files. If you need to do so, please contact Microsoft Support for additional instructions.

[MartijnVeldman]

I have the exact same problem. And Cisco Tac is pointing to Microsoft. And Microsoft is allready searching for 2 days now.

[mannychandra]

We had to create a content filter to  strip the IronPort-PHdr and also a new incoming mail flow policy to disable the outbreak filter for select senders who are having the issue

The PHDR header is added by outbreak filter whenever it scans a message, this is part of the normal ESA processing, nevertheless, it is well known that the length of the header may cause delivery problems. By creating a content filter, we deal with the header; However, bear in mind that content filters are evaluated before Outbreak filters, if the Outbreak filter is enabled on the outgoing mail policies, IronPort-PHdr cannot be removed, hence only option would be to disable the outbreak filter in the policy, and leave the content filter to remove the IronPort-PHdr header. 

The above is documented in the bug below;
This is being tracked under: https://bst.cloudapps.cisco.com/bugsearch/bug/CSCuw09558/?rfs=iqvred

[msepic]

Thumbs up! That's a pretty good solution if you only have problems with senders from specific domains and do not have any complicated policy setup. Same/similar bug is also tracked here: 

https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvx34718

I can confirm that workaround suggested by Microsoft is working.

[taagaard · Accepted solution]

Did you find a solution for this ? I have a similar problem with 13.5.1b277 - this one apparently adds the Ironport-Data header.

[ac513]

Our issue disappeared when our appliances were updated to 13.7.0-093. TAC stated it was a bug in the CASE engine, but ours was up-to-date all along, so if that' true or not, no idea.

We just got upgraded again to 14.0.0-692 to fix a different bug (malformed URLs for Cisco Secure Proxy), so here's hoping the large Ironport header bug doesn't come back.

[taagaard]

Well - just to keep you guys updated on this.

My issues was a very long Ironport-Data header, that was set outbound by ESA and then rejected by Office365 tenant.

Opened a TAC case, and after 3 weeks of going in circles with support - I have given up !

Bottom line is :

The Ironport-Data header is being appended by the CASE engine. Only on emails with a "high" number of recipients.

Cisco recommends just to remove the header with an outbound filter - but have failed to explain why it is actually there, and what consequences it might have to remove it.

Good luck if you decide to open a support case on this

[ac513]

@taagaard  After experiencing the same bug, it's been resolved on the last two versions of AsyncOS (spanning both13 and 14, see my previous comment) that our appliances have been on, regardless of the CASE engine versions or definition updates. (CASE is still the issue, but something about it specific to some versions of AsyncOS seems to be at fault)

What version is running on your ESA? I'd ask for an out-of-band upgrade to be scheduled if you're a hosted customer, as they will schedule this.  Don't let TAC off that easy if they're the ones wanting to be lazy.

[taagaard]

That's funny - I was asking very precisely if this was fixed in newer software builds, and the support guy said no.

I am running on-prem appliances version 12.5.1 build 277.

Seems like the TAC "engineers" are paid by the hour - not by the number of cases they actually solve

[ac513]

We're still seeing this same symptom with various vendors' emails, and it has put us between a rock and a hard place. We use outbreak filters, so we can't strip the header. We're an Exchange Online customer, so we can't modify transport configs to allow these large headers.  Affected emails don't have any super obvious problem like lengthy To: fields to trigger the unfolded header bug, so we can't tell vendors what they can change, if anything, to alleviate or at least not exacerbate this bug.

I opened a TAC case to see if there were any other undocumented workarounds, to attach our case to the bug, and also if they have an ETA they could share (if possible, I understand they may not be at liberty to promise/share anything). That whole ordeal was more than useless. First response just parroted back to me what I already said & acknowledged in the ticket description, ignoring any of my other questions.  When I responded back saying as much, basically all they said is that I can follow the bug's notifications. (duh, I'd hope most people know this)  Never so much as a "no there's no other workarounds", or "no, we don't have an ETA", much less any actual answers.

Silly us for wanting our email to work, I guess?  It's really hard to keep a completely professional attitude when such a high profile bug is out there, and support is mostly noncommunicative.

[MartijnVeldman]

As far as I know there are 2 workarounds.

1 Create an incoming mail policy for the recipient or sender, and disable Outbreak filters for this Policy, adn strip the Ironportphd header with a filter.

2 Have the sender use the BCC instead of To: ... 

I am currently running AsyncOS 14.0.0-692, and I thought/hoped It is wasresolved in this version, as this version is not mentioned in the Bug description..