Hi everyone,
With Gergely, we've been communicating directly and I got also looped into the TAC case that was raised for this issue, leading the investigation. Following Gergely's permission, I'd like to share the explanation we arrived to.
The root cause of the problem was the fact that the input traffic rules on the Checkpoint firewall were misconfigured - they did not allow unicast OSPF LSU packets (retransmissions) to be accepted. Hence, when either of the Catalyst switches started sending unicast LSU retransmissions to the firewall, it would silently drop them, causing the switch to give up after 25 retransmissions and drop the adjacency.
However, the reason why the Catalyst switches engaged into the unicast retransmissions was much more involved. Analyzing the packet captures performed on the Catalyst switch supervisors (using the "monitor capture ... control-plane both"), we could see that the firewall successfully acknowledged all LSAs flooded from the switches as multicasts, and the corresponding LSAck packets from the firewall were all received by both switches properly. There was no apparent reason for the switches to start the unicast retransmissions because all LSAs flooded from the switches were acknowledged by the firewall and no LSAck packets were lost. The majority of our investigation focused on explaining these apparently unjustified retransmissions initiated at random moments by either of the Catalyst switches.
Eventually, the behavior of the switches was identified as expected. In 2011, tracked in CSCtu25818 for OSPFv2 and CSCtx15575 for OSPFv3, Cisco implemented a scalability enhancement to the IOS OSPF implementation to prevent retransmissions caused by late processing of LSAck packets. The gist of this enhancement is to have the IOS OSPF router process put the incoming unprocessed non-hello OSPF packets into two software queues: LSAcks go into one queue (called high priority queue) while DBDs, LSRs and LSUs go into the other queue (called low priority queue). Whenever the IOS OSPF process dequeues packets, it always first dequeues packets from the high priority queue that exclusively holds LSAcks. Only if the high priority queue is empty, the IOS OSPF process dequeues the low priority queue. Essentially, for the non-hello packets, the IOS OSPF router process always processes incoming LSAcks first, and only processes other incoming packets if there are no more LSAcks to process. This is not just a proprietary quirk: RFC 4222 specifically endorses prioritizing the processing of incoming Hello and LSAck packets over the rest of OSPF packets.
This behavior can lead to an interesting scenario which was indeed occurring in Gergely's customer's network:
- The switch A (DR or BDR) bursts out a large amount of LSUs to 224.0.0.5
- Both the switch B (BDR or DR) and the firewall receive these LSUs.
- The firewall promptly sends back a number of LSAcks to 224.0.0.6, acknowledging the LSAs received in the LSUs.
- Switch B may have processed a few of the received LSUs in the meantime, but after receiving the burst of LSAcks from the firewall, it will process these first as they are placed into the high priority queue. Only after there are no more LSAcks to process, it will continue processing the remaining LSUs from the low priority queue.
- As a result, some of the LSAs received and acknowledged by the firewall are not yet installed in the LSDB of switch B and put on the retransmit queues for OSPF neighbors when the LSAcks from the firewall are processed; these LSAs are still waiting in the low priority queue on switch B.
- Consequently, when switch B processes LSAcks sent by the firewall, LSAs that already are in the LSDB on switch B will be removed from the retransmit list for the firewall if they are on the list; for the rest of the acknowledged LSAs that are not in the switch B's LSDB, switch B does nothing.
- After switch B processes all received LSAcks, it returns to the low priority queue and processes the remaining received LSUs. The LSAs in these LSUs will be installed into the LSDB and put on the retransmit list for the firewall as usual (but not flooded to the firewall again because switch A as the DR/BDR did the flooding already). As a result of putting these LSAs on the retransmit list, switch B will wait for acknowledgements from the firewall – but the firewall already sent them and so has no reason to send them again; it does not even know that switch B waits for the acknowledgements.
- Consequently, switch B will eventually start the unicast retransmissions toward the firewall – and since the firewall drops the unicast OSPF retransmissions, switch B will eventually give up and drop the adjacency.
This completely explains the reason why either of the Catalyst switches - based on event timing - started the unicast retransmissions despite the firewall acknowledging all received LSAs. Correcting the input rules on the firewall to permit incoming unicast OSPF LSU packets resolved the issue, and we know that the retransmissions from the Catalyst switches are justified and not a result of a misbehavior.
As a side note, as we had to deal with large amounts of debug outputs, the following configuration was very useful on the Catalyst switches as it
1) prepends every logging message with a monotonously increasing sequence number allowing to identify any lost or rate-limited logging message
2) enables persistent logging of system messages into files in the bootflash:
3) increases the size of the internal memory buffer to hold logging messages before they are sent to their individual logging destinations to prevent losing logging messages due to overfilling this internal buffer
4) increases the rate limit for the logging messages to prevent losing logging messages due to rate limiting
configure terminal
service sequence-numbers
logging persistent filesize 1048576 immediate url bootflash:/logs
logging queue-limit 100000
logging rate-limit 7000
end
In fact, at the beginning of our investigation, we were losing debugging messages in the logs and that has misled me into believing that the Catalyst switches were incompletely processing received LSAcks. In reality, the received LSAcks were always processed completely but the debug messages were rate-limited which made the impression that the processing of certain LSAcks was only partial. That was a red herring.
Hopefully this will be of interest and help to anyone who might be facing similar issues or spurious apparently inexplicable retransmissions in IOS/IOS XE OSPF.
Best regards,
Peter
