There is a known ARP issue on some platforms (so far I have seen this issue on C3750 & C3560 platforms with 12.2(52)SE code) when using OSPF network type point-to-point, the mac address of the next hop is unresolved on point-to-point ethernet causing a delay resolving the next hop address and installation of the next hop address in CEF forwarding table. Because of this issue Cisco has modified newer codes from 12.2(33) to use combination of unicast (exchanging DB info) and multicast (exchange of hello) instead of pure multicast OSPF adjacency is formed!
You may have the following ACL below inbound on the interface used for ospf adjacency using point-to-point network type and OSPF is working beatifully until you decided to upgrade to a recent IOS code for example on WS-C3750G-24TS and noticed OSPF no longer able to form adjacency and stuck in "EXSTART" and your deny ACL statement is now denying OSPF packet and you check your logs and noticed the logs.
Perhaps by analysing the logs, it does look like MTU issue and it's not. As describe above Cisco has made a change to newer IOS codes (I have only noticed this behavior on switch platforms such as 3750 & 3560s) now using unicast & multicast. This is a know issue. see bug CSCsv89643 (The bug details is being updated to include the codes mentioned in this doc)
ACL
access-list 102 permit ospf any host 224.0.0.5
Logs Output
Oct 4 17:43:24.248 BST: OSPF: Rcv DBD from 150.2.2.2on GigabitEthernet1/0/25 seq 0xE6 opt 0x52 flag 0x7 len 32 mtu 1500 state EXSTART
Oct 4 17:43:24.248 BST: OSPF: First DBD and we are not SLAVE
Oct 4 17:43:38.014 BST: OSPF: Killing nbr 150.2.2.2 on GigabitEthernet1/0/25 due to excessive (25) retransmissions
Oct 4 17:43:38.014 BST: OSPF: 150.2.2.2 address 192.168.1.2 on GigabitEthernet1/0/25 is dead, state DOWN
Oct 4 17:43:38.014 BST: %OSPF-5-ADJCHG: Process 110, Nbr 150.2.2.2 on GigabitEthernet1/0/25 from EXSTART to DOWN, Neighbor Down: Too many retransmissions
Workaround
1, Remove "ip ospf network point-to-point" from the interface, this will convert back to multicast.
2, Convert ACL to allow unicast for OSPF. See this https://www.cisco.com/en/US/tech/tk365/technologies_q_and_a_item09186a008012d8f7.shtml#qa11
Please note that this issue describe in the document is NOT a bug but a feature removed from newer codes because of the ARP issue described above when using point-to-point mulicast.
