I have numerous sites with an ISR1 onsite as the local site router connecting each location to our MPLS cloud and Internet. We started dropping local Internet services at these sites last Fall and were putting in a CheckPoint FW between the PE and our ISR. The ISR was connected to the CheckPoint via 2 connections. To accomodate these additional connections we added a HWIC-4ESW to each site router as we turned up local Internet services. The "inside" interface of the CP was connected to one interface on the HWIC. A DMZ inteface on the CheckPoint was connected to another interface on the HWIC. The first connection (site router to "inside" CP interface) was on the L2 switchport with an access VLAN and SVI in the Default VRF. The other connection to a DMZ interface on the CP was terminated on a L2 switchport with an access VLAN and SVI in a VRF dedicated to guest traffic (wired and wireless behind the site router). Simple config. The CP bridged the gap between the corporate traffic in the default VRF and the Internet. Same for the guest traffic in the Guest VRF and the Internet. No NAT happened on the router; it was all in the CP. Very simple.
Every few weeks this setup would experience issues. I would be unable to pass traffic between the ISR and the CP. Connecting via the Internet to the CP I was able to run tcpdump and see inbound CDP packets on both physical interfaces connected to the HWIC. The CP would have ARP entries for the Cisco and the CP would respond to ARP requests from the Cisco but the Cisco never saw the ARP replies coming in. Ie, the CP had ARP entries for the Cisco but not visa versa. Sometimes this only applied to the Guest interface and not the site router to FW "inside" interface. Bouncing the interfaces wouldn't help. Rebooting the CheckPoint wouldn't help. Removing the entire VRF config, interface memeberships, etc and readding it all didn't help. The only fix was to reboot the ISR1. This happened every 2-4 weeks. It also happened at every single site I had an ISRv1 at. All of those sites with 2821s are running 15.1(4)M2. All with HWIC-4ESW modules to get the required port count. None of our sites with ISR2s had trouble (2921, 3925, or 3945).
I assumed it was just a weird issue between the Cisco and the CP. Perhaps even a L2 autonegotiation issue between the HWIC-4ESW and the CP that would crop up from time to time (I've seen weirder issues before, even on Cisco to Cisco connections). We recently starting cabling around the CheckPoints and removing them from service due to more issues than I care to recount here. I implemented NAT on the ISR and eliminated the CP altogether. Today one of the sites we did this at lost local Internet. We rebooted the provider's cable modem and ultimately called on their support folks. I bounced the port on my ISR1 with them on the phone. They hit the CM several times as well. It started looking like the familiar issue all over again so I bounced the ISR1. When it came up the problem was gone, just like before when I was connected to the CheckPoint.
I don't have any sites with ISR1s that have the local Internet handoff connected to an onboard port. And I don't have any sites with ISR2s and
EHWIC-4ESG modules that are having this issue. Is this a known issue with the code rev, ISR1, or HWIC? I haven't been able to find anything on it yet.
Thanks
