Could be multiple reasons would need to see the router outputs may help , can you post the show interfaces from dialer and physical , do you receive any alerts in logs
Are the mac and arp still in tact for those ints when the issue occurs or do they go incomplete/timeout etc
Had a similar issue before it was an ISP problem in the end used an ip sla to far end of the link to keep interesting traffic flowing that stopped our freezes but you should try and determine first why its happening