Did you notice any micro burst on the link to the remote site? That may result in delay of the icmp packets from solarwinds? You can use snmp traffic grapher and monitor the link utilization in secs , for example 5 sec or 10 secs to find any micro bust.
Did you see any output drops on the interfaces? Any qos implemented on the links?
I guess u are using icmp for measuring latency, you need to take note that icmp packets are not high priority packets, so it may take time when the CPU is busy.
How about ur applications? Users facing any latency issue?
I have precisely the same problem, and it is also at only one branch office. We are using a 2811 and a stack of 3560s at the branch. Cisco TAC tells me there is nothing wrong with the configs on either. Our QoS is set up properly. I'm pulling my hair out.