Hi. I had similar issue with a couple of remote sites with Internet Explorer as a browser. From my perspective it is related with network bandwidth/delay. Since we started to use FF (which gives significantly greater caching/performance), the problem has been disappeared.
I would recommend to try FF and load the page in debug mode to check timings to load gadgets.