Have you tried the ‘Unified CM Phones With Mismatched Load’ report under Cisco Unified Reporting (https://IPOfCUCM:8443/cucreports)? This is fairly accurate in my experience. Also you could do a device search under RTMT (go to CallManager / Device Search / Phone). Otherwise as you mention there are third party tools like the Uplinx Report Tool for Cisco UC (http://www.uplinx.com/reporttool.htm) which http to every phone for you and compile load information amongst other things. It’s not free unfortunately and the demo version’s phone report is limited to 30 phones.
I’m assuming the phones that fail to upgrade are running version 8.3(3) at least? (minimum required to go to 9.2(1)) as you say they upgrade sucessfully after a few resets. I’ve seen 6961 phones refuse to upgrade at all unless they’re power cycled, haven’t seen this behaviour with 79xx phones though. Are the phones that are failing to upgrade on remote sites? We see this quite a lot, TFTP is not the best method to copy data over long distances as it gets slower as the RTT from the TFTP server increases. You could play with peer firmware sharing – this in theory should allow phones of the same type to get firmware off a local phone which has upgraded successfully.
You're not doing anything wrong, it happens and yes it's frustrating! Thankfully Cisco seems to be moving away from TFTP as a method to distribute firmware.
I did review the cureports but it seems that the mismatch load option is not availalable in 6.1(2). I am very happy to hear that there is a tool in subsequent releases though and I will have to check that out. I did note the interim 8.3(3) requirement in the release notes but that did not apply since we were already at 8.3(4). All the phones are in the same campus and I did not see any indicators that the TFTP server (pub) was stressed. To prove that, I would still get the same result doing as few as 5; really, even had the same doing multiple resets on same phone (even put the image in device config on a few with same result).
Thank you for the Uplinx, cureporter, and peer TFTP notes. Will defintely look at the last item given that we have WAN link constraints on another project.