The most important is which IOS ver. is selected and the required feature is fully tested before the upgrade. The router should be upgradeable remotely. However, you have to practice the procedure to prevent any disconnection. Or you need to verify the modem dial-up access is workable during the upgrade.
The reason I highlight the above technical issue that is because it impact the plan.
If you ensure all technical issue is well-prepared. You can only need to plan the schedule, for your industry, I believe the upgrade cannot be carried in office hour. So carefully planning is a must and may require a on-site staff to help to power-recycle the router if there is a need.
Are the routers located gloablly ? This question is for the timezone and support issue only.
I suggest to have three phases :
1) Phase 1 - pilot upgrade the non-critical sites to veridy the step and IOS is ok. You can determine how many sites to be the pilot sites.
2) Phase 2 - Upgrade the core sites to ensure it is running smoothly after the upgrade.
4) Phase 3 - Upgrade the remaning sites to complete the upgrade project.
The length of each phase that depends on your site location, support staff schedule. For 1000 routers, it is around 250 routers per month. And 10 routers per night, it is quite aggressive. You have to design different group of routers that fall-into different upgrade procedure and plan. Some routers may be able to group to same upgrade schedule for more than 10 routers at a time.
If you do not have local staff or resources to assist the upgrade operation. I recommend to get one vendor which have the coverage of all sites to assist you for any fall-back or troubleshooting issue during the upgrade, or even outsource the upgrade project to them.
Hppe this helps.
