You should migrate the database before or after you upgrade TMS, it's the same number of steps either way, so it's up to you in what order you want to do it. Regarding your comment about installing TMS on a clean server, take the below things into consideration (taken from the TMS release notes and latest install guide):
- The Complete/Custom choice has been removed to simplify the installation. The installer contains all necessary dialogs in one install path. This means you can no longer change the database location during an upgrade; if required this must be done using TMS Tools before upgrading.
- SQL Server is no longer installed as part of the Cisco TMS install.
- If there is not an SQL database present on the server when installing Cisco TMS, you will be prompted to install Microsoft SQL Server.
You'll need to install TMSPE 1.0 in order to migrate the TMS Agent data into TMSPE, then you can upgrade TMS and TMSPE to their latest releases. Just be sure to check all the release notes and latest install guides for any gotchas as well knowing any changes in behavior for what you're upgrading/installing (such as what's already noted above with installing newer version of TMS).