It depends on your security policies and requirements.
You can secure access to the terminal server so that only certain staff can access this device. This in turn limits the people who can access your ASA. You can also log who accessed the device and when and what they did when they connected to the ASA.
However by connecting a modem directly to the ASA you have none of these security features available.
For a large organisation I think this would be something you would only use as a temporary measure to fix a short term problem rather than leave in place permanently.