Heads Up :
The post you are writing will appear in a public forum. Please ensure all content is appropriate for public consumption. Review the employee guidelines for the community here.
Had the same problem. Just needed to use the ip tacacs source-interface command and source the packets from my loopback so they weren't coming from the serial.