Hi NCC, thank you for using our forum, my name is Luis I am part of the Small business Support community.
When you set a priority in QoS you have to choose which service going to receive high priority or low priority, so if you set high priority for VoIP, and low priority for the rest you also will see slow Internet because all resources of the bandwidth will be focused in VoIP. Now, you could set some bandwidth prioritizes in order to provide high quality in your calls and also some services in Internet.
I hope you find this answer useful
Greetings,
Luis Arias.
Cisco Network Support Engineer.