If your router does not allow to configure outbound firewall rules, you supply regular user accounts for your users, not admin accounts, and you configure the computers' network settings to obtain them automatically via DHCP from the router. Regular user accounts cannot change the network settings and cannot install most VPN clients which is another favored way to circumvent restrictions.
"doing it on windows firewall on a computer... don't see any router options for it."
The Windows firewall is on the computer, of course, not on the router. But even if you use the Windows firewall for this, admin users can change it again, same as the network settings. So it stays with that you supply only regular user accounts for your users.