A LAN is a broadcast domain which means when there are too much broadcast traffic on the subnet
nobody can transmit till the broadcast as been received and process by all parties. also you're using ethernet which uses CSMA/CD
and hence with a chatty server like you have it would always tie up the medium.
The router has nothing to do with this slowness since it never get that much inbound traffic anyway.
Solution, use a switch and put the server in a different subnet, if possible.