Jitter is a problem here (as well as the packet loss) That maximum jitter of 297 is huge. You should be aiming for jitter to be a fraction of that.
Jitter is caused when packets are delayed.
Check that QoS is correctly configured on your DSL link. If this is happening like clock-work, it could be that someone/thing is sending/receiving large IP packets, and these are delaying the RTP packets. You need to make sure that your DSL interface is configured to correctly fragment these large IP packets so as not to delay the RTP packets.
The point of a jitter buffer is to smooth out the variation in transmition time/delay of packets. Sure, you can have a 1000ms jitter buffer. But that would mean that the phone has to store 1000ms of voice data before playing it back to the person on the phone. If both ends of the VoIP call have 1000ms jitter buffers, that means it would take over 2 seconds for the person at the other end to respond to a simple "Hello".
If memory serves, you want jitter to be less than 40ms. So your max jitter of just under 300ms is way too high.
Once packets start arriving outside the jitter buffer window, the phone treats the packet as lost, and so either has to guess what the contents might have been, or just give silence. This is when quality starts falling off a cliff.