I've been working on this issue at 2 sites, and you might be seeing a very real internet problem called Buffer Bloat.
Short story, is there are only a few firewalls that I know of that can deal with it. One is the Astaro and I have reduced it in my DD-WRT buffers by reducing the Q sizes.
ifconfig wlan0 txqueuelen 16
ifconfig eth0 txqueuelen 50
You have to experiment and change the values on every interface on your DD-WRT router.
This really won't be fixed until they actually fix it in the Linux kernels they use to run a lot of the home routers we use today. The RFC is at
http://queue.acm.org/detail.cfm?id=2209336.
Essentially, if you router is busy, your TCP packets (other traffic) don't get correctly throttled resulting in your real time UDP packets being buffered way to long.
Buffer bloat is well discussed on the internet and regardless of your bandwidth or latency values, you may still have issues.