My DSL sucks ass due to packet loss. Sometimes latency times get pretty crappy as well. Check your packet loss rate. Run a continuous ping to www.yahoo.com (from dos prompt "ping -t www.yahoo.com". If you're packet loss rate is above 3%, all of your TCP connections (browsing, file share) are gonna be slow.
As more people have gone to DSL in my area, the system has become over-subscribed. I can't watch a video on youtube without it lagging unless it is after 11PM. It is really time for me to switch to cable modem.
ETA:
Quoted:
Its taking 50-100ms just to make it out of quests network. Not very good at all. You can have a fast network connection but still have lag. Sounds like this is the case.
-Foxxz
This is pretty crappy. If you're latency times are better at certain times of the day, then your DSL system is probably over-subscribed. A faster router isn't going to help that much in your case since you're getting thru your piece of the puzzle in <1ms. Can't do much better than that.