This is a multi-step troubleshooting procedure that I am afraid I can't fully answer with out more details but hopefully I can get you off to a good start. Run some traceroutes to multiple websites and try to see if you can figure out any consistent pattern of where the latency is coming from. for example from command prompt you would type: tracert www.yahoo.com
You will now see latency at each hop. Run them to different places, compare and see if it is having latency at the same hops.
If you can narrow down where the latency is, you can address fixing the proper thing. Could be the router, could be to the isp's node, could be on the backbone completely un-fixable, etc. If you call you ISP with the traceroute info they can help you determine where the problem lies usually.
Hope that helps.
additional info edit:
One other thing to keep in mind... the hop with the problems isn't necessarily the device but narrows it to the devices on either end of the hop or the connection between them. What device is hooked to the modem could be the problem, or the cord between the two. ethernet cables often go bad. try reversing the cable (the end that's plugged in the modem plug into the other device and vice versa). if that is fixing it the cord is going bad, and certain wires within it are functioning better than others. next i'd replace the cord. you can pick up a cord for a couple bucks from walmart or www.newegg.com so that would be the cheapest fix. NIC's go bad almost as often, and are the next cheapest thing to replace- if you are using a router, the NIC should not be the problem though- so depending if you have a router or not you might replace the NIC. If you are using a router, try a direct connect to the computer and reboot the modem and computer, see if that fixes it. If it does either replace your router or contact the router vendor for support if it is offered. If removing the router doesnt fix it (or you aren't using one) and you have replaced the ethernet cord and nic- it's got to be the modem.
here are a couple things you can try with the modem, if they don't work and you are sure its not the cord or device connected to the modem I suggest you talk to the ISP and they will possibly replace the modem unless you own it.
If you have cable internet: Do you have a router in use? If so remove the router from the equation, reboot the modem and computer and leave it this way for about 24 hours or more. Then hook things back up with the router, reboot the modem, router and computer. It could be somehow you got a funky IP address or dns servers assigned and this is a simple way to change that and fix the problem by getting a new IP assigned. This will not work for DSL.
If you have cable internet, but no router, is there another computer in the home you could use for 24 hours? Hook the different computer to the modem and follow the same procedure rebooting things listed above each time you change the setup.
Regardless of what type of ISP you have, check the modem's functionality after it has been off and fully cooled for a few hours. Unplug the power to the modem for a couple hours and try what its like when plugged back in. If this seems to resolve it, the modem could be slightly overheating and functionality is reduced. This can be fixed by placing the modem in a better ventilated area, or near a fan, or in some way keeping it a bit cooler.