I don't know about running multiple VoIP routers... but I do have Vonage, and tried to set it up like this:
Cable modem -> Linksys Router -> Vonage Router keeping my computers on the first router. The Vonage router did not like that, and I would not get a dialtone with that setup. I had to ditch my old router and reconfigure my server, port forwarding and all that hoohah to make it work. It's a great router that they provide, but yeah, you've only got two phone sockets. I believe you could plug a splitter into one of those sockets if you needed to.
Otherwise, if you want to have multiple phones in other rooms, check out Uniden's cordless phone systems. They will allow you to have up to 4 handsets on the same system, and I believe each of the handsets will actually act as a 'repeater' to extend the range of the whole system, if say, your base is on the other side of the house, and you have a handset between you and the base station. Plus their phones have all the cool features (caller ID, voicemail indicator, name/phone number storage...). I have a set with two handsets, and it works great. I'm going to get another one or two handsets to expand it. One for the bedroom and one for the garage.