I currently have my closet-lab gear on my home network but I want that stuff on a different subnet.  This is easy enough if I get a second NIC for my main rig but I want internet available on that subnet as well so I can do downloads/updates/etc in the VMs I have running on my server.  It is my understanding that I can't do that with just a second NIC.  I also have my main rig set up with the vSphere Client so I can administrate my ESXi host over LAN which I'd like to keep.  So I'm thinking I still need a second nic, I just can't figure out how to get internet on that second network.
My current setup is an sb6121 modem > WRT54gs router  + DD-WRT
mega > main rig / server(wired) and laptop/livingroom desktop/android devices(wireless).  My closet lab gear so far is a dell poweredge 860 running ESXi v5.5, a sonicwall tz 150, a firebox x5 edge, and a cisco 1711.  I also have a spare wrt54 running DD-WRT micro but I may use that as a repeater for my home network.
What would be the best way to accomplish what I'm trying to do?  
 

I'm not really clear on what you are asking.  The only issue with a multihomed host is that you can only have a single default gateway on a device, so regardless of how many NICs or secondary addresses you have, only one will be used for default route.   The new network will need a gateway to leave its subnet.

You're asking about multiple subnets, but you're not saying anything about VLANs, so I'm going to assume that you are talking about running multiple networks within a single layer2 segment.  To do that, you'll have to add secondary network address(es) to your layer3 device to act as the gateway address for the new network.

If you were running multiple VLANs, I'd suggest 802.1q subinterfaces for each network.

From the NAT device out to the Internet, it's easy to add multiple networks to the hide NAT, or if your Internet gateway device can't handle that, just hide NAT your new network(s) behind the single NATed network, for example if your internet gateway only NATs 192.168.1.0/24, then hide your 10.1.1.0/24 behind an available 192.168.1.x/24 address.

What I'm wanting to do is have two separate networks.  Network 1 = home network.  Network 2 = closet lab network.  I want both to have internet connectivity.  I want my main rig to have network connectivity on both, but use my default gateway/home network for internet connectivity.  The reason I want my rig to have connectivity to Network 2 is so I can locally administrate my ESXi host(dell PE 860) and any other hardware on Network 2.  I know I'll need a second NIC for that.
The hardware I have to work with is a Motorola sb6121 cable modem, wrt54gs running DD-WRT mega, wrt54gs running DD-WRT micro, cisco 1711, sonicwall tz 150, and firebox x5 edge.  I have no switches but can probably pick one up if necessary.  My wrt54gs is my current router but it's a v1 so it's not capable of VLAN.  The wrt54g v6 is a RAM-neutered version running DD-WRT micro so I either can't do VLANs with that micro install or I can but it'll be too much for that low amount of RAM to handle which will make it drop connections/services.





 

Well, you don't have to multihome you main host, if you don't want too, you can just make the second network routed via the first gateway or add a static route to a new router host as long as it's on the same network.  If you want to add a second NIC, then your main host will make the routing decision (specifically a "connected network" in the local routing table.  Honestly, 6 of 1, half dozen of the other..excluding cost.

So, the only switch you have are the on board switch ports on the linkSys?  You do this pretty easy if I'm visualizing  the parts correctly.  Depends on how far you want to take it, but there are a couple of ways to do it.  You can use the spare linkSys as the gateway for the new network and hide the new network behind your legacy network address space, then set up a port forwarding for connectivity into the new network; telnet to outside IP:23 and hit your esx host, then use it as a jump box for the new network if you're going to have multiple hosts.  Nice thing there is you don't have to change anything on your legacy network to provide NATing to the internet.

Or, you could use you cisco 1711(providing it has the Ethernet WAN) and give the outside IP on your legacy network, configure conditional NATing with source static overload behind the WAN interface, then you'd be able to just configure a static route on your main PC for the new network with the next hop address of the C1711 WAN IP.  No second NIC required, just using the router to route.  Both networks on the C1711 would be connected when it's acting as the gateway for the new network, so no complicated routing there.

Lol, clear as mud!

Edit: I do something similar on my home network, but for a different reason.  My outside firewall is an ASA 5505 and I can only support 10 hosts on the base license.  So I hung a WLAN behind a single address on my LAN.  This way all my WLAN hosts appear as a single IP to my ASA and I protect my LAN hosts from being accessed via WLAN connected hosts.  It's like a guest WLAN only no guests are welcome.  Inside the WLAN network you can battle with other WLAN hosts, but not  my all important desktop pc or game computer...or my Fort Knox NAS so full of precious booty, so to speak.  The whole thing exists to reduce IP use on the ASA and provide physical diversity from my LAN.  I am paranoid, but I have 3 Cisco 1200 APs to provide contiguous overlapping coverage even out by the pool, pretty much a necessity for us work from home people, so it's secure, but accessible from outside the house.  You break into it you'll see cell phones, Roku, blu ray player, iPad, game councils, etc.

Is there any reason I couldn't use something like one of those layer-3 netgear switches with VLAN capability to do this?
Example...






Modem > switch > router > home network(integrated NIC on rig, everything else wireless)





 


                 


                 


                 


                 


                 


                 


                 


                 --------------\/






               
                 


                 


                 


                 


                 


                 


                 


                 


                   


                 --------------->>>> router > closet lab network(secondary NIC on rig, firewalls, servers, etc)
I tried to explain that to my buddy and he said I can't do that because of my modem but I don't know if he didn't understand me, or I wasn't understanding him.  If I can do it this way I wouldn't have to buy anything but a switch and and NIC, and I may just be able to get those from work.

 
 
 
 

 

Well, it really depends on your service provider.  Most non-commercial/non-business providers only give you a single Internet routeable IP address.  Back in the day, they used to restrict you to a single regestered MAC-Address, so if you changed devices you had to call them.

If your service provider gave you multiple IPs, then maybe....BUT, this is the "responsible Internet Citizen" answer, if you don't need 2 to accomplish your minimum requirements, don't waste the IP.

That kinda boils down what I was trying to explain.  You really only need 1 Internet routeable IP and it's up to the deceive doing your NATing.  Think about this: at work 100's if not 1000's of RFC 1918 networks can be hidden behind a single outside IP.  The NAT device tracks and identifies each outbound stream so it knows to whom to send the specific return traffic.  An Enterprise class firewall or router can do that with ease.  The little SoHo LinkSys stuff isn't really geared for that, so as a work around, you can hide your new network behind your old network.  This is the time I'd be scribbling on the whiteboard and I might go spank out a quick Visio later but I'd have to open the work laptop....gasp....on the weekend....VERBOTEN!

You might be able to use the single Cisco 1711 to replace everything, but I'd have to see what the 1711 supports.  You might be able to create 2 VLAN interfaces and then assign the switchports to one of the 2 VLANs....then NAT both networks behind the outside WAN address.  Be careful if you put that Cisco out dirty on the Internet... ACLs and vty access-class would be a must.

This is what I'm thinking.  Here you can use a second router/firewall device to hide the new network behind your old, Legacy network.  Internet access is easy, and if you don't want to dual home your Main PC, you can just set up port forwarding on the new router to allow whatever access you want (ie telnet, SSH, http, etc.).  This is nice and simple and you don't have to go too deep to slap it in and get it working.  You can use a SoHo LinkSys for this, or a firewall, or even that C1711.  If you hide NAT the new network (10.1.1.0/24 for the sake of the example) behind the legacy network (192.168.1.0/24 in the example), then you don't have to do anything at all to your legacy router, it will just think that the 192.168.1.254/24 is a single host and not a hide NAT for a whole network.  Easy as pie!

If I'm understanding that diagram correctly you have the secondary router plugged in to the primary router.  Don't I need to set up a VLAN for that port the secondary router is plugged in to?  If so, can't do that.  Primary router is incapable of VLAN due to hardware, and secondary router is incapable of VLAN due to software.

You can use a second nic on your desktop to access a separate network.  If you are using windows 7, follow these Instructions.  You will need to configure your default gateway on your internet facing nic and then configure static routes for the other nic.

As for sharing internet connection, right click on you internet facing nic and select properties.  From there goto sharing tab and enable ICS.

Nope, no need to create a VLAN .  Look at the IP addresses, the secondary router's outside is a member of the first routers "inside" network.  The second router will treat the 192.168.1.254/24 interface the same as your Internet router treats its outside interface and hide NAT the new network behind the old network IP.  

Plug the new routers WAN/Outside interface into the old router's inside switch port.  You want the new router to be layer 2 adjacent with your main PC.  That's what the Ethernet "tube" represents, same tube, same Ethernet segment, same LAN.  You will have 2 physically diverse LANs that reside on the router switch ports themselves.

A VLAN is a way to break one physical switch into 2 or more virtual switches or "broadcast domains". You can't cross from one VLAN into another without using a gateway or bridge device.  In this example, you don't have any VLAN, just 2 unique LANs.

So a friend took a look at my sonicwall and it's some sort of all-in-one unit which made it a great device for what I needed.  I set it up behind my modem, patch cable from lan to lan with my wrt54gs, run lab network on another lan port on the sonicwall and then create all the rules and whatnot.  This also requires some config changes in the wrt54gs but I never got that far.  After trying for an hour or so I just couldn't get the sonicwall to talk to the cable modem.





I then realized this could be a great time to familiarize myself with my firebox x5 edge.  Pretty much using it in the same way I would have used the sonicwall except now I have net connectivity, the ability to run an optional network from the OPT port, etc.  Home network is back online behind the wrt54gs which is behind the firebox which is behind the modem.  I'm currently administrating VMs on my server from my main rig without using that second nic.  There is a page in the firebox config specifically for setting rules between my trusted network(ports 0-6) and the optional network(port OPT) which I'll need to play around with so the only connectivity allowed is my rig to the server.





Here is the setup...







 

Cool.  Ya, good ecposure to the concept of DMZ'ing.   Treat you 10/8 as a DMZ network and control access to and from your inside network with explicit permits.

Good job!  If your buddy did it for you, you should study it then write erase, reload and rebuild it!

Train like you fight!

Did this one all on my own.



Right now I'm going over all the rules between trusted and optional network on the firebox.  Gonna try to lock them out from each other but still allow my rig to connect to that network.  If I can't do that it'll be due to limitations of the firebox and I'll just use my rig's second nic to do it.  No biggie.



 

I have two network at home with a ASA. One is my home network. the other is my test network. The ASA handles the rules between the networks and NOT for both outbound. I only allow some very basic traffic for management and DNS traffic to pass between them.