Right now I have a pair of Atoms, each with two NICs, one runs pfsense and the other runs untangle bridged behind it.

I am considering picking up a quad-core Bay Trail MB and a four-port NIC (don't ask, I don't want to virtualize NICs) and running a single esxi system with pfsense and UT in VMs.

I like the idea of thinning out the hardware closet a little, and think the overall power consumption will be lower (although not by a whole lot) while the overall performance will be better.  I also think the system has enough ooomph to possibly accept a third VM at some point, maybe a home automation system running linux or something similarly lightweight.  So there is the added benefit of having a VM server available for other use, possibly.  In addition, the final system will have a 5th NIC, and would allow me to run a completely separate guest network, DMZ, whatever.  Point is there would be an extra interface for play time later on, vs what I have now.

Right now if a system fails, I can network around it and keep the network up.  If the entire VM machine goes down, I lose it all.  Its a home network so it would be a minor inconvenience at worst, but still.  

I had run an esxi system a year or so ago but only ever used one VM and never played with it much.  Biggest wonder is if I can easily grab a copy of the VM's file and toss it on my NAS.  So in the event of a hard disk failure on the VM server, I can replace the disk, move the VM files back, and be back up in short order.  Not even sure this is possible, I am hypothesizing only, I have little experience here.

Comments welcome.

ESXi is way overkill for a one-host system; VMWare Workstation seems a much better fit for what you are trying to accomplish.
Unless you have unusually high availability requirements for a home network, I'd just mirror the drives in the host system to cover the most likely failure scenario and go with it.  If something else goes seriously wrong with the host system, well, fix it.

If for some reason you do have a real need for something approaching 24x7 availability and already own ESXi, then build a two-way cluster.  That's what it for.  You won't save any floor space over the current setup, but you'll add a lot of capacity and improve availability.

Thanks for the reply.

The esxi software I played with last year booted off a USB disk, and then was managed by an app on my desktop PC.  And it was free.  Is that particular software no longer available for free?

Dunno, I'm not familiar with what VMWare offers in the way of trials, etc.  It's possible they released one of the older versions for free/eval use; I know there's a version of VMWare Server out there like that.

As I understand it (and I'm not the virtualization expert... just a Linux guy who happens to use VMWare both at work and home) ESXi is VMWare's enterprise virtualization software, part of the vSphere suite, which is neither free nor particularly simple to administer.  Looking around on the VMWare site now, though, it looks like they do in fact have a version of the ESXi hypervisor available free... it must just be the bare-bones virtualization engine without all the added managment tools or something.   Looking further...

There is still a free ESXi. I use it in parts of my data center.

Don't quit half way.

I think you're confusing ESXi with Vsphere.  ESXi is bare-metal, for servers, and is entirely appropriate for even one-host setups.  Workstation is for desktop use, and runs on top of a host operating system.  OP should go for ESXi for what he's doing.  (Vsphere, VMWare's management suite, is expensive, needs a lot of infrastructure to run, and is NOT well-suited to single-host setups.)

You can get a free trial of VMWare products, they generally have all features activated, and expire in a limited amount of time.  You can also get a free license that will not expire, and which still has plenty of features for what you want, OP.

As for whether you should virtualize your firewall, go for it.  Like you said, it's a home system.  And if you have solid hardware, ESXi is awfully stable.  You can boot ESXi off of a regular disk, or a USB flash drive.

You are correct; I was confusing ESXi with vSphere.  I appreciate the lesson (seriously, no sarcasm whatsoever).  VMWare's product line has always been a little difficult for me to grasp.

It's not just you.
Vsphere.  Vcenter.  VCloud.  Vswitch.  VMotion.  Vthis, Vthat, their naming is horrible, and it doesn't help that their official descriptions often give you no clue as to what the particular technology actually does.

"Introducing new VWidget!  Cut costs, and improve ROI by leveraging widget synergy to enhance shared resources!"  

Kind of a roundabout way of doing things but it's a quick and dirty method.

if you dont want to go through lengths to setup failover for your firewalls you could just setup too independent firewalls, then if an esxi host crashes and you lose your primary firewall just change the gateway (on your clients) to the other firewall that's running on the other host.

I've always thought the idea of using xenclient with a synchronizer to manage two light weight hosts running virtual firewalls would be neat.

OK, this is getting to be a little more than I expected, I may have bitten off more than I can chew.

My back-in-the-day ESXi system was an Intel microATX mb, intel chipset (obviously), Intel NIC, and E8500 CPU.  Based on all that I have been reading, it seems that I literally happened to fall into a system that fully supported vt-x and vt-d.  ESXi never gave any errors during install and ran like a top during the time I used it.  

I am only just learning what those terms mean.

So I've been looking at CPUs that support both of those.  And in the desktop (no mobile) category, I can only find some i5s, some i7s, and all Xeons.  But the cheapest/slowest i5 is still $150 on e-gay, which is more than I spent on my entire present pfsense system.  Not to mention it is vastly overkill for two VMs who simply pass packets.

I also read that hypervisors that are run on CPU/chipsets that do NOT support vt-x and/or vt-d can do it in software, emulating it, albeit slower.  Sometimes.

Now the questions:

- if I want to virtualize a firewall, and provide access to a four-port NIC to the virtual machines, do I need vt-d, or can it be emulated?
- same question for vt-x
- will ESXi sufficiently emulate the two above conditions, and/or one or the other, assuming the answer to the first two questions, or either, was yes

Essentially I have a Celeron G550T and a Pentium G3220 sitting around waiting for systems to be built around them.  Neither support vt-x or vt-d according to Intel.  However both are more than adequate, performance-wise, for what I want to do.  And both draw infinitely less power and an i5/7/xeon, unless I want a "-T" model at added expense.

Assuming one of the above two CPUs works, then I have to find a CHIPSET that supports vt-x and vt-d, as apparently not all chipsets do.  

Then I have to load special drivers for ESXi, as nearly all mini-ITX 1155/1150 boards I am seeing have either Realtek 8111 or Intel 214/217 network controllers, or Boradcomm or Atheros, non of which are supported out-of-box by the free version of ESXi.

When I read about "white box" virtual machine hosts, am I to presume folks are using hardware that fully supports virtualization?  Or will any modern, fast enough CPU/MB combo do it, albeit with emulation?

I am learning a lot in reading, which is fun, however I am getting frustrated trying to figure out exactly how to build a super low power, COMPATIBLE system for ESXi.  Maybe I should just keep my two Atoms and move on.

You only need those technologies if you want to pass an entire PCI-E device on to ESXi, IIRC.  Even if you don't have them, you can create 4 vSwitches, tie one to each of your four physical ports, and then connect your VMs as needed to one or more of the vSwitches.  Totally doable, totally legit.

Don't get me wrong, simply trunking VLANs over a single port (or LAG) to segregate data is even more cool, and offers even more flexibility (as well as potentially reduced port and switch costs), but you may not have a VLAN-aware switch.

Seriously, look into XenClient with a Synchronizer.  Get 2 generic v-pro laptops, install the client hypervisor.  Build the firewall images on the syncronizer push them to the clients manage the images from the "data center" and have two virtual firewalls on physical appliances.

My only hang up with this, besides the fact that I haven't worked with VLANs in over 10 years, is the wonder about network security.  I figured four physical interfaces would be more secure than virtual ones, figuring that a virtual NIC may be easier to compromise.  Or am I being paranoid?

It's like everything else in your network - if you're doing really dumb stuff, like putting management interfaces on public networks and using default passwords, etc., sure, it's a down side.  But if you're taking reasonable security precautions, then you're probably being overly paranoid about not using them.  

Still, like I said, attaching a physical switch to each NIC, and each NIC to it's own vSwitch is legit, and will work fine if it makes you feel better.