Keep coming back to the vt-d thing.  Want to run pfsense and untangle on VMs, but some posts I read elsewhere say virtualizing the NICs is OK, some say I can't live without vt-d.  One guy recommended a Xeon.  An 80w Xeon.  For a home firewall.  Well, right now a pair of Atom D2500s are running pfsense and untangle, I just want to consolidate machines and save some power, heat, and space.  I'd like to run a very small linux VM also.  I don't think a $250 Xeon is necessary for a home firewall.  If it is, I'll stick with the Atoms!

I have at my disposal presently a Celeron G550T (1155) and Pentium G3220 (1150).  Both are dual core, the Celeron runs cooler and with less power.  

I am also looking at Kabini, the Athlon 5350 specifically, the quad-core 25W AM1 cpu in a cheap Asrock board (that supports AMD's version of vt-d).

I have a quad-gig Intel NIC that I will be using.  

So the question is, how badly will performance suffer if my NICs are "virtualized" and I'm not using direct i/o pass through?  The Kabini looks interesting, but it basically ranks with the J1900 Atom (which I would have already bought but the only mITX boards with the J1900 and a full-length PCIe slot are $200+ Supermicros).  Just wondering what kind of performance I might be looking at, although it does offer direct i/o pass through.  I would run 3 VMs and give each VM two CPUs.  Oversubscribing CPU cores is said to be semi-problematic for pfsense -- if I run the G550T or G3220 I'll do one core per VM.  So that might be another bump in the road.

And then there is the option of going to a low-end i3 for two real and two fake cores.  

Too much for me to digest without help.

I've never used that software before but I don't think your performance will suffer with regular old virtual NICs. Best way is to just try it but for a home firewall I don't see this being much of an issue. If anything the performance issues would come from processing traffic and any IDS it might run than anything NIC related. I just assume a home firewall won't be seeing much inbound traffic other than maybe web and torrent unless you're hosting something.

I am running several ESXi clusters at work. Virtual NIC performance is not an issue on any of the machines.

I'm not 100% certain it will even let you run ESXi without VT-d.

ESXi does not require VT-d.  You only need to enable the virtualization technology setting.



I'm honestly not even completely sure what VT-d is or what it does when used with ESXi.  Never enabled or needed it before and I've built well over a hundred VMware hosts for clients.



 

Is this just for your home network?  If you're not pushing a large
corporation's worth of data then pfSense, if acting only as a
router/gateway, only needs a single core and maybe 512MB RAM.  I'm
currently running 2 cores / 512MB RAM and I have it running Snort,
logging, and a freeBSD Teamspeak3 server.  I've yet to see my CPU usage
go over 60%.  It doesn't really need much.





I've no experience
with Untangle so I can't help you there.  No idea how it interfaces with
pfSense or what resources it consumes, nor what would be involved with it being paired with pfSense on an ESXi host as far as the virtual networking goes.  I'm sure there are plenty of guides out there on doing this.





If you plan to run those plus a linux VM, then you're probably better off going with the quad core.  VMware claims the ability to over-allocate CPU, RAM, and even disk space(if thin provisioned) but in practice it has caused problems, at least in my experience, which is why I don't do it.





Virtual
networking on ESXi is what makes running pfSense on it so awesome and
easy.  For over a year I ran pfSense in a VM on a PowerEdge 2950
alongside 3-4 other VMs.  I gave it 1 core, 512MB RAM, and 120GB vHDD.  
The host only had 2 physical NICs.  I assigned them as WAN/vswitch0 and
LAN1/vswitch1.  All my other VMs were placed on vswitch1.  Later on I
ended up adding a dual GB intel NIC to it which gave me LAN2/vswitch2
and LAN3/vswitch3.  The other VMs were for game servers so I was pushing a decent amount of traffic back and forth.  My only bottleneck was disk i/o.



 

1. I/O device assignment. This feature allows an administrator to assign I/O devices to VMs in any desired configuration.

2. DMA remapping. Supports address translations for device DMA data transfers.

3. Interrupt remapping. Provides VM routing and isolation of device interrupts.

4. Reliability features. Reports and records system software DMA and interrupt erros that may otherwise corrupt memory of impact VM isolation.

I'm running over 100 VM hosts and 5000 VMs at work on AMD procs so I'm not all too familiar with VT-d.  Enable the virtualization setting in the BIOS and you should be good to go.

I'll have to read up on it as I have 96 Cisco UCS blades sitting in boxes waiting to be setup over the next few months.  

As far as over-subscribing vCPUs.  It will all depend on how much use they see.  You could run in to high CPU Ready times which could cause performance issues.  I'd recommend only assigning 1 vCPU per VM to start with.

Unless you are running a ton of traffic at home I wouldn't  expect you'd run in to any problems.

You're actually the first person I've met who used AMD procs in their hosts.

I knew they existed, but I've never seen one either.

First time I've worked anywhere that used them as well and it wasn't my choice.  One of the many great things about UCS is that they only come with Intel processors.

I have some of their 1U rack mount units at the office and really like them.

 
I have ESXI 5.0 and 5.1 running on several HP DL585s with AMD Opteron CPUs. They work fine except when devs write code with Intel CPUs in mind, resulting in apps that run poorly or not at all. After learning about this the hard way, I now get servers only with Intel CPUs.

Same.  I've never personally heard of AMD hosts in production.

As to the OP, you don't need VT-d for your home setup.  You'll be fine without it for what you want to do.

So my first AMD Opteron ESXi host today.  First time for everything.

I'm curious about performance and stability.  Do you have any roughly equivalent Intel hosts that you can compare it to?

I'm told the server has been very reliable, and is actually being repurposed as a hyper-v host which is why it was in my hands.  Hardware specs are as follows:

2x Opteron 4234 - 6 cores @ 3.099Ghz

4x Samsung 840 Series 128GB SSD - RAID5

2x Seagate Barracuda 7200.12 500GB 7200 RPM - RAID1

32GB - 4x 8GB ECC Registered DIMMS

ASUS KCMA-D8

A quick glance at my distributor inventory if I were to sell the above parts I'd get a rough price of $2875.69

If I kept everything the same but swapped for intel chips and board of equivalent performance (lets say, E5-2430v2 and DBS2400SC2) the price goes to $3549.80

Personally, working with this host over the last couple of months, I've noticed no significant problems in testing or in production and the performance is what I would expect of a host of this spec.  I've run several benchmarks on it while calculating IOPS and CPU load and stuff and I've been pretty impressed, especially now that I have a rough idea of what it cost in front of me.  Would I buy 1000 of them and fill a data center?  I don't think I would, because everything I've ever seen in benchmarking shows at scale Opterons take a pretty big hit on performance*, and power draw under heavy virtualization load which negates their lower acquisition costs.

BUT

What about situations where scaled up performance doesn't matter?  Such as the SMB space in the 1-5 physical servers with lighter virtualization load.  I think that could show some benefit because power draw costs aren't really a huge concern, and at lighter loads they perform just as well as the Xeons and are damn near half the price.  You could even couple that with my other discussion about the reduced licensing overhead of Hyper-V vs VMware and go budget on the hardware in this market segment and bring TCO down further.

Conclusion?  In Big Enterprise Xeon makes way more sense, in SMB markets AMD could be a real viable option if you did it right.

*Note: There is data out there that suggests that Linux KVM based virtualization performs better on Opterons than Xeons.

Appendix A

Background on this particular customer is that the company is hemorrhaging cash like crazy, and is in the midst of squeezing blood from everywhere, and I've been brought in to slim down IT overhead as much as possible while the IT staff that's still there can keep them running day to day.  Which kinda sucks ass for me, because they laid off pretty much anyone who knew what they were doing because they were the most expensive, and they did it in the middle of some major projects, like this VMware to Hyper-V conversion, and a full domain Forrest migration that is now half done with no documentation, and a Exchange migration.

The observations about suitability for enterprise/datacenter vs. SMB is what I expected to hear although I never considered power draw (which for those of you that may not know is always a major factor in datacenter ops.  Power is expensive, very expensive).  That was a very interesting insight and the kind of thing that will trip a senior manager's trigger when it comes time to spec and budget for the datacenter.

Your current customer challenge sounds like a real shit show.  Those circumstances are a hell of a time for them to gut their internal IT team and *could* wind up costing them more than the payroll liability (liability since they are probably cash flow negative/poor) of one or two senior staff that could assist a contract heavy lifter like you in finishing the work properly and keeping the risk lower.

This type of shit is exactly where most companies really fuck up.  They either "manage" IT at the ops or general executive level (read: incompetents in charge of IT strategy and tactics) or use an under qualified and skilled technology head (read: cheap) who mainly serves to say yes a lot to directives and no a lot to IT spending (usually to be compliant with directive!).  

Of course, I can tell some private stories about a larger company like say, HP, that did exactly the same thing on a vastly larger scale and are still recovering years after that CIO was shit-canned....

But I'm digressing on my pet peeve with the way IT is handled in much of the business world regardless of the size of the enterprise.

Anyway, it seems like you wouldn't lose much going the AMD route in a small-ish business.  Are they going to virtualize Exchange into the Hyper-V deployment or is it somewhere else (or managed)?

Power is indeed expensive, especially at large scales.  As for the customer, they kept the department head and a few junior techs to handle tier 1 stuff, the poor department head stays busy enough just keeping up with patching and general maintenance and such.  But, I'm helping him out there in a sense, he's signed on for an RMM platform we resell that will allow him to actively monitor things and give him tighter control over patch deployment to kinda optimize his workflow more.  The plus side is, this department head owned the computer shop my boss hung out in when he was 9 years old.  Needless to say they go waaaaaaay back, so my boss and I are keeping the billable hours to a pretty manageable level as a favor to him, it's a pretty sweetheart deal.  The guy is smart but hasn't been a engineer in a while and definitely knows when he's in over his head, and asked for help and my boss likes the guy so here I am.

As for Exchange it is currently already virtualized, and they've only had the VM for about 2 years I think.  I did suggest that when the time comes to upgrade to Exchange 2016 or whatever is out after they make ROI that I do sell hosted exchange and that I'd be happy to give him TCO comparison numbers and he seemed pretty interested in that.