"The Beast" build. More fun with Dual Xeon's . Still rocking it as my main pc. - Page 1

[ARCHIVED THREAD] - "The Beast" build. More fun with Dual Xeon's . Still rocking it as my main pc. (Page 1 of 2)

Posted: 4/12/2023 10:39:38 PM EDT

Almost done collecting the pieces of what will be a decently overkill yet not too outdated "Workstation" build.

The goal is to have fun building it and using it for a while until it will replace the dual E5-2670 V2 board in my Proxmox server.

This will be the third dual Xeon built that I have done, the first was a dual socket 2011 using E5 V1 and V2 cpu's, the second was a dual socket 2011-3 board with two E5-2667 V3's

My secret, if you can call it that, is to use a new board with used ram and cpu's that are being retired from service and that are much MUCH cheaper than the original prices new and this go around I am using a dual socket 3647 board since there are much more options at a better prices on used Xeons vs used Epyc cpu's.

Dual LGA 3647 Asrock Rack motherboard:

With cpu coolers I am a bit limited since this motherboard will eventually end up in a 4U server case I can't go crazy on a couple of huge coolers and to be honest there are very few socket LGA 3647 coolers at all.

I first thought of a pair of Noctua NH-D9 DX-3647 4U (LGA3647) or the louder Supermicro 4 U air coolers but after searching I ended up buying a pair of new OEM Asetek enterprise liquid coolers with 120 mm radiators that will fit both the Enthoo Pro case that I will use first and the 4U case this setup will eventually end up it.

I bought them off Ebay for $75.00 each and seem to be my best option.

Test fitting them on my board, not screwed down but just laying on top of the socket protectors..

The radiators. 120 x ~50 mm thick.

I have enough DDR 4 ram for testing but I plan on getting 12 x 16 gigs of ECC ram to fill all slots.

For cpu's I am leaning towards a pair of first gen Xeon Gold 6148 20 cores that boost up to 3.7 ghz but I am on the lookout for gen two cpu's also.

Also need to get a pair of m.2 drives.

I already have a pci-e sound card that's much better than the onboard sound, plus a pci-e usb adapter to give me four more usb ports since this board is limited in usb ports.

Posted: 4/12/2023 10:49:19 PM EDT

[#1]

For a little perspective, the last build from 2018 ish.

The first build that is currently in my 4U server case with 128 gigs of ram and dual E5-2610 V2 cpu's running ProxMox, getting long in the tooth since the last version of ProxMox was a pain to install.

Edit: Found a pic of the first build with two E5-2670 V1 cpu's and two RX 480's.

Posted: 4/12/2023 10:58:24 PM EDT

[#2]

I've got a Xeon E5-2620-V3 machine that I put together a while back. It was used for various things but didn't support the 10 card I wanted to use it for. Ended up building a Ryzen 3 machine as a virtualization host. Been thinking about building a workstation with the X89 board with the Xeon...

The last big servers I built were second gen AMD EPYC's (dual 12 cores and a single 24) for a virtualization cluster at my last job. Worked great for what it was used for.

Posted: 4/12/2023 11:00:56 PM EDT

[#3]

What are you using it for?

Posted: 4/12/2023 11:06:03 PM EDT

[#4]

Quote History

Quoted:
I've got a Xeon E5-2620-V3 machine that I put together a while back. It was used for various things but didn't support the 10 card I wanted to use it for. Ended up building a Ryzen 3 machine as a virtualization host. Been thinking about building a workstation with the X89 board with the Xeon...

The last big servers I built were second gen AMD EPYC's (dual 12 cores and a single 24) for a virtualization cluster at my last job. Worked great for what it was used for.

View Quote

I do not work in IT so I just build them for fun and to learn new things on.

Computers have probably been my biggest hobby since the mid 1980's and I really enjoy building pc's

Newer hardware has much better performance BUT at a much higher price, so for the third I have went with the best bang per buck going with a combination of used and new and two or three generations old.

Posted: 4/12/2023 11:14:01 PM EDT

[#5]

Quote History

Quoted:
What are you using it for?

View Quote

Killer Linux Workstation.

Vm's and Linux gaming mostly plus what ever new thing that interests me.

Mostly for fun until I retire the current board in my Proxmox VM server so eventually it will end up there.

Posted: 4/13/2023 12:01:16 AM EDT

[#6]

Love workstation builds like this, there's something special about a big dual cpu/gpu system

Posted: 4/13/2023 2:29:15 AM EDT

[#7]

Nice progress

Check out this VM gamer

Dual 4090 With One 13900K Unraid tutorial. Dual PC Set Up Using One CPU.

https://www.youtube.com/watch?v=W2-a2XtMdt8

Posted: 4/13/2023 9:55:10 PM EDT

[#8]

Just experimenting with the new case I purchased for my AM5 build just to see if this case would work.

It's a Geometric Future Lohan case and mounts the motherboard 90 degrees from usual., the company that builds this case is owned by InWin.

The fans I will use on the radiators are pretty powerful and move alot of air.

I like the placement of the radiators except I don't think the airflow would be ideal.

Plus I would lose the number 7 pci-e slot.

Yep will stick with the planned Enthoo Pro case.

Still waiting on a few things to be able to swap the AM5 setup into this case.

Trying this case out with this board has answered a few of my questions, would probably work alot better if just using air coolers for the cpu's.

Posted: 4/16/2023 9:49:45 PM EDT

[#9]

Now that that the Enthoo Pro case is freed up I went ahead and mounted up the motherboard.

Looks like I will need to 3d print a couple of offset fan adapters to allow the rad's and fans to clear the ram and power connectors.

Posted: 4/17/2023 9:24:12 PM EDT

[#10]

3 D printing the first of two offset mounting brackets.

Posted: 4/18/2023 5:42:09 AM EDT

[#11]

The bracket gives me the clearance for the power connectors but the thickness moves it over the top rearmost ram slot if I mount both rad's to the top of the case.

If I mount one to the top and one exhausting out the rear that gives me the most clearance.

This mounting setup allows me to use a 140 mm fan at the top.

EDIT: That is not the ram I will be running, just using it for testing purposes. I also have another couple of fan adapters with offset that I will try out.

Posted: 4/18/2023 5:52:22 AM EDT

[#12]

I'll print one of these tonight, might actually work better than version one and allow me to mount both at the top of the case.

Posted: 4/19/2023 7:35:04 PM EDT

[#13]

Well I printed the new offset adapter but it's not going to work for me.

It does get the rad away from the ram but it also would make me have to remove the radiators to access the ram.

Posted: 4/19/2023 11:20:01 PM EDT

[#14]

Just did not work like I wanted.

Printing a much thinner one, 120 mm to 140 mm adapter, with a little more offset in two directions to give me a little bit of flexibility.

If that does not work i'll use the one I printed first, this one.

The way I have it set up now might be the way I use it, this is the only config that actually clears everything and lets me add or remove ram without removing anything.

It currently clears the tall ram, the ecc ram I plan on using is not as tall.

Also I might print another of the first adapters and use on the lh side rad to move it a bit more away from the motherboard.

But i'm still fiddling with it.

Posted: 4/20/2023 12:47:57 AM EDT

[#15]

The new thinner offset adapter works great!

Had to use three different types of screws but is the best so far.

Clears the test ram.

The other radiator clears everything with plenty of room.

I will print another thin adapter tomorrow night and test fit the other rad at the top also, it just might have enough room to clear everything.

Posted: 4/21/2023 12:08:33 AM EDT

[#16]

Finally mounted the radiators where I am satisfied.

The two thin 140 mm to 120 mm offset adapters worked nicely. Looks much cleaner than one on top and one in the rear.

Using an old GTX 960 for clearance testing, still undecided on a video card.

Looks like it just barely clears the ram slot.

Posted: 4/21/2023 8:51:02 PM EDT

[#17]

A little light bulb went off a few minutes ago. I realized that the 120 - 140 mm bracket did not have to go between the rad and the bottom side of the top of the case but it could also mount on the top side of the case and still work.

So once again removing and remounting the brackets and rad's I found out it that it works and gives me a little bit more room between the motherboard and the rad/fan assemblys.

Gained about four millimeters, does not sound like much but it moves the fan mounting screws a little bit farther away from the ram.

Hopefully this is enough but if not I can use a different pair of screws that are shorter and mount only through the closet part of the fan to the radiator.

A bit more room for the power connectors.

Looks a little bit nicer.

And yes the top case panel does clear it....barely.

Posted: 4/23/2023 7:34:48 PM EDT

[#18]

Ordered two Xeon Gold 6148 20 core cpu's.

https://www.intel.com/content/www/us/en/products/sku/120489/intel-xeon-gold-6148-processor-27-5m-cache-2-40-ghz/specifications.html

I decided on this one because it's the best bang for the buck of the used first gen scaleable Xeons, the ones that are just a little bit more powerful are alot more.

It seems this cpu was a very popular one for HPC servers and Intel sold a lot of them hence the low prive compared to the others in its performance range.

Going by the benchmarks I have seen the Gold 6148's have about three times the performance over the E5-2670 V2 cpu's in my server.

Posted: 4/25/2023 10:06:44 PM EDT

[#19]

Received my Intel 2TB m.2 ssd and mounted the Big Ass m.2 cooler to it just for the hell of it, will have to be careful on the choice of a video card to be able to use it.

Yea the cooler is overkill but that's the whole point of this build.

Posted: 4/26/2023 7:08:56 PM EDT

[#20]

Cpu's are in a day earlier than I expected.

Posted: 4/26/2023 11:51:18 PM EDT

[#21]

Cpu's are mounted and its starting to come together. I'll use my regular DDR 4 ram until I get my ECC ram.

Barely enough room to clear the power connector.

Posted: 4/29/2023 5:41:58 PM EDT

[#22]

Assembled and ready for the first boot.

Threw in 64 gigs of ram and an older GTX 1060 video card for now.

Also mounted a pci-e sound card and a HSB 3.0 adapter to give me four more USB ports since the motherboard is lacking in that area.

Posted: 4/29/2023 8:47:20 PM EDT

[#23]

It's not booting into the bios but i can log into the BMC, so now trouble shooting mode.

Posted: 4/29/2023 9:14:15 PM EDT

[#24]

Looks like one of the cpu's is bad.

One of cpu's refuses to boot when it is in either socket or by itself, the other is fine.

So I contacted the seller but it looks like he is out of stock now.

Posted: 4/29/2023 10:24:55 PM EDT

[#25]

Booo. Hope he gets more in stock soon.

Posted: 4/29/2023 11:00:27 PM EDT

[#26]

Quote History

Quoted:
Booo. Hope he gets more in stock soon.

View Quote

I'm ok with getting a refund on one cpu.

There are other sellers with them.

On another note, one of the ram slots is no longer seeing the stick of ram.

Now i'm installing Windows 11.

Posted: 4/30/2023 10:20:17 AM EDT

[#27]

Seller already replied and pays shipping on returns so the bad cpu is going back.

I ordered another one from a different seller for $20.00 less.

Should be here at the end of the week.

Posted: 5/4/2023 7:56:32 PM EDT

[#28]

My ram finally came, 12 DDR4 ecc 2666 mhz 16gb sticks for 192 gigs of ram, so i'm testing them two at at time with MemTest.

Posted: 5/5/2023 7:39:01 PM EDT

[#29]

The replacement cpu came in and I installed it and all the 12 sticks of ram.

Posted: 5/5/2023 11:54:56 PM EDT

[#30]

I did not think I would ever get the damn thing to boot, turns out the seller sent me four sticks of DDR4 2666 ecc ram and eight sticks of DDR4 2933 ecc ram.

Did not think it would matter since the bios showed them all running at 2666 which is the fastest my cpu's will run and I have mixed and matched ram before and never had an issue other than running at the slowest rams speed.

But nothing but problems, so I tried just the eight sticks of the faster ram and it still showed 2666 speed but I could still could not get it to boot either the m.2 with the windows install or a usb flash drive.

So I used just the four sticks of 2666 and it boots.

Talk about a PITA!!

The funny thing is the next gen cpu's that will work on my motherboard do use the faster 2933 ram and when I tested two sticks with just the single cpu they ran fine.

Oh well it boots now.

80 threads.

Edit: Some Cinebench 23R runs. The multicore run with 80 threads ...lol

Posted: 5/6/2023 11:36:11 AM EDT

[#31]

Well it looks like one of the ram slots is bad, runs and boots fine until I populate that one certain slot.

With regular DDR4 ram it pops up an error that the ram in that spot did not pass test on a cold boot.

With ECC server ram it does not show that error but locks up.

I can see the ram and it shows all the info when I log into the BMC but refuses to actually boot until I remove the ram from that slot.

So I contacted the Mobo's tech support and will go from there.

What sucks is if I would not have been delayed a week I would have found this out within NewEgg's return period.

But as of now its running pretty dam nicely on 176 gigs of ram.

So this puts this project on hold again so I am running memtest just to give the pc something usefull to do.

Posted: 5/8/2023 4:07:45 AM EDT

[#32]

I was just laying in bed thinking about the ram issue and I thought maybe I had two bad cpu's so I got up and swapped them out to see if the "bad ram slot" moved to the ram controlled by the other socket.

So I swapped the cpu's, now the ram slot works.

Shows the full 192 gig's.

Oh well now I'm running memtest and testing all the ram.

Posted: 5/8/2023 7:15:12 PM EDT

[#33]

Memtest found zero errors so far.

Posted: 5/9/2023 10:39:16 PM EDT

[#34]

Well the memory error is back, CPU 0 ram slot E1.

As long as the slot is unoccupied it runs great.

So Asrock Rack wanted a detailed pic of the socket and I just emailed them.

Hopefully they will tell me to send it in.

Posted: 5/12/2023 8:55:10 PM EDT

[#35]

Looks like I found my problem.

One of the socket pins are bent.

I will fix it in the am after I get a good nights sleep.

Posted: 5/13/2023 12:55:35 PM EDT

[#36]

I had purchased a "dental pick" set from harbor freight yesterday and using a big draftsman lighted magnifying glass in series with a smaller magnifying glass let me see the pins pretty good.

I was able to "snap" the pin back into place.

So I reinstalled the motherboard back into the case and reassembled everything and cleaned up a the wiring a bit.

After lunch i'll fire it up and see what happens.

Posted: 5/13/2023 1:30:32 PM EDT

[#37]

Ram issue is fixed.

Sees all 192 gigs of ram, does not lock up with a stick in slot E1 anymore.

Running memtest now.

Posted: 5/13/2023 2:36:23 PM EDT

[#38]

Nice that it's a problem you can fix yourself.

Posted: 5/13/2023 7:16:26 PM EDT

[#39]

Quote History

Quoted:
Nice that it's a problem you can fix yourself.

View Quote

Yep, I would have to send it in otherwise.

No telling how long that would take.

So far so good, memtest finished the first pass with zero errors.

Also no ecc errors in the BMC log like there was before.

Posted: 5/14/2023 10:44:58 AM EDT

[#40]

Still going strong.

Posted: 5/14/2023 3:48:19 PM EDT

[#41]

Posted: 5/14/2023 7:31:49 PM EDT

[#42]

Finally passed with flying colors.

Now maybe I can go ahead and finish it.

Posted: 5/18/2023 8:03:58 PM EDT

[#43]

There is an issue with the cooling fans on the radiators, when the pc is idle they would slow down and stop causing an cpu fan error.

I played around with fan setting in the bios and nothing really made a difference.

These fans are pretty powerful and draw a bit of current and have a minimum speed of 700 rpm.

So I think when the pc is idle and the cpu's are not doing anything the board drops the voltage to below the point that the fans need at the slowest speed.

When I stressed the cpu's the temps never went above 55 c and the fans never even ran anywhere to their max speed.

So I ordered new fans, I went with a couple of pressure optimized 120 mm x 15 mm thick fans, 10 mm less than the more powerful fans, that draw alot less voltage at low speeds.

These are rated less airflow but should still have plenty and give me a bit more clearance.

Posted: 5/20/2023 11:13:45 PM EDT

[#44]

Posted: 5/28/2023 12:07:00 AM EDT

[#45]

Went and swapped everything into the new case, should have done this since the beginning since everything went together so well.

New thin fans on the radiators work perfectly.

I need to get a 90 degree adapter for the USB 3.0 connctor on the motherboard.

~~Thinking about removing the blue USB 3.0 cable and move the pci-e usb card and the sound card one slot farther away from the video card.~~ Removed it and now the video card has another slot worth of room.

I have just enough usb connectors without the blue cable.

Ended up using a 1000 w evga platinum power supply because the 1600 watt power supply's power cord does not fit into the hole in the case.

Also just installed Ubuntu 23.04.