HYEPYC - 2022 Server build-log

[Pri]

Small update today. I have decided to cancel the Asrock motherboard (ROMED8-T2) and instead purchase a Supermicro motherboard (H12SSL-NT).

There are a few reasons for this which I'll now detail but before that, both boards are very similar offering about the same capability. I do believe the Supermicro to be superior in a few ways though. Below is a photo of it (not mine obviously, as mine is in transit).



Both boards are for EPYC 7002 (ROME) and 7003 (MILAN) CPU's and support 2TB of RAM, dual M.2 SSD's, lots of PCIe slots, dual 10Gb networking etc. However there are some things the Supermicro offers that the Asrock doesn't beyond just easier availability.

1. The Asrock has some performance issues with Milan. Asrock appears to be aware of this and is telling people a new hardware revision is coming that will resolve it. This isn't great. I think it's power related. The Supermicro has no such issue, it has a very strong power setup for EPYC supporting the 280 Watt TDP with ease and has dual 8-pin CPU connectors while the Asrock has 8 + 4 Pin.

2. The Supermicro has upwards facing front I/O pins (power & reset buttons, activity LED's etc) while the Asrock has them turned 90 degrees. I used some calipers to size up the front I/O headers for my case and they're 14mm while the board only provides 8mm of space.. this means I will not be able to plug these into the board as my 4U case leaves only 2mm of room between the case wall and the motherboard.

As the Asrock board has IPMI I did consider only using that to start the server and forgoing the activity LED's but this was less than desirable. Ultimately I decided I wanted the front I/O connected making the Asrock a non-starter. I wasn't able to source any jumper wires that would have worked, even the smallest ones I found were 12mm and that's still too large.

3. The Supermicro has better placed connectors. Its USB 3.0 internal header is placed in a better location and its PWM fan headers are spread out around the board including one near the rear where I have two fans that will be on a Y splitter. This in general makes cabling neater.

4. The Supermicro has 2 extra USB-A ports on the back I/O shield. This is convenient for me, the Asrock had 1 x USB-C and 2 x USB-A. I really don't need USB-C at this time on my server.

5. The Supermicro allows higher utilisation of the EPYC CPU's PCIe lanes. On the Asrock board you have two M.2 slots and two sets of SATA ports. If you want to use all of those you lose the 2nd PCIe slot on the board entirely. On the Supermicro you don't lose any slots.

6. The Supermicro has more compatible NVMe connectors. The board has two connectors intended for cables near the CPU socket which are both PCIe 4.0 x8. These are actually the same connector as on my 9500-8i HBA card. The Asrock by contrast has two OCulink connectors with a rare corner locking mechanism, getting cables for this connector is impossible and they're half the bandwidth of the ports the Supermicro has.

These ports allow for either NVMe SSDs to be connected (two x4 or one x8 per connector) or for eight SATA drives per connector. This is quite flexible and Supermicro does provide a cable in the package to connect eight hard drives (SATA) to one of these connectors. Like noted above these do not steal bandwidth from one of the PCIe slots so you can use all 7 slots while also using these two connectors.

7. The Supermicro has support for longer M.2 SSD's which are more common in the enterprise due to higher capacities needing more NAND flash packages or for capacitor banks which allow the SSD to write out its memory to NAND during a power outage. This isn't really a major problem for me but it's nice to know I can pickup enterprise SSD's and use them very easily on the Supermicro regardless of their length.

8. And finally perhaps the most damning on the Asrock is the board has an unfinished BIOS and Asrock appears to not care. I read a lot of reviews from customers of this board complaining of the BIOS not working correctly with missing options. The last stable update is from almost one year ago and their beta update only adds MILAN-X functionality (Milan CPU's with 3D VCache).

So those are all the reasons I've switched motherboard. I'm expecting to receive my Supermicro around the same time as the Asrock one which is to say the start of August. So the waiting continues but at-least I'm getting a much better motherboard.

[Pri]

Today Amazons Prime Day started and I was able to get some absolute insane deals. Two SN850's for £177 each (usually £260) and 4 x 18TB Western Digital drives for £222 each (usually £305).

So I actually bought an extra 2TB SSD instead of only one and an extra hard disk drive instead of only 3. Insanely good deals. Really paid to stay up for the prime day starting in the UK that's for sure.

At this point I now have every item for this server build either on the way or already here.

[Pri]

Received my Hard Disks yesterday. They're currently being tested, I didn't take any photos of those as I think we've all seen hard drives before nothing too interesting.

But I did receive my SSD's today which have Western Digital's heatsinks on them and they do look quite interesting so here are those photos.





And I did want to take a photo of the sides because the heatsink arrangement is quite interesting. They have it raised off the SSD slightly and it's only physically attached to what I believe is the SSD controller and the DRAM chip, but not to the NAND flash (which likes to stay warm for optimum performance). Quite interesting that they didn't just connect it to the full length of the SSD like other makers.

Also pardon the flash photography it was required to illustrate the gap.

[Pri]

Thought I'd include a photo of one of the network cards I'm going to be installing in this system. I do actually have two of these cards but I'll probably just put one in. I'm going to be running a virtualised pfSense on this system as a backup in-case my physical pfSense system needs to be taken offline for some reason (hardware failure or some-such). So I'm using this card for that as I can pass it straight through to the virtual machine.

It is an Intel X540-T2 card (PCIe v2.1 x8) which supports 100Mb, 1Gb and 10Gb connections.

[Pri]

Received my CableMod cables today in a surprise shipment, the tracking for these were practically non-existent until late last night.

They do fit well and look nice. I think if I was ordering these again I'd go with 33cm long ones instead of 40cm as these are a little bit too long requiring me to curl them slightly however when I measured originally I added about 8cm more than I measured just in-case CableMod cut them short, in-fact they cut them longer than I specified (40cm) by 3cm (making them 43cm) so





But I am happy with them and they don't take up much room even with the slack, most importantly it keeps the front area clear so I can add and remove the hotswap fan in that position without rubbing up against the cables which is something I cannot do in my current server making removal of the fan while the system is powered on a bit dangerous as sudden power loss could cause drives to fall out of the raid array.

p.s. all the cables were pushed in fully after this photograph was taken, you may notice in the above photo some of the molex were only half-way in

[Pri]

Today my motherboard finally arrived. It has been an incredible wait for this part. I did switch from the Asrock ROMED8-T2 to this SuperMicro H12SSL-NT but it was well worth it. Enjoy the photos.




You may notice above there is only one SSD installed. This is because unfortunately the M.2 slots on this motherboard are too close together and Western Digital decided to make their heatsinks so wide that two of them can't be placed together on this motherboard. I will likely remedy this by using a PCIe card for the second SSD.








That is the build so far. There has been two issues. Firstly the M.2 situation above but also the fans. I did always intend to install Noctua fans in this system however the stock fans were going to be used for the first few weeks while I transitioned my data from my previous server.

That unfortunately will not be happening because these fans are the loudest fans I've ever heard. If you've ever used a 1300 Watt vacuum cleaner that might come close to the sound produced by these three fans. The are unbelievably loud. They are so loud that the wind they produce inside the case sounds like someone power washing a car from outside while you sit inside.

With them being so loud that you can hear them anywhere in our house even at 50% (which is the lowest they'll go when powered through this motherboard) .. well they had to go. So I've ordered three new Noctuas which are coming tomorrow and will go straight in simply so I can use the server without deafening my family.

I'll now leave you with some screenshots and benchmarks.

[Pri]

Today I received my new Noctua fans which are now inside the server and keeping things nice and silent. Very pleased with their performance as usual Noctua doesn't disappoint.

As a result I've shucked the remaining 18TB drives and placed them in the server, I've also installed and configured Unraid so we're off to the races. Right now I'm building parity on a single 18TB drive but two of the 18TB drives will be used for dual-parity eventually once I put my 7 x 10TB drives into the server in the week ahead.

I thought I'd show a few screenshots from the Unraid interface starting with the CPU screen and used cores/threads. As a reminder this is a 24 Core / 48 Thread CPU.



Below is what the main array tab looks like with the 4 x 18TB drives installed. At present the parity building has been running for 3 hours and the drives are only around 37-39c which is lower than they ran inside their external cases. The Noctua fans are doing a wonderful job.



This building process should take around 24 hours in total and you can see each disk is reading at about 217MB/s or 650 MB/s total while the parity disk is the only one receiving writes (the parity data) at 217MB/s. I'd say these 18TB drives are quite fast and absolutely silent I can't hear them making any sounds at all in-fact.

One last thing I found interesting is Unraid has a plugin called IPMI Tools which makes available many values from the servers built in BMC (Baseboard Management System). This lets you not only view readings like voltages, temperatures and fan speeds but also alter the fan curves based on temperatures and load conditions. Below is a screenshot showing all the readings it can monitor.



This is really good because it means my server can be even quieter thanks to the higher degree of fan control this plugin offers over the built in SuperMicro options and lets me view things from Unraid in a central location on the main dashboard instead of needing to access the SuperMicro IPMI Web interface. Quite neat.

[Pri]

This is likely the last photo of the server before it's placed in its permanent home in my 12U rack. The software is still being setup at the moment so it lives on my desk until then.

[Pri]

I installed the PCIe card with the SSD's today, they fit quite nicely in this



And I also did a little bit more cable management just to get more airflow to the expansion cards, put the front I/O ribbon cable a bit out of the way, shame these cables are so very long and hard to hide.

[Pri]

A photo of the Unraid dashboard with all my disks finally configured in the array.