just a heads up, finally ASUS w680 available.

[a.dresner]

Hi there, I took out the gpu and put it in the top slot. Did not even have to run it as gen3. Seems to be working. Thanks for the tip.

You might want to play with those top 2 slots, GPU in slot 1, HBA in slot 2 and test pass through :)

That’s I’m hoping to do, pass a GPU (windows) and HBA (truenas)

 

[oncdoc]

When the HBA is put into the second slot from the top, the ethernet does not work for some reason. Everything seems to work just fine on the top slot.

 

[a.dresner]

Thats so strange..the 2 built in Intel? what if you pass through the GPU in the second slot? does that effect the ethernet?

 

[Ericloewe]

There have been similar reports of PCIe wonkiness on Supermicro motherboards. I suspect Intel's base firmware package needs a round or two of polish, still.

 

[oncdoc]

There have been similar reports of PCIe wonkiness on Supermicro motherboards. I suspect Intel's base firmware package needs a round or two of polish, still.

thanks for your feedback on that issue. Makes me feel better that's it not just my board.

 

[oncdoc]

Thats so strange..the 2 built in Intel? what if you pass through the GPU in the second slot? does that effect the ethernet?

Once I took my GPU out I did not put it back in to test if it will work on the second pcie slot. Im inclined now to just keep the gpu out of the server and use the cpu graphics. I however could not get IOMMU to be enabled without the GPU in initially. Now thats its enabled Im done with it.

Let me know your results when you get to it.

Cheers.

 

[Klyoku]

Check wired zone for better prices

are you running these memories with the board?

Micron MTC20C2085S1EC48BA1 Memory 32GB DDR5 4800MHz UDIMM MEM-DR532L-CL01-EU48

Memory 32GB DDR5-4800MHz 288-pin Low Profile ECC UDIMM 2RX8 Supermicro Certified Server Memory - RoHS Compliant Supermicro P/N: MEM-DR532L-CL01-EU48

www.wiredzone.com

 

[a.dresner]

FYI I have read that MTC20C2085S1EC48BA1R works fine, so I ordered 2 sticks off eBay from Beach Audio. If you are in the states, you can order direct with Beach

are you running these memories with the board?

Micron MTC20C2085S1EC48BA1 Memory 32GB DDR5 4800MHz UDIMM MEM-DR532L-CL01-EU48

Memory 32GB DDR5-4800MHz 288-pin Low Profile ECC UDIMM 2RX8 Supermicro Certified Server Memory - RoHS Compliant Supermicro P/N: MEM-DR532L-CL01-EU48

www.wiredzone.com

 

[a.dresner]

Once I took my GPU out I did not put it back in to test if it will work on the second pcie slot. Im inclined now to just keep the gpu out of the server and use the cpu graphics. I however could not get IOMMU to be enabled without the GPU in initially. Now thats its enabled Im done with it.

Let me know your results when you get to it.

Cheers.

No doubt, when I get mine I'll be running it through as much testing as I can.. gonna test many different configs. I think ill have all the parts by mid March (international shipping taking time)

 

[oncdoc]

are you running these memories with the board?

Micron MTC20C2085S1EC48BA1 Memory 32GB DDR5 4800MHz UDIMM MEM-DR532L-CL01-EU48

Memory 32GB DDR5-4800MHz 288-pin Low Profile ECC UDIMM 2RX8 Supermicro Certified Server Memory - RoHS Compliant Supermicro P/N: MEM-DR532L-CL01-EU48

www.wiredzone.com

Decided to go with non ECC memory. Using a 32 GB right now and will upgrade

 

[a.dresner]

You might want to read about non Ecc with ZFS - you can loose data - it’s not advisable. Please reconsider :)

 

[Ericloewe]

ZFS is unlikely to be any worse than any other filesystem. The real point is that it's silly to not use ECC for something high-impact enough to be a server.

 

[omeganot]

Hey all,

I appreciate the discussions here and the comparison between the ASUS PRO WS W680-ACE board and the X12STL/X12STH boards. The discussion on PCIe slots and expandability for NAS-focused devices got me curious about what they offer, and what trade-offs you make.

With the X12STL, you get:
- 2 PCIe 4.0 8x
- 1 PCIe 4.0 4x
- 1 PCIe 3.0 2x

Seemingly, the PCIe 4.0 runs through the Xeon directly, but I'm not sure.

Wth the ASUS W680, you get (essentially):
- 2 PCIe 5.0 8x (through the CPU, when running two cards)
- 2 PCIe 3.0 4x (through the W680)
- 1 PCIe 3.0 1x (through the W680)

So with the W680 you downgrade one 4x slot to PCIe 3.0, and pick up a few PCIe 3.0 lanes. If by chance you can make use of PCIe 5.0, kudos to you.

What also stood out to me was the inclusion of an onboard SAS controller, which could, but doesn't have to, remove the need for a separate HBA card. I say could, because I have no idea what chip is providing that, and in the manual it says it runs via a PCIe 4.0 4x bus, supporting 4 drives. A few questions come up for me there, like:
- What SAS controller runs on PCIe 4.0 4x? Most I've seen are 8x lanes. Knowing what this board runs can help understand if it's even usable for TrueNAS.
- Is there really a 4 drive limitation? If it's SlimSAS (SFF-8654?) couldn't you run that to a SAS expander and add more drives? Seems at PCIe 4.0 4x there is plenty of bandwidth for more drives through an expander.
- Wouldn't using this card free up the need for one of the PCIe slots?
- Wouldn't using this built-in SAS create other PCIe lane limitations elsewhere on the board? This is often the case for consumer boards with NVMe drive slots. Using additional NVMe drives through the chipset often disables certain PCIe slots or SATA ports. I'm guessing this is similar, but can find no documentation on it.

I appreciate any insights and help you all may have.

 

[Ericloewe]

What also stood out to me was the inclusion of an onboard SAS controller

I don't think that's true. Asus' page for the non-IPMI version is broken at the moment, but the IPMI version says:

- 1 x SlimSAS Slot Support SlimSAS NVMe device (supports PCIe 4.0 x4 mode and up to 4 SATA devices)***

"SlimSAS" is a monstrously bad way for Asus to say "SFF-8654". It's a connector, there's no SAS controller here. From the system firmware, you can choose whether to configure four chipset I/O lanes as PCIe or SATA, and they're routed via the aforementioned SFF-8654 port.

 

[omeganot]

I don't think that's true. Asus' page for the non-IPMI version is broken at the moment, but the IPMI version says:

"SlimSAS" is a monstrously bad way for Asus to say "SFF-8654". It's a connector, there's no SAS controller here. From the system firmware, you can choose whether to configure four chipset I/O lanes as PCIe or SATA, and they're routed via the aforementioned SFF-8654 port.

That is a heck of a catch. Thanks for the clarification @Ericloewe and for teaching me a bit more about the connector. I was wondering what good providing 4 PCIe lanes over the connector was good for, but now I see it's just a way to get those lanes away from the board to... anything else that is smart enough to do something with them.

 

[Ericloewe]

I see it's just a way to get those lanes away from the board to... anything else that is smart enough to do something with them.

SSDs mostly, but any PCIe device will work with the right adapters.

 

[a.dresner]

I picked up one of these. DiLiVing SlimSAS 4X to 4*SATA，SFF-8654 38pin to 4xSATA(7pin) Cable, 100CM https://a.co/d/fqQZOgQ

Once I get all my parts I plan to do extensive testing as time permits on the board

My goals:
Proxmox
Truenas with HBA passthrough
Windows 11 VM with NVME, NIC and VGA pass through
Portainer

That’s a lot of passthrough lol

Worst case I’ll bail on proxmox and use truenas as my base - I’ll test both solutions

Will have to see how it all fits and plays out.. been very busy at work so only so much time for planning

 

[tsm37]

FYI I have read that MTC20C2085S1EC48BA1R works fine, so I ordered 2 sticks off eBay from Beach Audio. If you are in the states, you can order direct with Beach

Same. I ordered the same model from Beach Audio (2 x 32 GB), and the sticks work fine.

 

[tsm37]

I don't think that's true. Asus' page for the non-IPMI version is broken at the moment, but the IPMI version says:

"SlimSAS" is a monstrously bad way for Asus to say "SFF-8654". It's a connector, there's no SAS controller here. From the system firmware, you can choose whether to configure four chipset I/O lanes as PCIe or SATA, and they're routed via the aforementioned SFF-8654 port.

I can confirm this is true since I have the W680 IPMI mobo. The slimSAS can be switched between PCIE mode or SATA mode, each mode at x4. If you change the slimSAS to SATA model, it supports four SATA drives.

 

[tsm37]

In general, Asus W680 is a decent motherboard and it gets sold out quick. Too bad the third and fourth x16 slots are PCIE3 (and not PCIE4). Also, for the first and second X16 slots, you can do 8x/8x bifurcation but not 4x/4x/4x/4x. I'm looking into the Asus Hyper M2 card, but based on my research and the notes on Asus website, I don't think I can use all four NVME M2 slots on the Asus Hyper card.