I have a Supermicro MBD-X11SCL-F-O motherboard with two Dell H310 Adapter HBA cards (successfully crossflashed to LSI IT firmware following instructions in this forum), and I would like to use its integrated Ethernet port (so, no expansion card into a PCIe slot for this). I would like to use this as a NAS with ZFS.
Out of curiosity, I visited all settings in its BIOS, and found the following:
Advanced > Chipset Configuration > PCH-IO Configuration > PCI Express Configuration > Peer Memory Write Enable = [Disabled].
The documentation does not tell much about it. Based on some googling (actually yahoo-ing as google stopped paginating its results), I think this is the Peer-to-peer Direct Memory Access feature. P2P DMA allows communication between PCI devices bypassing the computer's CPU and RAM.
I was wondering if this feature can have any benefit for me, so I thought through which pairs of devices could communicate to each other with this, and I came up with the following:
A) The HBA cards with each other. Counterargument: Though I don't know ZFS as deep as I would like to, it is software RAID, so I assume that all data goes through the CPU. In case of RAID-Z1 writing, two disks' contents are changed at the same time but not to the same bytes, so the CPU is need to xor. In case of a mirror, after writing to one disk A, the other disk, disk B, could get the data from the first but then writing to A and B could not happen in parallel, only one after the other.
B) A HBA card and the Ethernet port. Counterargument: Unlike the HBA cards, the Ethernet port is not directly connected to the CPU, so I'm not sure this feature is applicable to this connection at all. And even if it is, the data sent to the network must first be checked by ZFS (AFAIK not by xoring RAID-Z1 sectors or comparing mirrored sector but by calculating file hashes) but I'm not completely sure of that. If so, that involves the CPU.
Comparing bandwidths for performance bottlenecks would be a next step, but I'm not sure that using this feature is possible at all, so I've skipped that part.
What do you know about this? Is it useful for anything, or should I leave it turned off?
Out of curiosity, I visited all settings in its BIOS, and found the following:
Advanced > Chipset Configuration > PCH-IO Configuration > PCI Express Configuration > Peer Memory Write Enable = [Disabled].
The documentation does not tell much about it. Based on some googling (actually yahoo-ing as google stopped paginating its results), I think this is the Peer-to-peer Direct Memory Access feature. P2P DMA allows communication between PCI devices bypassing the computer's CPU and RAM.
I was wondering if this feature can have any benefit for me, so I thought through which pairs of devices could communicate to each other with this, and I came up with the following:
A) The HBA cards with each other. Counterargument: Though I don't know ZFS as deep as I would like to, it is software RAID, so I assume that all data goes through the CPU. In case of RAID-Z1 writing, two disks' contents are changed at the same time but not to the same bytes, so the CPU is need to xor. In case of a mirror, after writing to one disk A, the other disk, disk B, could get the data from the first but then writing to A and B could not happen in parallel, only one after the other.
B) A HBA card and the Ethernet port. Counterargument: Unlike the HBA cards, the Ethernet port is not directly connected to the CPU, so I'm not sure this feature is applicable to this connection at all. And even if it is, the data sent to the network must first be checked by ZFS (AFAIK not by xoring RAID-Z1 sectors or comparing mirrored sector but by calculating file hashes) but I'm not completely sure of that. If so, that involves the CPU.
Comparing bandwidths for performance bottlenecks would be a next step, but I'm not sure that using this feature is possible at all, so I've skipped that part.
What do you know about this? Is it useful for anything, or should I leave it turned off?
