Fastest way to transfer from Freenas another Freenas?

[jmcguire525]

I'm going to be adding storage and want to expand my 6x8TB pool to an 8x8TB (raidz2 on both). I also have a new build with 5x4TB drives that I can use for temp storage. The reason I want to move everything is to avoid having more parity disk than I need and I am limited to 8 sata ports, my only other option without moving everything is adding 2 new drives in a mirror.

Right now I only have about 14TB used, so I was thinking of setting the new system up in raidz1 then transferring everything over to it, destroy the pool in my large server and add two new 8TB drives to give me a new 8x8TB pool. After that I would move everything back to the new pool.

To save time I thought it would be best to pick up a couple 10gb ethernet cards, but I don't have a 10gb switch. What is the easiest way to initiate that transfer if I link the two servers directly?

I also have datasets with my jail config files, Plex being the most important, should I create a new datasets on the temp server and transfer everything within those datasets using smb or is there a better way?

Any suggestions on the proper method of doing this and recommendations on 10gb cards are appreciated.

 

[Elliot Dierksen]

If you look at the hardware recommendations, Chelsio is the 10G NIC of choice for FreeNAS. You can use a DAC/Twinax cable, or a fiber patch cord between them. That wouldn't require a switch.

 

[danb35]

As @Elliot Dierksen says, on the hardware side, a pair of Chelsio 10G NICs and a single cable between them will do it. Assign each of them an IP address in a different subnet from your main network. On the software side, ZFS replication.

 

[jmcguire525]

As @Elliot Dierksen says, on the hardware side, a pair of Chelsio 10G NICs and a single cable between them will do it. Assign each of them an IP address in a different subnet from your main network. On the software side, ZFS replication.

Would cheaper alternatives be fine for this one time task? I don't need 10gbe for normal use, just want to speed up this transfer. I was looking at these...

Mellanox 10gb Card

 

[danb35]

I've seen reports of Mellanox cards working, but haven't paid much attention to them. Chelsio T3-series cards are pretty cheap, though.

 

[Chris Moore]

Would cheaper alternatives be fine for this one time task? I don't need 10gbe for normal use, just want to speed up this transfer. I was looking at these...

Mellanox 10gb Card

I have used those Mellanox cards in my NAS. They do work, but there might be a faster way. Why do you say you are limited to eight drives?
Last time I copied data from one pool to another, I found it much, much faster to connect all the drives to a single system and copy the data from one pool to the other with both pools connected to the same computer. I had the drives mounted in the respective computers and just ran hard drive data cables between the two systems.

 

[Chris Moore]

Something like this might be an option and you could continue to use it going forward:

Drive Controller: SAS PCI-E 3.0 HBA LSI 9207-8i P20 IT Mode for ZFS FreeNAS - - US $44.88
https://www.ebay.com/itm/192639052923

Data Drive Cables: Mini SAS to 4-SATA SFF-8087 Multi-Lane Forward Breakout Internal Cable - - US $12.99
https://www.ebay.com/itm/371681252206

 

[jmcguire525]

Something like this might be an option and you could continue to use it going forward:

Drive Controller: SAS PCI-E 3.0 HBA LSI 9207-8i P20 IT Mode for ZFS FreeNAS - - US $44.88
https://www.ebay.com/itm/192639052923

Data Drive Cables: Mini SAS to 4-SATA SFF-8087 Multi-Lane Forward Breakout Internal Cable - - US $12.99
https://www.ebay.com/itm/371681252206

Thanks, that may be my best bet. I've read things about not using raid cards and will have to read up on using that controller, but that will be much easier if I can just attach all the drives during the transfer.

 

[Chris Moore]

Thanks, that may be my best bet. I've read things about not using raid cards and will have to read up on using that controller, but that will be much easier if I can just attach all the drives during the transfer.

I suggested a SAS HBA which is not a RAID card. A RAID card is a hardware RAID controller that has integrated RAM. The card I pointed out is flashed with the IT (Initiator Target) firmware that does not allow any type of RAID configuration and the card possesses no RAM. It is the same model controller I use in my NAS.

 

[Chris Moore]

If the description of that card is throwing you off, take a look at this one instead:
https://www.ebay.com/itm/HP-H220-66...-0-x8-Host-Bus-Adapter-Controlle/153224277440

 

[jmcguire525]

If the description of that card is throwing you off, take a look at this one instead:
https://www.ebay.com/itm/HP-H220-66...-0-x8-Host-Bus-Adapter-Controlle/153224277440

Sorry, I didn't mean to imply that it was a raid card, I just don't have any experience with pci cards in general and the only thing I did know was to stay away from raid cards. I'll definitely be looking into that was controller though.

 

[JustinClift]

Network card wise, these ones look to be the cheapest 10GbE capable Mellanox cards on US Ebay atm:

https://www.ebay.com/itm/Mellanox-M...etwork-Adapter-PCI-E-Low-Profile/132900243984

You'd need a cable to link them. Something like this (QSFP connector) would do:

https://www.ebay.com/itm/Mellanox-C...r-Hybrid-Cable-10GbE-10Gb-s-QSFP/123124528941

All up pricing would be about the same as the HP card @Chris Moore points out. The HP card approach would likely be faster than hooking up two boxes and going through that hassle though.

 

[Chris Moore]

Network card wise, these ones look to be the cheapest 10GbE capable Mellanox cards on US Ebay atm:

https://www.ebay.com/itm/Mellanox-M...etwork-Adapter-PCI-E-Low-Profile/132900243984

That is an InfiniBand adapter. Not a regular 10Gb network card. This is not what you want, even though it is cheap.
If you want to go with 10Gb networking, this hardware is known to work (by me) because I used a set of cards like this to connect my two NAS systems for them to sync with each other before I got my 10Gb switch:
https://www.ebay.com/itm/MNPA19-XTR...X-2-10Gbe-NIC-10GBe-3m-SFP-Cable/123529113623

Just understand this, there is a difference between the speed you will see over the network vs the speed you will see when the drives are directly connected to the same system. I have tried it both ways.

 

[JustinClift]

> That is an InfiniBand adapter. Not a regular 10Gb network card.

No. That is a Mellanox ConnectX-2 VPI card, same chipset as the one you're linking to even. ;)

VPI cards do both Ethernet and Infiniband. On FreeNAS, they do Ethernet (eg 10GbE, 40GbE, etc depending on model) as only the Ethernet driver is included.

 

[jmcguire525]

Just understand this, there is a difference between the speed you will see over the network vs the speed you will see when the drives are directly connected to the same system. I have tried it both ways.

Digging up on old thread bc I'm just now getting around to this... Chris do you remember about how long the data transfer took when the drives were wired directly to one system? (Just a ballpark number to give me an idea of what to expect) I'm trying to decide whether to add another 6x8tb vdev or take the time to transfer back and forth with the end result of a 12x8tb RaidZ2.