Two FreeNAS' - Looking to increase read/write performance (10GbE)

[eptesicus]

I'm running an Rsync task to sync data from NAS01 to NAS02 over a peer-to-peer 10GbE connection, so that the sync doesn't have to go over my 10GbE switch, and I'm getting the best possible performance, as I need to sync a lot of data. I'm currently peaking transfer over the dedicated 10GbE connection at 3.1 Gbps, and iperf confirms a pipe of at least 9 Gbps. I'm not sure how to better test file transfer from the storage pools between the NAS', but I feel as though I should get at least 5 or 6 Gbps transfer without using encryption. My NAS' specs are below.

NAS01:
FreeNAS-11.2-U6
Chassis: Chenbro NR40700
HBA: HP H220 flashed to IT-mode
CPU: E3-1220v6 3.0GHz
RAM: 32 GB
Autotune: Off
NICs: Mellanox ConnectX-3 Dual 10GbE
NIC 1: mlxen0 - 10.0.50.31 (LAN)
NIC 2: mlxen1 - 192.168.99.101 (peer-to-peer to NAS02)
Storage: 16x 10TB white-label WD Red drives in 2x 8-drive Z1 vdevs
Compression: LZ4
Dedup: Off
Rsync Module:

Rsync Task:

NAS02:
FreeNAS-11.2-U6
Chassis: Chenbro NR40700
HBA: HP H220 flashed to IT-mode
CPU: E3-1241v3 3.5GHz
RAM: 32 GB
Autotune: Off
NICs: Mellanox ConnectX-3 Dual 10GbE
NIC 1: mlxen0 - 10.0.50.33 (LAN)
NIC 2: mlxen1 - 192.168.99.102 (peer-to-peer to NAS01)
Storage: 40x 4TB HGST NAS drives in 5x 8-drive Z1 vdevs
Compression: LZ4
Dedup: Off
Rsync Module:

 

[dlavigne]

Were you able to improve this?

 

[eptesicus]

Were you able to improve this?

Unfortunately, no. I added 2 more zvols of 8x 10TB drives, ensuring plenty of free space, and performance is still much lower than expected...

 

[Elliot Dierksen]

rsync is never going to be the thing to max out your bandwidth. Try mounting one from the other and do a direct copy manually. You mention that you are using RAIDZ1 vdevs which is not recommend ever, but particularly not recommended with the large drives you are using. That isn't related to performance, but to the security of your data. Use iperf to measure network speed between the boxes. Assuming that is good, then we can look at any system or pool configuration related issues.

 

[eptesicus]

rsync is never going to be the thing to max out your bandwidth. Try mounting one from the other and do a direct copy manually. You mention that you are using RAIDZ1 vdevs which is not recommend ever, but particularly not recommended with the large drives you are using. That isn't related to performance, but to the security of your data. Use iperf to measure network speed between the boxes. Assuming that is good, then we can look at any system or pool configuration related issues.

Good idea with mounting shares to copy. Aside from using 'cp' via CLI, is there a good task in FreeNAS that can be used to automate the backup of data on a schedule like with the rsync tasks?

Regarding the Z1 vdevs... I did it this way because of the insane overhead of ZFS, and the amount of usable storage I lost if going with Z2 vdevs. With Z2 vdevs, I actually have more overhead than usable space, and that's using 8x drive Z2 vdevs. Also, I went with 8x drive vdevs for the ease of adding storage, as that, for me was the logical amount of drives I would add at a time. What would you recommend I do? I'm about to add a server with 7x disk shelves with 12x6TB in each to back everything up to a second time, as I may start from scratch.

I ran the iperf tests, noted in the OP, and got about 9 Gbps between each NAS.

 

[Elliot Dierksen]

rsynch is absolutely the right tool for subsequent backups. It spends a lot of time figuring out if it needs to copy which is why it doesn't max out the transfer speeds. When I have done this, I will do a manual copy the first time with a command something like this cd /dir-to-be-copied ; find . -print | cpio -pvmd /mnt/some-other-system. Then set up rsynch to keep the file set in synch.

I ran the iperf tests, noted in the OP, and got about 9 Gbps between each NAS.

That sounds like the underlying network infrastructure is good. Things frequently go back to old and simple things. You are only as fast as the slowest component. When you get a huge increase in the network speed, then things like disk speed, vdev construction, and fragmentation within the file system can now be slower than the network connection itself.

 

[NotFredEither]

Aside from using 'cp' via CLI, is there a good task in FreeNAS that can be used to automate the backup of data on a schedule like with the rsync tasks?

Since you're going from FreeNAS --> FreeNAS, why not use the fantastic built-in replication function? https://www.ixsystems.com/documentation/freenas/11.2-U7-legacy/storage.html#replication-tasks

 

[eptesicus]

Since you're going from FreeNAS --> FreeNAS, why not use the fantastic built-in replication function? https://www.ixsystems.com/documentation/freenas/11.2-U7-legacy/storage.html#replication-tasks

It was my understanding that with replication, only snapshots were replicated to the destination FreeNAS server. Is this not the case? In the event of a failure, I want to make sure I'm able to point servers to my backup NAS while restoring my primary.

 

[NotFredEither]

It was my understanding that with replication, only snapshots were replicated to the destination FreeNAS server. Is this not the case? In the event of a failure, I want to make sure I'm able to point servers to my backup NAS while restoring my primary.

I'm not sure what "only snapshots" implies in this case. You can set FreeNAS to make snapshots of your primary at almost any interval you like, and then the snapshot is immediately replicated to your backup server. That means your backup server contains an exact duplicate of your primary. Because ZFS knows exactly which sectors have changed from one snapshot to the next, when incremental snapshots are sent, they begin transmitting immediately (as opposed to rsync, which has to go through the entire filesystem to detect changes, adding time and read overhead). You can then roll back to any snapshot as necessary, or automatically delete them at a given age. It's quite powerful.

Regarding your concern about using the backup itself, this passage from the link I gave mentions that you can mount the backed-up copy as a clone:

The target dataset on the receiving system is automatically created in read-only mode to protect the data. To mount or browse the data on the receiving system, create a clone of the snapshot and use the clone. Clones are created in read/write mode, making it possible to browse or mount them. See Snapshots for more information on creating clones.

 

[eptesicus]

I'm not sure what "only snapshots" implies in this case. You can set FreeNAS to make snapshots of your primary at almost any interval you like, and then the snapshot is immediately replicated to your backup server. That means your backup server contains an exact duplicate of your primary. Because ZFS knows exactly which sectors have changed from one snapshot to the next, when incremental snapshots are sent, they begin transmitting immediately (as opposed to rsync, which has to go through the entire filesystem to detect changes, adding time and read overhead). You can then roll back to any snapshot as necessary, or automatically delete them at a given age. It's quite powerful.

Regarding your concern about using the backup itself, this passage from the link I gave mentions that you can mount the backed-up copy as a clone:

I'm following now. Thanks.

I got replication setup and running as of last night, and it's at 3%. However, speeds are slower than rsync and it's operating in spurts.

How is this now half the speed of rsync? What am I doing wrong here? Again, this is a direct 10GbE link between the two boxes. Nothing inbetween.

 

[kdragon75]

Yep, you need to forget about the network. You need to look at disk and CPU performance on both ends. Those look almost like 5 seconds bursts and if that's the case, I would look at the target side first. Try watching zpool iostat -v -i 1 (from my memory) and see if the disks are busy the whole time or if one is slower than the rest.

 

[eptesicus]

Yep, you need to forget about the network. You need to look at disk and CPU performance on both ends. Those look almost like 5 seconds bursts and if that's the case, I would look at the target side first. Try watching zpool iostat -v -i 1 (from my memory) and see if the disks are busy the whole time or if one is slower than the rest.

Thanks. I ended up replacing my 44x 4TB drives in the backup NAS02 with 16x 10TB drives (2x 8-drive Z1 zvols). The issue still exists. I'm watching 'zpool iostat -v 1' on both servers, and I'm thinking the issue exists on the source, NAS01. While every disk on NAS02 is constantly being written to (seeing no issues there), reading from the drives on NAS01 appears to be a bit of a mess, where there is a gap in performance every 5 seconds. Could this be likely a fragmentation issue on the NAS01 side?

Code:

                                           capacity     operations    bandwidth
pool                                    alloc   free   read  write   read  write
storagepool                              131T   159T     14      0  1.87M      0
  raidz1                                71.8T   742G      0      0      0      0
    gptid/32cfb661-b0ad-11e9-aa42-ac1f6b6656be      -      -      0      0      0      0
    gptid/338c3c84-b0ad-11e9-aa42-ac1f6b6656be      -      -      0      0      0      0
    gptid/3445580c-b0ad-11e9-aa42-ac1f6b6656be      -      -      0      0      0      0
    gptid/34fe57a7-b0ad-11e9-aa42-ac1f6b6656be      -      -      0      0      0      0
    gptid/35be77fe-b0ad-11e9-aa42-ac1f6b6656be      -      -      0      0      0      0
    gptid/36800f79-b0ad-11e9-aa42-ac1f6b6656be      -      -      0      0      0      0
    gptid/373e3223-b0ad-11e9-aa42-ac1f6b6656be      -      -      0      0      0      0
    gptid/37f96724-b0ad-11e9-aa42-ac1f6b6656be      -      -      0      0      0      0
  raidz1                                44.4T  28.1T      0      0   127K      0
    gptid/8fd3489a-b5ab-11e9-8a79-ac1f6b6656be      -      -      0      0  19.9K      0
    gptid/90aef942-b5ab-11e9-8a79-ac1f6b6656be      -      -      0      0  15.9K      0
    gptid/9182ebe1-b5ab-11e9-8a79-ac1f6b6656be      -      -      0      0  15.9K      0
    gptid/925c8582-b5ab-11e9-8a79-ac1f6b6656be      -      -      0      0  15.9K      0
    gptid/93d07d12-b5ab-11e9-8a79-ac1f6b6656be      -      -      0      0      0      0
    gptid/952d0953-b5ab-11e9-8a79-ac1f6b6656be      -      -      0      0  19.9K      0
    gptid/969117b9-b5ab-11e9-8a79-ac1f6b6656be      -      -      0      0  19.9K      0
    gptid/97f7fa2f-b5ab-11e9-8a79-ac1f6b6656be      -      -      0      0  19.9K      0
  raidz1                                7.80T  64.7T      9      0  1.25M      0
    gptid/b8101952-f75e-11e9-bf94-ac1f6b6656be      -      -     14      0   291K      0
    gptid/b9750334-f75e-11e9-bf94-ac1f6b6656be      -      -      7      0   143K      0
    gptid/baf302e9-f75e-11e9-bf94-ac1f6b6656be      -      -     16      0   319K      0
    gptid/bc55ae4c-f75e-11e9-bf94-ac1f6b6656be      -      -     14      0   259K      0
    gptid/bdb9e313-f75e-11e9-bf94-ac1f6b6656be      -      -      7      0   147K      0
    gptid/bf20ae92-f75e-11e9-bf94-ac1f6b6656be      -      -     14      0   259K      0
    gptid/c0855a56-f75e-11e9-bf94-ac1f6b6656be      -      -      9      0   183K      0
    gptid/c1f1d7a7-f75e-11e9-bf94-ac1f6b6656be      -      -      8      0   159K      0
  raidz1                                7.24T  65.3T      3      0   510K      0
    gptid/7f8b80c7-fa81-11e9-bf94-ac1f6b6656be      -      -      3      0  75.7K      0
    gptid/80f3abeb-fa81-11e9-bf94-ac1f6b6656be      -      -      2      0  51.8K      0
    gptid/8285a9c8-fa81-11e9-bf94-ac1f6b6656be      -      -      3      0  75.7K      0
    gptid/84130db6-fa81-11e9-bf94-ac1f6b6656be      -      -      2      0  51.8K      0
    gptid/857ab4f4-fa81-11e9-bf94-ac1f6b6656be      -      -      2      0  55.8K      0
    gptid/86ffebf0-fa81-11e9-bf94-ac1f6b6656be      -      -      3      0  71.7K      0
    gptid/885eba5b-fa81-11e9-bf94-ac1f6b6656be      -      -      2      0  55.8K      0
    gptid/89ff3a41-fa81-11e9-bf94-ac1f6b6656be      -      -      3      0  71.7K      0

                                           capacity     operations    bandwidth
pool                                    alloc   free   read  write   read  write
storagepool                              131T   159T     10    967  1.37M  4.86M
  raidz1                                71.8T   742G      0      0      0      0
    gptid/32cfb661-b0ad-11e9-aa42-ac1f6b6656be      -      -      0      3      0  16.0K
    gptid/338c3c84-b0ad-11e9-aa42-ac1f6b6656be      -      -      0      3      0  16.0K
    gptid/3445580c-b0ad-11e9-aa42-ac1f6b6656be      -      -      0      3      0  16.0K
    gptid/34fe57a7-b0ad-11e9-aa42-ac1f6b6656be      -      -      0      3      0  16.0K
    gptid/35be77fe-b0ad-11e9-aa42-ac1f6b6656be      -      -      0      3      0  16.0K
    gptid/36800f79-b0ad-11e9-aa42-ac1f6b6656be      -      -      0      3      0  16.0K
    gptid/373e3223-b0ad-11e9-aa42-ac1f6b6656be      -      -      0      3      0  16.0K
    gptid/37f96724-b0ad-11e9-aa42-ac1f6b6656be      -      -      0      3      0  16.0K
  raidz1                                44.4T  28.1T      0    298      0  1.55M
    gptid/8fd3489a-b5ab-11e9-8a79-ac1f6b6656be      -      -      0     32      0   344K
    gptid/90aef942-b5ab-11e9-8a79-ac1f6b6656be      -      -      0     33      0   332K
    gptid/9182ebe1-b5ab-11e9-8a79-ac1f6b6656be      -      -      0     37      0   376K
    gptid/925c8582-b5ab-11e9-8a79-ac1f6b6656be      -      -      0     33      0   348K
    gptid/93d07d12-b5ab-11e9-8a79-ac1f6b6656be      -      -      0     35      0   364K
    gptid/952d0953-b5ab-11e9-8a79-ac1f6b6656be      -      -      0     30      0   332K
    gptid/969117b9-b5ab-11e9-8a79-ac1f6b6656be      -      -      0     38      0   364K
    gptid/97f7fa2f-b5ab-11e9-8a79-ac1f6b6656be      -      -      0     34      0   348K
  raidz1                                7.80T  64.7T     10    333  1.37M  1.69M
    gptid/b8101952-f75e-11e9-bf94-ac1f6b6656be      -      -      2     44  55.9K   432K
    gptid/b9750334-f75e-11e9-bf94-ac1f6b6656be      -      -      9     42   180K   424K
    gptid/baf302e9-f75e-11e9-bf94-ac1f6b6656be      -      -      2     36  55.9K   416K
    gptid/bc55ae4c-f75e-11e9-bf94-ac1f6b6656be      -      -      3     31  71.9K   376K
    gptid/bdb9e313-f75e-11e9-bf94-ac1f6b6656be      -      -      9     40   188K   408K
    gptid/bf20ae92-f75e-11e9-bf94-ac1f6b6656be      -      -      2     37  51.9K   388K
    gptid/c0855a56-f75e-11e9-bf94-ac1f6b6656be      -      -      7     38   152K   392K
    gptid/c1f1d7a7-f75e-11e9-bf94-ac1f6b6656be      -      -      8     38   164K   380K
  raidz1                                7.24T  65.3T      0    334      0  1.62M
    gptid/7f8b80c7-fa81-11e9-bf94-ac1f6b6656be      -      -      0     33      0   376K
    gptid/80f3abeb-fa81-11e9-bf94-ac1f6b6656be      -      -      0     32      0   368K
    gptid/8285a9c8-fa81-11e9-bf94-ac1f6b6656be      -      -      0     36      0   404K
    gptid/84130db6-fa81-11e9-bf94-ac1f6b6656be      -      -      0     37      0   392K
    gptid/857ab4f4-fa81-11e9-bf94-ac1f6b6656be      -      -      0     41      0   412K
    gptid/86ffebf0-fa81-11e9-bf94-ac1f6b6656be      -      -      0     39      0   400K
    gptid/885eba5b-fa81-11e9-bf94-ac1f6b6656be      -      -      0     42      0   416K
    gptid/89ff3a41-fa81-11e9-bf94-ac1f6b6656be      -      -      0     37      0   376K

                                           capacity     operations    bandwidth
pool                                    alloc   free   read  write   read  write
storagepool                              131T   159T     74      0  9.12M      0
  raidz1                                71.8T   742G      0      0      0      0
    gptid/32cfb661-b0ad-11e9-aa42-ac1f6b6656be      -      -      0      0      0      0
    gptid/338c3c84-b0ad-11e9-aa42-ac1f6b6656be      -      -      0      0      0      0
    gptid/3445580c-b0ad-11e9-aa42-ac1f6b6656be      -      -      0      0      0      0
    gptid/34fe57a7-b0ad-11e9-aa42-ac1f6b6656be      -      -      0      0      0      0
    gptid/35be77fe-b0ad-11e9-aa42-ac1f6b6656be      -      -      0      0      0      0
    gptid/36800f79-b0ad-11e9-aa42-ac1f6b6656be      -      -      0      0      0      0
    gptid/373e3223-b0ad-11e9-aa42-ac1f6b6656be      -      -      0      0      0      0
    gptid/37f96724-b0ad-11e9-aa42-ac1f6b6656be      -      -      0      0      0      0
  raidz1                                44.4T  28.1T     18      0  2.14M      0
    gptid/8fd3489a-b5ab-11e9-8a79-ac1f6b6656be      -      -      4      0   323K      0
    gptid/90aef942-b5ab-11e9-8a79-ac1f6b6656be      -      -      4      0   299K      0
    gptid/9182ebe1-b5ab-11e9-8a79-ac1f6b6656be      -      -      4      0   307K      0
    gptid/925c8582-b5ab-11e9-8a79-ac1f6b6656be      -      -      3      0   299K      0
    gptid/93d07d12-b5ab-11e9-8a79-ac1f6b6656be      -      -      4      0   327K      0
    gptid/952d0953-b5ab-11e9-8a79-ac1f6b6656be      -      -      4      0   303K      0
    gptid/969117b9-b5ab-11e9-8a79-ac1f6b6656be      -      -      4      0   327K      0
    gptid/97f7fa2f-b5ab-11e9-8a79-ac1f6b6656be      -      -      3      0   295K      0
  raidz1                                7.80T  64.7T     54      0  6.85M      0
    gptid/b8101952-f75e-11e9-bf94-ac1f6b6656be      -      -     13      0   965K      0
    gptid/b9750334-f75e-11e9-bf94-ac1f6b6656be      -      -     11      0  1005K      0
    gptid/baf302e9-f75e-11e9-bf94-ac1f6b6656be      -      -     12      0   977K      0
    gptid/bc55ae4c-f75e-11e9-bf94-ac1f6b6656be      -      -     13      0  1.06M      0
    gptid/bdb9e313-f75e-11e9-bf94-ac1f6b6656be      -      -     14      0  1001K      0
    gptid/bf20ae92-f75e-11e9-bf94-ac1f6b6656be      -      -     11      0  1001K      0
    gptid/c0855a56-f75e-11e9-bf94-ac1f6b6656be      -      -     12      0  1.05M      0
    gptid/c1f1d7a7-f75e-11e9-bf94-ac1f6b6656be      -      -     12      0  1009K      0
  raidz1                                7.24T  65.3T      0      0   128K      0
    gptid/7f8b80c7-fa81-11e9-bf94-ac1f6b6656be      -      -      0      0  19.9K      0
    gptid/80f3abeb-fa81-11e9-bf94-ac1f6b6656be      -      -      0      0  19.9K      0
    gptid/8285a9c8-fa81-11e9-bf94-ac1f6b6656be      -      -      0      0  19.9K      0
    gptid/84130db6-fa81-11e9-bf94-ac1f6b6656be      -      -      0      0  16.0K      0
    gptid/857ab4f4-fa81-11e9-bf94-ac1f6b6656be      -      -      0      0  16.0K      0
    gptid/86ffebf0-fa81-11e9-bf94-ac1f6b6656be      -      -      0      0  16.0K      0
    gptid/885eba5b-fa81-11e9-bf94-ac1f6b6656be      -      -      0      0      0      0
    gptid/89ff3a41-fa81-11e9-bf94-ac1f6b6656be      -      -      0      0  19.9K      0

                                           capacity     operations    bandwidth
pool                                    alloc   free   read  write   read  write
storagepool                              131T   159T      7      0  1022K      0
  raidz1                                71.8T   742G      0      0      0      0
    gptid/32cfb661-b0ad-11e9-aa42-ac1f6b6656be      -      -      0      0      0      0
    gptid/338c3c84-b0ad-11e9-aa42-ac1f6b6656be      -      -      0      0      0      0
    gptid/3445580c-b0ad-11e9-aa42-ac1f6b6656be      -      -      0      0      0      0
    gptid/34fe57a7-b0ad-11e9-aa42-ac1f6b6656be      -      -      0      0      0      0
    gptid/35be77fe-b0ad-11e9-aa42-ac1f6b6656be      -      -      0      0      0      0
    gptid/36800f79-b0ad-11e9-aa42-ac1f6b6656be      -      -      0      0      0      0
    gptid/373e3223-b0ad-11e9-aa42-ac1f6b6656be      -      -      0      0      0      0
    gptid/37f96724-b0ad-11e9-aa42-ac1f6b6656be      -      -      0      0      0      0
  raidz1                                44.4T  28.1T      0      0      0      0
    gptid/8fd3489a-b5ab-11e9-8a79-ac1f6b6656be      -      -      0      0      0      0
    gptid/90aef942-b5ab-11e9-8a79-ac1f6b6656be      -      -      0      0      0      0
    gptid/9182ebe1-b5ab-11e9-8a79-ac1f6b6656be      -      -      0      0      0      0
    gptid/925c8582-b5ab-11e9-8a79-ac1f6b6656be      -      -      0      0      0      0
    gptid/93d07d12-b5ab-11e9-8a79-ac1f6b6656be      -      -      0      0      0      0
    gptid/952d0953-b5ab-11e9-8a79-ac1f6b6656be      -      -      0      0      0      0
    gptid/969117b9-b5ab-11e9-8a79-ac1f6b6656be      -      -      0      0      0      0
    gptid/97f7fa2f-b5ab-11e9-8a79-ac1f6b6656be      -      -      0      0      0      0
  raidz1                                7.80T  64.7T      2      0   383K      0
    gptid/b8101952-f75e-11e9-bf94-ac1f6b6656be      -      -      2      0  55.9K      0
    gptid/b9750334-f75e-11e9-bf94-ac1f6b6656be      -      -      1      0  31.9K      0
    gptid/baf302e9-f75e-11e9-bf94-ac1f6b6656be      -      -      1      0  35.9K      0
    gptid/bc55ae4c-f75e-11e9-bf94-ac1f6b6656be      -      -      2      0  55.9K      0
    gptid/bdb9e313-f75e-11e9-bf94-ac1f6b6656be      -      -      1      0  39.9K      0
    gptid/bf20ae92-f75e-11e9-bf94-ac1f6b6656be      -      -      2      0  55.9K      0
    gptid/c0855a56-f75e-11e9-bf94-ac1f6b6656be      -      -      2      0  55.9K      0
    gptid/c1f1d7a7-f75e-11e9-bf94-ac1f6b6656be      -      -      2      0  51.9K      0
  raidz1                                7.24T  65.3T      4      0   639K      0
    gptid/7f8b80c7-fa81-11e9-bf94-ac1f6b6656be      -      -      3      0  75.8K      0
    gptid/80f3abeb-fa81-11e9-bf94-ac1f6b6656be      -      -      7      0   144K      0
    gptid/8285a9c8-fa81-11e9-bf94-ac1f6b6656be      -      -      4      0  95.8K      0
    gptid/84130db6-fa81-11e9-bf94-ac1f6b6656be      -      -      8      0   164K      0
    gptid/857ab4f4-fa81-11e9-bf94-ac1f6b6656be      -      -      7      0   152K      0
    gptid/86ffebf0-fa81-11e9-bf94-ac1f6b6656be      -      -      9      0   176K      0
    gptid/885eba5b-fa81-11e9-bf94-ac1f6b6656be      -      -      7      0   144K      0
    gptid/89ff3a41-fa81-11e9-bf94-ac1f6b6656be      -      -      9      0   176K      0

                                           capacity     operations    bandwidth
pool                                    alloc   free   read  write   read  write
storagepool                              131T   159T     19      0  2.47M      0
  raidz1                                71.8T   742G      0      0      0      0
    gptid/32cfb661-b0ad-11e9-aa42-ac1f6b6656be      -      -      0      0      0      0
    gptid/338c3c84-b0ad-11e9-aa42-ac1f6b6656be      -      -      0      0      0      0
    gptid/3445580c-b0ad-11e9-aa42-ac1f6b6656be      -      -      0      0      0      0
    gptid/34fe57a7-b0ad-11e9-aa42-ac1f6b6656be      -      -      0      0      0      0
    gptid/35be77fe-b0ad-11e9-aa42-ac1f6b6656be      -      -      0      0      0      0
    gptid/36800f79-b0ad-11e9-aa42-ac1f6b6656be      -      -      0      0      0      0
    gptid/373e3223-b0ad-11e9-aa42-ac1f6b6656be      -      -      0      0      0      0
    gptid/37f96724-b0ad-11e9-aa42-ac1f6b6656be      -      -      0      0      0      0
  raidz1                                44.4T  28.1T      0      0      0      0
    gptid/8fd3489a-b5ab-11e9-8a79-ac1f6b6656be      -      -      0      0      0      0
    gptid/90aef942-b5ab-11e9-8a79-ac1f6b6656be      -      -      0      0      0      0
    gptid/9182ebe1-b5ab-11e9-8a79-ac1f6b6656be      -      -      0      0      0      0
    gptid/925c8582-b5ab-11e9-8a79-ac1f6b6656be      -      -      0      0      0      0
    gptid/93d07d12-b5ab-11e9-8a79-ac1f6b6656be      -      -      0      0      0      0
    gptid/952d0953-b5ab-11e9-8a79-ac1f6b6656be      -      -      0      0      0      0
    gptid/969117b9-b5ab-11e9-8a79-ac1f6b6656be      -      -      0      0      0      0
    gptid/97f7fa2f-b5ab-11e9-8a79-ac1f6b6656be      -      -      0      0      0      0
  raidz1                                7.80T  64.7T      0      0      0      0
    gptid/b8101952-f75e-11e9-bf94-ac1f6b6656be      -      -      0      0      0      0
    gptid/b9750334-f75e-11e9-bf94-ac1f6b6656be      -      -      0      0      0      0
    gptid/baf302e9-f75e-11e9-bf94-ac1f6b6656be      -      -      0      0      0      0
    gptid/bc55ae4c-f75e-11e9-bf94-ac1f6b6656be      -      -      0      0      0      0
    gptid/bdb9e313-f75e-11e9-bf94-ac1f6b6656be      -      -      0      0      0      0
    gptid/bf20ae92-f75e-11e9-bf94-ac1f6b6656be      -      -      0      0      0      0
    gptid/c0855a56-f75e-11e9-bf94-ac1f6b6656be      -      -      0      0      0      0
    gptid/c1f1d7a7-f75e-11e9-bf94-ac1f6b6656be      -      -      0      0      0      0
  raidz1                                7.24T  65.3T     19      0  2.47M      0
    gptid/7f8b80c7-fa81-11e9-bf94-ac1f6b6656be      -      -     15      0   319K      0
    gptid/80f3abeb-fa81-11e9-bf94-ac1f6b6656be      -      -      9      0   256K      0
    gptid/8285a9c8-fa81-11e9-bf94-ac1f6b6656be      -      -     14      0   319K      0
    gptid/84130db6-fa81-11e9-bf94-ac1f6b6656be      -      -     10      0   274K      0
    gptid/857ab4f4-fa81-11e9-bf94-ac1f6b6656be      -      -     10      0   286K      0
    gptid/86ffebf0-fa81-11e9-bf94-ac1f6b6656be      -      -     10      0   233K      0
    gptid/885eba5b-fa81-11e9-bf94-ac1f6b6656be      -      -     11      0   248K      0
    gptid/89ff3a41-fa81-11e9-bf94-ac1f6b6656be      -      -     10      0   237K      0

                                           capacity     operations    bandwidth
pool                                    alloc   free   read  write   read  write
storagepool                              131T   159T     72  1.02K  9.11M  9.58M
  raidz1                                71.8T   742G      0      0      0      0
    gptid/32cfb661-b0ad-11e9-aa42-ac1f6b6656be      -      -      0      0      0      0
    gptid/338c3c84-b0ad-11e9-aa42-ac1f6b6656be      -      -      0      0      0      0
    gptid/3445580c-b0ad-11e9-aa42-ac1f6b6656be      -      -      0      0      0      0
    gptid/34fe57a7-b0ad-11e9-aa42-ac1f6b6656be      -      -      0      0      0      0
    gptid/35be77fe-b0ad-11e9-aa42-ac1f6b6656be      -      -      0      0      0      0
    gptid/36800f79-b0ad-11e9-aa42-ac1f6b6656be      -      -      0      0      0      0
    gptid/373e3223-b0ad-11e9-aa42-ac1f6b6656be      -      -      0      0      0      0
    gptid/37f96724-b0ad-11e9-aa42-ac1f6b6656be      -      -      0      0      0      0
  raidz1                                44.4T  28.1T     13    377  1.75M  2.54M
    gptid/8fd3489a-b5ab-11e9-8a79-ac1f6b6656be      -      -      7     48   248K   547K
    gptid/90aef942-b5ab-11e9-8a79-ac1f6b6656be      -      -      7     45   280K   523K
    gptid/9182ebe1-b5ab-11e9-8a79-ac1f6b6656be      -      -      7     47   232K   523K
    gptid/925c8582-b5ab-11e9-8a79-ac1f6b6656be      -      -      7     45   240K   499K
    gptid/93d07d12-b5ab-11e9-8a79-ac1f6b6656be      -      -      7     55   288K   551K
    gptid/952d0953-b5ab-11e9-8a79-ac1f6b6656be      -      -      7     52   236K   531K
    gptid/969117b9-b5ab-11e9-8a79-ac1f6b6656be      -      -      7     48   268K   535K
    gptid/97f7fa2f-b5ab-11e9-8a79-ac1f6b6656be      -      -      7     43   236K   511K
  raidz1                                7.80T  64.7T     38    346  4.87M  5.58M
    gptid/b8101952-f75e-11e9-bf94-ac1f6b6656be      -      -      7     47   667K   979K
    gptid/b9750334-f75e-11e9-bf94-ac1f6b6656be      -      -      7     40   723K   947K
    gptid/baf302e9-f75e-11e9-bf94-ac1f6b6656be      -      -      8     48   743K   971K
    gptid/bc55ae4c-f75e-11e9-bf94-ac1f6b6656be      -      -      7     43   727K   943K
    gptid/bdb9e313-f75e-11e9-bf94-ac1f6b6656be      -      -      7     43   699K   959K
    gptid/bf20ae92-f75e-11e9-bf94-ac1f6b6656be      -      -      7     46   703K   955K
    gptid/c0855a56-f75e-11e9-bf94-ac1f6b6656be      -      -      7     52   671K   991K
    gptid/c1f1d7a7-f75e-11e9-bf94-ac1f6b6656be      -      -      7     51   711K   963K
  raidz1                                7.24T  65.3T     19    325  2.50M  1.45M
    gptid/7f8b80c7-fa81-11e9-bf94-ac1f6b6656be      -      -      6     37   355K   347K
    gptid/80f3abeb-fa81-11e9-bf94-ac1f6b6656be      -      -      8     36   347K   343K
    gptid/8285a9c8-fa81-11e9-bf94-ac1f6b6656be      -      -      6     37   343K   351K
    gptid/84130db6-fa81-11e9-bf94-ac1f6b6656be      -      -      7     36   316K   335K
    gptid/857ab4f4-fa81-11e9-bf94-ac1f6b6656be      -      -      7     42   359K   375K
    gptid/86ffebf0-fa81-11e9-bf94-ac1f6b6656be      -      -      7     40   351K   367K
    gptid/885eba5b-fa81-11e9-bf94-ac1f6b6656be      -      -      6     36   359K   355K
    gptid/89ff3a41-fa81-11e9-bf94-ac1f6b6656be      -      -      8     29   347K   320K

                                           capacity     operations    bandwidth
pool                                    alloc   free   read  write   read  write
storagepool                              131T   159T      0      0      0      0
  raidz1                                71.8T   742G      0      0      0      0
    gptid/32cfb661-b0ad-11e9-aa42-ac1f6b6656be      -      -      0      3      0  15.4K
    gptid/338c3c84-b0ad-11e9-aa42-ac1f6b6656be      -      -      0      3      0  15.4K
    gptid/3445580c-b0ad-11e9-aa42-ac1f6b6656be      -      -      0      3      0  15.4K
    gptid/34fe57a7-b0ad-11e9-aa42-ac1f6b6656be      -      -      0      3      0  15.4K
    gptid/35be77fe-b0ad-11e9-aa42-ac1f6b6656be      -      -      0      3      0  15.4K
    gptid/36800f79-b0ad-11e9-aa42-ac1f6b6656be      -      -      0      3      0  15.4K
    gptid/373e3223-b0ad-11e9-aa42-ac1f6b6656be      -      -      0      3      0  15.4K
    gptid/37f96724-b0ad-11e9-aa42-ac1f6b6656be      -      -      0      3      0  15.4K
  raidz1                                44.4T  28.1T      0      0      0      0
    gptid/8fd3489a-b5ab-11e9-8a79-ac1f6b6656be      -      -      0      0      0      0
    gptid/90aef942-b5ab-11e9-8a79-ac1f6b6656be      -      -      0      0      0      0
    gptid/9182ebe1-b5ab-11e9-8a79-ac1f6b6656be      -      -      0      0      0      0
    gptid/925c8582-b5ab-11e9-8a79-ac1f6b6656be      -      -      0      0      0      0
    gptid/93d07d12-b5ab-11e9-8a79-ac1f6b6656be      -      -      0      0      0      0
    gptid/952d0953-b5ab-11e9-8a79-ac1f6b6656be      -      -      0      0      0      0
    gptid/969117b9-b5ab-11e9-8a79-ac1f6b6656be      -      -      0      0      0      0
    gptid/97f7fa2f-b5ab-11e9-8a79-ac1f6b6656be      -      -      0      0      0      0
  raidz1                                7.80T  64.7T      0      0      0      0
    gptid/b8101952-f75e-11e9-bf94-ac1f6b6656be      -      -      0      3      0  15.4K
    gptid/b9750334-f75e-11e9-bf94-ac1f6b6656be      -      -      0      3      0  15.4K
    gptid/baf302e9-f75e-11e9-bf94-ac1f6b6656be      -      -      0      3      0  15.4K
    gptid/bc55ae4c-f75e-11e9-bf94-ac1f6b6656be      -      -      0      3      0  15.4K
    gptid/bdb9e313-f75e-11e9-bf94-ac1f6b6656be      -      -      0      3      0  15.4K
    gptid/bf20ae92-f75e-11e9-bf94-ac1f6b6656be      -      -      0      3      0  15.4K
    gptid/c0855a56-f75e-11e9-bf94-ac1f6b6656be      -      -      0      3      0  15.4K
    gptid/c1f1d7a7-f75e-11e9-bf94-ac1f6b6656be      -      -      0      3      0  15.4K
  raidz1                                7.24T  65.3T      0      0      0      0
    gptid/7f8b80c7-fa81-11e9-bf94-ac1f6b6656be      -      -      0      3      0  15.4K
    gptid/80f3abeb-fa81-11e9-bf94-ac1f6b6656be      -      -      0      3      0  15.4K
    gptid/8285a9c8-fa81-11e9-bf94-ac1f6b6656be      -      -      0      3      0  15.4K
    gptid/84130db6-fa81-11e9-bf94-ac1f6b6656be      -      -      0      3      0  15.4K
    gptid/857ab4f4-fa81-11e9-bf94-ac1f6b6656be      -      -      0      3      0  15.4K
    gptid/86ffebf0-fa81-11e9-bf94-ac1f6b6656be      -      -      0      3      0  15.4K
    gptid/885eba5b-fa81-11e9-bf94-ac1f6b6656be      -      -      0      3      0  15.4K
    gptid/89ff3a41-fa81-11e9-bf94-ac1f6b6656be      -      -      0      3      0  15.4K

                                           capacity     operations    bandwidth
pool                                    alloc   free   read  write   read  write
storagepool                              131T   159T      0      0      0      0
  raidz1                                71.8T   742G      0      0      0      0
    gptid/32cfb661-b0ad-11e9-aa42-ac1f6b6656be      -      -      0      0      0      0
    gptid/338c3c84-b0ad-11e9-aa42-ac1f6b6656be      -      -      0      0      0      0
    gptid/3445580c-b0ad-11e9-aa42-ac1f6b6656be      -      -      0      0      0      0
    gptid/34fe57a7-b0ad-11e9-aa42-ac1f6b6656be      -      -      0      0      0      0
    gptid/35be77fe-b0ad-11e9-aa42-ac1f6b6656be      -      -      0      0      0      0
    gptid/36800f79-b0ad-11e9-aa42-ac1f6b6656be      -      -      0      0      0      0
    gptid/373e3223-b0ad-11e9-aa42-ac1f6b6656be      -      -      0      0      0      0
    gptid/37f96724-b0ad-11e9-aa42-ac1f6b6656be      -      -      0      0      0      0
  raidz1                                44.4T  28.1T      0      0      0      0
    gptid/8fd3489a-b5ab-11e9-8a79-ac1f6b6656be      -      -      0      0      0      0
    gptid/90aef942-b5ab-11e9-8a79-ac1f6b6656be      -      -      0      0      0      0
    gptid/9182ebe1-b5ab-11e9-8a79-ac1f6b6656be      -      -      0      0      0      0
    gptid/925c8582-b5ab-11e9-8a79-ac1f6b6656be      -      -      0      0      0      0
    gptid/93d07d12-b5ab-11e9-8a79-ac1f6b6656be      -      -      0      0      0      0
    gptid/952d0953-b5ab-11e9-8a79-ac1f6b6656be      -      -      0      0      0      0
    gptid/969117b9-b5ab-11e9-8a79-ac1f6b6656be      -      -      0      0      0      0
    gptid/97f7fa2f-b5ab-11e9-8a79-ac1f6b6656be      -      -      0      0      0      0
  raidz1                                7.80T  64.7T      0      0      0      0
    gptid/b8101952-f75e-11e9-bf94-ac1f6b6656be      -      -      0      0      0      0
    gptid/b9750334-f75e-11e9-bf94-ac1f6b6656be      -      -      0      0      0      0
    gptid/baf302e9-f75e-11e9-bf94-ac1f6b6656be      -      -      0      0      0      0
    gptid/bc55ae4c-f75e-11e9-bf94-ac1f6b6656be      -      -      0      0      0      0
    gptid/bdb9e313-f75e-11e9-bf94-ac1f6b6656be      -      -      0      0      0      0
    gptid/bf20ae92-f75e-11e9-bf94-ac1f6b6656be      -      -      0      0      0      0
    gptid/c0855a56-f75e-11e9-bf94-ac1f6b6656be      -      -      0      0      0      0
    gptid/c1f1d7a7-f75e-11e9-bf94-ac1f6b6656be      -      -      0      0      0      0
  raidz1                                7.24T  65.3T      0      0      0      0
    gptid/7f8b80c7-fa81-11e9-bf94-ac1f6b6656be      -      -      0      0      0      0
    gptid/80f3abeb-fa81-11e9-bf94-ac1f6b6656be      -      -      0      0      0      0
    gptid/8285a9c8-fa81-11e9-bf94-ac1f6b6656be      -      -      0      0      0      0
    gptid/84130db6-fa81-11e9-bf94-ac1f6b6656be      -      -      0      0      0      0
    gptid/857ab4f4-fa81-11e9-bf94-ac1f6b6656be      -      -      0      0      0      0
    gptid/86ffebf0-fa81-11e9-bf94-ac1f6b6656be      -      -      0      0      0      0
    gptid/885eba5b-fa81-11e9-bf94-ac1f6b6656be      -      -      0      0      0      0
    gptid/89ff3a41-fa81-11e9-bf94-ac1f6b6656be      -      -      0      0      0      0
[/ICODE]

 

[Robert Thomspon]

Since you're going from FreeNAS --> FreeNAS, why not use the fantastic built-in replication function? https://www.ixsystems.com/documentation/freenas/11.2-U7-legacy/storage.html#replication-tasks

Shit, if i could get SSH to actually connect to my other machine in the replication tab, I'd happily use it...

 

[eptesicus]

I see that the 5 second spike in reads only occurs on all of the disks in a single vdev. All of the other disks in the other 3 vdevs are reading/writing as they're supposed to.

Edit: It's the raidz1-2 vdev. I'm trying to look at issues for that particular vdev, but I guess I don't know the right keywords to Google.

Example of drives in vdevs other than raidz1-2:

Example of drives in the raidz1-2 vdev:

 

[Mannekino]

What kind of monitoring/graphing tool is that from the screenshots?

 

[NotFredEither]

I see that the 5 second spike in reads only occurs on all of the disks in a single vdev. All of the other disks in the other 3 vdevs are reading/writing as they're supposed to.

Edit: It's the raidz1-2 vdev. I'm trying to look at issues for that particular vdev, but I guess I don't know the right keywords to Google.

Example of drives in vdevs other than raidz1-2:

Example of drives in the raidz1-2 vdev:

I'm risking getting out of my depth, but my first thought is that 5 seconds sounds an awful lot like the interval for synchronous writes. Do you have a fast SLOG on the receiving end? If not, your replication sync writes will be hitting the receiving pool twice; maybe your pool is fast enough that that isn't a factor.

If that's not it, can you look at the latencies of your drives on the sending end? From your screenshots above, it's true that your da16 is pulsing, but it's also transferring far more data than your da15. The stats you posted earlier show that the zpool is asymetrically filled--one raidz1 is nearly full, the second has the rest of the data, and the other two are almost empty. Is da16 the nearly-full raidz1? Is it possible that one of your drives in da16 is showing higher latency than the others and is slowing things down while the system waits on it?

 

[eptesicus]

What kind of monitoring/graphing tool is that from the screenshots?

This is netdata which can be enabled in Services.

I'm risking getting out of my depth, but my first thought is that 5 seconds sounds an awful lot like the interval for synchronous writes. Do you have a fast SLOG on the receiving end? If not, your replication sync writes will be hitting the receiving pool twice; maybe your pool is fast enough that that isn't a factor.

No SLOG on either servers, just spinning disks. Why will the replication writes hit the second NAS pool twice? If all SLOG is is a write cache, then shouldn't it go to disk a single time if the cache does not exist?

If that's not it, can you look at the latencies of your drives on the sending end? From your screenshots above, it's true that your da16 is pulsing, but it's also transferring far more data than your da15. The stats you posted earlier show that the zpool is asymetrically filled--one raidz1 is nearly full, the second has the rest of the data, and the other two are almost empty. Is da16 the nearly-full raidz1? Is it possible that one of your drives in da16 is showing higher latency than the others and is slowing things down while the system waits on it?

Disk latency:

Da16 should not be in the raidz1, but should be in raidz1-2. When I added drives to this, I put 8 in, created the first vdev, put another 8 in, created the second, and continued with the third and fourth the same. So, da16-23 should most definitely be in raidz1-2. Is there a good way to verify this? I was unable to find one.

Also, I disabled encryption on the replication task, and restarted it. Now speeds are up to 3.3 Gbps transfer, however, there's a lot more time between the bursts. I still have no idea why iperf says 9+ Gbps, but no transfers are ever actually above 3 +/- Gbps. I was told in other posts when using rsync that replication is faster and the better option, but for me, it's slower. This is a mess.

 

[NotFredEither]

No SLOG on either servers, just spinning disks. Why will the replication writes hit the second NAS pool twice? If all SLOG is is a write cache, then shouldn't it go to disk a single time if the cache does not exist?

In the most polite way possible, I'd suggest a re-read of the ZFS documentation before proceeding. It's crucial that one understands how the various parts of ZFS work together when trying to optimize for performance.

A good start:

24. ZFS Primer — FreeNAS®11.2-U7 User Guide Table of Contents

www.ixsystems.com

The ZFS ZIL and SLOG Demystified | TrueNAS - Welcome to the Open Storage Era

The ZIL and SLOG are two of the most misunderstood concepts in ZFS and hopefully this will clear things up

www.ixsystems.com

Aaron Toponce : ZFS Administration, Appendix A- Visualizing The ZFS Intent LOG (ZIL)

pthree.org

Da16 should not be in the raidz1, but should be in raidz1-2. When I added drives to this, I put 8 in, created the first vdev, put another 8 in, created the second, and continued with the third and fourth the same. So, da16-23 should most definitely be in raidz1-2. Is there a good way to verify this? I was unable to find one.

You can see how the disks are arranged from the GUI at Storage->Pools->Your pool->Status. Just FYI, you can certainly add vdevs like you've done, but be aware that ZFS does not re-balance data that's already been written, so while you'll gain storage space, performance gains are more complicated. I'd refer you to the documentation for a full description, e.g.

Chapter 22. The Z File System (ZFS)

ZFS is an advanced file system designed to solve major problems found in previous storage subsystem software

www.freebsd.org

Also, I disabled encryption on the replication task, and restarted it. Now speeds are up to 3.3 Gbps transfer, however, there's a lot more time between the bursts. I still have no idea why iperf says 9+ Gbps, but no transfers are ever actually above 3 +/- Gbps.

If you have hardware encryption support, the added overhead should be minimal--the documentation says "line speed". Then again, I don't have a 10Gbps line to test, so maybe you're reaching speeds where the encryption overhead becomes significant. YMMV.

I was told in other posts when using rsync that replication is faster and the better option, but for me, it's slower. This is a mess.

Replication is the correct way to do what you're wanting to do; the initial transfer is huge and time consuming, but subsequent updates are quick as long as your dataset is changing by small fractions. I think it's important to understand that ZFS's primary goal is data integrity, so replication receives are synchronous writes. The lack of a SLOG makes me suspect that you're seeing the effect of this, and I'd suggest adding one if possible to the receiving machine.