Is ZFS a slow FS?

[danb35]

Wow this tutorial is awesome! The person who wrote it must be a smart guy. Any idea who it was?

It was, as the landing page for that guide says, (former) user @UncleFester here, who hasn't been seen for a couple of years. I just had the idea to put it on a Wiki, since his idea was that it would be edited and updated by the community.

 

[osharko]

At least in the past the built-in switch of a FritzBox was not fast. I remember not being able to get 1 Gbps, although that was probably about 10 years ago. Can you try another one? Any cheap unmanaged switch from e.g. Netgear should do.

I got three test of transfer over LAN, passing over 3 fritzbox nodes. The tests hasn't overcome the 110MB/s (~900mpbs) but I think it's a reasonable speed. Bye the way, here are the results:

- FIRST with a 6GB film transfer:

Code:

Results obtained by executing the following copy from My Host to NAS:
rsync --progress -av osharko@Koinusame:/data/Data/Multimedia/BigFile ./BigFile

Show : SSD

receiving incremental file list
BigFile
6,488,831,438 100% 110.67MB/s 0:00:55 (xfr#1, to-chk=0/1)

sent 43 bytes received 6,490,415,731 bytes 90,775,045.79 bytes/sec
total size is 6,488,831,438 speedup is 1.00

Show : MIRRORED POOL (while some qbittorent moving operation in progress)

receiving incremental file list
BigFile
6,488,831,438 100% 110.97MB/s 0:00:55 (xfr#1, to-chk=0/1)

sent 43 bytes received 6,490,415,731 bytes 90,775,045.79 bytes/sec
total size is 6,488,831,438 speedup is 1.00

Show : SLOW POOL (qbittorent active but with no internet connection)

receiving incremental file list
BigFile
6,488,831,438 100% 110.49MB/s 0:00:56 (xfr#1, to-chk=0/1)

sent 43 bytes received 6,490,415,731 bytes 114,874,615.47 bytes/sec
total size is 6,488,831,438 speedup is 1.00

Show : ZRAID1 POOL not tested due to badblocks test in progress

N/A

- SECOND with a 50GB file transfer created with dd**:

Code:

Results obtained by executing the following copy from My Host to NAS:

rsync --progress -av osharko@Koinusame:/data/Data/Multimedia/50GB ./50GB

Show : SSD

receiving incremental file list
50GB
53,687,091,200 100% 111.06MB/s 0:07:40 (xfr#1, to-chk=0/1)

sent 43 bytes received 53,700,198,495 bytes 112,697,163.77 bytes/sec
total size is 53,687,091,200 speedup is 1.00

Show : MIRRORED POOL (while some qbittorent moving operation in progress)

receiving incremental file list
50GB
53,687,091,200 100% 110.79MB/s 0:07:42 (xfr#1, to-chk=0/1)

sent 43 bytes received 53,700,198,495 bytes 112,461,148.77 bytes/sec
total size is 53,687,091,200 speedup is 1.00

Show : SLOW POOL (qbittorent active but with no internet connection)

receiving incremental file list
50GB
53,687,091,200 100% 111.07MB/s 0:07:40 (xfr#1, to-chk=0/1)

sent 43 bytes received 53,700,198,495 bytes 112

Show : ZRAID1 POOL not tested due to badblocks test in progress

N/A

- THIRD with the previous 50GB file transfered beetween 2 pools:

Show : SLOW TO MIRRORED

sending incremental file list
50GB
53,687,091,200 100% 1.40GB/s 0:00:35 (xfr#1, to-chk=0/1)

sent 53,700,198,503 bytes received 35 bytes 1,471,238,316.11 bytes/sec
total size is 53,687,091,200 speedup is 1.00

Show : MIRRORED TO SLOW

sending incremental file list
50GB
53,687,091,200 100% 1.43GB/s 0:00:34 (xfr#1, to-chk=0/1)

sent 53,700,198,503 bytes received 35 bytes 1,512,681,648.96 bytes/sec
total size is 53,687,091,200 speedup is 1.00

** dd command to create the big file

Code:

sudo dd if=/dev/zero of=/data/Data/Multimedia/50GB bs=25M count=2048 status=progress

Conclusion?

• Moving data from pc to nas, over ethernet, is fast as expected;
• Moving data inside NAS is too much fast that it become impossible to understand if it's a cache trick or other (I'll say that no harddisk can handle 1.3GB/s, expecially not a sata3 disk);
• Moving torrent with qbittorent is insanely slow. So now we may be sure that the title is terribly wrong and the problem is not about ZFS, but something about qbittorent? Dunno.

If someone knows a possible reason, it would be great.

 

[Stux]

The 50GB file is all zeros. It compresses to almost nothing.

Use dev/random instead of dev/zero

 

[chuck32]

• Moving data inside NAS is too much fast that it become impossible to understand if it's a cache trick or other (I'll say that no harddisk can handle 1.3GB/s, expecially not a sata3 disk);

Writing to RAM is always very fast and may exceed the HDD speeds.

• Moving torrent with qbittorent is insanely slow.

Speed on the formerly slowest drive seems to have improved as per your latest results. What do you mean by moving with torrent? I thought you also posted slow transfer speeds that used the pool but not qbittorrent?

All the speeds you report are pitiful, so something is clearly very wrong. The controller is one suspect. Serious misconfiguration is another (like using sync=always and/or recordsize=4k)

Are you referring to OPs test where they reached 250 MB/s copying from the 2 drive mirror to the raidz1 pool? Did you expect > 400 MB/s because the mirror should give two drives read performance and the raidz1 pool should be able to write that?

These are just 2 of the five new disk, because the long SMART is still ongoing on 3 of them:

Now badblocks is running on all the 5 disks, but I the SMART test already pointed out they're safe.

I didn't mean you should start badblocks in parallel to a long smart test ;) A smart test alone does not validate a HDD.

Do you want me to run the test on the slow disk? If yes, isn't it supposed to be dangerous?

You could run badblocks on that drive, too, but at this point I'm not sure this is the next logical step. If you mean potential data loss with dangerous then yes, you could lose data. From the link I posted:

badblocks also offers a non-destructive read-write test that (in theory) shouldn't damage any existing data, but if you do choose to run it on a production drive and suffer data loss, on your own head be it:

Personally I always ran badblocks on empty drives, so no experience there.

 

[osharko]

The 50GB file is all zeros. It compresses to almost nothing.

Use dev/random instead of dev/zero

Never used the random stream, infact I had the clue that it may use a kind of cache. Thank you.
This time I stopped all the apps before the test, and through iostat 2 checked the disks utilization, only when all the disks (except for the os's) where at 0 utilizzation I started the test. And I create 2 different file from /dev/random for the two rsync test, just to avoid some caching.

Show : SLOW HDD

receiving incremental file list
50GB
53,687,091,200 100% 78.43MB/s 0:10:52 (xfr#1, to-chk=0/1)

sent 43 bytes received 53,700,198,494 bytes 82,173,218.88 bytes/sec
total size is 53,687,091,200 speedup is 1.00

Show : MIRRORED POOL

receiving incremental file list
50GB
53,687,091,200 100% 86.22MB/s 0:09:53 (xfr#1, to-chk=0/1)

sent 43 bytes received 53,700,198,495 bytes 90,328,340.69 bytes/sec
total size is 53,687,091,200 speedup is 1.00

Speed on the formerly slowest drive seems to have improved as per your latest results. What do you mean by moving with torrent? I thought you also posted slow transfer speeds that used the pool but not qbittorrent?

I noticed the problem with qbittorrent change position feature, where I was moving the downloaded data to the Multimedia library.
Moving from the Mirrored Pool to the slow lead to 125MB/s speed and the viceversa was even faster. So I'm assuming that the OS uses cache during rsync but not when i move data with qbittorent (at this point, it's a better test).

You could run badblocks on that drive, too, but at this point I'm not sure this is the next logical step. If you mean potential data loss with dangerous then yes, you could lose data. From the link I posted:

Personally I always ran badblocks on empty drives, so no experience there.

Yep, as pointed out from the guide, it should be safe but there's no assurance.

 

[ChrisRJ]

/dev/random requires a lot of CPU cycles. Depending on the system you might be testing the CPU and not the disks.

 

[osharko]

/dev/random requires a lot of CPU cycles. Depending on the system you might be testing the CPU and not the disks.

It have been generated from my Desktop pc (32GB DDR4, Ryzen 7 5800X), the point was just create a random and uncachable file to transfer over network.

Step by step I'm moving all the file from the old slow drive to the new, and i'm noticing that the transfer is becoming faster. At this point, I think that previously the random hdd utilization was too much high and for that even the sequential transfer was compromized.

 

[osharko]

I've finally completed the migration from the old pool to the newest.
I've to say that qbittorent migration are immediate and I've found no other bottlenecks.
The only problem was the slow pool where I was leaning to much.

Thank you very much to everyone for the knoledge sharing and the help.

Really appreciate.