Hey,
I've been troubleshooting by googling and researching here for a couple of weeks, trying not to be 'that guy'. I've made a little progress, but need a hand!
I've had a 4-bay Drobo for a while now that it's time to replace. It has recently given me some heard stopping moments ("All your drives are broken!" red lights, to which the solution was to physically disassemble the thing, optimistically blow out some dust, put it back together again and it miraculously works - not the most confidence-inspiring thing).
So, I was after a system that I can leave on, and automate back up to - Time Machine & Full-disk backups from Macs, photos, videos, etc etc. Ultimately I'd also like to get similar set-ups for the rest of my family so we all have off-site backup with encrypted replication between sites so that the data is safe, but there's no nosing in each others' stuff because one needs boundaries. But that's for later! I want to get my setup working first...
Full of optimism, I picked up:
First 'attempt' - 13-25 MB/s speeds
I installed TrueNAS on the mSATA drive on the internal USB 2 port, before putting any other drives in. This seems of note as it asked me if I wanted to create a swap partition to which my fresh eyes said 'of course, that sounds like a good idea!'. I then installed the rest of the drives, created a raidz1 pool with the IronWolf drives and left the SSD unconfigured.
It worked - I could copy, but only at 25MB/s. I poked about, disabling compression, atime and encryption, trying autotune - all with no effect.
Trying iperf gave me ~950 Mb/s (so theoretically ~115 MB/s) and 'dd' ing a 30GB file generated from /dev/random between the SSD and the main pool gave me 250 MB/s each way with 'zpool iostat'. It spiked to 400 going to the main pool, but I guess that's parity or some such.
At this point I discovered that two of the bays are SATA3, the other two and the ODD port are SATA2. Whilst not outstanding, it's ~SATA2 speed and perfectly in-line with what I hope the link-aggregated 1GbE connections will be, so all good there.
I tried SMB, I tried SCP, I tried AFP - all gave me 25MB/s. Hurrumph.
There was a little grasping at straws here and noticed that the USB mSATA drive was HOT. I saw a comment about swap. 25MB/s seemed suspiciously near half the throughput of a USB 2.0 link (25MB/s write + 25MB/s read ~= 480Mb/s (/ 8 ~=60MB/s). Was it copying everything to swap on the slow boot drive and then copying it to the pool? [I realise this is probably not how ZFS works and it is maybe a coincidence, but...]
[NB at this point reinstalling was faster than learning about managing swap on FreeBSD. Hey, my wife is expecting me to lay the new floor - there's only so much time in the world!]
I reinstall (with the Ironwolf drives already in), *it doesn't ask me about swap this time* and lo and behold...
Second 'attempt' - 45 MB/s speeds
Squinting a bit, that's double - and half a gigabit link. Check me out, troubleshooting all on my own. Still a way short, though. Clearly TrueNAS or ZFS don't like swap on a slow SSD! Or something.
Since then I've been tinkering to get up up towards the 100 MB/s mark. It hasn't budged. I've tried using NFS, WebDav, AFP, an SCP copy, SMB. I've tried autotune. The highest my CPU seems to go is about 15% utilisation. My ram usage varies between 3.6 and 8GB (of the 16 total) for services, but not more. The 'iperf' and 'dd' / 'pool iostat' results are the same.
Maybe that's okay and I just live with that. No need for link aggregation. It just seems a bit... weak. I'm sure it'll give me all the reliability and simple replication stuff that I need. I just don't have that inner glow of 'Cool, that's sorted!'.
Any ideas?
(NB I've also tried the Samsung SSD that (I think) I configured as an SLOG drive - and that made no difference, so I removed it and it's sitting unused. As I don't seem to be able to boot from that SATA, I'll probably remove it & use it for something else, unless you guys have any ideas!)
I've been troubleshooting by googling and researching here for a couple of weeks, trying not to be 'that guy'. I've made a little progress, but need a hand!
I've had a 4-bay Drobo for a while now that it's time to replace. It has recently given me some heard stopping moments ("All your drives are broken!" red lights, to which the solution was to physically disassemble the thing, optimistically blow out some dust, put it back together again and it miraculously works - not the most confidence-inspiring thing).
So, I was after a system that I can leave on, and automate back up to - Time Machine & Full-disk backups from Macs, photos, videos, etc etc. Ultimately I'd also like to get similar set-ups for the rest of my family so we all have off-site backup with encrypted replication between sites so that the data is safe, but there's no nosing in each others' stuff because one needs boundaries. But that's for later! I want to get my setup working first...
Full of optimism, I picked up:
- an HP Microserver Gen8 on eBay that had 16GB of ECC ram (it had a Celeron, I replaced it with a Xeon E3-1220L).
- 4x shiny new 4TB Ironwolf drives (balancing ambition against being in trouble with my other half!).
- 1x 512GB Samsung SATA SSD, and some kit to install it in the optical bay ("I'm not 100% sure it will be useful, but I can use it in something else if it isn't".
- a usb -> mSATA adapter so I can boot from a 120GB Kingston mSATA drive I have, with the internal **USB 2.0** port.
First 'attempt' - 13-25 MB/s speeds
I installed TrueNAS on the mSATA drive on the internal USB 2 port, before putting any other drives in. This seems of note as it asked me if I wanted to create a swap partition to which my fresh eyes said 'of course, that sounds like a good idea!'. I then installed the rest of the drives, created a raidz1 pool with the IronWolf drives and left the SSD unconfigured.
It worked - I could copy, but only at 25MB/s. I poked about, disabling compression, atime and encryption, trying autotune - all with no effect.
Trying iperf gave me ~950 Mb/s (so theoretically ~115 MB/s) and 'dd' ing a 30GB file generated from /dev/random between the SSD and the main pool gave me 250 MB/s each way with 'zpool iostat'. It spiked to 400 going to the main pool, but I guess that's parity or some such.
At this point I discovered that two of the bays are SATA3, the other two and the ODD port are SATA2. Whilst not outstanding, it's ~SATA2 speed and perfectly in-line with what I hope the link-aggregated 1GbE connections will be, so all good there.
I tried SMB, I tried SCP, I tried AFP - all gave me 25MB/s. Hurrumph.
There was a little grasping at straws here and noticed that the USB mSATA drive was HOT. I saw a comment about swap. 25MB/s seemed suspiciously near half the throughput of a USB 2.0 link (25MB/s write + 25MB/s read ~= 480Mb/s (/ 8 ~=60MB/s). Was it copying everything to swap on the slow boot drive and then copying it to the pool? [I realise this is probably not how ZFS works and it is maybe a coincidence, but...]
[NB at this point reinstalling was faster than learning about managing swap on FreeBSD. Hey, my wife is expecting me to lay the new floor - there's only so much time in the world!]
I reinstall (with the Ironwolf drives already in), *it doesn't ask me about swap this time* and lo and behold...
Second 'attempt' - 45 MB/s speeds
Squinting a bit, that's double - and half a gigabit link. Check me out, troubleshooting all on my own. Still a way short, though. Clearly TrueNAS or ZFS don't like swap on a slow SSD! Or something.
Since then I've been tinkering to get up up towards the 100 MB/s mark. It hasn't budged. I've tried using NFS, WebDav, AFP, an SCP copy, SMB. I've tried autotune. The highest my CPU seems to go is about 15% utilisation. My ram usage varies between 3.6 and 8GB (of the 16 total) for services, but not more. The 'iperf' and 'dd' / 'pool iostat' results are the same.
Maybe that's okay and I just live with that. No need for link aggregation. It just seems a bit... weak. I'm sure it'll give me all the reliability and simple replication stuff that I need. I just don't have that inner glow of 'Cool, that's sorted!'.
Any ideas?
(NB I've also tried the Samsung SSD that (I think) I configured as an SLOG drive - and that made no difference, so I removed it and it's sitting unused. As I don't seem to be able to boot from that SATA, I'll probably remove it & use it for something else, unless you guys have any ideas!)