Offsite backup suggestions

[Chris Moore]

If I'd had this suggestion earlier I might have gone this way... I ended up with an LSI HBA...

This statement makes NO sense, you asked your question TODAY, I answered within six hours of your question. How much earlier ??
Should the answer have been provided before the question was asked?

 

[NASbox]

The thing you have here, in the RAIDZ2 with four drives, effectively, two drives are storage and two drives are redundancy, that is not very space efficient. Normally, you would want to have a minimum of five drives (obviously you can do it with four) but it works better with five or six or more up to around 10 but ZFS will let you do even more. The way you have it, this should only be giving you about 10.2TB of usable space
Then with the RAIDZ pool, effectively, one disk is data and the other is parity so it is working like a mirror. Normally with RAIDZ you would want to have three drives at the minimum and again it will let you do what you did, but that is not optimal. The way you have it, this should only be giving you about 2.6TB of usable space
You can't add more drives to an existing vdev, but you could add another vdev to an existing pool. I would suggest that if you are interested in adding more drives, that you should copy all the data off your device, destroy the existing pool and create a new pool using six 6TB drives at RAIDZ2, which would give you about 20TB of usable storage and that might allow you to totally eliminate the 3TB drives and simply create datasets inside your main storage pool for everything to be.

This would be more economical of the storage space and power consumption would be more efficient as you would be spinning two 6TB drives instead of two 3TB drives.

Thanks. I'll keep that in mind when I expand. I'll likely keep the small pool as is since it's almost offline archive. It stays spun down most of the time and I'll bet it spends more time doing monthly scrubs than actual work!

I put four 5TB drives in the external enclosure and they each show up as separate drives to FreeNAS. It works as if each drive were directly connected to a SATA port and as I said, I have been using this for years with no trouble. I created a RAIDZ1 pool on those four drives which gives me about 12TB of storage. When I connect it to make my backup I can do a zpool import just the same way you were talking about doing with a single drive. It works just the same, you just have to wait for FreeNAS to detect all four drives. Then I run my rsync from the command line that synchronizes the data on my NAS such that any changes are copied to the external pool. Once I have done all the operations that I have in mind to do, like scrubbing the pool and smart tests of the drives etc. Then I can do the zpool export and shutdown the external enclosure.

Does this use 4 SATA ports or are you using some sort of port multiplexing?
How do you cable externally to connect the drive enclosure-Just a regular eSATA cable or some special multi-channel cable of some sort?

I am confused by why you bother to ask a question when you have not only already decide on something else, but it sounds like you have already procured the hardware to implement that solution.

I might still have an application for this type of arrangement in the future (and when I finish my rebuild and document what I have done for the benefit of the community, I'll likely reference this post as well). When I was doing my original research, it was very clear there were a lot of noobs like me that were struggling with a practical home backup solution. (Enterprise use makes redundant mirroring a no brainer!) This looks like a very good "simple/workable" solution that more people should know about.

This is the same model SAS controller that I use and it works fine with hot-swapping drives.

Quick question about procedure-is this procedure correct?
Plug in hot swap carrier, wait for drive to spin up, create or import pool, do work on drive, export/unmount and remove the hot swap carrier.

Are there special bios settings required for hot swap on that controller or other FreeNAS settings required?

Thanks again for the input.

 

[Chris Moore]

Does this use 4 SATA ports or are you using some sort of port multiplexing?
How do you cable externally to connect the drive enclosure-Just a regular eSATA cable or some special multi-channel cable of some sort?

The adapter card I listed earlier in the thread and the enclosure both support port multiplier technology which allows the four drives in the external enclosure to all share the single eSATA connection to the server. Because the speed of mechanical drives is slower than SATA 6Gig speed, the connectivity is more than adequate for the drives to be fully engaged in the backup activity. It works well. It will not be properly mounted if it is connected to the server at power up. The server needs to already be on when this is connected.

Quick question about procedure-is this procedure correct?
Plug in hot swap carrier, wait for drive to spin up, create or import pool, do work on drive, export/unmount and remove the hot swap carrier.

Are there special bios settings required for hot swap on that controller or other FreeNAS settings required?

No special settings required. I have all of my drives plugged into that type of controller in a rack chassis with all hot-swap bays. I can pull a drive when I need to and put a new one in.
There is actually another thread about that topic on the board. Take a look: https://forums.freenas.org/index.ph...-mounted-in-a-hot-swap-bay.56919/#post-399710

 

[danb35]

The adapter card I listed earlier in the thread and the enclosure both support port multiplier technology which allows the four drives in the external enclosure to all share the single eSATA connection to the server.

SATA port multipliers are very strongly discouraged. SAS expanders, OTOH, are commonly recommended and work well.

 

[Chris Moore]

SATA port multipliers are very strongly discouraged

Why? The drivers are built into the base OS.

 

[danb35]

Rather than wade through all the threads, check out the "site results" tab here: https://forums.freenas.org/index.php?search/45672419/&q=sata+port+multipliers&t=post&o=date

 

[NASbox]

This statement makes NO sense, you asked your question TODAY, I answered within six hours of your question. How much earlier ??
Should the answer have been provided before the question was asked?

I can understand your confusion... I think I implied, but may not have explicitly stated that I originally started my search for a better backup methodology back in March 2017 after Coral was released as the new production version (before it was pulled in favor of v11)

I did a lot of reading through the forum and started my own thread https://forums.freenas.org/index.php?threads/new-backup-options-for-freenas-corral.51600/ which led me to the solution that I finally landed on. I read various posts (can't quote references now) that said that most of the cheap HBAs were a problem waiting to happen, and AFAICR that port multipliers didn't work well in FreeNAS either as danb35 just mentioned:

SATA port multipliers are very strongly discouraged. SAS expanders, OTOH, are commonly recommended and work well.

However, as is the case with forums, while they can be a great knowledge base, they also get out of date, and there is no "garbage collection" process to remove obsolete stuff and incomplete information can sometimes make it hard to understand what is actually happening. You have managed to find something that does work reliably so it's worth documenting clearly for the benefit of others. Your caveat about booting the server before plugging in the backup box is an important part of the equation.

As an aside (if anybody cares):
Since I ended up buying an LSI HBA and needing a large case, I decided that it only made sense to offload some tasks to the FreeNAS box. In round numbers (CDN-Multiply by .667 for approximate USD at time of purchase-Canada is more expensive, and choice is also a bit restricted) $200 for the large case with extra hard drive cages, $100 for the LBA and $200 in mounting hardware for 4 backup drives (About 25% more than a well made USB enclosure). This new configuration can accommodate 14 3.5" drives, the removable drive and 4 SSDs. (I would need another HBA or Motherboard to expand beyond a total of 14 drives, but I don't anticipate doing that any time soon.) Another $100 for a i7 and $15 for a 2 port intel NIC (used from Kijijii) should be pleanty to run Own/NextCloud, some web based CRM & Invoicing software and maybe some other things that I haven't yet thought of inside my own firewall.)

Complements to The FreeNAS Devs - they know what they are doing and do things "right" rather than just doing things-been super stable and reliable since 2012 which is why I'm still on 8.3. Looking forward to finally catching up and getting some of the new features in 11+.

 

[Chris Moore]

Rather than wade through all the threads, check out the "site results" tab here: https://forums.freenas.org/index.php?search/45672419/&q=sata+port+multipliers&t=post&o=date

Edit: Original post was Aug 11, 2017; by May 21018, this solution was not working. See later posts in the thread.

You might want to look through those results yourself. Many of the problems people had were down to hardware that was not properly compatible. I did good research and the hardware I selected is compatible and it has been working for me for several years. I have it plugged in and running right now. Initially it was how I expanded my pool beyond the internal drives I could put inside a desktop computer. Once upon a time, it even worked properly through a reboot, but when they went from BSD 9x to 10x something changed in the way the OS loads. It appears initializes the drives and even shows all the external drives as detected and ready in the log, then it resets the interface and tries to detect the drives again. This causes the external port multiplier to go offline. It must be power cycled after whatever it is that the OS does. This is why the enclosure can't be connected at boot. The port multiplier only works on first connect, not on a reconnect without a reset. So, to make it work, the system must be booted, then the external enclosure powered on, then the connection made so that it is only initialized once.
I had to do a horrible amount of work on my own to figure this out because everyone just says it is not supported, but it is part of the SATA interface standard. The only way it is not supported is if BSD is doing something wrong in their implementation of the hardware driver. This hardware (by the way) works in Linux and Windows too. Any operating system that properly implements the SATA interface supports this and to say that it isn't supported is like saying that electricity will only flow through a wire if it is coated in red plastic. The problems that people started having with this technology is because of some change that was made in how the underlying OS initializes devices or because the hardware they had was not properly implementing the technology. When I first started using FreeNAS, it was perfectly reliable and remained so until they changed the OS and it is still reliable as long as it is not connected when FreeNAS boots. Honestly, it could be something that FreeNAS is doing because I have not tried it with a plain vanilla installation of BSD. It does work perfectly in Linux though, so it isn't the hardware.
Also, there is this: https://forums.freebsd.org/threads/47548/#post-265616
and this: https://forums.freebsd.org/threads/58725/#post-335883

 

[Chris Moore]

You might want to look through those results yourself. Many of the problems people had were down to hardware that was not properly compatible. I did good research and the hardware I selected is compatible and it has been working for me for several years. I have it plugged in and running right now. Initially it was how I expanded my pool beyond the internal drives I could put inside a desktop computer. Once upon a time, it even worked properly through a reboot, but when they went from BSD 9x to 10x something changed in the way the OS loads. It appears initializes the drives and even shows all the external drives as detected and ready in the log, then it resets the interface and tries to detect the drives again. This causes the external port multiplier to go offline. It must be power cycled after whatever it is that the OS does. This is why the enclosure can't be connected at boot. The port multiplier only works on first connect, not on a reconnect without a reset. So, to make it work, the system must be booted, then the external enclosure powered on, then the connection made so that it is only initialized once. I had to do a horrible amount of work on my own to figure this out because everyone just says it is not supported, but it is part of the SATA interface standard. The only way it is not supported is if BSD is doing something wrong in their implementation of the hardware driver. This hardware (by the way) works in Linux and Windows too. Any operating system that properly implements the SATA interface supports this and to say that it isn't supported is like saying that electricity will only flow through a wire if it is coated in red plastic. The problems that people started having with this technology is because of some change that was made in how the underlying OS initializes devices or because the hardware they had was not properly implementing the technology. When I first started using FreeNAS, it was perfectly reliable and remained so until they changed the OS and it is still reliable as long as it is not connected when FreeNAS boots. Honestly, it could be something that FreeNAS is doing because I have not tried it with a plain vanilla installation of BSD. It does work perfectly in Linux though, so it isn't the hardware.
Also, there is this: https://forums.freebsd.org/threads/47548/#post-265616
and this: https://forums.freebsd.org/threads/58725/#post-335883

Almost a year later and I think I should provide an update on this. I don't use a SATA port multiplier any more. I don't know why, but it stopped working reliably. I have another solution for my backup pool now.

 

[NASbox]

Almost a year later and I think I should provide an update on this. I don't use a SATA port multiplier any more. I don't know why, but it stopped working reliably. I have another solution for my backup pool now.

So what do you do now.

 

[Chris Moore]

So what do you do now.

I have a larger chassis and the backup pool is now in the same chassis with the main pool. I don't know if there was a fault with the external enclosure or if there was some change to the operating system, but the backup pool was having problems staying online. Disks were dropping out and the pool would become degraded. I put those drives internal and they have been perfectly reliable for a couple months. I am thinking it was the external enclosure, but I don't know.

 

[pro lamer]

EDIT: Some of the below is obsolete as Chris mentioned later.

The Rosewill product says it supports drives up to 5TB so if you think you need bigger drives, you might try this product which says it is tested with drives up to 6TB.

https://www.newegg.com/Product/Product.aspx?Item=N82E16817707364
I have not used the StarTech unit but it looks very similar except I believe that it just has slots for the drives with no need to use trays.

The interface card I use is this: https://www.newegg.com/Product/Product.aspx?Item=N82E16816124063

This is good and bad news then ;) Or rather: very good and a bit bad. Good because there is a card known to work, thanks to Chris. Bad because if one wants to use bigger HDDs then they need to test other enclosures on their own. And good again - at least they are halfway through checking their eSATA solution (since they know the card is compatible with FreeNAS/FreeBSD).

Will the HDDs burn-in tests found in our forums (https://forums.freenas.org/index.php?resources/hard-drive-burn-in-testing.92/ or https://forums.freenas.org/index.php?resources/solnet-array-test.1/) apriopriate in terms of enclosures tests? Should they be run for longer time to test the enclosure better? Or some other types of tests would be recommended?

 

[Chris Moore]

This is good and bad news then ;) Or rather: very good and a bit bad. Good because there is a card known to work, thanks to Chris. Bad because if one wants to use bigger HDDs then they need to test other enclosures on their own. And good again - at least they are halfway through checking their eSATA solution (since they know the card is compatible with FreeNAS/FreeBSD).

Will the HDDs burn-in tests found in our forums (https://forums.freenas.org/index.php?resources/hard-drive-burn-in-testing.92/ or https://forums.freenas.org/index.php?resources/solnet-array-test.1/) apriopriate in terms of enclosures tests? Should they be run for longer time to test the enclosure better? Or some other types of tests would be recommended?

I made another comment on this subject after you liked my post yesterday. The situation has changed for me. I am not using this solution at this time because it stopped working. Times change. I don't know if it stopped working because of some change to the operating system or if my hardware became defective after years of service. Either way, it was dropping drives offline and the backup pool went degraded one week and a couple weeks later two drives dropped out at once taking the pool completely offline. I pulled the drives and put them internal in my server and the drives are working fine. I have not done testing with the enclosure to see if the fault was within the hardware of the enclosure. Perhaps I can do some testing later this week. Still, I can't recommend this solution.

There are other ways to connect an external enclosure, but they would likely be more expensive. What are you looking to do?

 

[pro lamer]

What are you looking to do?

I'm considering a future backup mirror drives attachment - carrying both of them in one enclosure (hopefully some smaller one) might be convenient (or not. Depending on the shape of a bag ;) )
or at least the enclosure will take SATA connectors wear.

Perhaps I can do some testing later this week.

To me it's not a hurry but would be grateful in future (it may help my future plans).

On the other hand I've come across a new thread about an interface card. I haven't checked if their card supports port multiplying but I think they may like some ideas on burn in testing a card too... not sure how to refer to the threads each other yet. Maybe I should move one of my questions there (the question about burn in testing a card) since that thread seems more appropriate. By moving I mean a strikethrough here and posting there - wouldn't you mind? (I am new to forums in general and don't know what solution would be kind or rude in this case).

if the fault was within the hardware of the enclosure

Exactly, sth might have happened to the card, cabling (or software might have changed - as you mentioned already).