Rebuild or Start New?

[redham]

I am at cross roads right how and I am seeking some advice on my next move.

My first "network device", is a Lenovo TS440, with 8 x 3tb drives on LSI SAS-3265-8i on raid 6, 24GB ECC Memory, Xeon E3-1245v3, W7 Ultimate.

This device is primarily used for PLEX Serving, Utorrents and just network accessible drive sharing. Two problems I have been getting is my electricity was inching much high than I would like and about every 2 weeks or so one of the drives in the raid was going down. I have fiddled with drive issues more than 10 times already and went through the usual trouble shooting (i.e. reloaded drivers, even upgraded with 3265-8i with battery BU). Somehow a I/O issue seems to cause one of the drives to be thrown out of the raid. It has actually destroyed on drive before which I had to replace. I think I have too many devices connected to the MB. The OS is on a separate Transcend 1tb SSD. I can see my system just hang sometimes. I didn't want to invest too much time into it as the power consumption was heavy and causing my wife to redline about the electricity bill and I need to migrate the files off of this system first (fair notice, never on power save).

My interim system:
Intel Skull Canyon i7KYK version with WD MY book DUO 16Tb. as the Plex server. Not really a NAS, just something to keep my plex running.

Just picked up Terra Master F4-220 for $200 and loaded it with 4x8tb reds and testing this little guy.

I like the features of the little Terra Master but it is pretty cheap and this concerns me if hardware were to go down.

My remain requirements are really:
Plex serving, with some 4K transcoding. Dont really want that many 4K files stored
Utorrents
Centralized File storage to support 3 users and 5 Computers
Website hosting for my small business
Android support to back up my phone pictures.



Now that I have migrated my files off the TS440 do should I?

1. Strip down the TS440 rebuild with FreeNAS and put it on power save, I now have a total of 8 x 8tb reds but I don't want to put it on the TS440 and have it kill one of them.

2. Get a new motherboard and CPU, would like to reuse the 24 ECC memory if possible, and start a new NAS.

3. Look into more reliable NAS brand like Synology DS1019+ or DS1819+. This 4 port link aggregation is unique. Are there mb's out there that can offer 2 port aggregation for self builds?


Any comments would be welcomed. Thx

 

[Chris Moore]

After reading all this, please let us know if you have more questions. We love to help.

Somehow a I/O issue seems to cause one of the drives to be thrown out of the raid

Exactly what kind of drive are you using? That is a more likely problem that that you have too many drives attached.

Just picked up Terra Master F4-220 for $200 and loaded it with 4x8tb reds and testing this little guy.

Sorry.

I like the features of the little Terra Master but it is pretty cheap and this concerns me if hardware were to go down.

It should scare the mess out of you if you are thinking to rely on it for anything that is important.

Strip down the TS440 rebuild with FreeNAS and put it on power save

FreeNAS doesn't really do any power saving. Your electricity cost would likely be close to the same as it is the same hardware.

I now have a total of 8 x 8tb reds but I don't want to put it on the TS440 and have it kill one of them.

The server is not the problem. It is more likely that the drives you had before were the problem, but any hard drive can die at any time without warning, which is the reason we use some form of array to give us protection from drive failure.

Get a new motherboard and CPU, would like to reuse the 24 ECC memory if possible, and start a new NAS.

Nothing wrong with the IBM. Did you have it on a UPS? Power fluctuations can cause all sorts of problems, but that server is more than capable of running an array of eight drives. You can attach hundreds of drives to a server through SAS, depending on the controller, as many as 1024 drives on a single controller card.

Look into more reliable NAS brand like Synology DS1019+ or DS1819+.

This question is kind of like going to a Honda dealer and asking what kind of Toyota they recommend.
There is nothing particularly good or reliable about Synology. The file system they use is actually not as good as the ZFS file system that FreeNAS uses.

This 4 port link aggregation is unique.

There is nothing remotely unique about link aggregation and it is not unique to Synology. There are many, many server system boards that offer two or four ports of integrated network connectivity, but even if they didn't, a good server will have expansion card slots that would allow you to add connectivity. Total non-issue, as long as you make good hardware choices.

Please review this reading list and learn about ZFS, FreeNAS and the hardware we suggest using with it. If you want to use the IBM server you have, you would need to ditch that hardware RAID controller in favor of a SAS HBA because FreeNAS does software RAID through the ZFS file system.

Terminology and Abbreviations Primer
https://forums.freenas.org/index.php?threads/terminology-and-abbreviations-primer.28174/

Slideshow explaining VDev, zpool, ZIL and L2ARC
https://forums.freenas.org/index.ph...ning-vdev-zpool-zil-and-l2arc-for-noobs.7775/

Why not to use RAID-5 or RAIDz1
https://www.zdnet.com/article/why-raid-5-stops-working-in-2009/

Hardware Recommendations Guide Rev. 1e) 2017-05-06
https://forums.freenas.org/index.php?resources/hardware-recommendations-guide.12/

Hardware Recommendations by @cyberjock - from 26 Aug 2014 - and still valid
https://forums.freenas.org/threads/hardware-recommendations-read-this-first.23069/

Don't be afraid to be SAS-sy
https://forums.freenas.org/index.php?resources/don't-be-afraid-to-be-sas-sy.48/

Confused about that LSI card? Join the crowd ...
https://www.ixsystems.com/community/threads/confused-about-that-lsi-card-join-the-crowd.11901/

The 'Hidden' Cost of Using ZFS for Your Home NAS
https://louwrentius.com/the-hidden-cost-of-using-zfs-for-your-home-nas.html

Building, Burn-In, and Testing your FreeNAS system
https://forums.freenas.org/index.php?resources/building-burn-in-and-testing-your-freenas-system.38/

Github repository for FreeNAS scripts, including disk burnin
https://forums.freenas.org/index.ph...for-freenas-scripts-including-disk-burnin.28/

solnet-array-test (for drive / array speed) non destructive test
https://forums.freenas.org/index.php?resources/solnet-array-test.1/

Useful Commands
https://forums.freenas.org/index.php?threads/useful-commands.30314/#post-195192

 

[Chris Moore]

PS. If you want to give FreeNAS a try, this is the SAS controller I suggest:
https://www.ebay.com/itm/HP-H220-6G...0-IT-Mode-for-ZFS-FreeNAS-unRAID/162862201664
That vendor provides a quality product that they test for proper operation prior to shipping. I have bought from them myself.

No, you can't use the hardware RAID controller you have.

 

[redham]

TY for the reply CM. Whoops somehow I double quoted. Responses below.

After reading all this, please let us know if you have more questions. We love to help.

Exactly what kind of drive are you using? That is a more likely problem that that you have too many drives attached.

Exactly what kind of drive are you using? That is a more likely problem that that you have too many drives attached.

The drives are a mixed bag of drives. Some I have already and some new. At the moment they are 2 Green, 2 Seagate (I know... didn't went to spend the money the 2 greens with 2 other greens), 3 reds and 1 black. I have these 8 drives connected to the LSI board. The Boot os drive, CDRW and hot swap bay are attached to the MB directly.

Drive fails have been going on for a year already. I just didn't want to plow more money into it, especially if I was going to migrate to 8tb drives.
I dirty deduction deduction is that: 1. I have too many devices attached to the MB with some kind of conflict. 2. Bad OS SSD drive that is causing the system to stall for a fraction of a second causing the raid to destabilize 3. One or two crappy drives in raid causing the system to hang for a split second? Is this possible?

It should scare the mess out of you if you are thinking to rely on it for anything that is important.

I needed a cheap interim solution to help me move forward with moving my data. It is better than the WD my book duos I have.

FreeNAS doesn't really do any power saving. Your electricity cost would likely be close to the same as it is the same hardware.

Are there not power management feature like sleep modes? Drive sleep?

Nothing wrong with the IBM. Did you have it on a UPS? Power fluctuations can cause all sorts of problems, but that server is more than capable of running an array of eight drives. You can attach hundreds of drives to a server through SAS, depending on the controller, as many as 1024 drives on a single controller card.

I stopped using UPSs years ago. It actually cost me more money in electricity to keep those things humming. I may consider getting a small one again just for this server as I have noticed than I have been getting the occasional brownouts. Riddle me this, I live 10 blocks away from the municipal facility.

This question is kind of like going to a Honda dealer and asking what kind of Toyota they recommend.
There is nothing particularly good or reliable about Synology. The file system they use is actually not as good as the ZFS file system that FreeNAS uses.

Reliable was a bad choice of wording. Perhaps a simpler plug and play solution was what I was looking for.

Please review this reading list and learn about ZFS, FreeNAS and the hardware we suggest using with it. If you want to use the IBM server you have, you would need to ditch that hardware RAID controller in favor of a SAS HBA because FreeNAS does software RAID through the ZFS file system.

If I can reuse the Ts440 that would be ideal for now. I will order the SAS HBA card you recommended today since it is cheap. The main reasons I would like to stay with them are 1. chassis and drive cradles are pretty nice 2. I have 2 of them (one just for replacement parts) 3. If I could find a way to swap / upgrade the MB and CPU in these it would be great but I think the connectors and mounts are proprietary.

I did find your posting on a recommended build with 512gb ram build which looks very appealing. I will try to shorten the learning curve with setting up a stable system first before i plunge bigger ticket items.


Terminology and Abbreviations Primer
https://forums.freenas.org/index.php?threads/terminology-and-abbreviations-primer.28174/

Slideshow explaining VDev, zpool, ZIL and L2ARC
https://forums.freenas.org/index.ph...ning-vdev-zpool-zil-and-l2arc-for-noobs.7775/

Why not to use RAID-5 or RAIDz1
https://www.zdnet.com/article/why-raid-5-stops-working-in-2009/

Hardware Recommendations Guide Rev. 1e) 2017-05-06
https://forums.freenas.org/index.php?resources/hardware-recommendations-guide.12/

Hardware Recommendations by @cyberjock - from 26 Aug 2014 - and still valid
https://forums.freenas.org/threads/hardware-recommendations-read-this-first.23069/

Don't be afraid to be SAS-sy
https://forums.freenas.org/index.php?resources/don't-be-afraid-to-be-sas-sy.48/

Confused about that LSI card? Join the crowd ...
https://www.ixsystems.com/community/threads/confused-about-that-lsi-card-join-the-crowd.11901/

The 'Hidden' Cost of Using ZFS for Your Home NAS
https://louwrentius.com/the-hidden-cost-of-using-zfs-for-your-home-nas.html

Building, Burn-In, and Testing your FreeNAS system
https://forums.freenas.org/index.php?resources/building-burn-in-and-testing-your-freenas-system.38/

Github repository for FreeNAS scripts, including disk burnin
https://forums.freenas.org/index.ph...for-freenas-scripts-including-disk-burnin.28/

solnet-array-test (for drive / array speed) non destructive test
https://forums.freenas.org/index.php?resources/solnet-array-test.1/

Useful Commands
https://forums.freenas.org/index.php?threads/useful-commands.30314/#post-195192

 

[redham]

I assume the HP H220 board can be used with your 512GB RAM build?

One correction. I mis-typed the raid card I have. it is a LSI-SAS 9265-8i

 

[Chris Moore]

I assume the HP H220 board can be used with your 512GB RAM build?

Sure. No problem.

 

[IQless]

as many as 1024 drives on a single controller card.

Is this depending on the controller on the card? I thought it was 255 or something. Might be old information I run with :P

 

[Chris Moore]

Is this depending on the controller on the card? I thought it was 255 or something. Might be old information I run with :p

It depends on which controller card you are looking at.
If I recall correctly, the LSI 9207-8i chipset cards are able to support a maximum of 256 drives by way of SAS Expanders.
I have a server at work that has a LSI 9305-16i 12Gb/s SAS HBA and it is able to connect up to 1024 devices.
It is just a matter of what hardware you throw at it, but the system board doesn't dictate the number of drives you can use.

 

[redham]

Sure. No problem.

Looks like I will be "Start New" just pulled the trigger on the MB, RAM, CPU and Cooler for about $340

The rest of the components, I think I should be able to salvage the rest out of my TS440. Maybe I can fit the SM board into the Lenovo case.

In any case I will return the Terra Master and recoup some of the $200.00.

This seems to be much more sensible value for what I am getting. Now I will have work with the power consumption and learning curve on setting up the software.

 

[redham]

Sure. No problem.

Still working on parts to come in form my new build. I did get the motherboard in and noticed that there are no USB 3.o ports.

Will this board work? Or would you have another board you could suggest. Drivers a can be big PITA with these little boards and I just wanted to make sure I get one that will work.

https://www.amazon.com/dp/B00FPIMJE...h_pd_luc_ac_nb_02_01_t_img_lh#customerReviews

 

[Chris Moore]

I did get the motherboard in and noticed that there are no USB 3.o ports.

Why do you want USB 3.0?

 

[redham]

Why do you want USB 3.0?

Take advantage of USB 3.0 external drives for backup. Direct computer connection for faster xfer rates.

 

[Chris Moore]

Take advantage of USB 3.0 external drives for backup. Direct computer connection for faster xfer rates.

Different people have tried to connect hard drives by USB 3.0 since the introduction of the technology, including me, and it has never worked reliably with FreeNAS. If you want a backup, don't use USB, get a hot-swap drive bay like this:
https://www.ebay.com/itm/5-25-Tray-Less-Sata-Hot-Swap-Hard-Drive-Bay/132998742657
Then plug the drive direct to a SATA or SAS port in the server. USB is just asking for problems.

 

[redham]

Finally all the parts and the time to put the new server. I seem to be having a bit of an issue loading Freenas and Mirror boot drives.

The major components are based off Chris Moore High end build in the Resources section.

I went through the mother board bios (raid config) and configured 2 x 60 ssd's in mrror. The raid was configured correctly.
Booted Freenas via cd rom and it gave me options to load to Drive 0, Drive 1 or RAID (for some reason doesn't show the name that i gave the array when I set up. Free loads completely and then prompts for a reboot. On reboot it will not recognize the RAID Mirror. When I enter the RAID config, the array is in "Rebuild" status. The legend on the bottom states that a "Rebuild" status will be rebuild by the OS. This doesn't make sense as it will no boot to the OS.

Update:
Pulled the drives out reformatted them on a W7 with GPT. Ran through the raid setup and the Frenass loading again. I am now getting a GPT partition table error on Secondary drive. Switching out of RAID now and just going to try to load on one SSD to just get the system up and running.

Another Update:

Loaded onto one SSD... worked. No drive OS drive mirroring though :-(

 

[Apollo]

I doubt you are supposed to create the RAID array for the boot drives within the BIOS.
I think you need to set them to AHCI and once booting of the install media, you will be asked how many disk you want to allocate for Freenas. That's when you will be given the option of selecting mirror.

The reason you need OS support to rebuild the array is most likely because you have a software based RAID controller and it requires a CPU to perform the calculation.

 

[Chris Moore]

I went through the mother board bios (raid config) and configured 2 x 60 ssd's in mrror. The raid was configured correctly.

As Apollo said, you are not supposed to use any hardware RAID with FreeNAS, not even for the boot pool. FreeNAS creates a ZFS mirror on the boot media and the hardware RAID you have enabled is creating a problem.

Never use any hardware RAID of any kind with FreeNAS.

 

[Chris Moore]

Loaded onto one SSD... worked. No drive OS drive mirroring though :-(

Here is a step-by-step guide that should help you with configuring FreeNAS:

Uncle Fester's Basic FreeNAS Configuration Guide
https://www.familybrown.org/dokuwiki/doku.php?id=fester:intro

 

[Chris Moore]

Booted Freenas via cd rom

Also, you should be using the IPMI management interface, not a physical CD-drive. Please review this video for more information:

https://www.youtube.com/watch?v=cPv_ikws7A0

 

[redham]

Also, you should be using the IPMI management interface, not a physical CD-drive. Please review this video for more information:

https://www.youtube.com/watch?v=cPv_ikws7A0

TY Apollo and CM for responding.

Just to give you the whole picture of where I am at on this build.

1. Re-purposed one of my TS440 cases because of the 8 bay drive holders.
2. Installed a Corsair New (old) HX1000W I had laying around (giving me problems, but that's another story)
3. Got all the header led's and power buttons working, except for stupid thermal? connector.
4. Added 4 port USB(non3.0) header board and downgraded the front 3.0 USB headers and used a pin converter to connect to the MB
5. Re-purposed the 2 SAS cables that came with TS440, Lenovo part 03x3988, each cable plugs right into the 4 bay back planes
6. CD rom and 2 60gb ssd's are connect directly to the MB, Sata mode AHCI
7. Created a bootable image via instructions from download link:
https://www.ixsystems.com/documentation/freenas/11.2/freenas.html
8. Loaded FreeNas with by booting up via CD rom
9. Selected ada0 60gb drive (SSD drive connected to MB)
10. Set root password
11. Selected Boot via BIOS (The MB supports UEFI, should I selected Boot via UEFI instead?)
12. Server is boots and loads to Console setrup screen (physical monitor connected), I am able to remote in via IP address as well

CM, If I understand you correctly, I need get the 2 ssd's onto the HP card? Which would mean getting another sas cable to have on one of the raid ports to connect to both backplanes and another to connect to the 2 ssds' and the bare drive removable drivebay.

When figure out my PSU problem, I would like to assign as static ip address. At the stage of where I am, can I go through bios, IMPI and enter a static IP address as per your youtube link?

 

[Chris Moore]

CM, If I understand you correctly, I need get the 2 ssd's onto the HP card?

No.
They connect to the SATA ports on the system board, you just don't configure hardware RAID. It needs to be set to AHCI mode like @Apollo said.
FreeNAS handles the mirror in software.

(The MB supports UEFI, should I selected Boot via UEFI instead?)

No.