Large 70+TB CIFS/NFS File Server Advice

[clahti]

Hello All:

I am building two large CIFS/NFS file servers, they will both have active volumes that will be replicated to one another (cross-replication), Here is my initial design ideas:

Dell PowerEdge 2950
Two Dual Core 2.4GHz CPU
32GB RAM
6X2TB 7200RPM SAS drives RAID 5
Two MD1200 Disk Arrays with 24X3TB 7200RPM SAS Drives RAID 5
10G Network

My Intention is to expose the 36TB as CIFS storage and the 10TB as NFS storage. This will be used in a mixed Windows/Linux environment. My questions are:

• is that enough RAM?
• hardware or software RAID?
• would I benefit from an SSD for dedicated ZIL? How large?
• would Quad Cord processors be better?
• what 10G NIC's are supported under FreeNAS?
  

Any additional performance tips would be greatly appreciated. I know that the 2950 servers are old, but I have lots of them. So if money were reasonably no object what would you do? I already own the MD1200 array shelves, 2950 servers. I would need to buy the drives. Is there any reason to use SAS over SATA with this many spindles? Also I cannot tell if OZC RevoDrive PCI express SSD works under FreeNAS, that would be the IDEAL cache drive I think, here is the link http://www.amazon.com/dp/B005F30IYU/?tag=ozlp-20

I am wondering if these would perform well WD Red 3TB http://www.newegg.com/Product/Product.aspx?Item=N82E16822236344 they are certainly cheap compared to SAS, like half the price.

Thanks!

/Christian

 

[cyberjock]

My questions are:

• is that enough RAM?
• hardware or software RAID?
• would I benefit from an SSD for dedicated ZIL? How large?
• would Quad Cord processors be better?
• what 10G NIC's are supported under FreeNAS?
  

1. No way to know. How much RAM you will need depends on a lot of factors that are hard to describe in a forum. If your systems can't use more than 32GB of RAM or if you aren't ready to buy more RAM if needed, then you are making a bad choice. I wouldn't build a system with 70TB of data that can't use at least 128GB of RAM. My 30TB zpool required 20GB of RAM just to perform decently for a single user.
2. Again, a choice you have to make depending on how much you like/dislike hardware and software. But don't do both at the same time. I'm happy with ZFS myself.
3. Again, depends on a lot of factors. This is stuff you should research and understand the need, how much, etc. But I wouldn't add a ZIL right off the bat. They don't work like you think and therefore might not help as much as you want.
4. Maybe. Depends on many factors. If your CPUs become the limiting factor, then yes.
5. I know Intel ones are. Other than that, you should look at the FreeBSD 8.3 hardware support list and search the forums for the best answer to your question.

The SAS and SATA debate is ... still up for debate. I'm content with my SATA drives and I wouldn't change it even if I had the option to.

As for the OCZ drive, I don't have any information. I'd use google to see if anyone has commented on it working in FreeBSD. I'm pretty sure it doesn't behaving like a SATA controller and requires its own special drivers. If this is the case, then you'll need to know if FreeBSD has drivers. If it does then its probably not included with FreeNAS and you'll have to compile them yourself.

 

[clahti]

Thank you very much for responding, I have done quite a bit more research and have answered a few of my own questions. The servers top out at 64GB memory, I hope that is sufficient. I am going to go with SATA and ZFS software raid, I think an SSD for ZIL is a good idea since I may be a bit starved for memory. I am still researching :)

 

[cyberjock]

I think an SSD for ZIL is a good idea since I may be a bit starved for memory.

Remember how I said the ZIL doesn't do what you think it does(everyone thinks this at first)? The statement I just quoted proves that you do not understand the ZIL at all. There is no relationship at all between having a ZIL and a performance increase from starved memory. :)

 

[louisk]

Since you already have some of the hardware, I would suggest you create a sample system and see how it stands up to your needs. This should tell you the answers to all your questions.

All that said, if you intend to be capable of pushing more than around 4G of traffic, you will almost assuredly need both a zil and an l2arc on SSD.