Sprint
Explorer
- Joined
- Mar 30, 2019
- Messages
- 72
Hi all
So I've been lurking for a while now, doing lots of reading up, and some experimenting, as I was fairly new to FreeNas when I started.
For the last 3 or 4 years, my storage has been entirely handled by my Synology DS1815+, and an offsite DS411+ii backup. Superb boxes, yes a little under powered but did the job flawlessly. As my storage needs have grown though, I have for the last year at least started thinking about how I was going to grow my storage to match (I am doing more and more video editing these days)... It looked like I was going to have to just stump up for a mid 4 figure Synology NAS to get the number of bays i wanted as well as the ability to go 10Gb and rack mount..... but....
I had a stroke of luck. I'm a systems administrator by day (have been for 3 years now, but came from a 9 year stint in Cisco networking) so now look after a small VM environment. Our old platform finally got retired and replaced, and while in the data centre helping to de-rack it all for disposal, I mentioned that it was a shame to throw so much kit away as it still had life in it..... Well, they then said I could take whatever i wanted (with the exception of storage devices, these HAD to be destroyed). Naturally.. I grabbed my screw driver and got to work!
Heres what I came away with....
4x Xeon E5-2630V4's (10c/20t)
2x Xeon E5 2478 (8c/16t)
4x Xeon X5675 (4c/8t)
16x 32Gb 2400Mhz ECC DDR4 dimms
and 72x 16Gb 1333Mhz ECC DDR3 dimms.
At the time I had no plans, so when they said 'fill your boots' I grabbed what i felt i might be able to make use of from a handful of the blades. They had been brought over time as our capacity needs grew, which is why there is such an age spread. But already I was thinking 'why by a NAS, I might as well build one, I now have most the parts, I knew what FreeNas was, I'd seen plenty of videos about it, but had always dismissed it as buying heaps of ram was the biggest off putting cost for me... But with these parts, it suddenly seemed ludicrous not to make the most of it.
After some initial reading, I had originally thought I'd build three servers, a dual CPU unit in a 1 or 2U server with minimal storage running ESXi to replace my existing VM server (an old 2nd Gen i5 desktop i re-purposed), a second single CPU 4U machine for a FreeNas server to act as my file server and iSCSI store over 10Gb... However I quickly realised this was going to cost a bomb as server motherboards for these CPUs i had weren't cheap. The Dual socket one i was looking at was over £300, so buying two or three very quickly was putting me off the idea again!! (And a 3rd for offsite backups, which I am only now starting to plan, just trying to find the right cheap used server motherboard to work with at least one of the Xeons and ram I have spare).
Anyway, this was a few months back, and after a little experimenting and alot of reading, I found an article which talked about virtualising FreeNas... I did alot more reading, but suddenly the answer seemed clear...
So I ended up building a dual Xeon E5-2630V4 server with 256Gb of DDR4 Ram in an old re-purposed HTPC case. (I have a 24 drive chassis on order but its taken nearly a month to arrive... any day now), to which I installed ESXi 6.7, as well as a pair of LSI 9207-8i's (I'll need a 3rd before I'm finished). These are passed through to the FreeNas VM directly so FreeNas has direct access to the drives. When the machine is powered on, it boots into ESXi. Only the FreeNas VM is visible, i boot that, and once booted and all the storage and iSCSI services are up, the other VMs storage become available, and I fire these up. Genius! The only thing that doesn't work is CPU temp monitoring, but ESXi does this so I suppose I have to live with that. Anyway, I'm sure I found the article here, so thank you all for saving me from building 3 separate servers. Now I only need 2.
So yes, my journey continues. At the moment I only have 5x2Tb Greens installed (spares I had lying around) and am using it for iSCSI and a handful of non important SMB shares. Once my chassis arrives, I'll start playing musical data as i shuffle data around on my Synology Nas to allow me to unmount the secondary array of 5x4Tb Reds (I'll order a 6th) and install those into FreeNas (RaidZ2). Then again, shuffle more data round to free up the 7x8Tb reds (again, planning to order an 8th drive and deploy in RaidZ2) and get these moved into the FreeNas box. Still trying to decide what todo with the existing pair of 512Gb 860Pro SSDs i used as caches in the Synology.... I have a pair of 256Gb SanDisk SSDs I'm using as L2ARC... I had though about using the 860 Pros as a slog, but the more i read, the less I think I'll get any benefit from them? Eventually I'll put a 10Gb NIC in as well as one into my Desktop, and I can then video edit directly off the NAS, and I still have 8 Bays in the chassis available for future drives so I'm nicely future proofed!
Though I'd include a handful of pictures too, as everyone loves pictures :)
Just a small selection of the stuff I saved from the skip.
The motherboard I settled on... using that right hand PCIe slot is going to be a challenge!!
I tried test fitting the LSI card... no chance...
Coolers on... hard to find coolers that would clear one another as the sockets are SO close...
Once fired up, I threw together a VM just to run Cinebench for the lol's! Impressive numbers for "old" hardware!
And this never gets old!...
Its abit of a mess at the moment, just waiting on the chassis, then I can do it properly! Can't wait to get it all properly bolted in!
This is what really impressed me... this is a disk benchmark from a VM running on my old i5 server with the C Drive on a iSCSI volume on my Synology NAS over gigabit... (no idea why it keeps loading in sideways)...
but now, as the iSCSI server and VMs are on the same internal vSwitch, and traffic never leaves the box.. these are the new numbers... Mind... BLOWN.... (again, not sure why its loading sideways)
Confused why the writes are faster than the reads?.... Something else to read up on :)
So yes, once my case arrives, if people are interested, I'll document and provide pictures. Massive thanks to all for teh information I've already gleamed... thought it was about time i said hello, and thank you :)
Sprint
So I've been lurking for a while now, doing lots of reading up, and some experimenting, as I was fairly new to FreeNas when I started.
For the last 3 or 4 years, my storage has been entirely handled by my Synology DS1815+, and an offsite DS411+ii backup. Superb boxes, yes a little under powered but did the job flawlessly. As my storage needs have grown though, I have for the last year at least started thinking about how I was going to grow my storage to match (I am doing more and more video editing these days)... It looked like I was going to have to just stump up for a mid 4 figure Synology NAS to get the number of bays i wanted as well as the ability to go 10Gb and rack mount..... but....
I had a stroke of luck. I'm a systems administrator by day (have been for 3 years now, but came from a 9 year stint in Cisco networking) so now look after a small VM environment. Our old platform finally got retired and replaced, and while in the data centre helping to de-rack it all for disposal, I mentioned that it was a shame to throw so much kit away as it still had life in it..... Well, they then said I could take whatever i wanted (with the exception of storage devices, these HAD to be destroyed). Naturally.. I grabbed my screw driver and got to work!
Heres what I came away with....
4x Xeon E5-2630V4's (10c/20t)
2x Xeon E5 2478 (8c/16t)
4x Xeon X5675 (4c/8t)
16x 32Gb 2400Mhz ECC DDR4 dimms
and 72x 16Gb 1333Mhz ECC DDR3 dimms.
At the time I had no plans, so when they said 'fill your boots' I grabbed what i felt i might be able to make use of from a handful of the blades. They had been brought over time as our capacity needs grew, which is why there is such an age spread. But already I was thinking 'why by a NAS, I might as well build one, I now have most the parts, I knew what FreeNas was, I'd seen plenty of videos about it, but had always dismissed it as buying heaps of ram was the biggest off putting cost for me... But with these parts, it suddenly seemed ludicrous not to make the most of it.
After some initial reading, I had originally thought I'd build three servers, a dual CPU unit in a 1 or 2U server with minimal storage running ESXi to replace my existing VM server (an old 2nd Gen i5 desktop i re-purposed), a second single CPU 4U machine for a FreeNas server to act as my file server and iSCSI store over 10Gb... However I quickly realised this was going to cost a bomb as server motherboards for these CPUs i had weren't cheap. The Dual socket one i was looking at was over £300, so buying two or three very quickly was putting me off the idea again!! (And a 3rd for offsite backups, which I am only now starting to plan, just trying to find the right cheap used server motherboard to work with at least one of the Xeons and ram I have spare).
Anyway, this was a few months back, and after a little experimenting and alot of reading, I found an article which talked about virtualising FreeNas... I did alot more reading, but suddenly the answer seemed clear...
So I ended up building a dual Xeon E5-2630V4 server with 256Gb of DDR4 Ram in an old re-purposed HTPC case. (I have a 24 drive chassis on order but its taken nearly a month to arrive... any day now), to which I installed ESXi 6.7, as well as a pair of LSI 9207-8i's (I'll need a 3rd before I'm finished). These are passed through to the FreeNas VM directly so FreeNas has direct access to the drives. When the machine is powered on, it boots into ESXi. Only the FreeNas VM is visible, i boot that, and once booted and all the storage and iSCSI services are up, the other VMs storage become available, and I fire these up. Genius! The only thing that doesn't work is CPU temp monitoring, but ESXi does this so I suppose I have to live with that. Anyway, I'm sure I found the article here, so thank you all for saving me from building 3 separate servers. Now I only need 2.
So yes, my journey continues. At the moment I only have 5x2Tb Greens installed (spares I had lying around) and am using it for iSCSI and a handful of non important SMB shares. Once my chassis arrives, I'll start playing musical data as i shuffle data around on my Synology Nas to allow me to unmount the secondary array of 5x4Tb Reds (I'll order a 6th) and install those into FreeNas (RaidZ2). Then again, shuffle more data round to free up the 7x8Tb reds (again, planning to order an 8th drive and deploy in RaidZ2) and get these moved into the FreeNas box. Still trying to decide what todo with the existing pair of 512Gb 860Pro SSDs i used as caches in the Synology.... I have a pair of 256Gb SanDisk SSDs I'm using as L2ARC... I had though about using the 860 Pros as a slog, but the more i read, the less I think I'll get any benefit from them? Eventually I'll put a 10Gb NIC in as well as one into my Desktop, and I can then video edit directly off the NAS, and I still have 8 Bays in the chassis available for future drives so I'm nicely future proofed!
Though I'd include a handful of pictures too, as everyone loves pictures :)
Just a small selection of the stuff I saved from the skip.
The motherboard I settled on... using that right hand PCIe slot is going to be a challenge!!
I tried test fitting the LSI card... no chance...
Coolers on... hard to find coolers that would clear one another as the sockets are SO close...
Once fired up, I threw together a VM just to run Cinebench for the lol's! Impressive numbers for "old" hardware!
And this never gets old!...
Its abit of a mess at the moment, just waiting on the chassis, then I can do it properly! Can't wait to get it all properly bolted in!
This is what really impressed me... this is a disk benchmark from a VM running on my old i5 server with the C Drive on a iSCSI volume on my Synology NAS over gigabit... (no idea why it keeps loading in sideways)...
but now, as the iSCSI server and VMs are on the same internal vSwitch, and traffic never leaves the box.. these are the new numbers... Mind... BLOWN.... (again, not sure why its loading sideways)
Confused why the writes are faster than the reads?.... Something else to read up on :)
So yes, once my case arrives, if people are interested, I'll document and provide pictures. Massive thanks to all for teh information I've already gleamed... thought it was about time i said hello, and thank you :)
Sprint
Last edited: