My Dream System (I think)

[joeschmuck]

Yes, RAID1 (mirror) of two SSDs. One is an Crucial M4, the other is a Plextor M7V. The M4 has a lot of hours on it, I've owned it for many years and was my primary drive in my main computer. I remember when the M4 took a dive, after so many runtime hours the firmware crashed. After that it would only run for 1 hour at a time after you applied power to it. The firmware was fixed within a few weeks but it sucked having that problem. The M4 is the slowest drive of the two and may be the one holding me back. If that is true then I'll be happy to live with it. The read speed seems fair.

 

[joeschmuck]

Here is an update with respect to some more benchmarks...

I have placed a clean Windows 7 64-bit VM on my FreeNAS as an iSCSI (Note: I've never done iSCSI so if there are some optimizations to be made, I'm unaware). I ran Crystal Diskmark and ATTO, of with I feel ATTO is more representative of the true I/O. Note that the read speed drops off significantly once the cache is not able to keep up while the writing is super fast because of the write cache. To be honest, I kind of like this iSCSI thing and I'm now considering removing my 1.5TB single hard drive in the ESXi server (used as a datastore) and then just use this iSCSI as my datastore.

 

[Black Ninja]

Yes, RAID1 (mirror) of two SSDs. One is an Crucial M4, the other is a Plextor M7V. The M4 has a lot of hours on it, I've owned it for many years and was my primary drive in my main computer. I remember when the M4 took a dive, after so many runtime hours the firmware crashed. After that it would only run for 1 hour at a time after you applied power to it. The firmware was fixed within a few weeks but it sucked having that problem. The M4 is the slowest drive of the two and may be the one holding me back. If that is true then I'll be happy to live with it. The read speed seems fair.

Ohh they are not even the same drives. That's not a great idea is my opinion , but let see:
Crucial M4 writes (if 128GB) around 180MB/s , and Plextor M7V it seems faster , but it's been hold back by Crucial M4... I would say it's not unusual the have this write speed.

With Samsung 2x850 Pro's in raid0 (which are priced super great) you can expect some "unholy" speeds. Here is with just 1 drive and with 2 in raid0 My favorites by the way

 

[Black Ninja]

Here is an update with respect to some more benchmarks...

I have placed a clean Windows 7 64-bit VM on my FreeNAS as an iSCSI (Note: I've never done iSCSI so if there are some optimizations to be made, I'm unaware). I ran Crystal Diskmark and ATTO, of with I feel ATTO is more representative of the true I/O. Note that the read speed drops off significantly once the cache is not able to keep up while the writing is super fast because of the write cache. To be honest, I kind of like this iSCSI thing and I'm now considering removing my 1.5TB single hard drive in the ESXi server (used as a datastore) and then just use this iSCSI as my datastore.

View attachment 13021 View attachment 13022

I also don't have iSCSI experience but I was eager to play with for a long time. I had the impression that was too slow and kind of didn't want to have a storage outside the physical machine especially as a block storage , so I decide to just keep it with raid for Esxi. But now I got my eyes sparkle again.

I just finish my new masterpiece (pfsense router) install it , and now I am recovering my media-center , but that will not take long. And it's time to invest most time in redoing the ESXI host. The problem is: my Esxi host is 1U chassis and I can either have a 10Gb nic or Raid card, or need new chassis ( unless I go for iSCSI and don't need raid card) so that's why I jumped when you mention iscsi. I am interested in your results.


P.S On Atto and Crystalmark comment: they are both good. Crystal mark will give more accurate speed on large sequential read/write, where atto could some times trick you to believe in some speed you don't actually have. I'll see if I can get my test screenshot on my Intel 530 480GB. Here they are !
So how fast you think my drive can write large files?

 

[Black Ninja]

I wonder does you esxi see the datastore as SSD ? I would assume it's detected as regular disk, despite that underneath the raid card are actually SSDs that makes the volume ?


P.S. I know there are ways to change this but is it worth doing so ?

 

[joeschmuck]

The mirrored pair (datastore1) on the RAID card are seen as SSDs. See screen capture.

 

[Black Ninja]

The mirrored pair (datastore1) on the RAID card are seen as SSDs. See screen capture.

View attachment 13181

And that was out of the box. You didnt have to change to SDD manualy ?

 

[joeschmuck]

Nope, right out of the box. Although I would have changed it to SSD if that is possible, but first I'd have to figure out how to do that.

 

[Black Ninja]

It won't take you long to figure how, I am sure.
It could be cause it's more HBA/Raid sort of raid card or the esxi 6.0 vs 5.5, not sure.

P.S. I am suppriced you have so much ram in it ?! I remember you mention being "cheap" on hardware

 

[joeschmuck]

For this server I went with the maximum amount of RAM that I thought I'd ever need. I am not even scraping the surface of the systems capabilities but that is due to my own ignorance when it comes to networking VMs and such. I'd really like to be able to offer an easy way for my family to log into a windows VM and it work without any issues. I know it's not difficult to do but the first think is it must be easy.

Actually, I wouldn't mine a nice Linux OS that everyone could use. It must be very cheap though.

But right now my project is modifying my Tacoma headlights with Morimoto D2S projector lamps, they look sharp. I hopefully will finish these up this weekend, it will depend on the weather. I'll post a few photos when I'm done.

 

[Black Ninja]

For this server I went with the maximum amount of RAM that I thought I'd ever need.

It's not a bad decision, it just since it's CPU is limited to 4 core unlike 2011 Socket, I would do 32 GB with expectation of around 15 or even 20 VMs to be able to hadnle before run out of cpu power, but still not a bad thing if cache is spearable.

I am very concern about the power usage of my servers, I wonder what is the power consumption of your system as it is ?

I'll post a few photos when I'm done.

I was gonna ask you about it You mention it before and I got more curious what would it look like.

 

[joeschmuck]

Power usage is 54.76KW for last month (73.6 watts/hour average) running normally for me. This is running FreeNAS, Sophos, and Ubuntu. Ubuntu is running BOINC and the processor is set for 25% usage of 2 vcores. Power drops to ~68 watts when I shutdown Ubuntu. So not a lot of power used for running BOINC but it does crunch the numbers fast enough for my purposes.

Now in all fairness, most of the power is used by the motherboard, CPU, and RAM. The hard drives suck the rest of the power up. And while I didn't measure it, I suspect that the RAID card sucks a little more than 3 watts of power, it gets pretty warm.

And here is a shot of the passenger side headlamp assembly before I mount it, ensure the projector alignment, and then sealing the poly-carbonate lens cover. Hopefully tomorrow will be a good day to finish this job. The other headlight is ready to be tested as well.

 

[Black Ninja]

73.6 W is not bad. I am just thinking and comparing for my results with older hardware. Your half power goes to your HDD, you save a power from having non server power supply and fans. I don't know when I become obsess with power consumption , but It's more for the reason to be able to run more servers in given "power home budget" I would like me server rack to stay in 1000W power budget.

I this sophos your router or it's just a UTM ?

This headlight will still have the glass on top of it right ? It will look nice when finished I am sure, that's why I'll not mention my disbelieving in projection headlights.

 

[joeschmuck]

I this sophos your router or it's just a UTM ?

Yes, it's my router as well. I have a cable modem, it's just a simple pass-through device, provides two internet IP addresses. Right now I'm only using one of those IPs, I still need to work on getting the second one operational again but what I want to do is have a single UTM manage both connections (shouldn't be hard) and then the most difficult thing is to use one Ethernet cable directly from the Cable Modem to the server and having ESXi use both IPs. Right now I can only obtain one IP. I was hoping to not need a second Ethernet port to get a second MAC address. I'll figure it out.

This headlight will still have the glass on top of it right ?

Yes, but it's a hard clear plastic these days.

that's why I'll not mention my disbelieving in projection headlights. :)

Well I'm hoping they work well without blinding on-coming drivers. I'm a perfectionist when dealing with this kind of stuff so I take my time and do it right the first time. If these turn out to be crap, then I'll cry just a little inside.

I would like me server rack to stay in 1000W power budget.

OUCH! 1000 watts is a lot of power. You should ask yourself if you really need that much equipment. Maybe it's time to put up a few solar panels.

 

[Black Ninja]

I have a cable modem, it's just a simple pass-through device, provides two internet IP addresses.

Oh so your ISP is giving you 2 IPs ? You mean to external IPs right ?

I want to do is have a single UTM manage both connections (shouldn't be hard)

That's something similar perhaps the what I am trying to do with pfsense. My purpose is to have 2 separate networks , where one has open ports for outside access and another that is closed and separate, but I don't have 2 IPs and still have much more to learn how to properly and securely do that.

It's seems you have shortage on ports , I have 4 on my router , do you have 3 total on your esxi (where your router resides), from what I count ? I would take out the Intel CT nic that you have and put a quad port intel nic, they are $50 or less on ebay. For Esxi you'll need as much nics as you can if you as me.

Maybe it's time to put up a few solar panels.

I love the idea, I would of done it already if I didn't have "an association" to deal with where I live.

 

[joeschmuck]

Oh so your ISP is giving you 2 IPs ? You mean to external IPs right ?

Yes, two internet exposed IPs, distinctly different. I have achieved using both IPs by using a 5 port switch and a second LAN port and then that shows two MAC addresses to the modem and thus I have two IPs. This feels wasteful and I also need to run two instances of the UTM but it at least does work. It may be the only way to make it work but I'm looking to see if that is true. I've been told I can't do what I want but maybe I can figure it out.

 

[ChriZ]

My best guess would be that you need another network interface for your sophos.
There is also a feature called "additional addresses". Perhaps that can do something..
Not really sure.

 

[Black Ninja]

Yes, two internet exposed IPs, distinctly different. I have achieved using both IPs by using a 5 port switch and a second LAN port and then that shows two MAC addresses to the modem and thus I have two IPs. This feels wasteful and I also need to run two instances of the UTM but it at least does work. It may be the only way to make it work but I'm looking to see if that is true. I've been told I can't do what I want but maybe I can figure it out.

That is really great. I don't know how you got them, but our Cable provider Comcast don't allow more than 1 ip per residential Customer. Are your IP's static by the way ?

You have to run two instances but at lest they are virtual so you don't waste another machine. I am not sure how it works on sophos but on pfsense you can have another WAN interface for failover from another provider , or use it to create another separate network. Perhaps you should add second WAN interface and use it for whatever purpose you like if sophos allows it. Is this free version of sophos with 50 lan-IPs limit ?

 

[joeschmuck]

Metrocast provides two IPs as part of their normal service, I suspect for folks who don't have a router so two computers can connect, however if they don't have a firewall, watch out! The IPs are dynamic however so long as I never disconnect them, well the IP stays static just like most ISPs.

True, another VM like the UTM is nothing to the system I have.

Yes, the free version of Sophos has a 50 IP address limit. I have have 26 IPs assigned, that includes 4 DHCP IPs for friends which visited 4 days ago. The IP limit has to deal with those which are controlled by Sophos or listed in the configuration. If I have a static IP address of say 192.168.1.205 and it was for my laptop, so long as I didn't tie it to a definition like identifying the IP as "My Laptop", then it's not counted in the 50 limit. But I do define all of mine and I'm not close to using 50 IPs. Defining the IPs helps when looking at a log, a name makes things much more readable.

Sophos does have the ability to allow a second WAN and failover and lots of stuff, but from what I can tell, not what I want to do.

 

[Black Ninja]

Metrocast provides two IPs as part of their normal service, I suspect for folks who don't have a router so two computers can connect, however if they don't have a firewall, watch out! The IPs are dynamic however so long as I never disconnect them, well the IP stays static just like most ISPs.

That's very nice of them. I dont see any information about how many ips they give on their web site, but maybe they don't want to say there. I am not sure how I cable modem is getting 2 external IPs. From what I know your cable modem will get one extrenal IP from their DHCP server , if you register your modem mac with them first ?So based on your modem mac , IPS assign dynamic IP to it. How that work with 2 IPs ans 1 modem is still not very clear to me.

Sophos does have the ability to allow a second WAN and failover and lots of stuff, but from what I can tell, not what I want to do.

What is it in your case that you are trying to do ?