Register for the iXsystems Community to get an ad-free experience and exclusive discounts in our eBay Store.

Does More RAM help iSCSI performance?

Western Digital Drives - The Preferred Drives of FreeNAS and TrueNAS CORE
Status
Not open for further replies.

Deadringers

Member
Joined
Nov 28, 2016
Messages
41
Hey all,

I'm planning to use FreeNAS as an iSCSI store - essentially just a SAN for a few servers.

I'm not seeing anything specific with regards to the question: "does more ram help in iSCSI performance"?

I see general stuff like "the more RAM the better".

Can someone tell me if I have more RAM it'll use it to Cache the iSCSI reads and writes?
 

Deadringers

Member
Joined
Nov 28, 2016
Messages
41
Looks like I didn't google hard enough:

"
For iSCSI, install at least 16 GB of RAM if performance is not critical, or at least 32 GB of RAM if good performance is a requirement.
"
 

darkwarrior

Senior Member
Joined
Mar 29, 2015
Messages
336
Hi there,
the amount RAM definitely matters a lot fscking lot.
I will try to do a TL;DR of my years of research:
At the end of the day it all depends what you want to do with your system and if you're requesting it to be fast, "zippy" or "as fast as it possibly could be"...
I'm using iSCSI zVols to host all my data at home and have "only" 20GB of RAM and would wish it to be magnitudes faster sometimes ...
Searching for a cheap Xeon E5 build right now to get better speeds and less delays when browsing folders.

First of all:
- Read the forum post below and any other posts that a research for "iSCSI" "zVOL" or "Blockstorage" comes up with
https://forums.freenas.org/index.ph...d-why-we-use-mirrors-for-block-storage.44068/
- Don't choose a Hardware plattform that will limit your RAM to 32GB or 64GB (mistake I did ...)
- Edit: Praise the ZFS gods to have mercy

If you want nicely behaving iSCSI performance for something that matters to you and you "will get kicked in the butt if doesn't", then I would consider the following:
1. install 64GB of RAM
2. Add a +/- 250GB L2ARC SSD to the pool
3. Keep some cash
4. measure the performance, if performance = 'Meh' => buy more RAM + add more L2ARC
5. ???
6. Profit
:cool:

BTW: don't forget that striped mirrors are the only way to go with blockstorage, if you don't want to commit sepukku afterwards

Have fun :p
 

Spearfoot

He of the long foot
Moderator
Joined
May 13, 2015
Messages
2,478
You are definitely on the right track! I suggest installing the maximum RAM your system will support before even considering adding an L2ARC device.
 

Deadringers

Member
Joined
Nov 28, 2016
Messages
41
I think it'll have enough performance?

2 x 10 core Xeon (2015 gen)
128 GB RAM
4-8 2 TB SSDs

Should I consider 256 GB ram?
 

Spearfoot

He of the long foot
Moderator
Joined
May 13, 2015
Messages
2,478
I think it'll have enough performance?

2 x 10 core Xeon (2015 gen)
128 GB RAM
4-8 2 TB SSDs

Should I consider 256 GB ram?
It's hard to say... With a 128GB of RAM and an SSD-based pool, you're likely to find that network speed turns out being the bottleneck. Are you running 1 gigabit, or 10+?
 

Deadringers

Member
Joined
Nov 28, 2016
Messages
41
Looking to use 10Gb.

Edit:
With jumbo frames
 
Joined
Jan 13, 2016
Messages
129
Will you use COPPER 10gigabit or optical 10gigabit link betwene the Storage and Servers?
 

Deadringers

Member
Joined
Nov 28, 2016
Messages
41
Twinax - so Copper really.

But 10g is 10g, doesn't matter if it's optical or copper.
 
Joined
Jan 13, 2016
Messages
129
Twinax - so Copper really.

But 10g is 10g, doesn't matter if it's optical or copper.

Well it does, on copper LAG is higher than on optical, in terms of ms, on Optical Cabel you do not have this problem, and you have to create separate vlan for iscsi, because bradcast will kill iscsi performance if it is on the same vlan as ipcameras, servers, stations.

If you will have 20+ vm-s on 1 datastore served by freenas over iscsi, you will have the same problem as i did, crapy performance. I went with COPPER, than changed to Fiber Optic + separate vlan, wich then i changed to Fiber Chanel, and, even that FC is not officialy supported by freenas ( only truenas) FC beats all, it is "a öne setup and forghet it". Very stable, reliable and fast.

+ FC 4 gbs is cheap nowdays, so are FC swiches
 

Deadringers

Member
Joined
Nov 28, 2016
Messages
41
Well it does, on copper LAG is higher than on optical, in terms of ms, on Optical Cabel you do not have this problem, and you have to create separate vlan for iscsi, because bradcast will kill iscsi performance if it is on the same vlan as ipcameras, servers, stations.

If you will have 20+ vm-s on 1 datastore served by freenas over iscsi, you will have the same problem as i did, crapy performance. I went with COPPER, than changed to Fiber Optic + separate vlan, wich then i changed to Fiber Chanel, and, even that FC is not officialy supported by freenas ( only truenas) FC beats all, it is "a öne setup and forghet it". Very stable, reliable and fast.

+ FC 4 gbs is cheap nowdays, so are FC swiches
LAG?

Do you mean latency?

If so the is no difference for any user when talking about the distances we are, e.g. in the same rack or building.

If you're talking about a few km or more then it's worth it.

With regards to broadcast traffic, yes it's worth having separate VLANs.
 

Deadringers

Member
Joined
Nov 28, 2016
Messages
41
PS - Copper in fact has lower "latency" than fiber. :)

e.g. it can send signals faster than optical can over the same distance.
 
Joined
Jan 13, 2016
Messages
129
In to answer to the original post, yes, more RAM will help performance, i wold go for minimum 32gb, ( i use 64 gb )

Regarding type of connection used, i don't want to be too offensive, but:

"
Performance is the main historical reason IT professionals prefer Fibre Channel over Ethernet; however, some may read that statement and disagree. Ethernet offers 40 Gb and Fibre Channel is only 16 Gb. If Ethernet has more bandwidth, how could Fibre Channel have better performance? The answer is bandwidth is not the primary performance consideration for some applications. If an application is looking for low latency, Fibre Channel will win over Ethernet almost every time. A look at the design of the two protocols will explain why.

Fibre Channel design assumes very short connections that are never longer than a Kilometer and usually much shorter. In contrast, Ethernet networks can stretch around the world. Due to this design difference, Fibre Channel can assume that all frames make it to the other side, where Ethernet assumes that many of them will not make it. This means Fibre Channel doesn’t have to do as much error checking and re-transmitting as Ethernet does. This translates into significantly lower latency numbers.

Another low latency device that is quite popular is Flash. Fibre Channel offers a better latency match to Flash than Ethernet does. Perhaps one has to look no further to see the reason behind this resurgence in Fibre Channel. If a customer has a latency-sensitive application, they are going to consider Flash as their storage medium. And if they are going to use Flash, they will want a low-latency protocol to communicate with their storage – and Fibre Channel meets the bill."

source: https://nexenta.com/why-fibre-channel-resurging
 

Deadringers

Member
Joined
Nov 28, 2016
Messages
41
I think we've gotten confused here between Ethernet (which is over Fiber or copper) and fibre channel.

Yes Fibre channel has "lower latency", but in reality it's not really an issue.
 
Joined
Jan 13, 2016
Messages
129
PS - Copper in fact has lower "latency" than fiber. :)

e.g. it can send signals faster than optical can over the same distance.

Copper cannot mach optics never, it is only cheaper.
10gigabit and more over copper wire was developed because fiber optic was expensive and copper is cheaper.

In real life usage, in enterprise apliances, optical is used, first of all it does not suffer from EFI Interference, and so on.
For Home use and Small Buisness optical is overkill. But if you have the opportunity tu buy used optical hardware, do it, there is a diffrence.
 

Deadringers

Member
Joined
Nov 28, 2016
Messages
41
Copper cannot mach optics never, it is only cheaper.
10gigabit and more over copper wire was developed because fiber optic was expensive and copper is cheaper.

In real life usage, in enterprise apliances, optical is used, first of all it does not suffer from EFI Interference, and so on.
For Home use and Small Buisness optical is overkill. But if you have the opportunity tu buy used optical hardware, do it, there is a diffrence.

You need to work on your definitions.

Copper has (technically) lower latency than fibre.

What you're saying about "copper cannot match optics", I think you mean bandwidth.

Bandwidth is not latency. (which is what you were talking about to begin with).
 

Evi Vanoost

Member
Joined
Aug 4, 2016
Messages
91
From a purely technical perspective, copper will be ever so slightly faster because you don't need to do the conversion from copper-optical and optical-copper at every switch/router, that is really microseconds (the standard specs a maximum of 2.5 microseconds) but they can add up, fiber on the other hand is less susceptible to electrical noise and interference which CAN cause serious latency issues in certain environments. For most environments however, these considerations are negligible (is your server room or wiring next to large generators or motors?)

If you do end up doing really long runs or runs through hard to reach places, fiber may be better for the long term, once you get to 40 or 100Gbps+ office-building-distance networks, using high quality fiber will become inevitable - for most of us 40Gbps is only slated for one of the future upgrades, 100Gbps is slowly entering the lives of the Google's and Facebook's of the world.
 

Deadringers

Member
Joined
Nov 28, 2016
Messages
41
Yeah I think we got completely side tracked here.

Fibre/Copper @ 10Gb is EXACTLY the same* for 99.99% of all deployments under a certain length.

Same bandwidth, latency etc etc.

10Gb is 10Gb.

When you go over a certain length, you need fibre or else you have to have some kind of copper repeater to keep the signal from being lost.


*Technically it's not the same, but the difference is not noticeable by you, me or 99.99% of all applications and OS as we're talking sub ms difference in latency.
 
Status
Not open for further replies.
Top