NEED Advice with ISCSI BUILD

Status
Not open for further replies.

orm1server

Cadet
Joined
Sep 13, 2018
Messages
9
Hello,

I've tried searching for this and can't get a straight answer it seems but this is my build plans

(ALREADY OWN)
AMD A10 CPU
ASUS A88X-PRO MOBO
32GB DDR3-2400
VARIOUS SATA2 AND SATA3 HDDS

(GOING TO BUY)
LSI SAS 9261
SSD FOR L2 CACHE (IF NEEDED)
(2) HP 81E 8G FIBER
Matching HDDS for RAID CARD


Plans

Want a Storage system for an ESXI 6.7 FREE. So ISCSI to my knowledge is only option. Since I don't have a switch that supports the fiber and I only have 1 esxi server atm. I want to run the fiber between freenas and esxi without a router or switch in between/involved.

This is a home lab/home business set up with at Max 8hdds at this time.

I could use any suggestions and guides to maximize performance and set up ISCSI without a router/switch (direct connection) low budget is a thing but am also about doing it right and to allow growth. Want to stick with fiber if not too difficult.

Thanks in advance,
Steve
 

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,176

kdragon75

Wizard
Joined
Aug 7, 2016
Messages
2,457

orm1server

Cadet
Joined
Sep 13, 2018
Messages
9
Horrible choice. You need an SAS HBA, like the SAS 9207 or SAS 9300, and not a RAID controller.
Can you explain the difference? I'm shooting for a hardware raid and not software raid. I'm using sata drives. Can you elaborate why a horrible choice please
 

orm1server

Cadet
Joined
Sep 13, 2018
Messages
9
The free hypervisor does not support vStorage APIs so you might as well use NFS since you can't leverage any of the VAAIs.
From googling it appears esxi free supports iscsi. I'm not try to use vmotion or anything like that. Just use as a vmfs storage as my server is running out of storage space
 

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,176

rvassar

Guru
Joined
May 2, 2018
Messages
971
Can you elaborate why a horrible choice please

FreeNAS uses the ZFS filesystem developed at Sun Microsystems, and open sourced by the Open Solaris project prior to the 2010 acquisition by Oracle. It implements RAID at the filesystem "Pool" level. The hardware RAID controller actually hides information about the disks that ZFS uses to do it's job. You won't actually miss the HW RAID controller, because ZFS does a much better job managing the disks directly, and offers ZFS unique features like copy-on-write, ZVol's, snapshots, compression & deduplication, all of which can have a huge impact on a VM datastore.

The free ESXi does support iSCSI, but you can also use NFS shares. The VAAI stuff is not just for vMotion, FreeNAS supports coupled snapshots, where the hypervisor and the NAS cooperate to generate a VM snapshot.

I also need to point out that AMD CPU's have been reported to be problematic in the past. I have no current information on them. The A10 dedicates a lot of silicon to a GPU, which will be completely unused by FreeNAS. The A10 and your motherboard do not support ECC memory, which is recommended for stability, etc...

I have a similar setup currently waiting for a SFP+ cable to arrive to connect a pair of 10Gbe NIC's. The problem with bonded ethernet is the cable complexity and it doesn't scale unless you have multiple active connections.
 

Mlovelace

Guru
Joined
Aug 19, 2014
Messages
1,111
From googling it appears esxi free supports iscsi. I'm not try to use vmotion or anything like that. Just use as a vmfs storage as my server is running out of storage space
Yes, it supports iSCSI. I never said it didn't. The fiber channel card you listed don't support iSCSI. The vmfs datastores you're wanting to make won't support VAAI was the point I was making. So the need for iSCSI over NFS with freeNAS is irrelevant. I would suggest doing some research.
 

orm1server

Cadet
Joined
Sep 13, 2018
Messages
9
Yes, it supports iSCSI. I never said it didn't. The fiber channel card you listed don't support iSCSI. The vmfs datastores you're wanting to make won't support VAAI was the point I was making. So the need for iSCSI over NFS with freeNAS is irrelevant. I would suggest doing some research.
Ok thank you. I didn't realize the fiber card didn't support iscsi. I'll look into that.
 

orm1server

Cadet
Joined
Sep 13, 2018
Messages
9
FreeNAS uses the ZFS filesystem developed at Sun Microsystems, and open sourced by the Open Solaris project prior to the 2010 acquisition by Oracle. It implements RAID at the filesystem "Pool" level. The hardware RAID controller actually hides information about the disks that ZFS uses to do it's job. You won't actually miss the HW RAID controller, because ZFS does a much better job managing the disks directly, and offers ZFS unique features like copy-on-write, ZVol's, snapshots, compression & deduplication, all of which can have a huge impact on a VM datastore.

The free ESXi does support iSCSI, but you can also use NFS shares. The VAAI stuff is not just for vMotion, FreeNAS supports coupled snapshots, where the hypervisor and the NAS cooperate to generate a VM snapshot.

I also need to point out that AMD CPU's have been reported to be problematic in the past. I have no current information on them. The A10 dedicates a lot of silicon to a GPU, which will be completely unused by FreeNAS. The A10 and your motherboard do not support ECC memory, which is recommended for stability, etc...

I have a similar setup currently waiting for a SFP+ cable to arrive to connect a pair of 10Gbe NIC's. The problem with bonded ethernet is the cable complexity and it doesn't scale unless you have multiple active connections.


Ok. Makes sense to allow freenas to handle raid now. May I ask which 10gbe nics you're using
 

orm1server

Cadet
Joined
Sep 13, 2018
Messages
9
Horrible choice. You need an SAS HBA, like the SAS 9207 or SAS 9300, and not a RAID controller.
So I see a lsi 9260 on supported hardware list. For VMware. Do I want to avoid raid controllers all together and let free Nas handle it or just avoid the 9261. Many thanks
 

HoneyBadger

actually does care
Administrator
Moderator
iXsystems
Joined
Feb 6, 2014
Messages
5,110

rvassar

Guru
Joined
May 2, 2018
Messages
971
Ok. Makes sense to allow freenas to handle raid now. May I ask which 10gbe nics you're using

Planning to use... Old EOL'ed Mellanox ConnectX-2's I picked up off eBay for $20 each. I just got notified my cable has arrived!

As for the RAID controllers... Many of us here have LSI SAS RAID controllers that have been intensionally flashed with firmware that disables the RAID functions. SAS is electrically compatible with SATA. So I have fanout cables that connects the 2 SAS ports on the card to 8 SATA drives, and I can use cheap inexpensive SATA disks with ZFS & FreeNAS.
 

orm1server

Cadet
Joined
Sep 13, 2018
Messages
9
Planning to use... Old EOL'ed Mellanox ConnectX-2's I picked up off eBay for $20 each. I just got notified my cable has arrived!

As for the RAID controllers... Many of us here have LSI SAS RAID controllers that have been intensionally flashed with firmware that disables the RAID functions. SAS is electrically compatible with SATA. So I have fanout cables that connects the 2 SAS ports on the card to 8 SATA drives, and I can use cheap inexpensive SATA disks with ZFS & FreeNAS.
Ok may I ask the exact model number of those nics? Are they on the approved hardware? Do they require any extra config to set up(drivers, mods etc)

Also I don't have a switch or router that supports thia fast of Sfp can I run fiber between these 2 nics directly without a router or switch in between (for now)? If so how would I config it? On esxi and freenas
 

rvassar

Guru
Joined
May 2, 2018
Messages
971
Ok may I ask the exact model number of those nics? Are they on the approved hardware? Do they require any extra config to set up(drivers, mods etc)

I just drove to my office, and I don't have them in front of me. If you'd asked a few minutes earlier... Here's a link:

https://www.ebay.com/itm/RT8N1-0RT8...0GBe-ETHERNET-NIC-SERVER-ADAPTER/132756940343

I haven't plugged them in yet, but I'm under the impression the drivers come in from FreeBSD (the base OS used by FreeNAS) in the FreeNAS 11.x branch, and they'll just appear as a useable network interface. Some version prior (9.x?) they are not supported. You're literally asking me about these on the day my SFP+ cable arrived. :)

I don't have a switch or router that supports thia fast of Sfp can I run fiber between these 2 nics directly without a router or switch in between (for now)? If so how would I config it? On esxi and freenas

I'm planning to do just exactly this, but I'm going to use a SFP+ to SFP+ "twinax" cable, since the hosts are right next to each other. They have lower latency than fiber, and are not as easy to damage. I currently have a pair of Intel 1GbE NIC's configured with static IP's as a "desktop SAN". I have a VM on the ESX host that routes to the SAN addresses when needed. I'm planning on simply substituting the the 10GbE NIC's and retaining the address space in use, etc... In order to use iSCSI you need to make sure the ESX Hypervisor itself has a vNIC and IP address on the vSwitch attached to the 10GbE physical NIC.

As for having a switch... Another option used by some people here that need a mix of hosts connected to the NAS is to get another PC chassis, and dual port 10GbE NIC's and quad port 1GbE NIC's, and build a homebrew 10GbE <--> 1GbE switch using VyOS (Linux variant specialized for networking). You can search Youtube for a 3 part video on home 10GbE network for more info. I haven't decided if I want to run a third chassis, so I'm sticking with the local host to host SAN network for the moment. Note the dual port 10GbE NIC's require a PCIe x8 slot. Consumer boards often have x4's and x16's, and x16's that are actually wired x4 because they're expected to be used with SLI GPU pairs. So it's kind of a pain to get all the minutiae correct. There are 10GbE switches now in the $200 range, but they're RJ45 Cat 6a copper, and I'd need different NIC's.

The nice thing about FreeNAS, you can live like a king on old cheap hardware!
 
Joined
May 10, 2017
Messages
838
I haven't plugged them in yet, but I'm under the impression the drivers come in from FreeBSD (the base OS used by FreeNAS) in the FreeNAS 11.x branch, and they'll just appear as a useable network interface. Some version prior (9.x?) they are not supported. You're literally asking me about these on the day my SFP+ cable arrived.

FreeNAS includes the driver for these NICs since at least 9.10
 

kdragon75

Wizard
Joined
Aug 7, 2016
Messages
2,457
@orm1server no offence but I think you need to step back and read this book BEFORE you buy anything for your home lab.
 

orm1server

Cadet
Joined
Sep 13, 2018
Messages
9
@orm1server no offence but I think you need to step back and read this book BEFORE you buy anything for your home lab.
No offense taken. But yes learning this from a book is one thing and works for some but learning hands on is so much better Imo. I'm not a fan of boring old text. I thought the forums were to help with this. I have a fully functional esxi server now it's the freenas iscsi I'm un sure of as I've never worked with freenas. May I ask in all politeness what I've said or not said that makes you think a book is more helpful then a forum?

Many thanks
Steve
 
Status
Not open for further replies.
Top