ZIL (ZFS Intent Log) do I need it or not?

[liukuohao]

Ok guys,

I need some advice here.

After reading some information on the guide and internet, I have decided to start
a project of building FreeNAS for storing CCTV recording footage, using iSCSI protocol.

The type of RAID will be using ZFS RAID 1+0.
The benefit of using RAID 1+0 are as followings
1) Easy of adding more disk for increasing storage.
2) Faster write speed compares to ZFS RAIDZ & RAIDZ2.

There is 1 question that I failed to answer so far.....
Do I need ZIL (ZFS intent log) for my application?

According to: http://forums.freenas.org/threads/slideshow-explaining-vdev-zpool-zil-and-l2arc-for-noobs.7775/

Cyberjock recommends that ZIL can be completely avoid if necessary, since
recovery process will be complex to configure when you really need to recover
some data. At this point, I cannot tell whether the CCTV surveillance software
is doing synchronous writing or asynchronous.

Since my CCTV software will be doing mostly WRITE operations @ 90%, does having ZIL
improve the writing speed. Or it does not make any difference not having it.

Thank you.

 

[cyberjock]

The ZIL can increase performance. You asked the right question when you said...

At this point, I cannot tell whether the CCTV surveillance software is doing synchronous writing or asynchronous.

If the CCTV is doing asynchronous writes a ZIL won't matter.
If the CCTV is doing synchronous writes a ZIL may matter.

I say "may" because it depends. If your CCTV is writing only 1MB every 10 seconds and you are still plenty happy with the performance of your server, who cares? On the other hand, if its writing 30MB/sec as a bunch of sync writes then you are very likely to be very unhappy when the CCTV is being used. Big picture; its a matter of how fast your zpool is, how much data is actually being written, how its being written(sync or async), and how much you care about the CCTV video*.

* - I believe there's a setting that can make NFS convert all sync writes to async writes. If you don't care if the CCTV video is corrupted on a system crash or loss of power then that may be a good way to go.

The best advice I can give is to setup your CCTV and see how it does. I'm not familiar with your hardware/software but the rule for ZIL and L2ARC is to determine if you need it by testing it. FreeNAS 9.1 now includes a bunch of tools to help you determine if a ZIL or L2ARC could help performance. It's non-trivial to interpret the data from the tools and I haven't played with them much myself(yet).

It sounds like your cameras connect to a computer that then saves the video feeds. The only CCTV I have seen that had a setup like that didn't work well over network shares. The software couldn't handle it for whatever reason.

Also, something to consider. You should try not to fill a zpool too much. It also has no defrag tools, so writing large amounts of data to a small-ish zpool can cause fragmentation that becomes so significant that the zpool can't perform even small tasks in a reasonable period of time.

 

[liukuohao]

I cannot tell whether the CCTV surveillance software
is doing synchronous writing or asynchronous.

Actually, I got response from the CCTV surveillance software technical support people.
One of the technical person confirmed today, that its software will ensure that video
data is written to the physical hard disk (carry out by the Operating System), before
the software can proceed to do any task or process any job. So I guess, this kind of
WRITE operation is classified as SYNCHRONOUS write. :)

 

[liukuohao]

I went back to the 8.3.1 Users Guide and read up page 112 of section 6.3.3.2, title: ZFS extra.

I read again on the bottom 3rd paragraph, where it say something about Synchronous Write.

It states that: " Application that do not do a lot synchronous write are less likely to benefit from
having dedicated ZIL devices."

....which raise a question what is Synchronous and Asynchronous Writes???
(as the guide did not specify to noobs, like me, to understand the difference)

I searched the good old google search, found a good site to read for noobs.:)
below is the link, for anyone who is interested in knowing more about the basic
stuff on ZFL synchronous write and ZIL:

http://constantin.glez.de/blog/2010/07/solaris-zfs-synchronous-writes-and-zil-explained

In summary, the difference between the 2 writes are here:

Synchronous Write = the software will issue a write call, requesting the OS to perform a write
to disk, the OS will grab the data and temporarily store it to RAM first, then wait till a certain
period of time has lapsed (because OS is allowing hard disk to get ready, since it has mechanical parts
which runs slower), the OS will then transferred all the accumulated data from the RAM
and writes the data to the hard disk. The software will ensure that data is written to a permanent
stable storage, before leaving and do other process. In other words, the software will not perform
other jobs, unless it had seen that data is written to physical hard disk, SSD or flash memory.

Asynchronous Write = the software ill issue a write call, requesting the OS to perform a write
to disk, the OS will grab the data and temporarily store it to RAM first, however, the software
will not wait for the OS to write the data on the hard disk. As such, the software can perform the
next task/job if so desires.

 

[liukuohao]

The best advice I can give is to setup your CCTV and see how it does. I'm not familiar with your hardware/software but the rule for ZIL and L2ARC is to determine if you need it by testing it. FreeNAS 9.1 now includes a bunch of tools to help you determine if a ZIL or L2ARC could help performance. It's non-trivial to interpret the data from the tools and I haven't played with them much myself(yet).

Yes, I need to test it first before I buy SSD for storing ZIL.

 

[liukuohao]

It sounds like your cameras connect to a computer that then saves the video feeds. The only CCTV I have seen that had a setup like that didn't work well over network shares. The software couldn't handle it for whatever reason.

Yes, you are right! Actually the CCTV surveillance software supports 2 method of storing video data:

1) DAS = Direct Access Storage using eSATA interface.

2) iSCSI = using SCSI commands over the network.

3) others = like, network share, USB external hard disk = not supported.

 

[liukuohao]

Also, something to consider. You should try not to fill a zpool too much. It also has no defrag tools, so writing large amounts of data to a small-ish zpool can cause fragmentation that becomes so significant that the zpool can't perform even small tasks in a reasonable period of time.

I understand that having ZIL running will cause a lot of disk fragmentation,
but......if ZIL is going store on a separate storage like = SSD, then there
will be no disk fragmentation on the actual Z pool( consist of hard disks in ZFS Raid 1+0
configuration), correct?

Is a ZFS RAID 1+0 configuration:
vdev 1 : 2TB = mirror = 2TB
vdev 2 : 2TB = mirror = 2TB
and vdev 1+ vdev 2 = ZFS RAID 1
actually available storage = 4TB - is this considered SMALLish??

 

[cyberjock]

I'm not sure I'd buy that their answer means they do sync writes. I'd test it for myself to see. That sounds more like someone that doesn't have a clue and is giving you a non-technical answer that sounds like its the "right" answer so you'll hang up the phone.

The ZIL doesn't do anything that will decrease the fragmentation.

The ZIL is nothing more than a non-volatile temporary storage for writes that haven't been stored on the zpool yet.

Size is entirely based on your personal needs. In high definition 1080p with many streams that could be filled very quickly.

 

[jgreco]

cctv software is generally written by the lowest bidder... in a third world country... by non-english speakers... who can't code in anything except php... need i go on :)

 

[liukuohao]

I am still learning here about ZFS, and struggling here.
May be I am not coming from an IT background,
that is why my learning curve is long! And never
ending!

The only reliable resource that I can fall back on is
the FreeNAS user guide + Google search.

Referring to the page 115 of the FreeNAS user guide v8.3.1, it states:

When extending a volume, ZFS supports the addition of virtual devices (vdevs) to an existing volume
(ZFS pool). A vdev can be a single disk, a stripe, a mirror, a RAIDZ1, RAIDZ2, or a RAIDZ3. Once a
vdev is created, you can not add more drives to that vdev ; however, you can stripe a new vdev (and its
disks) with the same type of existing vdev in order to increase the overall size of ZFS the pool. In other
words, when you extend a ZFS volume, you are really striping similar vdevs. Here are some examples:
• to extend a ZFS stripe, add one or more disks. Since there is no redundancy, you do not have to
add the same amount of disks as the existing stripe.
• to extend a ZFS mirror, add the same number of drives. The resulting striped mirror is a RAID
10.
• to extend a three drive RAIDZ1, add three additional drives. The result is a RAIDZ+0, similar
to RAID 50 on a hardware controller.
• to extend a RAIDZ2 requires a minimum of four additional drives. The result is a RAIDZ2+0,
similar to RAID 60 on a hardware controller.

So my goal here to create a RAID 1+0....... as I have highlighted above.

I have fitted a 3ware hardware RAID adapter on the system.
4 x 2TB WD hard disks connected to the adapter.

All of the 4 x 2TB hard disks are configured as JBOD/ single drive.
So in FreeNAS, when view disk, you will see da0, da1, da2, da3.

I used the ZFS volume manager to create a mirror for da0 & da1, I named it as MirrorA
which has a total disk space of 1.8TB

Again, I used the ZFS volume manager to create a mirror for da2 & da3, I named it as MirrorB
which has a total disk space of 1.8TB

The question is, how to create a single stripe ZFS volume on the 2 vdevs = Mirror A and B.

I did try using the volume manager, but no avail. Did I miss something important here in my
configuration steps?

 

[paleoN]

Did I miss something important here in my
configuration steps?

Yes. Assuming there is no data on MirrorB, delete the pool. Then you need to extend the MirrorA pool. Don't create a new pool.

 

[liukuohao]

Referring back to the latest FreeNAS guide v9.1.0

http://doc.freenas.org/index.php/Volumes#ZFS_Volume_Manager

When adding disks to increase the capacity of a volume, ZFS supports the addition of virtual devices, known as vdevs, to an existing ZFS pool. A vdev can be a single disk, a stripe, a mirror, a RAIDZ1, RAIDZ2, or a RAIDZ3. Once a vdev is created, you can not add more drives to that vdev; however, you can stripe a new vdev (and its disks) with the same type of existing vdev in order to increase the overall size of the ZFS pool. In other words, when you extend a ZFS volume, you are really striping similar vdevs. Here are some examples:

• to extend a ZFS stripe, add one or more disks. Since there is no redundancy, you do not have to add the same amount of disks as the existing stripe.

• to extend a ZFS mirror, add the same number of drives. The resulting striped mirror is a RAID 10.

• to extend a three drive RAIDZ1, add three additional drives. The result is a RAIDZ+0, similar to RAID 50 on a hardware controller.

• to extend a RAIDZ2 requires a minimum of four additional drives. The result is a RAIDZ2+0, similar to RAID 60 on a hardware controller.

The wording addition made me think that I need to add
the 2 vdevs together MirrorA & MirrorB to become
1 single ZFS stripe volume. But in actual fact you cannot do this.

 

[liukuohao]

Yes. Assuming there is no data on MirrorB, delete the pool. Then you need to extend the MirrorA pool. Don't create a new pool.

As paleoN said, I need to remove the MirrorB first, by doing "Detach" & "Mark disk as new (destroy data)".
Once this is done, da2 & da3 hard disks will no longer be part ZFS volume.

Then, I need to use "Volume to Extend" by pressing the down arrow triangle, in Volume
Manager, Select MirrorA, then press the "+" to add the available disks are da2 & da3.

 

[liukuohao]

I started all over again, and give a new name to the volume = CCTVstorage
Under CCTVstorage, there are 2 vdevs, Mirror-0 and Mirror-1
Now I have ZFS Raid 1+0 completed.:)

Now I have a total disk space of 3.6TB
(Mirror-0: 2TB = 2TB gives only 1.8TB)
(Mirror-1: 2TB = 2TB gives only 1.8TB)
CCTVstorage = Mirror-0 + Mirror-1 = 1.8TB + 1.8TB = gives 3.6TB disk space.
This is the actual stripe ZFS volume disk space.;)

Now I want to create a ZVOL and name it CCTVrecord.
But, how come I couldn't use the maximum of the disk space available?:(
However, if I typed in 1TB for the size of ZVOL, then there is no problem. Why?

 

[liukuohao]

Hmm...
I think I found someone in the forum has the same problem:
http://forums.freenas.org/threads/allocate-remaining-space-in-zvol-for-iscsi-use.3209/#post-13049

For ZFS volume, you just need to have some free space to work
around and for future upgrade.

I retyped the size of the volume to 3.5TB, and it prompted me I am out of space.
I retyped the size to 3.4TB also no avail.
I retyped the size to 3TB and no problem, FreeNAS accept it.