Beta 8.3 CIFS & Symantec Backup Exec 2012

[kkwoodbrey]

I am new to FreeNAS and not sure where to post for Beta input. I am in the process of building a backup storage system to use with Symantec Backup Exec 2012. My hardware is a Supermicro SuperStorage Server 6047R-E1R36N. I am running the 64bit, 8.3 Beta. I have set up a RAIDZ level 2 volume with a CIFS share. When Backup Exec has copied anywhere from 5 to 80GB of data, the CIFS share becomes unavailable and the FreeNAS web interface becomes unresponsive. The command line terminal interface remains active.

 

[cyberjock]

I'd highly recommend reading the forum rules and posting with your exact hardware(RAM, CPU, hard drive etc).

 

[kkwoodbrey]

Chassis: Supermicro CSE-847E16-R1K28LPB
Motherboard: Supermicro X9DRi-LN4F+ (Intel chipset C602)
CPU: Dual Xeon E5-2603
Memory: 6x 8GB Crucial Technology CT3CP102472BB1067 DDR3 PC3-8500 CL=7 Registered ECC DDR3-1066
SATA/SAS HBA: LSI Logic Controller Card SAS 9207-8I
Ethernet Adapters: 4x Onboard Intel i350 Gigabit (Only one in use for testing)
Hard Drive: 10x Western Digital 2TB WD20EARS-00J2GB0 SATA 64MB Cache
OS/APP: FreeNAS-8.3.0-BETA1-r12067-x64.iso

 

[cyberjock]

Thanks for the info!

First: I'd try running 8.2. Betas are usually pretty stable, but I never run them except for testing. In your case, since you are having problems I'd try 8.2 before trying to spend time "fixing" the issue.

Second: Have you run any diagnostics on your hard drives? If one of them starts having problems it can result in bizarre performance and reliability isssues. I have WD20EARS running with no problems so I know the drives should work fine.

You can try using iozone to see how your io is going on your pool. If one drive seems to be lagging behind the rest that may be a problem.

I know from personal experience with backing up across the LAN that some programs do not do it very well. Acronis is riddled with problems trying to backup to/recovering from the LAN. It's been an issue for 3 years in a row and alot of customers have left Acronis for alternate solutions because it doesn't work(including myself). Their forum is a good testament to their problems. Acronis has gotten some users to upgrade to 2011 and then to 2012 claiming they fixed the issue, but it hasn't been fixed. I won't be going back to Acronis until I can get a trial version to work flawlessly for LAN backups, assuming I ever have a problem with O&O backups(which work great over the LAN!)

What happens if you backup to a local hard drive and then copy the file to the server? If it works fine then the problem sounds to be related to Symantec's implementation of LAN backups. How big will the backup file be when the backup is complete? You aren't trying to backup across wifi I hope ;)

 

[kkwoodbrey]

I originally tried the 8.2 release. It did not support the Intel i350 gigabit network adapters so I moved to the 8.3 beta. I pulled these drives from another storage unit which was working with no problems. I will try a straight Windows copy to see what happens. I am backing up up 100GB for a test but will be backing up several TB each night in production. I am using one of the Ethernet interfaces on the motherboard currently but will use link aggregation of all four once I get past this issue.

 

[kkwoodbrey]

They Windows copy stopped after it had copied about 8GB. The share was no longer active. The FreeNAS web interface also stopped responding. I can still get into the command line interface on the terminal. Are there any logs that I should look at on the FreeNAS server? When I shutdown the FreeNAS server from the command line I get "Waiting for PIDS 2526" and it hangs there for a long time then it finally shuts down.

 

[cyberjock]

Ouchies. So the problem is probably on the server end.

Ideas of things you may be able to try:

Any chance you can try a different NIC to see if it works? Any older Intel NIC should just work if you plug it in.

Another idea is to try copying the file while logged into the server and type zpool iostat -v 1. This lets you watch the reads/writes to the array every second. If a bunch of drives are completing their reads/writes and one is lagging behind that drive may be failing.

Have you setup SMART monitoring of your drives? If not, that may tell you if a drive is having problems.

Have you tried doing a DD write to your pool? The manual gives examples of using DD to determine the zpool performance. Do a 100GB file and see how fast it goes and if it finishes. If you get good performance and it finishes it's likely not your zpool that's the problem.

You don't have compression or deduplication enabled do you?

Just to make sure I'm not making bad assumptions.. you are using CIFS for the windows share, correct?

 

[kkwoodbrey]

I don't have any nic cards that will fit the server.

No compression or dedup and I am using CIFS for the Windows share. I will try the three disk ideas that you suggested but it will later today since I need to put a new faucet and water filter in the kitchen sink and trailer hitch on my wife's car.

 

[kkwoodbrey]

I am running DD write tests currently and am having no problems. I have written 20GB, 100GB, 200GB and 500GB files. I am thinking that maybe the Intel i350 support might be an issue. The i350 is new in this release and both the network share and the web interface freeze at the same time. Any ideas on tools or logs that I can look at to see if the network interface is failing?

 

[Stephens]

"ifconfig" from the shell of the FreeNAS box and "ping" from the either device. I think you're right as to the cause.

 

[cyberjock]

You could try using iperf. It's included in FreeNAS. You could also try pinging www.google.com or another machine from your FreeNAS server after it freezes.

In all seriousness your configuration doesn't have anything out of the ordinary hardware-wise except your NIC. Trying another NIC is my first guess for your problems at this point. I always keep a spare Intel NIC around the house myself. It is possible your NIC is bad. Has it worked properly in other systems or is it brand new?

 

[kkwoodbrey]

ifconfig shows all is well and active. I can successfully ping 192.168.88.197 and 127.0.0.1 from the console. I can also successfully ping 192.168.88.197 from other computers but both the CIFS share and the FreeNAS web interface have frozen. After waiting for a few minutes the CIFS share comes back.

The (4) i350 interfaces are on the motherboard (brand new). I will try another one of those and then I will have to buy a low profile card to try.

 

[cyberjock]

After the CIFS share comes back does the web interface too?

Do you have any kind of tuning setup on your machine? Any settings you manually entered to try to speed up CIFS or ZFS? Perhaps enabling the auto-tune(or disabling it if it is enabled) may help. Its suppose to help optimize the settings for your hardware but some people have had some issues with it both enabled or disabled.

CIFS is single threaded. I know that is a major limitation of Samba. You are running a 1.8Ghz CPU. For CIFS more Mhz is much more important than more cores. But even still I wouldn't expect your system to act the way it is with a slower CPU. I'd just expect less than ideal performance if you were CPU limited. I run an older i3-530 (2.93Ghz and I can max out 2 Gb NICs at the same time over CIFS).

Yeah, my only good idea is to try a different NIC. It could be that the i350 just isn't well supported on FreeBSD 8.3.

 

[freibuis]

I originally tried the 8.2 release. It did not support the Intel i350 gigabit network adapters so I moved to the 8.3 beta. I pulled these drives from another storage unit which was working with no problems. I will try a straight Windows copy to see what happens. I am backing up up 100GB for a test but will be backing up several TB each night in production. I am using one of the Ethernet interfaces on the motherboard currently but will use link aggregation of all four once I get past this issue.

I cant even boot my machine with the ISO with the intel I350. panic "invalid cluster type" <- igb driver issue


and when it does boot the system is so unstable with it.I believe the problem you are having is with the network card. I have the same issue. I pulled the card put in an old INTEL card . no problems..

:( I Think the 64bit IGB driver might be the issue...

 

[kkwoodbrey]

I even put in another NIC and the server would not work with it. I can find no way in the BIOS to disable the four onboard i350 interfaces. I agree that the drivers for them are the issue. Is there any way to report this to the developers? The documentation says that the i350 chipset should be supported in 8.3. I have three of these servers that each have the four onboard i350 interfaces so I really need to make this work.

 

[Stephens]

Yes, there is a way to report this to the developers. Look at the 8.3 documentation under Trac Database (10.2).

 

[kkwoodbrey]

The developers resolved my issue. They had me add a tunable:

kern.ipc.nmbclusters

Value 262144

 

[Izo]

Hi There,
We have the same server (supermicro 6047R-E1R36N with 36x2TB disks, intel chipset c602, 64 GB RAM) and run FreeNAS v8.3 beta1 for the same reason (v8.2 do not support the NIC). Our problem is that the FreeNAS do not see the disks. how we can resolve this problem? should we use tunables? what should be our preferred configuration. in general we want something like hardware RAID1 but it seems that FreeNAS offers better solutions which i do not understand very well (i am very new in this things)?
Regards,
Izo

 

[kkwoodbrey]

The RAID controller that comes with the server will not allow the operating system to see and control the individual disks. We replaced the RAID controller with an HBA controller (LSI Logic Controller Card SAS 9207-8I) which gives you direct access to the disks. We are using the FreeNAS RAIDZ level 2 and it is working very well.