Hello from a computer room in SW London.. 11.2-U2 upgrade... No module named 'site' (I'm on-site! ;-))

[ReubenC]

fun and games with FreeNAS on a sunny February Saturday afternoon... don't get me started as to why the IPMI on this SuperMicro SRX-4200G isn't working, hence I'm on-site whilst FreeNAS reports No module named 'site' - the irony isn't lost! :)

upgrade from 11.2-RELEASE-U1 didn't go smoothly, ended up having to manually select 11.2-U2 from the Boot tab, that wasn't sensible! so here I am on-site trying to select the previous boot env. and all that happens is the menu recycles... I might be reinstalling and restoring that config backup I took!

middlewared not running
cannot open /usr/local/share/pcbsd/scripts/functions.sh: No such file or directory

Anyhow, this is not the right group for introductions - am newish (1yr) to FreeNAS where we elected to try it with a customer vs. Windows Storage Server (am a 20+ yr Windows veteran and ex-Microsoft PSS engineer in the days of NT4 and what became Windows 2000... a long time ago!)

 

[Basil Hendroff]

Ouch! I assume you're working on a production system (h/w specs would be helpful. More info at https://forums.freenas.org/index.php?threads/updated-forum-rules-12-5-18.45124/). If you're reinstalling FreeNAS, it may be prudent to downgrade to 11.1-U7 as long as you haven't upgraded the pool with new feature flags. See post https://forums.freenas.org/threads/possible-new-install-whitch-version.74072/#post-513985. There appear to be a number of issues with 11.2, in particular U2.

 

[ReubenC]

Hi Seymour, thank you for swift reply on this fine Saturday!

Being a Brit I made a cup of tea and voila! or rather by Jove! its back online! I came back with said cuppa and made another attempt at the boot environments menu... booted with the previous 11.2-U1 - tea!!
Yes, its production and yes I've noted a few issues since 11.2 (services needing manual start, sambad issues, false positives on network connectivity, etc.), I have 11.1-U6.3 and 11.1-U6 available as previous, but not 11.2-U7. Right now/today I'm going to leave it (the 6 Nations won't delay KO for me! ;-)

As for spec, it is:
CORAID SRX4200-G6 is based on the ‘SuperChassis 847E16-R1K28LPB’
Supermicro X8DTH-iF’ motherboard
1x Intel Xeon X5550, 48GB RAM
Intel Pro/1000 Quad Port LP Srv Adaptor
5x 8 Port LSI SAS 9207-8i SGL Internal 6Gb/s SATA+SAS, PCIe 3.0 HBA
72TB RAW (63TB 21x 3TB SATA + 9TB 15x 600GB 15K SAS) Storage Server
2x USB keys for boot
pools configured as:

NAME SIZE ALLOC FREE EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
Vol1-Fast 8.12T 1.01T 7.11T - 0% 12% 1.00x ONLINE /mnt
Vol2-Storage1 32.6T 18.0T 14.6T - 1% 55% 1.00x ONLINE /mnt
Vol3-Storage2 24.5T 11.4T 13.1T - 19% 46% 1.00x ONLINE /mnt

FYI, I've got upgrades planned for an SATA DOM for boot and an Intel 10GB NIC (there is 10GB elsewhere inc.).

It's primary role is serving iSCSI LUNS as Hyper-V CSV's to a Windows Server 2019 RC VM host that is secondary to a primary Windows Server 2016 Hyper-V platform. Secondary role is SMB hosting for an Archives store. Essentially its not running primary workloads hence I've been pretty relaxed!

Would appreciate your thoughts on making 11.1-U6.3 active and no, I have not 'upgraded' the volumes, so retrograding is possible.

Thanks again and very best!

Reuben

 

[Basil Hendroff]

I think if it's working as you expect, then there's probably no reason to rollback. If there are nagging issues that were not present under 11.1, then you may want to revert back to U6.3. Once you're back there, you may want to consider if U7 provides any additional benefit. See the changelog at https://www.ixsystems.com/blog/library/freenas-11-1-u7/

 

[ReubenC]

I think if it's working as you expect, then there's probably no reason to rollback. If there are nagging issues that were not present under 11.1, then you may want to revert back to U6.3. Once you're back there, you may want to consider if U7 provides any additional benefit. See the changelog at https://www.ixsystems.com/blog/library/freenas-11-1-u7/

Thank you! good suggestion, we rolled back and voila SAMBA working again (which is helpful as we have a DFS presented Archive library on the box), later updated to FreeNAS-11.1-U7.
So.. what's the latest on 11.2 - still hold off for now..?

 

[Basil Hendroff]

11.2 is okay for most, but sudden death for others. It feels a bit like Russian roulette atm. For the lowdown, refer to the thread Lost Data in Upgrade to 11.2. There's also a worrying database lock issue that appeared in 11.2, that, as of today, has reared its ugly head in 11.1. Only two reported instances to date though. For more information, refer to the thread OperationalError: database is locked after 11.2-RELEASE upgrade (31f889bbf)

 

[ReubenC]

11.2 is okay for most, but sudden death for others. It feels a bit like Russian roulette atm. For the lowdown, refer to the thread Lost Data in Upgrade to 11.2. There's also a worrying database lock issue that appeared in 11.2, that, as of today, has reared its ugly head in 11.1. Only two reported instances to date though. For more information, refer to the thread OperationalError: database is locked after 11.2-RELEASE upgrade (31f889bbf)

Yikes! those data loss threads are ugly!
Backups!! we have VM replication to another host, are backing up from our guest VM's nightly to a remote Synology NAS, and before we tried 11.2 we did VM exports to a local 10TB monster USB (slow!), but that's sometimes an exception not the rule...
Clearly mult-method & target backups must be the rule (regardless of data loss stories) - belt & braces!

Fingers crossed no issues appearing in our 11.1-U7 env. It can stay as is until clear upgrade reasons/benefits with plenty of field testing, and, of course, solid backups! was planning a SATA-DOM & 10GB NIC upgrade this weekend (along with an old HP iLO to 'resolve' (replace!) the IPMI)) - think these can wait too!

Thanks again Seymour, you have likely avoided us some more on-site shenanigans!

very best, Reuben

 

[Basil Hendroff]

My pleasure Reuben. There are a bunch of 11.2 issues that are troubling FreeNAS users and developers alike atm. The problem is that the conditions that cause these issues to arise remain elusive. I've begun tracking some of the more problematic issues (refer to the tail end of My Bookshelf in my signature block). Personally, I have not seen evidence of any of these issues on my backup 11.2 systems. The majority of 11.2 users are also probably okay, but for the small number of affected users, the impact is high to catastrophic.

I wouldn't want to be moving to 11.2 on my production systems anytime soon and certainly not until developers have a good handle on the worst of these issues. This could take upwards of six months or longer, and is largely dependent on developers continuing to receive quality feedback from the community so that the causes can be identified and solutions devised.