date program parameters changed between FreeNAS and TrueNAS, breaking script

zoomzoom

Guru
Joined
Sep 6, 2015
Messages
677
This is likely simple to fix , and am unsure when the change occurred, but after more than an hour yesterday trying different things while reviewing the date man page, I can't figure out what the correct parameter formatting should be, as I keep getting error output from:
  • This snippet is from a customized version of report.sh:
    Code:
    # Problematic parameters:
      ## date -j -f "%Y-%b-%e_%H:%M:%S" "$scrubDate" "+%s"
    
    # Full snippet:
    statusOutput="$(zpool status "$pool")"
    
      if [ "$(echo "$statusOutput" | grep "scan" | awk '{print $2}')" = "scrub" ]; then
        scrubRepBytes="$(echo "$statusOutput" | grep "scan" | awk '{print $4}')"
        scrubErrors="$(echo "$statusOutput" | grep "scan" | awk '{print $10}')"
    
        # Convert time/datestamp format presented by zpool status, compare to current date, calculate scrub age:
          scrubDate="$(echo "$statusOutput" | grep "scan" | awk '{print $17"-"$14"-"$15"_"$16}')"
          scrubTS="$(date -j -f "%Y-%b-%e_%H:%M:%S" "$scrubDate" "+%s")"
          currentTS="$(date "+%s")"
          scrubAge=$((((currentTS - scrubTS) + 43200) / 86400))
          scrubTime="$(echo "$statusOutput" | grep "scan" | awk '{print $8}')"
      fi


  • Results with date errors:
    Code:
    date:   illegal option -- 1
    usage:  date [-jnRu] [-d dst] [-r seconds|file] [-t west] [-v[+|-]val[ymwdHMS]]
                 [-I[date | hours | minutes | seconds]]
                 [-f fmt date | [[[[[cc]yy]mm]dd]HH]MM[.ss]] [+format]
    
    date:   illegal option -- 0
    usage:  date [-jnRu] [-d dst] [-r seconds|file] [-t west] [-v[+|-]val[ymwdHMS]]
                 [-I[date | hours | minutes | seconds]]
                 [-f fmt date | [[[[[cc]yy]mm]dd]HH]MM[.ss]] [+format]
    
    date:   illegal option -- 0
    usage:  date [-jnRu] [-d dst] [-r seconds|file] [-t west] [-v[+|-]val[ymwdHMS]]
                 [-I[date | hours | minutes | seconds]]
                 [-f fmt date | [[[[[cc]yy]mm]dd]HH]MM[.ss]] [+format]
    
    date:   illegal option -- 0
    usage:  date [-jnRu] [-d dst] [-r seconds|file] [-t west] [-v[+|-]val[ymwdHMS]]
                 [-I[date | hours | minutes | seconds]]
                 [-f fmt date | [[[[[cc]yy]mm]dd]HH]MM[.ss]] [+format]
    
    date:   illegal option -- 0
    usage:  date [-jnRu] [-d dst] [-r seconds|file] [-t west] [-v[+|-]val[ymwdHMS]]
                 [-I[date | hours | minutes | seconds]]
                 [-f fmt date | [[[[[cc]yy]mm]dd]HH]MM[.ss]] [+format]
    
    date:   illegal option -- 0
    usage:  date [-jnRu] [-d dst] [-r seconds|file] [-t west] [-v[+|-]val[ymwdHMS]]
                 [-I[date | hours | minutes | seconds]]
                 [-f fmt date | [[[[[cc]yy]mm]dd]HH]MM[.ss]] [+format]
 
Last edited:

Tony-1971

Contributor
Joined
Oct 1, 2016
Messages
147
Hello,
Probably is better if you check your script line by line.
For example on my Core System:
Code:
[tony@freenas-sm /]$ statusOutput="$(zpool status "tank-big")"
[tony@freenas-sm /]$ echo "$statusOutput" | grep "scan"
  scan: scrub repaired 0B in 08:10:00 with 0 errors on Sun Apr  3 08:10:01 2022
[tony@freenas-sm /]$ echo "$statusOutput" | grep "scan" | awk '{print $17"-"$14"-"$15"_"$16}'
-08:10:01-2022_

The last output is probably wrong.
Best Regards
 

Patrick M. Hausen

Hall of Famer
Joined
Nov 25, 2013
Messages
7,776
@Tony-1971 is right. sh -x <scriptfile> helps with that.
 

anodos

Sambassador
iXsystems
Joined
Mar 6, 2014
Messages
9,554
Not exactly what you've asked for, but you guys can also use this to get output in JSON:

midclt call pool.query

or if you convert to a python script you can use py-libzfs to get info directly:

Code:
>>> import libzfs
>>> lz = libzfs.ZFS()
>>> for p in lz.pools:
...     print(f'name: {p.name}, scrub_start: {p.scrub.start_time}')
... 
name: dozer, scrub_start: 2022-03-20 07:00:06
name: freenas-boot, scrub_start: 2022-04-11 10:45:00
name: tank, scrub_start: 2022-03-20 07:00:00

In this case start_time and end_time will be datetime objects.

For that matter you can also import our python middleware client:
Code:
>>> from middlewared.client import Client
>>> Client().call('pool.query')
[{'id': 1, 'name': 'dozer', 'guid': '14182928993082600117', 'encrypt': 0, 'encryptkey': '', 'path': '/mnt/dozer', 'status': 'ONLINE', 'scan': {'function': 'SCRUB', 'state': 'FINISHED', 'start_time': datetime.datetime(2022, 3, 20, 7, 0, 6, tzinfo=datetime.timezone.utc), 'end_time': datetime.datetime(2022, 3, 20, 18, 29, 11, tzinfo=datetime.timezone.utc), 'percentage': 0.0, 'bytes_to_process': 5939729899520, 'bytes_processed': 5944882798592, 'bytes_issued': 0, 'pause': None, 'errors': 0, 'total_secs_left': None}, 'topology': {'data': [{'type': 'RAIDZ1', 'path': None, 'guid': '809643658987360320', 'status': 'ONLINE', 'stats': {'timestamp': 234056299062900, 'read_errors': 0, 'write_errors': 0, 'checksum_errors': 0, 'ops': [0, 140523, 9268897, 0, 0, 0, 0], 'bytes': [0, 1406599168, 148291072000, 0, 0, 0, 0], 'size': 11957188952064, 'allocated': 6039472496640, 'fragmentation': 18, 'self_healed': 0, 'configured_ashift': 12, 'logical_ashift': 9, 'physical_ashift': 12}, 'children': [{'type': 'DISK', 'path': '/dev/gptid/6f6fafaf-b464-11e8-be2b-d0509964a926', 'guid': '7891567635167139394', 'status': 'ONLINE', 'stats': {'timestamp': 234056299152667, 'read_errors': 0, 'write_errors': 0, 'checksum_errors': 0, 'ops': [0, 46701, 3090400, 0, 0, 0, 0], 'bytes': [0, 468914176, 49432285184, 0, 0, 0, 0], 'size': 0, 'allocated': 0, 'fragmentation': 0, 'self_healed': 0, 'configured_ashift': 12, 'logical_ashift': 9, 'physical_ashift': 12}, 'children': [], 'device': 'ada1p2', 'disk': 'ada1', 'unavail_disk': None}, {'type': 'DISK', 'path': '/dev/gptid/7308cbf9-b464-11e8-be2b-d0509964a926', 'guid': '14095304525142238280', 'status': 'ONLINE', 'stats': {'timestamp': 234056299232106, 'read_errors': 0, 'write_errors': 0, 'checksum_errors': 0, 'ops': [0, 46934, 3089870, 0, 0, 0, 0], 'bytes': [0, 469647360, 49430106112, 0, 0, 0, 0], 'size': 0, 'allocated': 0, 'fragmentation': 0, 'self_healed': 0, 'configured_ashift': 12, 'logical_ashift': 9, 'physical_ashift': 12}, 'children': [], 'device': 'ada3p2', 'disk': 'ada3', 'unavail_disk': None}, {'type': 'DISK', 'path': '/dev/gptid/76af066c-b464-11e8-be2b-d0509964a926', 'guid': '2554344526982436024', 'status': 'ONLINE', 'stats': {'timestamp': 234056299303124, 'read_errors': 0, 'write_errors': 0, 'checksum_errors': 0, 'ops': [0, 46888, 3088627, 0, 0, 0, 0], 'bytes': [0, 468037632, 49428680704, 0, 0, 0, 0], 'size': 0, 'allocated': 0, 'fragmentation': 0, 'self_healed': 0, 'configured_ashift': 12, 'logical_ashift': 9, 'physical_ashift': 12}, 'children': [], 'device': 'ada4p2', 'disk': 'ada4', 'unavail_disk': None}], 'unavail_disk': None}], 'log': [], 'cache': [], 'spare': [], 'special': [], 'dedup': []}, 'healthy': True, 'status_detail': 'Some supported features are not enabled on the pool. The pool can still be used, but some features are unavailable.', 'autotrim': {'value': 'off', 'rawvalue': 'off', 'parsed': 'off', 'source': 'DEFAULT'}, 'encryptkey_path': None, 'is_decrypted': True}, {'id': 2, 'name': 'tank', 'guid': '18265342295712180351', 'encrypt': 0, 'encryptkey': '', 'path': '/mnt/tank', 'status': 'ONLINE', 'scan': {'function': 'SCRUB', 'state': 'FINISHED', 'start_time': datetime.datetime(2022, 3, 20, 7, 0, tzinfo=datetime.timezone.utc), 'end_time': datetime.datetime(2022, 3, 20, 7, 0, tzinfo=datetime.timezone.utc), 'percentage': 0.0, 'bytes_to_process': 3829760, 'bytes_processed': 3829760, 'bytes_issued': 0, 'pause': None, 'errors': 0, 'total_secs_left': None}, 'topology': {'data': [{'type': 'DISK', 'path': '/dev/gptid/e1880a9f-5451-11eb-92d4-d0509964a926', 'guid': '14145385741218639868', 'status': 'ONLINE', 'stats': {'timestamp': 234041909879292, 'read_errors': 0, 'write_errors': 0, 'checksum_errors': 0, 'ops': [0, 243, 129, 0, 0, 0, 0], 'bytes': [0, 1789952, 2084864, 0, 0, 0, 0], 'size': 246960619520, 'allocated': 3829760, 'fragmentation': 0, 'self_healed': 0, 'configured_ashift': 12, 'logical_ashift': 9, 'physical_ashift': 0}, 'children': [], 'device': 'ada0p2', 'disk': 'ada0', 'unavail_disk': None}], 'log': [], 'cache': [], 'spare': [], 'special': [], 'dedup': []}, 'healthy': True, 'status_detail': None, 'autotrim': {'value': 'off', 'rawvalue': 'off', 'parsed': 'off', 'source': 'DEFAULT'}, 'encryptkey_path': None, 'is_decrypted': True}]
>>> 


For cases of using middleware API, you can refer to our API docs to see what you can get out of it. If you want to parse JSON from a shell script you can use the `jq` command to do this. Trivial example from our installer: https://github.com/truenas/truenas-installer/commit/31d0a84c8454dad87ab4e99a7833fdb4810cf799
 

anodos

Sambassador
iXsystems
Joined
Mar 6, 2014
Messages
9,554
If you use py-libzfs in an application (other than just scripting), then you'll probably want to get data within a context manager, e.g.
Code:
with libzfs.ZFS() as lz:

So that you don't have a resource leak.

Just giving some ideas for enhancement of scripts :D
 

zoomzoom

Guru
Joined
Sep 6, 2015
Messages
677
Top