SOLVED [X-Post] What's the CloudSync API call's time format?

Joined
Dec 28, 2020
Messages
18
Cross-posting to here from the Operation section! Sorry if that's very gauche, but I think this is the better home for the question.

I'm trying to use the API to retrieve some data about my cloud sync tasks, including the start date and end date. I'm a little befogged by the time stamps I'm seeing, though, and wanted to check if anyone had any insight. Here's an example json blob returned after calling /api/v2.0/cloudsync

Code:
{'id': 113,
 'description': 'PUSH my data',
 'direction': 'PUSH',

 # -- snip --

 'locked': False,
 'job': {'id': 1234,
  'method': 'cloudsync.sync',
  'arguments': [113],
  'logs_path': '/tmp/middlewared/jobs/4196.log',
  'logs_excerpt': '2022/06/17 00:00:04 INFO  : \nTransferred:   \t         0 / 0 Bytes, -, 0 Bytes/s, ETA -\nElapsed time:         1.0s\n\n2022/06/17 00:00:05 INFO  : \nTransferred:   \t         0 / 0 Bytes, -, 0 Bytes/s, ETA -\nChecks:                51 / 2341, 2%\nElapsed time:         2.0s\nChecking:\n * fils/a.bin: checking\n... 2783 more lines ...\n * : checking\n * files/n.bin: checking\n\n2022/06/17 00:03:24 INFO  : There was nothing to transfer\n',
  'progress': {'percent': 100,
   'description': 'checks: 19853 / 19853',
   'extra': None},
  'result': None,
  'error': None,
  'exception': None,
  'exc_info': None,
  'state': 'SUCCESS',
  'time_started': {'$date': 1655449200272},
  'time_finished': {'$date': 1655449404621}}}


What's the time format for 1655449200272 and 1655449404621? It doesn't seem to match the epoch, and the duration itself seems longer than the job actually took.
Thanks!
 
Joined
Dec 28, 2020
Messages
18
A savvy co-worker has solved my problem! The $date values are in milliseconds since the epoch (1 January 1970, UTC).
 
Joined
Aug 10, 2016
Messages
28
I've just spent some precious time figuring this out too. General epoch timestamp in *nix world is 10-digit long, while TrueNAS uses 13-digit long, which includes milliseconds. You either truncate the value to 10digits or divide by 1000 to use it with date -r "timestamp" +Format
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
General epoch timestamp in *nix world is 10-digit long,

This is incorrect. It is a time_t, which is conventionally a 32 bit signed integer. Because it is an integer, it starts at 0 (one digit long), and only reached its current 10 digits long on September 9th, 2001. Writing code that assumes this to be a ten-digit quantity is a bad idea, as it will malfunction on dates before that point. Of course, 2^31-1 is 2147483647, which means that a 32 bit signed time_t will never gain an 11th digit. However, due to the impending obsolescence issue in 2038 when 2^31-1 is reached, there are proposals to extend this to a 64 bit signed integer, and at some point (in November 20 2286 if I did it correctly) it will roll to 11 digits.

Please do the world a favor and handle your data types correctly.
 
Joined
Aug 10, 2016
Messages
28
This is incorrect. It is a time_t, which is conventionally a 32 bit signed integer. Because it is an integer, it starts at 0 (one digit long), and only reached its current 10 digits long on September 9th, 2001. Writing code that assumes this to be a ten-digit quantity is a bad idea, as it will malfunction on dates before that point. Of course, 2^31-1 is 2147483647, which means that a 32 bit signed time_t will never gain an 11th digit. However, due to the impending obsolescence issue in 2038 when 2^31-1 is reached, there are proposals to extend this to a 64 bit signed integer, and at some point (in November 20 2286 if I did it correctly) it will roll to 11 digits.

Please do the world a favor and handle your data types correctly.
Well, thanks for the clarification, BUT....
In practical terms, let's consider this job snippet ( yeah, it's a short test job ) :
midclt call cloudsync.query | jq ".[1].job"
Code:
{
  "id": 6163,
  "method": "cloudsync.sync",
  "arguments": [
    2
  ],
  "logs_path": "/tmp/middlewared/jobs/6163.log",
  "logs_excerpt": "2023/01/19 11:06:58 INFO  : Starting bandwidth limiter at 1kBytes/s\n2023/01/19 11:06:58 INFO
                            : There was nothing to transfer\n2023/01/19 11:06:58 INFO  : \nTransferred:   \t 
                            0 / 0 Bytes , -, 0 Bytes/s, ETA -\nChecks:                 1 / 1, 100%\nElapsed time:       
                            0.2s\n\n[Post-script] at: pluralization is wrong\n[Post-script] Job 52 will be executed
                            using /bin/sh\n",
  "progress": {
    "percent": 100,
    "description": "checks: 1 / 1",
    "extra": null
  },
  "result": null,
  "error": null,
  "exception": null,
  "exc_info": null,
  "state": "SUCCESS",
  "time_started": {
    "$date": 1674137218818
  },
  "time_finished": {
    "$date": 1674137218824
  }
}


Now let's take time_started["$date"] and time_finished["$date"] ar arguments to date command:
date -r $(midclt call cloudsync.query | jq '.[1].job.time_started["$date"]') Thu Apr 19 01:20:18 -03 55021

and

date -r $(midclt call cloudsync.query | jq '.[1].job.time_finished["$date"]') Thu Apr 19 01:20:24 -03 55021

The "divide by 1000" I assumed as truth coming for other forum topic, but it really isn't true, the number turns into float

Another possibility is:
date -r $(midclt call cloudsync.query | jq '.[1].job.time_started["$date"]/1000|floor') Thu Jan 19 11:06:58 -03 2023

Just wondering what the correct approach would be to convert the .[1].job.time_started["$date"] into usable data for date command ( date -r TIMESTAMP +FORMAT )

Here's an example of how dividing by 1000 is a widespread practice:
https://community.cloudera.com/t5/Support-Questions/convert-timestamp-to-date-format/td-p/228353
and another one:
https://stackoverflow.com/questions/58699093/use-jq-to-format-output-and-convert-timestamp

So, how should we properly deal with this conversion ?
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
The "divide by 1000" I assumed as truth coming for other forum topic, but it really isn't true, the number turns into float

Right. The consequences of unconsidered side effects. You have a weird variation on time_t, which is an integer value. At some point, someone did some variation on printf("%d%03d", time_t, msec) to generate the string you're dealing with. It is probably best to unwind it in a similar fashion, stripping off the three trailing digits and reimporting the integer, maybe via strtol() or sscanf() if we were writing this in C.

Don't you have substring operations in JavaScript? There should be some clever way to extract "all but the last three characters" and also avoid unnecessary type conversions. That's really what I'm suggesting here; I was horrified at the nonportability of the implication of just assuming it to be 10 characters.
 
Top