[SOLVED] ACME Cert Failed to perform route53 challenge Signature Expired

[kyeotic]

I'm trying to get a LetsEncrypt cert and running into an error on the final step.

Code:

[EFAULT] Failed to perform route53 challenge for 'local.kye.dev' domain: [EFAULT] Failed to get Hosted zones with provided credentials :An error occurred (SignatureDoesNotMatch) when calling the ListHostedZones operation: Signature expired: 20230303T110322Z is now earlier than 20230303T190722Z (20230303T191222Z - 5 min.)

I've tried running the list-buckets command from the AWS CLI using the same creds and it works. I've also verified that the system clock is accurate. Not sure whats going on.

Here is the full error log.

Code:

Error: Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/middlewared/plugins/acme_protocol_/authenticators/route53.py", line 60, in _find_zone_id_for_domain
    for page in paginator.paginate():
  File "/usr/lib/python3/dist-packages/botocore/paginate.py", line 255, in __iter__
    response = self._make_request(current_kwargs)
  File "/usr/lib/python3/dist-packages/botocore/paginate.py", line 332, in _make_request
    return self._method(**current_kwargs)
  File "/usr/lib/python3/dist-packages/botocore/client.py", line 357, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/lib/python3/dist-packages/botocore/client.py", line 676, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (SignatureDoesNotMatch) when calling the ListHostedZones operation: Signature expired: 20230303T110322Z is now earlier than 20230303T190722Z (20230303T191222Z - 5 min.)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/middlewared/plugins/acme_protocol_/authenticators/base.py", line 26, in perform
    perform_ret = self._perform(domain, validation_name, validation_content)
  File "/usr/lib/python3/dist-packages/middlewared/plugins/acme_protocol_/authenticators/route53.py", line 38, in _perform
    return self._change_txt_record('UPSERT', validation_name, validation_content)
  File "/usr/lib/python3/dist-packages/middlewared/plugins/acme_protocol_/authenticators/route53.py", line 84, in _change_txt_record
    zone_id = self._find_zone_id_for_domain(validation_domain_name)
  File "/usr/lib/python3/dist-packages/middlewared/plugins/acme_protocol_/authenticators/route53.py", line 71, in _find_zone_id_for_domain
    raise CallError(f'Failed to get Hosted zones with provided credentials :{e}')
middlewared.service_exception.CallError: [EFAULT] Failed to get Hosted zones with provided credentials :An error occurred (SignatureDoesNotMatch) when calling the ListHostedZones operation: Signature expired: 20230303T110322Z is now earlier than 20230303T190722Z (20230303T191222Z - 5 min.)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/middlewared/job.py", line 426, in run
    await self.future
  File "/usr/lib/python3/dist-packages/middlewared/job.py", line 461, in __run_body
    rv = await self.method(*([self] + args))
  File "/usr/lib/python3/dist-packages/middlewared/schema.py", line 1186, in nf
    res = await f(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/middlewared/schema.py", line 1318, in nf
    return await func(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/middlewared/plugins/crypto_/certificates.py", line 323, in do_create
    await self.middleware.call(f'certificate.{self.map_functions[create_type]}', job, data)
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1386, in call
    return await self._call(
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1346, in _call
    return await self.run_in_executor(prepared_call.executor, methodobj, *prepared_call.args)
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1249, in run_in_executor
    return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
  File "/usr/lib/python3.9/concurrent/futures/thread.py", line 52, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/lib/python3/dist-packages/middlewared/schema.py", line 1322, in nf
    return func(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/middlewared/plugins/crypto_/certificates.py", line 373, in create_acme_certificate
    final_order = self.middleware.call_sync('acme.issue_certificate', job, 25, data, csr_data)
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1416, in call_sync
    return methodobj(*prepared_call.args)
  File "/usr/lib/python3/dist-packages/middlewared/plugins/acme_protocol_/issue_cert.py", line 96, in issue_certificate
    self.handle_authorizations(job, progress, order, dns_mapping, acme_client, key)
  File "/usr/lib/python3/dist-packages/middlewared/plugins/acme_protocol_/issue_cert.py", line 139, in handle_authorizations
    self.middleware.call_sync(
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1416, in call_sync
    return methodobj(*prepared_call.args)
  File "/usr/lib/python3/dist-packages/middlewared/schema.py", line 1322, in nf
    return func(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/middlewared/plugins/acme_protocol_/challenge.py", line 29, in perform_challenge
    authenticator.perform(*self.get_validation_parameters(data['challenge'], data['domain'], data['key']))
  File "/usr/lib/python3/dist-packages/middlewared/plugins/acme_protocol_/authenticators/base.py", line 28, in perform
    raise CallError(f'Failed to perform {self.NAME} challenge for {domain!r} domain: {e}')
middlewared.service_exception.CallError: [EFAULT] Failed to perform route53 challenge for 'local.kye.dev' domain: [EFAULT] Failed to get Hosted zones with provided credentials :An error occurred (SignatureDoesNotMatch) when calling the ListHostedZones operation: Signature expired: 20230303T110322Z is now earlier than 20230303T190722Z (20230303T191222Z - 5 min.)

 

[kyeotic]

This turned out to be a system time issue after all. The system time shown in System Settings > General was correct, but the time shown by the shell was not. I had to forcefully set the system time and restart the ntp service to get it working.