Borg Backup to Rsync.net Recently Started Failing

[jde]

In April, 2018, I setup a borg backup script in an 11.1 jail which ran by a daily chron job. My script created a daily archive which backed up to my rsync.net account. Everything ran perfectly until recently when I noticed that the last daily archive stopped on 12/2/2018. From the jail shell, I attempted to manually create a borg archive, however, no luck. Thinking perhaps the EOL on FreeBSD 11.1 was potentially causing some issue, I resolved to wait on FreeNAS 11.2 to upgrade the jail and see if it would work.

A few days later on 12/6/18, I upgraded to FreeNAS 11.2, and created a new 11.2 jail. I was able to create a borg remote repository with

Code:

borg init --encryption=repokey xxxx@usw-s009.rsync.net:backup

However, as before, borg archive creation

Code:

borg create --debug xxxx.usw-s009.rsync.net:backup::daily1 /mnt

failed. I have tried running borg backups using the default rsync.net borg 0.29 server side version, and the --remote-path=borg1 option which calls borg 1.5.1 server side. I have even tried running borg from Ubuntu and had similar failures.

I'm thinking there might be a problem on rsync.net's server side implementation of borg. As I said, my backups ran perfectly for over six months.

Does anyone else on the forum do borg backups to rsync.net? If so, can you check to see if they are currently running successfully?

Freenas System:
FreeNas 11.2 Stable | Supermicro X10SLM+-F-O | Intel Xeon 1220 v3 | 16 GB (2 x 8 GB) Crucial DDR3 1.35V ECC | 5 x 3TB WD Red in RaidZ2 | SanDisk SSD Boot Drive | Seasonic SS 300ET 300W | Generic ATX full tower case

 

[jde]

As additional info, here is a log from the failure:

Code:

Remote: Fssh_packet_write_wait: Connection to 64.62.236.78 port 22: Broken pipe
RemoteRepository: 17.09 MB bytes sent, 3.84 kB bytes received, 33 messages sent
Connection closed by remote host
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/borg/archiver.py", line 152, in wrapper
    return method(self, args, repository=repository, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/borg/archiver.py", line 554, in do_create
    create_inner(archive, cache)
  File "/usr/local/lib/python3.6/site-packages/borg/archiver.py", line 518, in create_inner
    read_special=args.read_special, dry_run=dry_run, st=st)
  File "/usr/local/lib/python3.6/site-packages/borg/archiver.py", line 631, in _process
    read_special=read_special, dry_run=dry_run)
  File "/usr/local/lib/python3.6/site-packages/borg/archiver.py", line 631, in _process
    read_special=read_special, dry_run=dry_run)
  File "/usr/local/lib/python3.6/site-packages/borg/archiver.py", line 631, in _process
    read_special=read_special, dry_run=dry_run)
  File "/usr/local/lib/python3.6/site-packages/borg/archiver.py", line 605, in _process
    status = archive.process_file(path, st, cache)
  File "/usr/local/lib/python3.6/site-packages/borg/archive.py", line 1028, in process_file
    self.chunk_file(item, cache, self.stats, backup_io_iter(self.chunker.chunkify(fd, fh)))
  File "/usr/local/lib/python3.6/site-packages/borg/archive.py", line 956, in chunk_file
    item.chunks.append(chunk_processor(data))
  File "/usr/local/lib/python3.6/site-packages/borg/archive.py", line 944, in chunk_processor
    chunk_entry = cache.add_chunk(self.key.id_hash(data), data, stats, wait=False)
  File "/usr/local/lib/python3.6/site-packages/borg/cache.py", line 898, in add_chunk
    self.repository.put(id, data, wait=wait)
  File "/usr/local/lib/python3.6/site-packages/borg/remote.py", line 476, in do_rpc
    return self.call(f.__name__, named, **extra)
  File "/usr/local/lib/python3.6/site-packages/borg/remote.py", line 707, in call
    for resp in self.call_many(cmd, [args], **kw):
  File "/usr/local/lib/python3.6/site-packages/borg/remote.py", line 807, in call_many
    raise ConnectionClosed()
borg.remote.ConnectionClosed: Connection closed by remote host

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/borg/archiver.py", line 4434, in main
    exit_code = archiver.run(args)
  File "/usr/local/lib/python3.6/site-packages/borg/archiver.py", line 4366, in run
    return set_ec(func(args))
  File "/usr/local/lib/python3.6/site-packages/borg/archiver.py", line 152, in wrapper
    return method(self, args, repository=repository, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/borg/remote.py", line 630, in __exit__
    self.rollback()
  File "/usr/local/lib/python3.6/site-packages/borg/remote.py", line 476, in do_rpc
    return self.call(f.__name__, named, **extra)
  File "/usr/local/lib/python3.6/site-packages/borg/remote.py", line 707, in call
    for resp in self.call_many(cmd, [args], **kw):
  File "/usr/local/lib/python3.6/site-packages/borg/remote.py", line 807, in call_many
    raise ConnectionClosed()
borg.remote.ConnectionClosed: Connection closed by remote host

 

[jde]

As an update, the problem does not appear to be caused by rsync.net. I would like to point out that the folks at rsync.net were helpful and responsive in eliminating anything on their end being the problem. They did so even though the lost cost Borg only account I have with them does not officially include support. I noted yesterday that rsync.net recently dropped their Borg only accounts from $0.03/GB/Mo. To $0.02/GB/Mo.


My problem appears to be caused by something with my home ISP (at&t 100 Mbps fiber), or possibly my home firewall (pfsense). I was able to confirm the problem by attempting to create a Borg remote archive from my office (Comcast ISP) to my home machine (at&t) as the target. It failed in exactly the same way as when I tried to create a remote Borg archive from home to rsync.net. To eliminate the 11.2 jail and Borg 1.1.7 as the problems, I successfully ran a remote borg backup between two jails. I was also able to successfully create a remote Borg archive from my office (Comcast ISP) to rsync.net. I don't think my pfsense firewall is the problem as both my home and office use such a firewall with virtually identical settings.

Anyone know of any problems caused by at&t breaking Borg or ssh or the like?