Configuration Filetransfer

Edit on Gitlab

Table of Contents

The file transfer describes how local directories will be backed up

  • choose backup tool; incremental or full backup
  • where to write data (backup target)
  • set password for local encryption
  • more options
  • how to cleanup old backup data

Details to the the given config entries you find in the description for transfer.job.

Backup tool

You can decide between Restic (Default) and Duplicity.

bin = ...

bin = restic
type = auto

If you use Duplicity for servers it is suggested to run (daily) incremental backups and full backups on a specific day.

bin = duplicity
type = inc
full = WDM:3rd Fri
start-time-full = 23:0.

If you use Duplicity on a desktop client you maybe want to run incremental backups and a full backup if the last one is older than a wanted time.

bin = duplicity
type = auto
auto = 1M

Backup target

The definition of a backup target depends on the used backup tool (restic|duplicity). You need a bit knowledge about these tools.

We have tested the following targets

  • local directory/ mounted filesystem i.e. Samba share
    • Restic: /mnt/mounted_directory
    • Duplicity: file:///mnt/mounted_directory
  • ssh target (scp or rsync)
    • Restic: sftp:imlbackup@backup-target.example.com:/mnt/backupdata
    • Duplicity: rsync://imlbackup@backup-target.example.com//mnt/backupdata
    • Duplicity: scp://imlbackup@backup-target.example.com//mnt/backupdata
  • https
    • Restic: rest:https://user:pass@backup-target.example.com:8000/user/

Set your value in storage = ...

Example for Restic with SFTP transfer

storage = sftp:imlbackup@backup-target.example.com:/mnt/backupdata

Local encryption

Restic an Duplicity encrypt local data with a password before transferring them.

Deploy a host specific password with an orchestration tool (Ansible, Puppet, …) or on a manual installation keep a copy of it on a safe place. Without the password you cannot decrypt backup data after a filesystem crash.

passphrase = EnterYourSecretHere

More options

Automatic backup of samba shares - it works only if samba shares were detected. If no samba config is available it has no effect and shows no error.

sambashares = 1

When restoring data … this is the path for restored files:

restore-path = /restore

Prune, Verify

After a transfer you can apply a verify or prune action.

To execute them after each run (which is not needed) set its values to 0.

prune-after = 0
verify-after = 0

You can execute prune and verify only if it didn’t run for N days. Set an integer in days here:

prune-after = 3
verify-after = 7

You can disable it to run it as an extra job using verify.sh or prune.sh. You must disable prune-after if you use rest:https to a restic rest server with enabled append-only option.

prune-after = never
verify-after = 7

Duplicity settings

When writing backup data duplicity creates the target file and a index. By default the index is written to ~/.cache/duplicity/ (and to the backup target). If your /root partition has a limited size and/ or you have a lot of files (i.e. on a file server) then you can put the index files somewhere else:

duplicity_cachedir = [full path]

Duplicity with rsync needed an additional ssh-backend pexpect on Debian. It can be set with enabling:

# duplicity_ssh-backend = pexpect

On the backup target you can define the size of the chunks to write. This would allow to write the backup data to an additional medium with limited size (i.e. CD or DVD).

Set a size in MB

duplicity_volsize = 350

Restic settings

When writing backup data restic creates a local index. By default the index is written to ~/.cache/restic/. If your /root partition has a limited size and/ or you have a lot of files (i.e. on a file server) then you can put the index files somewhere else:

restic_cachedir = [full path]

Restic by default stores file indexes locally. To save space in restic_cachedir you can disable the file index but the backup will run much slower.

restic_nochache = false

The default verbose of IML backup for restic is 0. If you want to see unchanged/ new/ deleted files set it to “2”. But This generates larger logs in [installdir]/logs/.

restic_verbose = 0

A file transfer can be tagged. Each restic backup gets a defined tag

restic_tag = imlbackup

For restore you can copy files to the restore location. With restic you can mount the backup volume into the local filesystem using fuse. Then you can browse in the snapshots of the backup data and copy the files and dirs with linux commands.

The mountpoint for the backup storage is a full path (it will be created when selcting to mount in the restore).

restic_mountpoint = /mnt/restore

Cleanup options

Duplicity

Duplicity can set a value how long to keep backup data on the backup storage. It will keep all files of incremental and full backup to be able to restore data of the given min. availability.

Example if you defined a monthly full backup and keep backups for 6 monthes … then it keeps all files up to 6 monthes plus all older files up to the next full backup. You will find backup files up to 7 monthes.

  1 month    (for monthly full backup)
+ 6 monthes  (min. availability for restore)
---------------------------------------------
  7 monthes

The value duplicity_keep is a number and a letter for the time format.

s, m, h, D, W, M, or Y (indicating seconds, minutes, hours, days, weeks, months, or years respectively)

duplicity_keep = 6M

Restic

Restic works like a repository and uses deduplication. The first run of a file backup is a full backup and all following runs make an incremental backup.

There are 2 simple cleanup variants for restic forget:

  • The value for “keep-last” never deletes the last N most recent snapshots.
  • “restic_keep-within” keeps all created snapshots within a given period (eg. 1y5m7d2h)

If you need more control and want less snapshots as older they are:

You have the possibility to keep all younger backups and have the possibility to delete older backups but to keep N weekly, monthly and yearly snapshots.

Multiple values are allowed and work like an OR condition. See the docs for more details to the keep values of the forget command https://restic.readthedocs.io/en/latest/060_forget.html

# prune 
restic_keep-within = 6m
# restic_keep-last = 5
# restic_keep-hourly = 10
# restic_keep-daily = 30
# restic_keep-weekly = 12
# restic_keep-monthly = 12
# restic_keep-yearly = 10

Supported values are:

  • restic_keep-last [N] - keep the last n snapshots
  • restic_keep-(hourly|daily|weekly|monthly|yearly) [N] - keep then last n snapshots of a period
  • restic_keep-within [duration] - keep snapshots that are newer than duration (eg. 1y5m7d2h) relative to the latest snapshot
  • restic_keep-within-(hourly|daily|weekly|monthly|yearly) [duration] - keep snapshots of given period that are newer than duration (eg. 1y5m7d2h) relative to the latest snapshot
  • restic_keep-tag [taglist] - keep snapshots with this taglist (can be specified only one in imlbackup) (default [])

restic_keep-within = 6m