Scripts

Table of Contents

storage_helper.sh

The script should be alwys started with the non privileged user (imlbackup).

Without a parameter you get a help:

imlbackup@backup-host:~$ ./storage_helper.sh 

---------- :: STORAGE :: storage-connector :: ----------

SYNTAX:
storage_helper.sh [function]

  backupstatus
    show all servers and their backup times

  register [hostname]
    add a slot for a backup client

  unregister [hostname] [statuscode]
    remove slot for a backup client.
    The statuscode is an integer of the return code.

  setactive
    Reactivate an inctave backup repo.

  setinactive
    Mark a backup repo as inactive.
    This prevents errors of missing backup data
    if a host is obsolete and was shut down.

  status
    show current reserved slots

  usage [full|[filter to hostname]] - DEPRECATED
    show used diskspace
    you can set a string to show additionally details if these hosts
    if you add "full" then details of all hosts will be shown

What backup clients do

Before a backup client send its data it wants to register. ./storage_helper.sh register [hostname]

The exitcode is 0 if it got a free slot. And non-zero if all slots are in use (see iMaxConnections in config). The backup client will wait a random time until it gets a free slot to write its data.

If the backup is finished it sends ./storage_helper.sh unregister [hostname]

Admin tasks

See current activity

$ ./storage_helper.sh status

---------- :: STORAGE :: storage-connector :: ----------

STATUS: 0 existing connection(s). Allowed maximum is 10.

Backup status

You get a table of all servers that sent backup data. There are listed

  • last backup with start time, end time, duration, existcode
  • age of the last backup (to see if a system does not send data anymore)
  • size of backup data
  • Each line per server is colored depending on backup status or time of last sent data
$ ./storage_helper.sh backupstatus

---------- :: STORAGE :: storage-connector :: ----------

STATUS: 0 existing connection(s). Allowed maximum is 10.

This table shows the time and duration [s] of the last backup for each server.

  server                     | start                | end                  | duration |  rc | age [h] | size 
-----------------------------------------------------------------------------------------------------------------------------
. srv1.example.com            | 2022-09-19 23:04:14  | 2022-09-19 23:17:52  |      818 |   0 |      12 | 951G /netshare/restic-backup/srv1.example.com 
. srv2.example.com            | 2022-09-19 23:03:06  | 2022-09-19 23:18:28  |      922 |   0 |      12 | 955G /netshare/restic-backup/srv2.example.com
...

Legend
. OK | ? not started | R running | D disabled | E error

total : 26 servers
errors: 0

rc=0

deactivate a backup folder

If you delete a host but must keep its backup data then you can set it inactive. Then you don’t get an error because of missing incoming data.

To disable a backup target:

$ ./storage_helper.sh setinactive

---------- :: STORAGE :: storage-connector :: ----------

--- list of backup repositories
/netshare/restic-backup/srv1.example.com (active)
/netshare/restic-backup/srv1.example.com (active)
...
Repo to deactivate >

Enter the full path here, eg. /netshare/restic-backup/srv1.example.com and Return. Then comes an optional input for a hint - that is just for you and your team. Leave a message if the host is deprecated, the application was moved etc.

In the backup status the deactivated target is listed with D = Disabled and is gray.

BTW: the opposite way is possible to:

./storage_helper.sh setactive

rest_pruner.sh

!! This script is in version 0.1 - and work in progress !!

The pruner script is for restic rest server with append only option. It can prune all repositories on server side.

inc_config.sh

The inc_config.sh mus contain 3 variables for pruning

    # for prune on restic rest server
    prune_basedir=/netshare/restic-backup
    prune_params="--group-by paths,tags --prune --keep-within 180d --max-unused unlimited --max-repack-size 10G --cleanup-cache"
    prune_skipdays="7"
    prune_cachedir=${prune_basedir}/.cache_for_pruning

rest_pruner.cfg

To access the different local repositories we need the RESTIC_PASSWORD for each repository. The config file rest_pruner.cfg contains lines in the syntax

<USER>:<RESTIC_PASSWORD>

If a directory matches ${prune_basedir}/<USER> then it will be pruned.

You need a mechanism to create this file eg. by Ansible.

For securiy reasons this file must be owned by root:root and must have the permissions 0400.

ls -l rest_pruner.cfg
-r--------. 1 root root 159 Feb  1 13:35 rest_pruner.cfg

Syntax

------------------------------------------------------------------------------
  --------========###|    RESTIC REST PRUNER :: v0.4    |###=======--------
------------------------------------------------------------------------------

Pruner for restic rest server with append only option.
This script prunes all repositories on server side.

The config file [rest_pruner.cfg] contains <USER>:<RESTIC_PASSWORD>
If a directory matches /netshare/restic-backup/<USER> then it will be pruned.

Institute for Medical Education * University of Bern
GNU GPL 3.0


SYNTAX:
  rest_pruner.sh [OPTIONS] [FILTER]


OPTIONS:
  -h, --help           show help and exit.
  -a, --all            process all repositories, default: rest only + archives
  -d, --debug          show more infos
  -f, --force          force pruning; do not wait 7 days
  -s, --status         show pruning status with last prunes


PARAMETERS:
  FILTER               regex to filter directory list in
                       /netshare/restic-backup/*

EXAMPLES:
  rest_pruner.sh
                       Start pruning of all matching repositories

  rest_pruner.sh mail
                       Prune servers that match "mail",
                       eg. my-mailhub.example.com

  rest_pruner.sh --force mail
                       Prune servers that match "mail",
                       eg. my-mailhub.example.com

  rest_pruner.sh --status
                       Show statistics with last prunes, if it is archive or
                       running or on error.
                       It shows repositories with passwords and archives.
                       Other repositories are not visible until you
                       specify --all.

How does it work

It detects some requirements:

  • was it started by root? (The help is shown without being root)
  • permissions of rest_pruner.cfg
  • variables from inc_config.sh

It loops over the starting dir ${prune_basedir} and reads all its subdirectories. If a subdir matches a configuration entry (text before first “:” <USER>:<RESTIC_PASSWORD>) then the prune process will be started. It detects the username of the owner of the directory and executes

su - $_user - /bin/bash -c "<restic forget ...>"

Exitcodes:

  • 0 if the run was without errors.

  • 0 if a prune run of a reposaitory failed.

  • 1..3 if a requirement failed to start

The output of the prune command is written into

_last_prune/<USER>.log

If a job failed it is renamed to

_last_prune/<USER>.log.error