rzerres/dsnap-sync
1
0
Fork 0
Code Issues Pull Requests Releases 1 Activity

Markdown: update markup text to reflact changes for verson 0.5.7

Browse Source 
* Basic description in README.md
* Example section in /usr/share/doc/Examples.md
This commit is contained in:
Ralf Zerres
2018-07-15 16:04:38 +02:00
parent f461599b2e
commit 9628a9a623
2 changed files with 386 additions and 183 deletions
Download Patch File Download Diff File Expand all files Collapse all files

341
README.md
View File

@@ -1,5 +1,5 @@
<!-- dsnap-sync README.md -->
<!-- version: 0.5.5 -->
<!-- version: 0.5.7 -->
# dsnap-sync
@@ -10,238 +10,214 @@ It takes advantage of the specific snapshots functionality btrfs offers
and combines it with managemnet functionality of snapper.
`dsnap-sync` creates backups as btrfs-snapshots on a selectable target device.
Plug in and mount any btrfs-formatted device to your system. Supported devices
may be either local USB drives, but can be as well remote accessible RAID drives.
If possible the backup process will send incremental snapshots to the target drive.
If the snapshot will be stored on a remote host, it is secured with ssh.
Plug in and mount any btrfs-formatted device to your system. Supported targets
may be either local attached USB drives, as well as automountable remote host
RAIDs.
If possible the backup process will send incremental snapshots to the target
drive. If the snapshot will be stored on a remote host, the transport will be
secured with ssh.
The tool is implemented as a posix shell script, to keep the footprint small (dash).
`dsnap-sync` will support interactive and time scheduled backup runs.
The tool is implemented as a posix shell script (dash), to keep the footprint
small. `dsnap-sync` will support interactive and time scheduled backup processes.
Scheduling should be implemented as a pair of systemd service and timer-units.
The [example section](usr/share/doc/Examples.md#systemd)
will offer details as a reference point.
## Backup process
For a backup run, and per default `dsnap-sync` will iterate through all defined snapper
configurations found on your source system. If you prefer to just run on a specific
configuration per call, you are free to select it using the 'config' option `-c`.
The default `dsnap-sync` backup process will iterate through all defined snapper
configurations that are found on your source system. If you prefer to have single
processes for each configuration, you are free to define isolated systemd-units or
call `dsnap-sync` interactively while referencing the intended snapper configuration
(option `-c` or `--config`).
For each selected snapper configuration `dsnap-sync`
* will create an appropriate local snapshot and update the metadata
* will transfer the snapshot using btrfs-send to the target device
* will create and update the snapper configuration on the target
* will update the metadata on the target device
* will present/select target device informations
* will prepare snapper structures
* will perform the actual backup
(handle backupdir, handle snapper structures, handle btrfs send /btrfs recieve)
* will finalize the actual backup
(update snapper metadata for each source and target snapshot)
* will perform cleanup tasks
Usualy tools will document this proccess as a disk to disk (d2d) backup.
If possible `dsnap-sync` will levarage btrfs-send capabilities to only
send deltas. It will compare the snapshot data of the ongiong process with available
snapshot data on the target device.
Usualy other tools will document this proccess as a disk to disk (d2d) backup.
If possible `dsnap-sync` will levarage `btrfs send` capabilities to only
send deltas. It will compare snapshot data of the source snapshot with available
snapshot data on the target device. If a common snapshot id exists on source and
target, `dsnap-sync` will prepare the `btrfs send / btrfs receive` pipe to use
them respectively. This functionality dasticly reduces the time a sync process
will need to complete compared to a full backup process.
### Interactive backups
An interactive run will request you to select a mounted btrfs device.
You can pre-select the target drive via [command line options](https://github.com/rzerres/dsnap-sync#options).
Either use a UUID, a SUBVOLID or a TARGET name (read 'mount point').
An interactive process will gide you to select a mounted btrfs device.
You can pre-select the target drive via [command line options](./README.md#Options).
To uniquely define / select a target devices you either need to choose
* a pair of a btrfs UUID and SUBVOLID
* a TARGET name (read 'mount point')
This will asure, that `dsnap-sync` can distinguish backup processes that
have a commen source device, but save data to different target devices. As
an example it might be advisable, to save the project subvolume redundantliy
on to independent targets (disk and tape).
Before `dsnap-sync` will perform the backup, it will present the backupdir,
spit out the source and target locations will. You have to confirm or adapt
the given values. You may use commandline options to supress interaction
(e.g --noconfirm, --batch).
### Scheduled backups
A scheduled run will take all needed parameters from config options.
`dsnap-sync` does support systemd.timer units. Please refer to related paragraph [documenting systemd](https://github.com/rzerres/dsnap-sync#systemd).
A scheduled process should be defined as a systemd.unit. Inside the unit
definition the execution parameter will take the `dsnap-sync` call, appending
all needed parameters as config options. In combination with a corresponding
systemd.timer unit, you are able to finetune your backup needs.
The [example section](usr/share/doc/Examples.md#systemd)
will offer details as a reference point.
## Requirements
beside the shell itself, `dsnap-sync`relies on external tools to achieve its goal.
At run-time their availability is checked. Following tools are are used:
Beside the posix shell itself (e.g. `dash`), `dsnap-sync`relies on external
tools to achieve its goal. At run-time their availability is checked.
Following tools are used:
- awk
- btrfs
- findmnt
- sed
- snapper
- tee
- wc
optionaly tools
As an option, you can enrich interactive responses using
- notify-send
- pv
## Installation
### Building from source
`dsnap-sync` is a shell script. Thus no compilation is required.
To simplify correct target locations, this project uses a Makefile.
# make install
If your system uses a non-default location for the snapper
configuration file, specify it on the command line with
`SNAPPER_CONFIG`. For example, for Arch Linux use:
configuration defaults, specify the location with an environment variable
(`SNAPPER_CONFIG`).
Arch Linux/Fedora/Gentoo:
# make SNAPPER_CONFIG=/etc/conf.d/snapper install
Debian/Ubuntu:
# make SNAPPER_CONFIG=/etc/default/snapper install
The local snapper configuration will be extended to make use
of a new template 'dsnap-sync'.
The package is also available in the
<!-- [AUR](https://aur.archlinux.org/packages/dsnap-sync/). -->
### Using distribution packages
If available, you can install `dsnap-sync` as a precompiled package.
Please use your host software package manager.
<!--
* For ARCH-Linux
[AUR package](https://aur.archlinux.org/packages/dsnap-sync)
-->
<!-- For Debian
[deb package](https://packages.debian.org/dsnap-sync). -->
<!-- For Ubuntu
[deb package](https://packages.ubuntu.org/dsnap-sync). -->
## Options
Usage: dsnap-sync [options]
Options:
-b, --backupdir <prefix> backupdir is a relative path that will be appended to target backup-root
-d, --description <desc> Change the snapper description. Default: "latest incremental backup"
--label-finished <desc> snapper description tagging successful jobs. Default: "dsnap-sync backup"
--label-running <desc> snapper description tagging active jobs. Default: "dsnap-sync in progress"
--label-synced <desc> snapper description tagging last synced jobs.
Default: "dsnap-sync last incremental"
--color Enable colored output messages
-c, --config <config> Specify the snapper configuration to use. Otherwise will perform for each snapper
configuration. Can list multiple configurations within quotes, space-separated
(e.g. -c "root home").
--config-postfix <name> Specify a postfix that will be appended to the destination snapper config name.
-n, --noconfirm Do not ask for confirmation for each configuration. Will still prompt for backup
--batch directory name on first backup"
--nonotify Disable graphical notification (via dbus)
--nopv Disable graphical progress output (disable pv)
-r, --remote <address> Send the snapshot backup to a remote machine. The snapshot will be sent via ssh.
You should specify the remote machine's hostname or ip address. The 'root' user
must be permitted to login on the remote machine.
-p, --port <port> The remote port.
-s, --subvolid <subvlid> Specify the subvolume id of the mounted BTRFS subvolume to back up to. Defaults to 5.
-u, --uuid <UUID> Specify the UUID of the mounted BTRFS subvolume to back up to. Otherwise will prompt."
If multiple mount points are found with the same UUID, will prompt user."
-t, --target <target> Specify the mountpoint of the BTRFS subvolume to back up to.
--remote <address> Send the snapshot backup to a remote machine. The snapshot will be sent via ssh. You
should specify the remote machine's hostname or ip address. The 'root' user must be
permitted to login on the remote machine.
--dry-run perform a trial run where no changes are made.
-v, --verbose Be more verbose on what's going on (use multiple times to be more verbose).
--version show program version
Options:
-a, --automount <path> start automount for given path to get a valid target mountpoint.
-b, --backupdir <prefix> backupdir is a relative path that will be appended to target backup-root
-d, --description <desc> Change the snapper description. Default: "latest incremental backup"
--label-finished <desc> snapper description tagging successful jobs. Default: "dsnap-sync backup"
--label-running <desc> snapper description tagging active jobs. Default: "dsnap-sync in progress"
--label-synced <desc> snapper description tagging last synced jobs.
Default: "dsnap-sync last incremental"
--color Enable colored output messages
-c, --config <config> Specify the snapper configuration to use. Otherwise will perform for each snapper
configuration. Can list multiple configurations within quotes, space-separated
(e.g. -c "root home").
--config-postfix <name> Specify a postfix that will be appended to the destination snapper config name.
-n, --noconfirm Do not ask for confirmation for each configuration. Will still prompt for backup
--batch directory name on first backup"
--nonotify Disable graphical notification (via dbus)
--nopv Disable graphical progress output (disable pv)
-r, --remote <address> Send the snapshot backup to a remote machine. The snapshot will be sent via ssh.
You should specify the remote machine's hostname or ip address. The 'root' user
must be permitted to login on the remote machine.
-p, --port <port> The remote port.
-s, --subvolid <subvlid> Specify the subvolume id of the mounted BTRFS subvolume to back up to. Defaults to 5.
-u, --uuid <UUID> Specify the UUID of the mounted BTRFS subvolume to back up to. Otherwise will prompt."
If multiple mount points are found with the same UUID, will prompt user."
-t, --target <target> Specify the mountpoint of the BTRFS subvolume to back up to.
--remote <address> Send the snapshot backup to a remote machine. The snapshot will be sent via ssh. You
should specify the remote machine's hostname or ip address. The 'root' user must be
permitted to login on the remote machine.
--dry-run perform a trial run where no changes are made.
-v, --verbose Be more verbose on what's going on.
--version show program version
## First run
If you have never synced to the paticular target device (first run), `dsnap-sync`
will take care to create the necessary target file-structure to store the snapshot.
As an option you can prepend a backup-path.
will take care to create the necessary target filesystem-structure.
Before the sync job is started, source and target locations will be presented.
You have to confirm any further operation, or use defaults (option: noconfirm).
* btrfs-snapshots
## Example command line usage
On the target device the needed snapper structure is validated and build up as needed.
Aside the new filesystem path, `dsnap-sync` will use a snapper template
(`/etc/snapper/config-templates/dsnap-sync`) to create a new snapper target configuration.
To garantee unique configuration names, `dsnap-sync` take the source configuration name
and postfix it with targets hostname. You can adopt this behaviour with a config option
(`--config-postfix`).
The default `config-template` of dsnap-sync will inherit following snapper parameters:
* mark new snapshots as type 'single'
* mark new snaphosts with cleanup-algorithm 'timeline'
* apply config option 'CONFIG_TYPE=child'
* apply config option 'TIMELINE_CREATE=no'
* apply config option 'TIMELINE_CLEANUP=yes'
Please adapt the defaults, if your milage varies.
* btrfs-archive
### dsnap-sync to local target
Beside btrfs-snapshots, `dsnap-shot` will support another backup type:
btrfs-archive. If the target device is not a btrfs filesystem (e.g. ext4,
xfs, ltofs tapes), data are copied backupdir.
`dsnap-sync` will take the source snapshot-ID and create a corresponding directory
below targets backupdir. Inside this 'target-subdirectory' `dsnap-sync` will mimic the
snapper structure:
* the actual btrfs stream will we copied to a subdirectoy called `snapshot`
* the proccess metadata are saved to a file called `info.xml`
#### Default: no selections, run for all snapper configs
## Automounter
# dsnap-sync
`dsnap-sync` offer all mounted btrfs filesystems as valid process targets.
Since storage space on disks are very price efficient this days, environments
often use removable, external disks as additional backup targets. If the
external disks aren't mounted at boot time they can't be addressed by the
selection function. It's even advisable to not mount them all the time
(e.g prevent risks for malware encryption attacks).
#### Default: Select two configs, the backupdir and verbose output
# dsnap-sync --verbose --config root --config data2 --backupdir=toshiba_r700
#### Dry-run: Select config, select Target, as batchjob (--noconfirm)
# dsnap-sync -c root -s 265 --noconfirm --dry-run
### dsnap-sync to remote host
`dsnap-sync` will rely on ssh access to the target host. For batch usage make sure, that your
public key is accepted for remote login as user 'root'. You may have to adapt /root/.ssh/authorized_keys
on the target host.
On your target host, you should also verify the availability of a dsnap-sync config-template for snapper.
A template `dsnap-sync` is included in the package for your convenience.
#### Dryrun: Select remote host <ip/fqdn>, interactive, run for all configs
dsnap-sync --dry-run --remote 172.16.0.3
Selecting a mounted BTRFS device for backups on 172.16.0.3.
0) / (uuid=5af3413e-59ea-4862-8cff-304afe25420f,subvolid=257,subvol=/root)
1) /.snapshots (uuid=5af3413e-59ea-4862-8cff-304afe25420f,subvolid=258,subvol=/@snapshots-root)
2) /data2 (uuid=62a45211-9197-4a5f-aeaf-0ab803a42c32,subvolid=261,subvol=/data2)
3) /home (uuid=62a45211-9197-4a5f-aeaf-0ab803a42c32,subvolid=258,subvol=/home)
4) /data2/.snapshots (uuid=62a45211-9197-4a5f-aeaf-0ab803a42c32,subvolid=262,subvol=/@snapshots-data2)
5) /home/.snapshots (uuid=62a45211-9197-4a5f-aeaf-0ab803a42c32,subvolid=259,subvol=/@snapshots-home)
6) /var/lib/machines (uuid=2ba04452-74aa-44df-b1c7-74e0a70c6543,subvolid=260,subvol=/machines)
7) /var/lib/libvirt (uuid=2ba04452-74aa-44df-b1c7-74e0a70c6543,subvolid=261,subvol=/libvirt)
8) /data (uuid=2ba04452-74aa-44df-b1c7-74e0a70c6543,subvolid=257,subvol=/data)
9) /var/lib/machines/.snapshots (uuid=2ba04452-74aa-44df-b1c7-74e0a70c6543,subvolid=2121,subvol=/@snapshots-machines)
10) /data/.snapshots (uuid=2ba04452-74aa-44df-b1c7-74e0a70c6543,subvolid=258,subvol=/@snapshots-data)
11) /var/lib/dsnap-sync (uuid=753eba7a-41ce-49e0-b2e3-24ee07811efd,subvolid=420,subvol=/dsnap-sync)
x) Exit
Enter a number: 11
### Dry-run with given Target for snapper config 'home', no confirmations
#### Sync: Select config 'data2', remote host <ip/fqdn>, target '/data', as batchjob (--noconfirm)
# dsnap-sync --config data2 --remote 172.16.0.3 --target /data --noconfirm
## systemd
### service
[Unit]
Description=Run dsnap-sync backup
[Install]
WantedBy=multi-user.target
[Service]
Type=simple
ExecStart=/usr/bin/dsnap-sync --UUID 7360922b-c916-4d9f-a670-67fe0b91143c --subvolid 5 --noconfirm
### timer
[Unit]
Description=Run dsnap-sync weekly
[Timer]
OnCalendar=weekly
AccuracySec=12h
Persistent=true
[Install]
WantedBy=timers.target
## snapper template
###
# template for dsnap-sync handling
###
# subvolume to snapshot
SUBVOLUME="/var/lib/dsnap-sync"
# filesystem type
FSTYPE="btrfs"
# users and groups allowed to work with config
ALLOW_USERS=""
ALLOW_GROUPS="adm"
# sync users and groups from ALLOW_USERS and ALLOW_GROUPS to .snapshots
# directory
SYNC_ACL="yes"
# start comparing pre- and post-snapshot in background after creating
# post-snapshot
BACKGROUND_COMPARISON="yes"
# run daily number cleanup
NUMBER_CLEANUP="no"
# limit for number cleanup
NUMBER_MIN_AGE="1800"
NUMBER_LIMIT="10"
NUMBER_LIMIT_IMPORTANT="2"
# use systemd.timer for timeline
TIMELINE_CREATE="no"
# use systemd.timer for cleanup
TIMELINE_CLEANUP="no"
# dsnap-sync as timer unit
SNAP_SYNC_EXCLUDE="yes"
To link external disks in dynamically, but also asure a persistent naming
syntax, we can use them as auto-mountable targets. To wakeup the automount
proccess before parsing available target disks, append the target mount-point
as a config option to `dsnap-sync` (e.g: `--automount /var/backups/archive-disk1`).
The [example section](usr/share/doc/Examples.md#Automounter)
will offer details as a reference point.
## Contributing
@@ -251,7 +227,7 @@ TODO list. It might be already on the agenda.
## Related projects
I did fork from Wes Barnetts original work. I was aiming merged it back.
I did fork from Wes Barnetts original work. I intend to merged it back.
Beside the fact that this version doesn't use any bashisms, Wes did let me know,
that he doesn't have the time to review the changes appropriately to make it a merge.
Anyone willing to do so is invided.
@@ -259,7 +235,6 @@ Anyone willing to do so is invided.
Until that date, i will offer this fork for the public. To overcome any name clashes
i renamed it to dsnap-sync.
## License
<!-- License source -->

228
usr/share/doc/Examples.md Normal file
View File

@@ -0,0 +1,228 @@
# Examples
## Automounter
`dsnap-sync` will take advantage of systemd automount units to incorporate external,
removable disks into the selection process for target devices.
### btrfs disk
Format the disk with btrfs tools to prepare it as a target.
The following example will reference to a btrfs disk and mount a given subvolume.
### systemd automount units
#### mount unit: var-backups-archive\x2ddisk1.mount
[Unit]
Description=Backup - Archiv-Disk 1
Documentation=man:systemd.mount(5) man:mount.btrfs(8)
DefaultDependencies=yes
[Mount]
What=UUID=977b4ecf-be67-4643-84f5-10b368c24d25
Where=/var/backups/archive-disk1
Type=btrfs
Options=defaults,subvol=@archive-disk1,compress=lzo
[Install]
WantedBy=multi-user.target
#### automount unit: var-backups-archive\x2ddisk1.automount
[Unit]
Description=Automount Backup - Archive-Disk 1
Documentation=man:systemd.automount(5)
[Automount]
Where=/var/backups/archive-disk1
TimeoutIdleSec=45
[Install]
WantedBy=multi-user.target
## `dsnap-sync` command line usage
### backup to local target
#### Default: no selections, run for all snapper configs
# dsnap-sync
#### Default: Select two configs, the backupdir and verbose output
# dsnap-sync --verbose --config root --config data2 --backupdir=toshiba_r700
#### Dry-run: Select config, select Target, as batchjob (--noconfirm)
# dsnap-sync -c root -s 265 --noconfirm --dry-run
### backup to remote target
`dsnap-sync` will rely on ssh access to the target host. For batch usage
make sure, that your public key is accepted for a remote login as user
'root'. You may have to adapt /root/.ssh/authorized_keys on the target host.
On your target host, you should also verify the availability of a
`dsnap-sync` config-template for snapper. A template `dsnap-sync`
is included in the package for your convenience.
#### Dryrun: Select remote host <ip/fqdn>, interactive, run for all configs
dsnap-sync --dry-run --remote <fqdn/ip>
Select target disk...
0) /var/lib/snap-sync (uuid=b5915f47-38a8-4b37-8de6-c8b15d9aba5a,subvolid=257,subvol=/var/lib/@snap-sync)
1) /var/backups/archive-disk4 (uuid=137b4a36-6e17-487b-a328-29b0d1e020c3,subvolid=257,subvol=/@dws-archive-disk4)
2) /var/backups/archive-disk3 (uuid=c9fcbfd3-6a1f-4b43-8176-b6f326bf46c7,subvolid=257,subvol=/@dws-archive-disk3)
3) /var/backups/archive-disk1 (uuid=977b4ecf-be67-4643-84f5-10b368c24d25,subvolid=257,subvol=/@dws-archive-disk1)
4) /var/backups/archive-disk2 (uuid=b68c8a31-9878-4ec1-b7ed-948de2125285,subvolid=257,subvol=/@dws-archive-disk2)
x) Exit
Enter a number: 1
### Dry-run with given Target for snapper config 'data2', no confirmations
#### Sync: Select config 'data2', remote host <ip/fqdn>, target '/var/lib/snap-sync', as batchjob (--noconfirm)
# dsnap-sync --config data2 --remote <fqdn/ip> --target /var/lib/snap-sync --batch --verbose
## systemd
`dsnap-sync` will structure all scheduling tasks while using systemd units.
To perform a backup process for just a single snapper configuration at a
given time, you have to define a pair of a systemd service unit and a
corresponding systemd timer unit.
Below we define a generic `dsnap-sync service unit` that should be located
at /etc/systemd/system. Following call will reference this template:
`systemd enable dsnap-sync@data2.service`
### service unit: `dsnap-sync@.service
[Unit]
Description=dsnap-sync backup for target %i
[Install]
WantedBy=multi-user.target
[Service]
Type=simple
ExecStart=/usr/bin/dsnap-sync \
--config %i \
--uuid 7360922b-c916-4d9f-a670-67fe0b91143c \
--subvolid 5 \
--remote backup-host
--batch
### overriding service unit: `dsnap-sync@data2.service`
Please remember, that the template example encode a given target
UUID and SUBVOLID. If you want the unit to serve individual parameter,
you have to override the it like:
`systemd edit dsnap-sync@data2.service`
Define a service paragraph, clean out the ExecStart= parameter and
refine a new ExedStart= parameter with the intended.
[Service]
ExecStart=
ExecStart=/usr/bin/dsnap-sync \
--config %i \
--target /var/lib/dsnap-sync \
--remote my-backup-host
--batch
### timer unit: `dsnap-sync@.timer`
Below we define a generic `dsnap-sync timer unit` that should be located
at /etc/systemd/system.
[Unit]
Description=dsnap-sync weekly backup
[Timer]
OnCalendar=weekly
AccuracySec=12h
Persistent=true
[Install]
WantedBy=timers.target
Following call will reference this template:
`systemd enable dsnap-sync@data2.timer`
## snapper extensions
For any new `dsnap-sync` btrfs-snapshot a new target snapper structure is need,
if the target backups should be managable via snapper.
`dsnap-sync` will create this structure as needed. During the creation it
will reference to a template called `dsnap-sync`. Please adapt it
if your milage varies.
### `dsnap-sync` template
###
# snapper template for dsnap-sync handling
###
# subvolume to snapshot
SUBVOLUME="/var/lib/dsnap-sync"
# filesystem type
FSTYPE="btrfs"
# users and groups allowed to work with config
ALLOW_USERS=""
ALLOW_GROUPS="adm"
# sync users and groups from ALLOW_USERS and ALLOW_GROUPS to .snapshots
# directory
SYNC_ACL="yes"
# start comparing pre- and post-snapshot in background after creating
# post-snapshot
BACKGROUND_COMPARISON="yes"
# handele NUMBER_CLEANUP via systemd, if a timer unit is active
NUMBER_CLEANUP="yes"
# limit for number cleanup
NUMBER_MIN_AGE="1800"
NUMBER_LIMIT="52"
NUMBER_LIMIT_IMPORTANT="12"
# handle TIMELINE via systemd, if a timer unit is active
TIMELINE_CREATE="yes"
# create a systemd.timer unit to handle TIMELINE cleanup
TIMELINE_CLEANUP="yes"
# timeline settings
TIMELINE_MIN_AGE="1800"
TIMELINE_LIMIT_HOURLY="1"
TIMELINE_LIMIT_DAILY="14"
TIMELINE_LIMIT_MONTHLY="11"
TIMELINE_LIMIT_YEARLY="2"
# cleanup empty pre-post-pairs
EMPTY_PRE_POST_CLEANUP="yes"
# limits for empty pre-post-pair cleanup
EMPTY_PRE_POST_MIN_AGE="1800"
# uncomment to exclude this subvol when calling
# snap-sync as timer unit
# SNAP_SYNC_EXCLUDE="yes"
# Valid CONFIG_TYPE: archive, child, parent
# CONFIG_TYPE="archive" -> if synced, stream snapshot to a non btrfs filesystem
# CONFIG_TYPE="child" -> if synced, stream snapshot to a given CHILD_CONFIG name
CONFIG_TYPE="child"
#CHILD_CONFIG=<child config name>
#PARENT_CONFIG=<parent config name>