rzerres/dsnap-sync
1
0
Fork 0
Code Issues Pull Requests Releases 1 Activity

Markdown: update markup text to reflact changes for verson 0.5.7

Browse Source 
* Basic description in README.md
* Example section in /usr/share/doc/Examples.md
This commit is contained in:
Ralf Zerres
2018-07-15 16:04:38 +02:00
parent f461599b2e
commit 9628a9a623
2 changed files with 386 additions and 183 deletions
Download Patch File Download Diff File Expand all files Collapse all files

271
README.md
View File

@@ -1,5 +1,5 @@
<!-- dsnap-sync README.md --> <!-- dsnap-sync README.md -->
<!-- version: 0.5.5 --> <!-- version: 0.5.7 -->
# dsnap-sync # dsnap-sync
@@ -10,81 +10,133 @@ It takes advantage of the specific snapshots functionality btrfs offers
and combines it with managemnet functionality of snapper. and combines it with managemnet functionality of snapper.
`dsnap-sync` creates backups as btrfs-snapshots on a selectable target device. `dsnap-sync` creates backups as btrfs-snapshots on a selectable target device.
Plug in and mount any btrfs-formatted device to your system. Supported devices Plug in and mount any btrfs-formatted device to your system. Supported targets
may be either local USB drives, but can be as well remote accessible RAID drives. may be either local attached USB drives, as well as automountable remote host
If possible the backup process will send incremental snapshots to the target drive. RAIDs.
If the snapshot will be stored on a remote host, it is secured with ssh. If possible the backup process will send incremental snapshots to the target
drive. If the snapshot will be stored on a remote host, the transport will be
secured with ssh.
The tool is implemented as a posix shell script, to keep the footprint small (dash). The tool is implemented as a posix shell script (dash), to keep the footprint
`dsnap-sync` will support interactive and time scheduled backup runs. small. `dsnap-sync` will support interactive and time scheduled backup processes.
Scheduling should be implemented as a pair of systemd service and timer-units.
The [example section](usr/share/doc/Examples.md#systemd)
will offer details as a reference point.
## Backup process ## Backup process
For a backup run, and per default `dsnap-sync` will iterate through all defined snapper The default `dsnap-sync` backup process will iterate through all defined snapper
configurations found on your source system. If you prefer to just run on a specific configurations that are found on your source system. If you prefer to have single
configuration per call, you are free to select it using the 'config' option `-c`. processes for each configuration, you are free to define isolated systemd-units or
call `dsnap-sync` interactively while referencing the intended snapper configuration
(option `-c` or `--config`).
For each selected snapper configuration `dsnap-sync` For each selected snapper configuration `dsnap-sync`
* will create an appropriate local snapshot and update the metadata * will present/select target device informations
* will transfer the snapshot using btrfs-send to the target device * will prepare snapper structures
* will create and update the snapper configuration on the target * will perform the actual backup
* will update the metadata on the target device (handle backupdir, handle snapper structures, handle btrfs send /btrfs recieve)
* will finalize the actual backup
(update snapper metadata for each source and target snapshot)
* will perform cleanup tasks
Usualy tools will document this proccess as a disk to disk (d2d) backup. Usualy other tools will document this proccess as a disk to disk (d2d) backup.
If possible `dsnap-sync` will levarage btrfs-send capabilities to only If possible `dsnap-sync` will levarage `btrfs send` capabilities to only
send deltas. It will compare the snapshot data of the ongiong process with available send deltas. It will compare snapshot data of the source snapshot with available
snapshot data on the target device. snapshot data on the target device. If a common snapshot id exists on source and
target, `dsnap-sync` will prepare the `btrfs send / btrfs receive` pipe to use
them respectively. This functionality dasticly reduces the time a sync process
will need to complete compared to a full backup process.
### Interactive backups ### Interactive backups
An interactive run will request you to select a mounted btrfs device. An interactive process will gide you to select a mounted btrfs device.
You can pre-select the target drive via [command line options](https://github.com/rzerres/dsnap-sync#options). You can pre-select the target drive via [command line options](./README.md#Options).
Either use a UUID, a SUBVOLID or a TARGET name (read 'mount point'). To uniquely define / select a target devices you either need to choose
* a pair of a btrfs UUID and SUBVOLID
* a TARGET name (read 'mount point')
This will asure, that `dsnap-sync` can distinguish backup processes that
have a commen source device, but save data to different target devices. As
an example it might be advisable, to save the project subvolume redundantliy
on to independent targets (disk and tape).
Before `dsnap-sync` will perform the backup, it will present the backupdir,
spit out the source and target locations will. You have to confirm or adapt
the given values. You may use commandline options to supress interaction
(e.g --noconfirm, --batch).
### Scheduled backups ### Scheduled backups
A scheduled run will take all needed parameters from config options. A scheduled process should be defined as a systemd.unit. Inside the unit
`dsnap-sync` does support systemd.timer units. Please refer to related paragraph [documenting systemd](https://github.com/rzerres/dsnap-sync#systemd). definition the execution parameter will take the `dsnap-sync` call, appending
all needed parameters as config options. In combination with a corresponding
systemd.timer unit, you are able to finetune your backup needs.
The [example section](usr/share/doc/Examples.md#systemd)
will offer details as a reference point.
## Requirements ## Requirements
beside the shell itself, `dsnap-sync`relies on external tools to achieve its goal. Beside the posix shell itself (e.g. `dash`), `dsnap-sync`relies on external
At run-time their availability is checked. Following tools are are used: tools to achieve its goal. At run-time their availability is checked.
Following tools are used:
- awk - awk
- btrfs - btrfs
- findmnt - findmnt
- sed - sed
- snapper - snapper
- tee
- wc
optionaly tools As an option, you can enrich interactive responses using
- notify-send - notify-send
- pv - pv
## Installation ## Installation
### Building from source
`dsnap-sync` is a shell script. Thus no compilation is required.
To simplify correct target locations, this project uses a Makefile.
# make install # make install
If your system uses a non-default location for the snapper If your system uses a non-default location for the snapper
configuration file, specify it on the command line with configuration defaults, specify the location with an environment variable
`SNAPPER_CONFIG`. For example, for Arch Linux use: (`SNAPPER_CONFIG`).
Arch Linux/Fedora/Gentoo:
# make SNAPPER_CONFIG=/etc/conf.d/snapper install # make SNAPPER_CONFIG=/etc/conf.d/snapper install
Debian/Ubuntu:
# make SNAPPER_CONFIG=/etc/default/snapper install
The local snapper configuration will be extended to make use The local snapper configuration will be extended to make use
of a new template 'dsnap-sync'. of a new template 'dsnap-sync'.
The package is also available in the ### Using distribution packages
<!-- [AUR](https://aur.archlinux.org/packages/dsnap-sync/). --> If available, you can install `dsnap-sync` as a precompiled package.
Please use your host software package manager.
<!--
* For ARCH-Linux
[AUR package](https://aur.archlinux.org/packages/dsnap-sync)
-->
<!-- For Debian
[deb package](https://packages.debian.org/dsnap-sync). -->
<!-- For Ubuntu
[deb package](https://packages.ubuntu.org/dsnap-sync). -->
## Options ## Options
Usage: dsnap-sync [options] Usage: dsnap-sync [options]
Options: Options:
-a, --automount <path> start automount for given path to get a valid target mountpoint.
-b, --backupdir <prefix> backupdir is a relative path that will be appended to target backup-root -b, --backupdir <prefix> backupdir is a relative path that will be appended to target backup-root
-d, --description <desc> Change the snapper description. Default: "latest incremental backup" -d, --description <desc> Change the snapper description. Default: "latest incremental backup"
--label-finished <desc> snapper description tagging successful jobs. Default: "dsnap-sync backup" --label-finished <desc> snapper description tagging successful jobs. Default: "dsnap-sync backup"
@@ -112,136 +164,60 @@ The package is also available in the
should specify the remote machine's hostname or ip address. The 'root' user must be should specify the remote machine's hostname or ip address. The 'root' user must be
permitted to login on the remote machine. permitted to login on the remote machine.
--dry-run perform a trial run where no changes are made. --dry-run perform a trial run where no changes are made.
-v, --verbose Be more verbose on what's going on (use multiple times to be more verbose). -v, --verbose Be more verbose on what's going on.
--version show program version --version show program version
## First run ## First run
If you have never synced to the paticular target device (first run), `dsnap-sync` If you have never synced to the paticular target device (first run), `dsnap-sync`
will take care to create the necessary target file-structure to store the snapshot. will take care to create the necessary target filesystem-structure.
As an option you can prepend a backup-path.
Before the sync job is started, source and target locations will be presented. * btrfs-snapshots
You have to confirm any further operation, or use defaults (option: noconfirm).
## Example command line usage On the target device the needed snapper structure is validated and build up as needed.
Aside the new filesystem path, `dsnap-sync` will use a snapper template
(`/etc/snapper/config-templates/dsnap-sync`) to create a new snapper target configuration.
To garantee unique configuration names, `dsnap-sync` take the source configuration name
and postfix it with targets hostname. You can adopt this behaviour with a config option
(`--config-postfix`).
### dsnap-sync to local target The default `config-template` of dsnap-sync will inherit following snapper parameters:
#### Default: no selections, run for all snapper configs * mark new snapshots as type 'single'
* mark new snaphosts with cleanup-algorithm 'timeline'
* apply config option 'CONFIG_TYPE=child'
* apply config option 'TIMELINE_CREATE=no'
* apply config option 'TIMELINE_CLEANUP=yes'
# dsnap-sync Please adapt the defaults, if your milage varies.
#### Default: Select two configs, the backupdir and verbose output * btrfs-archive
# dsnap-sync --verbose --config root --config data2 --backupdir=toshiba_r700 Beside btrfs-snapshots, `dsnap-shot` will support another backup type:
btrfs-archive. If the target device is not a btrfs filesystem (e.g. ext4,
xfs, ltofs tapes), data are copied backupdir.
`dsnap-sync` will take the source snapshot-ID and create a corresponding directory
below targets backupdir. Inside this 'target-subdirectory' `dsnap-sync` will mimic the
snapper structure:
#### Dry-run: Select config, select Target, as batchjob (--noconfirm) * the actual btrfs stream will we copied to a subdirectoy called `snapshot`
* the proccess metadata are saved to a file called `info.xml`
# dsnap-sync -c root -s 265 --noconfirm --dry-run ## Automounter
### dsnap-sync to remote host `dsnap-sync` offer all mounted btrfs filesystems as valid process targets.
Since storage space on disks are very price efficient this days, environments
often use removable, external disks as additional backup targets. If the
external disks aren't mounted at boot time they can't be addressed by the
selection function. It's even advisable to not mount them all the time
(e.g prevent risks for malware encryption attacks).
`dsnap-sync` will rely on ssh access to the target host. For batch usage make sure, that your To link external disks in dynamically, but also asure a persistent naming
public key is accepted for remote login as user 'root'. You may have to adapt /root/.ssh/authorized_keys syntax, we can use them as auto-mountable targets. To wakeup the automount
on the target host. proccess before parsing available target disks, append the target mount-point
as a config option to `dsnap-sync` (e.g: `--automount /var/backups/archive-disk1`).
On your target host, you should also verify the availability of a dsnap-sync config-template for snapper. The [example section](usr/share/doc/Examples.md#Automounter)
A template `dsnap-sync` is included in the package for your convenience. will offer details as a reference point.
#### Dryrun: Select remote host <ip/fqdn>, interactive, run for all configs
dsnap-sync --dry-run --remote 172.16.0.3
Selecting a mounted BTRFS device for backups on 172.16.0.3.
0) / (uuid=5af3413e-59ea-4862-8cff-304afe25420f,subvolid=257,subvol=/root)
1) /.snapshots (uuid=5af3413e-59ea-4862-8cff-304afe25420f,subvolid=258,subvol=/@snapshots-root)
2) /data2 (uuid=62a45211-9197-4a5f-aeaf-0ab803a42c32,subvolid=261,subvol=/data2)
3) /home (uuid=62a45211-9197-4a5f-aeaf-0ab803a42c32,subvolid=258,subvol=/home)
4) /data2/.snapshots (uuid=62a45211-9197-4a5f-aeaf-0ab803a42c32,subvolid=262,subvol=/@snapshots-data2)
5) /home/.snapshots (uuid=62a45211-9197-4a5f-aeaf-0ab803a42c32,subvolid=259,subvol=/@snapshots-home)
6) /var/lib/machines (uuid=2ba04452-74aa-44df-b1c7-74e0a70c6543,subvolid=260,subvol=/machines)
7) /var/lib/libvirt (uuid=2ba04452-74aa-44df-b1c7-74e0a70c6543,subvolid=261,subvol=/libvirt)
8) /data (uuid=2ba04452-74aa-44df-b1c7-74e0a70c6543,subvolid=257,subvol=/data)
9) /var/lib/machines/.snapshots (uuid=2ba04452-74aa-44df-b1c7-74e0a70c6543,subvolid=2121,subvol=/@snapshots-machines)
10) /data/.snapshots (uuid=2ba04452-74aa-44df-b1c7-74e0a70c6543,subvolid=258,subvol=/@snapshots-data)
11) /var/lib/dsnap-sync (uuid=753eba7a-41ce-49e0-b2e3-24ee07811efd,subvolid=420,subvol=/dsnap-sync)
x) Exit
Enter a number: 11
### Dry-run with given Target for snapper config 'home', no confirmations
#### Sync: Select config 'data2', remote host <ip/fqdn>, target '/data', as batchjob (--noconfirm)
# dsnap-sync --config data2 --remote 172.16.0.3 --target /data --noconfirm
## systemd
### service
[Unit]
Description=Run dsnap-sync backup
[Install]
WantedBy=multi-user.target
[Service]
Type=simple
ExecStart=/usr/bin/dsnap-sync --UUID 7360922b-c916-4d9f-a670-67fe0b91143c --subvolid 5 --noconfirm
### timer
[Unit]
Description=Run dsnap-sync weekly
[Timer]
OnCalendar=weekly
AccuracySec=12h
Persistent=true
[Install]
WantedBy=timers.target
## snapper template
###
# template for dsnap-sync handling
###
# subvolume to snapshot
SUBVOLUME="/var/lib/dsnap-sync"
# filesystem type
FSTYPE="btrfs"
# users and groups allowed to work with config
ALLOW_USERS=""
ALLOW_GROUPS="adm"
# sync users and groups from ALLOW_USERS and ALLOW_GROUPS to .snapshots
# directory
SYNC_ACL="yes"
# start comparing pre- and post-snapshot in background after creating
# post-snapshot
BACKGROUND_COMPARISON="yes"
# run daily number cleanup
NUMBER_CLEANUP="no"
# limit for number cleanup
NUMBER_MIN_AGE="1800"
NUMBER_LIMIT="10"
NUMBER_LIMIT_IMPORTANT="2"
# use systemd.timer for timeline
TIMELINE_CREATE="no"
# use systemd.timer for cleanup
TIMELINE_CLEANUP="no"
# dsnap-sync as timer unit
SNAP_SYNC_EXCLUDE="yes"
## Contributing ## Contributing
@@ -251,7 +227,7 @@ TODO list. It might be already on the agenda.
## Related projects ## Related projects
I did fork from Wes Barnetts original work. I was aiming merged it back. I did fork from Wes Barnetts original work. I intend to merged it back.
Beside the fact that this version doesn't use any bashisms, Wes did let me know, Beside the fact that this version doesn't use any bashisms, Wes did let me know,
that he doesn't have the time to review the changes appropriately to make it a merge. that he doesn't have the time to review the changes appropriately to make it a merge.
Anyone willing to do so is invided. Anyone willing to do so is invided.
@@ -259,7 +235,6 @@ Anyone willing to do so is invided.
Until that date, i will offer this fork for the public. To overcome any name clashes Until that date, i will offer this fork for the public. To overcome any name clashes
i renamed it to dsnap-sync. i renamed it to dsnap-sync.
## License ## License
<!-- License source --> <!-- License source -->

228
usr/share/doc/Examples.md Normal file
View File

@@ -0,0 +1,228 @@
# Examples
## Automounter
`dsnap-sync` will take advantage of systemd automount units to incorporate external,
removable disks into the selection process for target devices.
### btrfs disk
Format the disk with btrfs tools to prepare it as a target.
The following example will reference to a btrfs disk and mount a given subvolume.
### systemd automount units
#### mount unit: var-backups-archive\x2ddisk1.mount
[Unit]
Description=Backup - Archiv-Disk 1
Documentation=man:systemd.mount(5) man:mount.btrfs(8)
DefaultDependencies=yes
[Mount]
What=UUID=977b4ecf-be67-4643-84f5-10b368c24d25
Where=/var/backups/archive-disk1
Type=btrfs
Options=defaults,subvol=@archive-disk1,compress=lzo
[Install]
WantedBy=multi-user.target
#### automount unit: var-backups-archive\x2ddisk1.automount
[Unit]
Description=Automount Backup - Archive-Disk 1
Documentation=man:systemd.automount(5)
[Automount]
Where=/var/backups/archive-disk1
TimeoutIdleSec=45
[Install]
WantedBy=multi-user.target
## `dsnap-sync` command line usage
### backup to local target
#### Default: no selections, run for all snapper configs
# dsnap-sync
#### Default: Select two configs, the backupdir and verbose output
# dsnap-sync --verbose --config root --config data2 --backupdir=toshiba_r700
#### Dry-run: Select config, select Target, as batchjob (--noconfirm)
# dsnap-sync -c root -s 265 --noconfirm --dry-run
### backup to remote target
`dsnap-sync` will rely on ssh access to the target host. For batch usage
make sure, that your public key is accepted for a remote login as user
'root'. You may have to adapt /root/.ssh/authorized_keys on the target host.
On your target host, you should also verify the availability of a
`dsnap-sync` config-template for snapper. A template `dsnap-sync`
is included in the package for your convenience.
#### Dryrun: Select remote host <ip/fqdn>, interactive, run for all configs
dsnap-sync --dry-run --remote <fqdn/ip>
Select target disk...
0) /var/lib/snap-sync (uuid=b5915f47-38a8-4b37-8de6-c8b15d9aba5a,subvolid=257,subvol=/var/lib/@snap-sync)
1) /var/backups/archive-disk4 (uuid=137b4a36-6e17-487b-a328-29b0d1e020c3,subvolid=257,subvol=/@dws-archive-disk4)
2) /var/backups/archive-disk3 (uuid=c9fcbfd3-6a1f-4b43-8176-b6f326bf46c7,subvolid=257,subvol=/@dws-archive-disk3)
3) /var/backups/archive-disk1 (uuid=977b4ecf-be67-4643-84f5-10b368c24d25,subvolid=257,subvol=/@dws-archive-disk1)
4) /var/backups/archive-disk2 (uuid=b68c8a31-9878-4ec1-b7ed-948de2125285,subvolid=257,subvol=/@dws-archive-disk2)
x) Exit
Enter a number: 1
### Dry-run with given Target for snapper config 'data2', no confirmations
#### Sync: Select config 'data2', remote host <ip/fqdn>, target '/var/lib/snap-sync', as batchjob (--noconfirm)
# dsnap-sync --config data2 --remote <fqdn/ip> --target /var/lib/snap-sync --batch --verbose
## systemd
`dsnap-sync` will structure all scheduling tasks while using systemd units.
To perform a backup process for just a single snapper configuration at a
given time, you have to define a pair of a systemd service unit and a
corresponding systemd timer unit.
Below we define a generic `dsnap-sync service unit` that should be located
at /etc/systemd/system. Following call will reference this template:
`systemd enable dsnap-sync@data2.service`
### service unit: `dsnap-sync@.service
[Unit]
Description=dsnap-sync backup for target %i
[Install]
WantedBy=multi-user.target
[Service]
Type=simple
ExecStart=/usr/bin/dsnap-sync \
--config %i \
--uuid 7360922b-c916-4d9f-a670-67fe0b91143c \
--subvolid 5 \
--remote backup-host
--batch
### overriding service unit: `dsnap-sync@data2.service`
Please remember, that the template example encode a given target
UUID and SUBVOLID. If you want the unit to serve individual parameter,
you have to override the it like:
`systemd edit dsnap-sync@data2.service`
Define a service paragraph, clean out the ExecStart= parameter and
refine a new ExedStart= parameter with the intended.
[Service]
ExecStart=
ExecStart=/usr/bin/dsnap-sync \
--config %i \
--target /var/lib/dsnap-sync \
--remote my-backup-host
--batch
### timer unit: `dsnap-sync@.timer`
Below we define a generic `dsnap-sync timer unit` that should be located
at /etc/systemd/system.
[Unit]
Description=dsnap-sync weekly backup
[Timer]
OnCalendar=weekly
AccuracySec=12h
Persistent=true
[Install]
WantedBy=timers.target
Following call will reference this template:
`systemd enable dsnap-sync@data2.timer`
## snapper extensions
For any new `dsnap-sync` btrfs-snapshot a new target snapper structure is need,
if the target backups should be managable via snapper.
`dsnap-sync` will create this structure as needed. During the creation it
will reference to a template called `dsnap-sync`. Please adapt it
if your milage varies.
### `dsnap-sync` template
###
# snapper template for dsnap-sync handling
###
# subvolume to snapshot
SUBVOLUME="/var/lib/dsnap-sync"
# filesystem type
FSTYPE="btrfs"
# users and groups allowed to work with config
ALLOW_USERS=""
ALLOW_GROUPS="adm"
# sync users and groups from ALLOW_USERS and ALLOW_GROUPS to .snapshots
# directory
SYNC_ACL="yes"
# start comparing pre- and post-snapshot in background after creating
# post-snapshot
BACKGROUND_COMPARISON="yes"
# handele NUMBER_CLEANUP via systemd, if a timer unit is active
NUMBER_CLEANUP="yes"
# limit for number cleanup
NUMBER_MIN_AGE="1800"
NUMBER_LIMIT="52"
NUMBER_LIMIT_IMPORTANT="12"
# handle TIMELINE via systemd, if a timer unit is active
TIMELINE_CREATE="yes"
# create a systemd.timer unit to handle TIMELINE cleanup
TIMELINE_CLEANUP="yes"
# timeline settings
TIMELINE_MIN_AGE="1800"
TIMELINE_LIMIT_HOURLY="1"
TIMELINE_LIMIT_DAILY="14"
TIMELINE_LIMIT_MONTHLY="11"
TIMELINE_LIMIT_YEARLY="2"
# cleanup empty pre-post-pairs
EMPTY_PRE_POST_CLEANUP="yes"
# limits for empty pre-post-pair cleanup
EMPTY_PRE_POST_MIN_AGE="1800"
# uncomment to exclude this subvol when calling
# snap-sync as timer unit
# SNAP_SYNC_EXCLUDE="yes"
# Valid CONFIG_TYPE: archive, child, parent
# CONFIG_TYPE="archive" -> if synced, stream snapshot to a non btrfs filesystem
# CONFIG_TYPE="child" -> if synced, stream snapshot to a given CHILD_CONFIG name
CONFIG_TYPE="child"
#CHILD_CONFIG=<child config name>
#PARENT_CONFIG=<parent config name>