From 9628a9a6230fbe73e25cd060d20ae2200c55c097 Mon Sep 17 00:00:00 2001 From: Ralf Zerres Date: Sun, 15 Jul 2018 16:04:38 +0200 Subject: [PATCH] Markdown: update markup text to reflact changes for verson 0.5.7 * Basic description in README.md * Example section in /usr/share/doc/Examples.md --- README.md | 341 ++++++++++++++++++-------------------- usr/share/doc/Examples.md | 228 +++++++++++++++++++++++++ 2 files changed, 386 insertions(+), 183 deletions(-) create mode 100644 usr/share/doc/Examples.md diff --git a/README.md b/README.md index da4defe..a96a26e 100644 --- a/README.md +++ b/README.md @@ -1,5 +1,5 @@ - + # dsnap-sync @@ -10,238 +10,214 @@ It takes advantage of the specific snapshots functionality btrfs offers and combines it with managemnet functionality of snapper. `dsnap-sync` creates backups as btrfs-snapshots on a selectable target device. -Plug in and mount any btrfs-formatted device to your system. Supported devices -may be either local USB drives, but can be as well remote accessible RAID drives. -If possible the backup process will send incremental snapshots to the target drive. -If the snapshot will be stored on a remote host, it is secured with ssh. +Plug in and mount any btrfs-formatted device to your system. Supported targets +may be either local attached USB drives, as well as automountable remote host +RAIDs. +If possible the backup process will send incremental snapshots to the target +drive. If the snapshot will be stored on a remote host, the transport will be +secured with ssh. -The tool is implemented as a posix shell script, to keep the footprint small (dash). -`dsnap-sync` will support interactive and time scheduled backup runs. +The tool is implemented as a posix shell script (dash), to keep the footprint +small. `dsnap-sync` will support interactive and time scheduled backup processes. +Scheduling should be implemented as a pair of systemd service and timer-units. +The [example section](usr/share/doc/Examples.md#systemd) +will offer details as a reference point. ## Backup process -For a backup run, and per default `dsnap-sync` will iterate through all defined snapper -configurations found on your source system. If you prefer to just run on a specific -configuration per call, you are free to select it using the 'config' option `-c`. +The default `dsnap-sync` backup process will iterate through all defined snapper +configurations that are found on your source system. If you prefer to have single +processes for each configuration, you are free to define isolated systemd-units or +call `dsnap-sync` interactively while referencing the intended snapper configuration +(option `-c` or `--config`). For each selected snapper configuration `dsnap-sync` -* will create an appropriate local snapshot and update the metadata -* will transfer the snapshot using btrfs-send to the target device -* will create and update the snapper configuration on the target -* will update the metadata on the target device +* will present/select target device informations +* will prepare snapper structures +* will perform the actual backup + (handle backupdir, handle snapper structures, handle btrfs send /btrfs recieve) +* will finalize the actual backup + (update snapper metadata for each source and target snapshot) +* will perform cleanup tasks -Usualy tools will document this proccess as a disk to disk (d2d) backup. -If possible `dsnap-sync` will levarage btrfs-send capabilities to only -send deltas. It will compare the snapshot data of the ongiong process with available -snapshot data on the target device. +Usualy other tools will document this proccess as a disk to disk (d2d) backup. +If possible `dsnap-sync` will levarage `btrfs send` capabilities to only +send deltas. It will compare snapshot data of the source snapshot with available +snapshot data on the target device. If a common snapshot id exists on source and +target, `dsnap-sync` will prepare the `btrfs send / btrfs receive` pipe to use +them respectively. This functionality dasticly reduces the time a sync process +will need to complete compared to a full backup process. ### Interactive backups -An interactive run will request you to select a mounted btrfs device. -You can pre-select the target drive via [command line options](https://github.com/rzerres/dsnap-sync#options). -Either use a UUID, a SUBVOLID or a TARGET name (read 'mount point'). +An interactive process will gide you to select a mounted btrfs device. +You can pre-select the target drive via [command line options](./README.md#Options). +To uniquely define / select a target devices you either need to choose + +* a pair of a btrfs UUID and SUBVOLID +* a TARGET name (read 'mount point') + +This will asure, that `dsnap-sync` can distinguish backup processes that +have a commen source device, but save data to different target devices. As +an example it might be advisable, to save the project subvolume redundantliy +on to independent targets (disk and tape). + +Before `dsnap-sync` will perform the backup, it will present the backupdir, +spit out the source and target locations will. You have to confirm or adapt +the given values. You may use commandline options to supress interaction +(e.g --noconfirm, --batch). + ### Scheduled backups -A scheduled run will take all needed parameters from config options. -`dsnap-sync` does support systemd.timer units. Please refer to related paragraph [documenting systemd](https://github.com/rzerres/dsnap-sync#systemd). +A scheduled process should be defined as a systemd.unit. Inside the unit +definition the execution parameter will take the `dsnap-sync` call, appending +all needed parameters as config options. In combination with a corresponding +systemd.timer unit, you are able to finetune your backup needs. +The [example section](usr/share/doc/Examples.md#systemd) +will offer details as a reference point. ## Requirements -beside the shell itself, `dsnap-sync`relies on external tools to achieve its goal. -At run-time their availability is checked. Following tools are are used: +Beside the posix shell itself (e.g. `dash`), `dsnap-sync`relies on external +tools to achieve its goal. At run-time their availability is checked. +Following tools are used: - awk - btrfs - findmnt - sed - snapper -- tee -- wc -optionaly tools +As an option, you can enrich interactive responses using - notify-send - pv ## Installation +### Building from source + +`dsnap-sync` is a shell script. Thus no compilation is required. +To simplify correct target locations, this project uses a Makefile. + # make install If your system uses a non-default location for the snapper -configuration file, specify it on the command line with -`SNAPPER_CONFIG`. For example, for Arch Linux use: +configuration defaults, specify the location with an environment variable +(`SNAPPER_CONFIG`). + Arch Linux/Fedora/Gentoo: # make SNAPPER_CONFIG=/etc/conf.d/snapper install + Debian/Ubuntu: + # make SNAPPER_CONFIG=/etc/default/snapper install + The local snapper configuration will be extended to make use of a new template 'dsnap-sync'. -The package is also available in the - +### Using distribution packages +If available, you can install `dsnap-sync` as a precompiled package. +Please use your host software package manager. + + + + + + ## Options Usage: dsnap-sync [options] - - Options: - -b, --backupdir backupdir is a relative path that will be appended to target backup-root - -d, --description Change the snapper description. Default: "latest incremental backup" - --label-finished snapper description tagging successful jobs. Default: "dsnap-sync backup" - --label-running snapper description tagging active jobs. Default: "dsnap-sync in progress" - --label-synced snapper description tagging last synced jobs. - Default: "dsnap-sync last incremental" - --color Enable colored output messages - -c, --config Specify the snapper configuration to use. Otherwise will perform for each snapper - configuration. Can list multiple configurations within quotes, space-separated - (e.g. -c "root home"). - --config-postfix Specify a postfix that will be appended to the destination snapper config name. - -n, --noconfirm Do not ask for confirmation for each configuration. Will still prompt for backup - --batch directory name on first backup" - --nonotify Disable graphical notification (via dbus) - --nopv Disable graphical progress output (disable pv) - -r, --remote
Send the snapshot backup to a remote machine. The snapshot will be sent via ssh. - You should specify the remote machine's hostname or ip address. The 'root' user - must be permitted to login on the remote machine. - -p, --port The remote port. - -s, --subvolid Specify the subvolume id of the mounted BTRFS subvolume to back up to. Defaults to 5. - -u, --uuid Specify the UUID of the mounted BTRFS subvolume to back up to. Otherwise will prompt." - If multiple mount points are found with the same UUID, will prompt user." - -t, --target Specify the mountpoint of the BTRFS subvolume to back up to. - --remote
Send the snapshot backup to a remote machine. The snapshot will be sent via ssh. You - should specify the remote machine's hostname or ip address. The 'root' user must be - permitted to login on the remote machine. - --dry-run perform a trial run where no changes are made. - -v, --verbose Be more verbose on what's going on (use multiple times to be more verbose). - --version show program version + + Options: + -a, --automount start automount for given path to get a valid target mountpoint. + -b, --backupdir backupdir is a relative path that will be appended to target backup-root + -d, --description Change the snapper description. Default: "latest incremental backup" + --label-finished snapper description tagging successful jobs. Default: "dsnap-sync backup" + --label-running snapper description tagging active jobs. Default: "dsnap-sync in progress" + --label-synced snapper description tagging last synced jobs. + Default: "dsnap-sync last incremental" + --color Enable colored output messages + -c, --config Specify the snapper configuration to use. Otherwise will perform for each snapper + configuration. Can list multiple configurations within quotes, space-separated + (e.g. -c "root home"). + --config-postfix Specify a postfix that will be appended to the destination snapper config name. + -n, --noconfirm Do not ask for confirmation for each configuration. Will still prompt for backup + --batch directory name on first backup" + --nonotify Disable graphical notification (via dbus) + --nopv Disable graphical progress output (disable pv) + -r, --remote
Send the snapshot backup to a remote machine. The snapshot will be sent via ssh. + You should specify the remote machine's hostname or ip address. The 'root' user + must be permitted to login on the remote machine. + -p, --port The remote port. + -s, --subvolid Specify the subvolume id of the mounted BTRFS subvolume to back up to. Defaults to 5. + -u, --uuid Specify the UUID of the mounted BTRFS subvolume to back up to. Otherwise will prompt." + If multiple mount points are found with the same UUID, will prompt user." + -t, --target Specify the mountpoint of the BTRFS subvolume to back up to. + --remote
Send the snapshot backup to a remote machine. The snapshot will be sent via ssh. You + should specify the remote machine's hostname or ip address. The 'root' user must be + permitted to login on the remote machine. + --dry-run perform a trial run where no changes are made. + -v, --verbose Be more verbose on what's going on. + --version show program version ## First run If you have never synced to the paticular target device (first run), `dsnap-sync` -will take care to create the necessary target file-structure to store the snapshot. -As an option you can prepend a backup-path. +will take care to create the necessary target filesystem-structure. -Before the sync job is started, source and target locations will be presented. -You have to confirm any further operation, or use defaults (option: noconfirm). +* btrfs-snapshots -## Example command line usage + On the target device the needed snapper structure is validated and build up as needed. + Aside the new filesystem path, `dsnap-sync` will use a snapper template + (`/etc/snapper/config-templates/dsnap-sync`) to create a new snapper target configuration. + To garantee unique configuration names, `dsnap-sync` take the source configuration name + and postfix it with targets hostname. You can adopt this behaviour with a config option + (`--config-postfix`). + + The default `config-template` of dsnap-sync will inherit following snapper parameters: + + * mark new snapshots as type 'single' + * mark new snaphosts with cleanup-algorithm 'timeline' + * apply config option 'CONFIG_TYPE=child' + * apply config option 'TIMELINE_CREATE=no' + * apply config option 'TIMELINE_CLEANUP=yes' + + Please adapt the defaults, if your milage varies. + +* btrfs-archive -### dsnap-sync to local target + Beside btrfs-snapshots, `dsnap-shot` will support another backup type: + btrfs-archive. If the target device is not a btrfs filesystem (e.g. ext4, + xfs, ltofs tapes), data are copied backupdir. + `dsnap-sync` will take the source snapshot-ID and create a corresponding directory + below targets backupdir. Inside this 'target-subdirectory' `dsnap-sync` will mimic the + snapper structure: + + * the actual btrfs stream will we copied to a subdirectoy called `snapshot` + * the proccess metadata are saved to a file called `info.xml` -#### Default: no selections, run for all snapper configs +## Automounter - # dsnap-sync +`dsnap-sync` offer all mounted btrfs filesystems as valid process targets. +Since storage space on disks are very price efficient this days, environments +often use removable, external disks as additional backup targets. If the +external disks aren't mounted at boot time they can't be addressed by the +selection function. It's even advisable to not mount them all the time +(e.g prevent risks for malware encryption attacks). -#### Default: Select two configs, the backupdir and verbose output - - # dsnap-sync --verbose --config root --config data2 --backupdir=toshiba_r700 - -#### Dry-run: Select config, select Target, as batchjob (--noconfirm) - - # dsnap-sync -c root -s 265 --noconfirm --dry-run - -### dsnap-sync to remote host - -`dsnap-sync` will rely on ssh access to the target host. For batch usage make sure, that your -public key is accepted for remote login as user 'root'. You may have to adapt /root/.ssh/authorized_keys -on the target host. - -On your target host, you should also verify the availability of a dsnap-sync config-template for snapper. -A template `dsnap-sync` is included in the package for your convenience. - -#### Dryrun: Select remote host , interactive, run for all configs - - dsnap-sync --dry-run --remote 172.16.0.3 - Selecting a mounted BTRFS device for backups on 172.16.0.3. - 0) / (uuid=5af3413e-59ea-4862-8cff-304afe25420f,subvolid=257,subvol=/root) - 1) /.snapshots (uuid=5af3413e-59ea-4862-8cff-304afe25420f,subvolid=258,subvol=/@snapshots-root) - 2) /data2 (uuid=62a45211-9197-4a5f-aeaf-0ab803a42c32,subvolid=261,subvol=/data2) - 3) /home (uuid=62a45211-9197-4a5f-aeaf-0ab803a42c32,subvolid=258,subvol=/home) - 4) /data2/.snapshots (uuid=62a45211-9197-4a5f-aeaf-0ab803a42c32,subvolid=262,subvol=/@snapshots-data2) - 5) /home/.snapshots (uuid=62a45211-9197-4a5f-aeaf-0ab803a42c32,subvolid=259,subvol=/@snapshots-home) - 6) /var/lib/machines (uuid=2ba04452-74aa-44df-b1c7-74e0a70c6543,subvolid=260,subvol=/machines) - 7) /var/lib/libvirt (uuid=2ba04452-74aa-44df-b1c7-74e0a70c6543,subvolid=261,subvol=/libvirt) - 8) /data (uuid=2ba04452-74aa-44df-b1c7-74e0a70c6543,subvolid=257,subvol=/data) - 9) /var/lib/machines/.snapshots (uuid=2ba04452-74aa-44df-b1c7-74e0a70c6543,subvolid=2121,subvol=/@snapshots-machines) - 10) /data/.snapshots (uuid=2ba04452-74aa-44df-b1c7-74e0a70c6543,subvolid=258,subvol=/@snapshots-data) - 11) /var/lib/dsnap-sync (uuid=753eba7a-41ce-49e0-b2e3-24ee07811efd,subvolid=420,subvol=/dsnap-sync) - x) Exit - Enter a number: 11 - - -### Dry-run with given Target for snapper config 'home', no confirmations - -#### Sync: Select config 'data2', remote host , target '/data', as batchjob (--noconfirm) - - # dsnap-sync --config data2 --remote 172.16.0.3 --target /data --noconfirm - -## systemd - -### service - - [Unit] - Description=Run dsnap-sync backup - - [Install] - WantedBy=multi-user.target - - [Service] - Type=simple - ExecStart=/usr/bin/dsnap-sync --UUID 7360922b-c916-4d9f-a670-67fe0b91143c --subvolid 5 --noconfirm - -### timer - - [Unit] - Description=Run dsnap-sync weekly - - [Timer] - OnCalendar=weekly - AccuracySec=12h - Persistent=true - - [Install] - WantedBy=timers.target - -## snapper template - - ### - # template for dsnap-sync handling - ### - - # subvolume to snapshot - SUBVOLUME="/var/lib/dsnap-sync" - - # filesystem type - FSTYPE="btrfs" - - # users and groups allowed to work with config - ALLOW_USERS="" - ALLOW_GROUPS="adm" - - # sync users and groups from ALLOW_USERS and ALLOW_GROUPS to .snapshots - # directory - SYNC_ACL="yes" - - # start comparing pre- and post-snapshot in background after creating - # post-snapshot - BACKGROUND_COMPARISON="yes" - - # run daily number cleanup - NUMBER_CLEANUP="no" - - # limit for number cleanup - NUMBER_MIN_AGE="1800" - NUMBER_LIMIT="10" - NUMBER_LIMIT_IMPORTANT="2" - - # use systemd.timer for timeline - TIMELINE_CREATE="no" - - # use systemd.timer for cleanup - TIMELINE_CLEANUP="no" - - # dsnap-sync as timer unit - SNAP_SYNC_EXCLUDE="yes" +To link external disks in dynamically, but also asure a persistent naming +syntax, we can use them as auto-mountable targets. To wakeup the automount +proccess before parsing available target disks, append the target mount-point +as a config option to `dsnap-sync` (e.g: `--automount /var/backups/archive-disk1`). +The [example section](usr/share/doc/Examples.md#Automounter) +will offer details as a reference point. ## Contributing @@ -251,7 +227,7 @@ TODO list. It might be already on the agenda. ## Related projects -I did fork from Wes Barnetts original work. I was aiming merged it back. +I did fork from Wes Barnetts original work. I intend to merged it back. Beside the fact that this version doesn't use any bashisms, Wes did let me know, that he doesn't have the time to review the changes appropriately to make it a merge. Anyone willing to do so is invided. @@ -259,7 +235,6 @@ Anyone willing to do so is invided. Until that date, i will offer this fork for the public. To overcome any name clashes i renamed it to dsnap-sync. - ## License diff --git a/usr/share/doc/Examples.md b/usr/share/doc/Examples.md new file mode 100644 index 0000000..cff081b --- /dev/null +++ b/usr/share/doc/Examples.md @@ -0,0 +1,228 @@ +# Examples + +## Automounter + +`dsnap-sync` will take advantage of systemd automount units to incorporate external, +removable disks into the selection process for target devices. + +### btrfs disk + +Format the disk with btrfs tools to prepare it as a target. +The following example will reference to a btrfs disk and mount a given subvolume. + +### systemd automount units + +#### mount unit: var-backups-archive\x2ddisk1.mount + + [Unit] + Description=Backup - Archiv-Disk 1 + Documentation=man:systemd.mount(5) man:mount.btrfs(8) + DefaultDependencies=yes + + [Mount] + What=UUID=977b4ecf-be67-4643-84f5-10b368c24d25 + Where=/var/backups/archive-disk1 + Type=btrfs + Options=defaults,subvol=@archive-disk1,compress=lzo + + [Install] + WantedBy=multi-user.target + +#### automount unit: var-backups-archive\x2ddisk1.automount + + [Unit] + Description=Automount Backup - Archive-Disk 1 + Documentation=man:systemd.automount(5) + + [Automount] + Where=/var/backups/archive-disk1 + TimeoutIdleSec=45 + + [Install] + WantedBy=multi-user.target + +## `dsnap-sync` command line usage + +### backup to local target + +#### Default: no selections, run for all snapper configs + + # dsnap-sync + +#### Default: Select two configs, the backupdir and verbose output + + # dsnap-sync --verbose --config root --config data2 --backupdir=toshiba_r700 + +#### Dry-run: Select config, select Target, as batchjob (--noconfirm) + + # dsnap-sync -c root -s 265 --noconfirm --dry-run + +### backup to remote target + +`dsnap-sync` will rely on ssh access to the target host. For batch usage +make sure, that your public key is accepted for a remote login as user +'root'. You may have to adapt /root/.ssh/authorized_keys on the target host. + +On your target host, you should also verify the availability of a +`dsnap-sync` config-template for snapper. A template `dsnap-sync` +is included in the package for your convenience. + +#### Dryrun: Select remote host , interactive, run for all configs + + dsnap-sync --dry-run --remote + Select target disk... + 0) /var/lib/snap-sync (uuid=b5915f47-38a8-4b37-8de6-c8b15d9aba5a,subvolid=257,subvol=/var/lib/@snap-sync) + 1) /var/backups/archive-disk4 (uuid=137b4a36-6e17-487b-a328-29b0d1e020c3,subvolid=257,subvol=/@dws-archive-disk4) + 2) /var/backups/archive-disk3 (uuid=c9fcbfd3-6a1f-4b43-8176-b6f326bf46c7,subvolid=257,subvol=/@dws-archive-disk3) + 3) /var/backups/archive-disk1 (uuid=977b4ecf-be67-4643-84f5-10b368c24d25,subvolid=257,subvol=/@dws-archive-disk1) + 4) /var/backups/archive-disk2 (uuid=b68c8a31-9878-4ec1-b7ed-948de2125285,subvolid=257,subvol=/@dws-archive-disk2) + x) Exit + Enter a number: 1 + +### Dry-run with given Target for snapper config 'data2', no confirmations + +#### Sync: Select config 'data2', remote host , target '/var/lib/snap-sync', as batchjob (--noconfirm) + + # dsnap-sync --config data2 --remote --target /var/lib/snap-sync --batch --verbose + +## systemd + +`dsnap-sync` will structure all scheduling tasks while using systemd units. +To perform a backup process for just a single snapper configuration at a +given time, you have to define a pair of a systemd service unit and a +corresponding systemd timer unit. + +Below we define a generic `dsnap-sync service unit` that should be located +at /etc/systemd/system. Following call will reference this template: + +`systemd enable dsnap-sync@data2.service` + +### service unit: `dsnap-sync@.service + + [Unit] + Description=dsnap-sync backup for target %i + + [Install] + WantedBy=multi-user.target + + [Service] + Type=simple + ExecStart=/usr/bin/dsnap-sync \ + --config %i \ + --uuid 7360922b-c916-4d9f-a670-67fe0b91143c \ + --subvolid 5 \ + --remote backup-host + --batch + +### overriding service unit: `dsnap-sync@data2.service` +Please remember, that the template example encode a given target +UUID and SUBVOLID. If you want the unit to serve individual parameter, +you have to override the it like: + +`systemd edit dsnap-sync@data2.service` + +Define a service paragraph, clean out the ExecStart= parameter and +refine a new ExedStart= parameter with the intended. + + [Service] + ExecStart= + ExecStart=/usr/bin/dsnap-sync \ + --config %i \ + --target /var/lib/dsnap-sync \ + --remote my-backup-host + --batch + +### timer unit: `dsnap-sync@.timer` + +Below we define a generic `dsnap-sync timer unit` that should be located +at /etc/systemd/system. + + [Unit] + Description=dsnap-sync weekly backup + + [Timer] + OnCalendar=weekly + AccuracySec=12h + Persistent=true + + [Install] + WantedBy=timers.target + +Following call will reference this template: + +`systemd enable dsnap-sync@data2.timer` + +## snapper extensions + +For any new `dsnap-sync` btrfs-snapshot a new target snapper structure is need, +if the target backups should be managable via snapper. + +`dsnap-sync` will create this structure as needed. During the creation it +will reference to a template called `dsnap-sync`. Please adapt it +if your milage varies. + +### `dsnap-sync` template + + ### + # snapper template for dsnap-sync handling + ### + + # subvolume to snapshot + SUBVOLUME="/var/lib/dsnap-sync" + + # filesystem type + FSTYPE="btrfs" + + + # users and groups allowed to work with config + ALLOW_USERS="" + ALLOW_GROUPS="adm" + + # sync users and groups from ALLOW_USERS and ALLOW_GROUPS to .snapshots + # directory + SYNC_ACL="yes" + + + # start comparing pre- and post-snapshot in background after creating + # post-snapshot + BACKGROUND_COMPARISON="yes" + + + # handele NUMBER_CLEANUP via systemd, if a timer unit is active + NUMBER_CLEANUP="yes" + + # limit for number cleanup + NUMBER_MIN_AGE="1800" + NUMBER_LIMIT="52" + NUMBER_LIMIT_IMPORTANT="12" + + # handle TIMELINE via systemd, if a timer unit is active + TIMELINE_CREATE="yes" + + # create a systemd.timer unit to handle TIMELINE cleanup + TIMELINE_CLEANUP="yes" + + # timeline settings + TIMELINE_MIN_AGE="1800" + TIMELINE_LIMIT_HOURLY="1" + TIMELINE_LIMIT_DAILY="14" + TIMELINE_LIMIT_MONTHLY="11" + TIMELINE_LIMIT_YEARLY="2" + + + # cleanup empty pre-post-pairs + EMPTY_PRE_POST_CLEANUP="yes" + + # limits for empty pre-post-pair cleanup + EMPTY_PRE_POST_MIN_AGE="1800" + + # uncomment to exclude this subvol when calling + # snap-sync as timer unit + # SNAP_SYNC_EXCLUDE="yes" + + # Valid CONFIG_TYPE: archive, child, parent + # CONFIG_TYPE="archive" -> if synced, stream snapshot to a non btrfs filesystem + # CONFIG_TYPE="child" -> if synced, stream snapshot to a given CHILD_CONFIG name + CONFIG_TYPE="child" + #CHILD_CONFIG= + #PARENT_CONFIG=