Feb 082013
 
About

Determine candidates and delete from a set of directories containing aging backups.
As a follow up to the release of sayebackup.sh last December, here’s a complimentary tool we’re using at Lanedo. Suppose a number of backup directories have piled up after a while, using sayebackup.sh or any other tool that creates time stamped file names:

 drwxrwxr-x etc-2010-02-02-06:06:01-snap
 drwxrwxr-x etc-2011-07-07-06:06:01-snap
 drwxrwxr-x etc-2011-07-07-12:45:53-snap
 drwxrwxr-x etc-2012-12-28-06:06:01-snap
 drwxrwxr-x etc-2013-02-02-06:06:01-snap
 lrwxrwxrwx etc-current -> etc-2012-12-28-06:06:01-snap

Which file should be deleted once the backup device starts to fill up?
Sayepurge parses the timestamps from the names of this set of backup directories, computes the time deltas, and determines good deletion candidates so that backups are spaced out over time most evenly. The exact behavior can be tuned by specifying the number of recent files to guard against deletion (-g), the number of historic backups to keep around (-k) and the maximum number of deletions for any given run (-d). In the above set of files, the two backups from 2011-07-07 are only 6h apart, so they make good purging candidates, example:

 $ sayepurge.sh -o etc -g 1 -k 3 
 Ignore: ./etc-2013-02-02-06:06:01-snap
 Purge:  ./etc-2011-07-07-06:06:01-snap
 Keep:   ./etc-2012-12-28-06:06:01-snap
 Keep:   ./etc-2011-07-07-12:45:53-snap
 Keep:   ./etc-2010-02-02-06:06:01-snap

For day to day use, it makes sense to use both tools combined e.g. via crontab. Here’s a sample command to perform daily backups of /etc/ and then keep 6 directories worth of daily backups stored in a toplevel directory for backups:

 /bin/sayebackup.sh -q -C /backups/ -o etc /etc/ && /bin/sayepurge.sh -q -o etc -g 3 -k 3

Let me know in the comments what mechanisms you are using to purge aging backups!

Resources

The GitHub release tag is here: backups-0.0.2
Script URL for direct downloads: sayepurge.sh

Usage
Usage: sayepurge.sh [options] sources...
OPTIONS:
  --inc         merge incremental backups
  -g <nguarded> recent files to guard (8)
  -k <nkeeps>   non-recent to keep (8)
  -d <maxdelet> maximum number of deletions
  -C <dir>      backup directory
  -o <prefix>   output directory name (default: 'bak')
  -q, --quiet   suppress progress information
  --fake        only simulate deletions or merges
  -L            list all backup files with delta times
DESCRIPTION:
  Delete candidates from a set of aging backups to spread backups most evenly
  over time, based on time stamps embedded in directory names.
  Backups older than <nguarded> are purged, so that only <nkeeps> backups
  remain. In other words, the number of backups is reduced to <nguarded>
  + <nkeeps>, where <nguarded> are the most recent backups.
  The puring logic will always pick the backup with the shortest time
  distance to other backups. Thus, the number of <nkeeps> remaining
  backups is most evenly distributed across the total time period within
  which backups have been created.
  Purging of incremental backups happens via merging of newly created
  files into the backups predecessor. Thus merged incrementals may
  contain newly created files from after the incremental backups creation
  time, but the function of reverse incremental backups is fully
  preserved. Merged incrementals use a different file name ending (-xinc).
See Also

Sayebackup.sh – deduplicating backups with rsync

Dec 012012
 
About

Due to popular request, I’m putting up a polished version of the backup script that we’ve been using over the years at Lanedo to backup our systems remotely. This script uses a special feature of rsync(1) v2.6.4 for the creation of backups which share storage space with previous backups by hard-linking files.
The various options needed for rsync and ssh to minimize transfer bandwidth over the Internet, time-stamping for the backups and handling of several rsync oddities warranted encapsulation of the logic into a dedicated script.

Resources

The GitHub release tag is here: backups-0.0.1
Script URL for direct downloads: sayebackup.sh

Example

This example shows creation of two consecutive backups and displays the sizes.

$ sayebackup.sh -i ~/.ssh/id_examplecom user@example.com:mydir # create backup as bak-.../mydir
$ sayebackup.sh -i ~/.ssh/id_examplecom user@example.com:mydir # create second bak-2012...-snap/
$ ls -l # show all the backups that have been created
drwxrwxr-x 3 user group 4096 Dez  1 03:16 bak-2012-12-01-03:16:50-snap
drwxrwxr-x 3 user group 4096 Dez  1 03:17 bak-2012-12-01-03:17:12-snap
lrwxrwxrwx 1 user group   28 Dez  1 03:17 bak-current -> bak-2012-12-01-03:17:12-snap
$ du -sh bak-* # the second backup is smaller due to hard links
4.1M    bak-2012-12-01-03:16:50-snap
128K    bak-2012-12-01-03:17:12-snap
4.0K    bak-current
Usage
Usage: sayebackup.sh [options] sources...
OPTIONS:
  --inc         make reverse incremental backup
  --dry         run and show rsync with --dry-run option
  --help        print usage summary
  -C <dir>      backup directory (default: '.')
  -E <exclfile> file with rsync exclude list
  -l <account>  ssh user name to use (see ssh(1) -l)
  -i <identity> ssh identity key file to use (see ssh(1) -i)
  -P <sshport>  ssh port to use on the remote system
  -L <linkdest> hardlink dest files from <linkdest>/
  -o <prefix>   output directory name (default: 'bak')
  -q, --quiet   suppress progress information
  -c            perform checksum based file content comparisons
  --one-file-system
  -x            disable crossing of filesystem boundaries
  --version     script and rsync versions
DESCRIPTION:
  This script creates full or reverse incremental backups using the
  rsync(1) command. Backup directory names contain the date and time
  of each backup run to allow sorting and selective pruning.
  At the end of each successful backup run, a symlink '*-current' is
  updated to always point at the latest backup. To reduce remote file
  transfers, the '-L' option can be used (possibly multiple times) to
  specify existing local file trees from which files will be
  hard-linked into the backup.
 Full Backups:
  Upon each invocation, a new backup directory is created that contains
  all files of the source system. Hard links are created to files of
  previous backups where possible, so extra storage space is only required
  for contents that changed between backups.
 Incremental Backups:
  In incremental mode, the most recent backup is always a full backup,
  while the previous full backup is degraded to a reverse incremental
  backup, which only contains differences between the current and the
  last backup.
 RSYNC_BINARY Environment variable used to override the rsync binary path.
See Also

Testbit Tools – Version 11.09 Release

Oct 012011
 
No Bugs

(Image: Mag3737)

 

And here’s another muffin from the code cake factory…

About Testbit Tools
The ‘Testbit Tools’ package contains tools proven to be useful during the development of several Testbit and Lanedo projects. The tools are Free Software and can be redistributed under the GNU GPLv3+.

This release features the addition of buglist.py, useful to aid in report and summary generation from your favorite bugzilla.

Downloading Testbit Tools
The Testbit Tools packages are available for download in the testbit-tools folder, the newest release is here: testbit-tools-11.09.0.tar.bz2

Changes in version 11.09.0:

  • Added buglist, a script to list and download bugs from bug trackers.
  • Added buildfay, a script with various sub commands to aid release making.
  • Fixed version information for all tools.
  • Added support to the Xamarin Bug Tracker to buglist.py.
  •  

    Feedback

    If you find this release useful, we highly appreciate your feature requests, bug reports, patches or review comments!

    See Also

    1. The Bugzilla Utility buglist.py – managing bug lists
    2. Wikihtml2man Introduction – using html2wiki

    Apr 092011
     

    BEAST/BSE version 0.7.4 is available for download at:

    BEAST is a music composition and modular synthesis application released as free software under the GNU LGPL that runs under Unix. Refer to the About page for more details.

    The 0.7.4 release integrates the bse-alsa package, several speedups, important bug fixes and translation updates.

    Please feel free to provide useful feedback or contribute on IRC, the mailing list and in the Wiki.

    TRANSLATORS: Please help us to improve the BEAST translation, just download the tarball, edit po/.po and email it to us or submit translations directly via the Beast page at Transifex.

    Overview of Changes in BEAST/BSE 0.7.4:

    • Renamed the project to Better Audio System / Better Sound Engine
    • Moved project website to: http://beast.testbit.eu/
    • Various build system fixes [stw,timj]
    • License fixups for some scripts [stw]
    • Fixed subnormal tests on AMD64 if SSE unit is in DAZ mode [stw]
    • Replaced slow resampler checks with a much faster resampling test [stw]
    • Performance improvements for various tests [stw]
    • GLib 2.28 unit test porting [stw]
    • Speed improvements for record field name [stw]
    • Fixed XRUNs in ALSA driver on 64bit systems [timj]
    • Added beast.doap [Jonh Wendell]
    • PO handling improvements.
    • Updated German translation.
    • Updated Norwegian bokmål translation [Kjartan Maraas]
    • Added e-Telugu translation [Veeven]

    Sep 102010
     

    BEAST/BSE and BSE-ALSA version 0.7.2 are available for download:

        http://beast.gtk.org/beast-ftp/v0.7/beast-0.7.2.tar.bz2
        http://beast.gtk.org/beast-ftp/v0.7/bse-alsa-0.7.2.tar.bz2
    

    BEAST is a music composition and modular synthesis application. The “Bedevilled” portion of the names has no religious background, please refer to the About page for more details.

        Homepage:           http://beast.gtk.org/
        Downloads:          http://beast.gtk.org/beast-ftp/
        Feedback:           http://mail.gnome.org/mailman/listinfo/beast/
    

    The 0.7.2 release provides new plugins and instruments, and a long list of bug fixes, improvements and translation updates.

    TRANSLATORS: Please help us to improve the BEAST translations at Transifex.

    Overview of Changes in BEAST/BSE 0.7.2:

    • Moved Beast/BSE to GNU LGPL, use AS-IS license for examples
    • Module changes and additions: ArtsCompressor – Relicensed to LGPL with permission from Matthias Kretz BseContribSampleAndHold – Relicensed to LGPL with permission from Artem Popov DavXTalStrings – Use deterministic random numbers for unit tests BseNoise – Improved random number generator
    • Switched to autogenerated ChangeLogs
    • Error bell can be muted in beast preferences dialog
    • Added multisample creation/editing command line tool: bsewavetool
    • Support adjustable volume, pitching and drum envelopes in .bsewave files
    • Added Retro Acoustic drum kit [Tim, Stefan]
    • New loadable Instruments/Effects:
      - BQS Bass Drum E8012 [Tim, Stefan]
      - BQS Slow Hum [Stefan, William DeVore]
      - FSM Fresh Water Bass instrument [Krzysztof Foltman]
      - FSM Growl Bass instrument [Krzysztof Foltman]
      - FSM Synth String Sweep [Krzysztof Foltman]
      
    • Added support for loading 32bit and 24bit PCM-format WAV files
    • Added support for gcc-4.4 and automake-1.10
    • Added support for guile-1.8, guile-1.6 remains as minimum requirement
    • Various fixes, improvements and much improved test coverage.
    • Bug fixes: #452604, #468229, #344388, #451086, #450724, #454121, #491552, #450490, #441936, #336766, #433431, #474332, #474244, #456879, #456408, #424897 [Tim Janik, Stefan Westerfeld]
    • Migrated translation support to use awk, sed and po/Makefile.am.
    • Updated German translation [Mario Blättermann]
    • Updated Italien translation [Michele Petrecca]
    • Updated Occitan translation [Yannig Marchegay]
    • Updated Brazilian Portugues translation [Leonardo Ferreira Fontenelle]
    • Updated British English translation [David Lodge]
    • Updated Spanish translation [Jorge Gonzalez]
    • Updated Slovenian translation [Andrej Znidarsic]
    • Updated Danish translation [Joe Hansen]
    • Updated French translation [Bruno Brouard]
    • Added Norwegian bokmal translation [Kjartan Maraas]
    • Added Ukrainian translation [Maxim V. Dziumanenko]

    Overview of Changes in BSE-ALSA 0.7.2:

    • Fixes for automake-1.10 builds
    • Moved Beast/BSE to GNU LGPL

    UPDATES: Updated translator instructions and uploaded a new release tarball with build fixes.

    Aug 202010
     

    Rapicorn 10.08.0 is available for download:

        http://rapicorn.org/dist/rapicorn/rapicorn-10.08.0.tar.bz2
    

    Rapicorn is an Experimental UI Toolkit. Most things in a toolkit implementation will benefit from proper application of modern technologies (e.g. pthreads, XCB, Cairo, compositing, IDL, XML notation, path evaluation, DSLs, unit tests, SVG). Rapicorn is developed on this base with the aim to significantly improve developer efficiency and user experience.

    This release contains a multitude of new features like Python bindings, an automated test suite, myriads of bug fixes and small improvements.

        Homepage:           http://rapicorn.org/
        Downloads:          http://rapicorn.org/dist/rapicorn/
        Feedback:           http://rapicorn.org/mailman/listinfo/rapicorn-list
    

    Updates in Rapicorn 10.08.0:

    • Introduced Cairo dependency.
    • Introduced C++ TR1 dependency (for shared_ptr, etc).
    • Added $RAPICORN environment variable to control logging and more.
    • Added PLIC, a pluggable IDL compiler.
    • Added Remote Object Programming Extension.
    • Added window and widget addressing functionality.
    • Allow signal disconnections by Id.
    • Added simple expression parser to evaluate property value assignments.
    • Added sample expression evaluator as: ui/tests/sinfextest –shell
    • Added XML UI file error reporting.
    • Changed packing properties into regular Item properties.
    • Merged h/v-scale and -align into Item pack properties.
    • Replaced hfill/vfill in Table by h/v-scale and -align properties.
    • Replaced *_attach in table by Item’s h/v-position and -span properties.
    • Got rid of the packer class entirely.
    • Added testing framework with macros and reference files.
    • Added ‘-x’ to rapidrun to auto-exit.
    • Added ‘–list’ to rapidrun to list gadgets from a GUI file.
    • Provide regular expression parser API in rapicorn-core.
    • Introduced a basic typed value system and model APIs.
    • Started list area/widget development.
    • Optimized resizing performance.
    • Cosmetic fixups in colors and gradients.
    • Implemented horizontal and vertical size groups.
    • Renamed string_to_cescape() and fixed string_to_cquote() quoting.
    • Generate internal C++ bindings (to IDL specifications).
    • Provide a main loop implementation in Python.
    • Generate external Python bindings.
    • Majorly extended test suite.
    • Optimized remote invocations and reference counting.
    • Started migrating interfaces from headers to IDL.
    • Hand optimized threading primitives and main loop handling.
    • Applied fixes for AMD64 compilation. [Stefan Westerfeld]
    • Updates to compile with GCC-4.4.
    • Linker optimizations to improve ELF relocations.
    • Loads of other cleanups and bug fixes.

    Apr 232010
     

    Doxer 10.04.0 is available for download:

        http://rapicorn.org/dist/doxer/doxer-10.04.0.tar.bz2
    

    Doxer is a documentation tool initiated originally for generating source code documentation. This is the first public release of Doxer as a separate package (it has been included in other packages previously).

    This release contains a wiki markup parser, a corresponding HTML generator, and a Drupal input format module. The markup syntax is designed to cover the feature set required to write source code documentation and general purpose documents. The parser and HTML generator have a strong focus on robustness to support the full range of user sophistication found on general purpose websites. An extensive test suite accompanies the development.

        Homepage:           http://rapicorn.org/doxer.html
        Download:           http://rapicorn.org/dist/doxer/
        Wiki Markup:        http://rapicorn.org/DoxerMarkup.html
        Feedback:           http://rapicorn.org/mailman/listinfo/rapicorn-list
    

    User visible features in Doxer 10.04.0:

    • Added Doxer wiki markup parser.
    • Added docs/DoxerMarkup.txt to describe the markup.
    • Added HTML 4.01 Transitional generator for wiki markup.
    • Added extensive Doxer wiki markup test suite.
    • Wrapped the wiki markup parser as Drupal 6 input format module.
    • Rewrote documentation extractor (Doxyscan).
    • Started new Doxyscan Html documentation generator.