Eddie On Everything » perl

Rotating Backup Directories using cp -al (hardlinks) to Save Disk Space

eddie — Sun, 15 Nov 2009 20:13:16 +0000

The copy command “cp -al” found on all versions of Unix/Linux creates what’s called a “hard link” to a file. The nice thing about this command is that it doesn’t create an actual copy of the file on disk – instead, it creates a “link” or pointer to the file data on the disk. Basically a “snapshot” of that directory in-time. The net result is that you can have 10 “copies’ of a 10G file that only take up a total of 10G.

This nifty behavior makes cp -al, when combined with rsync, ideal for backup systems. One can use the cp -al command to take a “snapshot” of a given directory tree at a given time, at the expense of very little additional disk space. I use this script in concert with my rsync_backup.pl script to keep 21 days of “snapshot” backups of each of my machines.


#!/usr/bin/perl
use POSIX;

# Rotates backup directories w/ cp -al (hardlinks)
# Deletes directories older than $KEEP_DAYS
# Runs each night ahead of backup process
# (c) 2009 eddie@eddieoneverything.com

$KEEP_DAYS=21;
$LOGFILE = "/var/log/rotate_backups";

@BACKUP_DIRS=(
   '/mnt/backup/hansel',
   '/mnt/backup/tiger',
   '/mnt/backup/june'
);

$ts = get_timestamp();

open hLOG, ">>$LOGFILE";
print hLOG "=" x 80, "\n";
print hLOG "Run START at " . `date` . "\n";
print hLOG "=" x 80, "\n";

## Do the rotation
print hLOG "Do today's rotation\n";
foreach $dir ( @BACKUP_DIRS ){
   print hLOG "\t" , `date`;
   $newfn = $dir ."_" .  $ts;
   $cmd = "cp -al $dir $newfn";
   print hLOG "Execute Command: $cmd\n";
   `$cmd`;
}

## Delete old directories
print hLOG "Delete Old Directories\n";
foreach $dir ( @BACKUP_DIRS ){
   $dir=~m/^(.+)\/(.+?)$/;
   $base= $1;
   $stub=$2;
   #print "dir is $dir\nBASE: $base\nSTUB:$stub\n";

   opendir hDIR, "$base" or die "can't open directory $base";
   @dirlist=grep { /^$stub\_/ && !/^$stub$/  } readdir(hDIR);
   closedir hDIR;

   foreach $d (@dirlist){
      print hLOG "\t" , `date`;
      $d=~/$stub\_([0-9]+)_([0-9]+)_([0-9]+)_.+$/;
      $year= $1;
      $month = $2;
      $day = $3;

      if (dirIsOlder($year, $month, $day)){
         #print "$d\n";
         $remove_dir= $base . '/' . $d;
         $cmd = "rm -Rf \"$remove_dir\"";
         print hLOG "Execute command: $cmd\n";
         `$cmd`;
      }else{
         print hLOG "Keep $d\n";
      }
   }
}
print hLOG "=" x 80, "\n";
print hLOG "Run END at " . `date` . "\n";
print hLOG "=" x 80, "\n";
close hLOG;

# ------------- Subroutines & functions -----------------

sub get_timestamp {
   ($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst) = localtime(time);
   $mon+=1;

   if ($mon < 10) { $mon = "0$mon"; }
   if ($mday < 10) { $mday = "0$mday"; }
   if ($hour < 10) { $hour = "0$hour"; }
   if ($min < 10) { $min = "0$min"; }
   if ($sec < 10) { $sec = "0$sec"; }
   $year=$year+1900;

   return $year . '_' . $mon . '_' . $mday . '__' . $hour . '_' . $min . '_' . $sec;
}

sub dirIsOlder{
   ($fyear, $fmonth, $fday) = @_;
   #print "Check $year - $month - $day \n";

   $now  = mktime(localtime());
   $then =  mktime (0, 0 , 0, $fday-1, $fmonth-1, $fyear-1900 , 0, 0);

   $diff_sec = $now - $then;
   $days_since = $diff_sec / 24 / 60 / 60;
   #print "n: $now. t: $then. Diff ($diff_sec) = $days_since\n";
   #print "$days_since $fyear-$fmonth-$fday\n";

   ## Subtract days from today

   if ($days_since > $KEEP_DAYS){
      return 1;
   }else{
      return 0;
   }
}

Using rsync & ssh to Backup Files Between Linux Machines (Perl)

eddie — Sun, 15 Nov 2009 19:27:32 +0000

This script performs a network backup using unix’s rsync command over ssh. It performs an incremental backup, which means that it only updates new files and files that have changed since the last backup, thus minimizing network traffic.

This script runs on the source machine (SOURCE) and connects to the backup machine (BACKUP). (SOURCE->BACKUP). It assumes a user named backup-SOURCE on the backup machine and connects as that user.

One advantage of using SOURCE->BACKUP setup rather than a BACKUP->SOURCE method is that if your BACKUP server is compromised, the attacker will not be able to gain access to all of your production machines.

There are admittedly disadvantages to running a SOURCE->BACKUP setup, rather than a BACKUP->SOURCE setup. One, you’ll need a copy of the script on each of the machines that you want to backup, making maintenance a bit more involved. Second, if your SOURCE machine becomes compromised, the attacker will be able to gain access to your backup server. This second issue can be mitigated by setting the permissions properly on the BACKUP machine. (This is why I use a separate user account for each machine that I backup.)

How to use this script:

Edit the configuration options in the script.
Add a user called “backup-{$SOURCE} to the BACKUP machine.
Set up the appropriate SSH keys for the user to bypass password prompts.
Set up a cron job to run the script nightly.
If you want Apple-ish “time machine style” backups, set up a script to rotate the backups on the BACKUP machine. I use this rotate_backups.pl script to make backups using cp -al (hardlinks).


#!/usr/bin/perl
# Filename: rsync_backup.pl
# Will back things up from one computer to another over SSH using rsync.
# Use in conjunction with a rotate script on the destination machine for
# time-machine-like incremental backups.
# Note: SSH SSL auth keys must be installed on each end to avoid passwd prompt
# Assumes a user named "backup-$THIS_COMPUTER" on remote machine.
# All output will go to Logfile
#
# (c) 2009 eddie@eddieoneverything.com 

use Data::Dumper;

# This computer's name.  Used to name the backup files.
$THIS_COMPUTER = "yoakam";

# The remote machine's URL or IP address.
$REMOTE_MACHINE = "127.0.0.1";

# The backup directory on the remote machine
$REMOTE_BACKUP_LOCATION="/mnt/backup";

# Log things
$LOGFILE = "/var/log/rsync_backup";

# Directories to backup.
@BACKUP_DIRS=(
   '/home',
   '/root',
   '/etc',
   '/var/log',
   '/var/lib/mysql',
   '/var/spool/cron'
);

# Directories to ignore.
@IGNORE_DIRS=(
   '/home/lost+found',
   '/home/tmp',
   '/home/virtfs'
);

#####################################################
########### END Configuration options ###############
#####################################################

open hLOG, ">>$LOGFILE";
$now = `date`;
print hLOG "-" x 80, "\n";
print hLOG "START BACKUP $THIS_COMPUTER at $now\n" ;
print hLOG "-" x 80, "\n";
# Get a list of dirs to back up
foreach (@BACKUP_DIRS){
   $dir = $_;
   opendir(DIR, $dir) || alert ("cant opendir $dir: $!");
   while ($file = readdir(DIR)){
      if ($file ne '.' && $file ne '..'){
         $my_file = $dir . "/" .  $file;
         if ( -d $my_file){
            if (grep {/$my_file/} @IGNORE_DIRS){
               print hLOG "Ignore Dir: $my_file\n";
            }else{
               print hLOG "Dir: $my_file\n";
               push @BACKUP_LIST, {'dir' => $dir, 'name'=>$file};
            }
         }else{
            print hLOG "file is $my_file -- \n";
            @arr_temp =  ($dir, $my_file);
            push @BACKUP_LIST, {'dir' => $dir, 'name'=>$file};
         }
      }
   }
   closedir DIR;
}

#print Dumper(@BACKUP_LIST);

#Back up each dir in the backup list
foreach $my_entry (@BACKUP_LIST){
   $dir = $my_entry->{'dir'};
   $b = $my_entry->{'name'};

   $cmd = "rsync -avz --delete-after -e ssh \"$dir/$b\" backup-$THIS_COMPUTER\@$REMOTE_MACHINE:\"/$REMOTE_BACKUP_LOCATION/$THIS_COMPUTER$dir\"";
   $now=`date`;
   print hLOG "\n$now\t$cmd ";
   #print "$cmd\n";
   `$cmd >> $LOGFILE 2>&1`;
}

$now = `date`;
print hLOG "-" x 80, "\n";
print hLOG "END BACKUP of $THIS_COMPUTER at $now\n" ;
print hLOG "-" x 80, "\n";
close hLOG;

###################### Subs #########################
sub alert(){
   $a=shift(@_);
   print hLOG "#############################\n";
   print hLOG "#############################\n";
   print hLOG "#############################\n";
   print hLOG "Cannot backup $a $!\n";
   print hLOG "#############################\n";
   print hLOG "#############################\n";
   print hLOG "#############################\n";
}

How to do a search and replace in multiple files using Perl

eddie — Wed, 21 Jan 2009 22:48:38 +0000

Over the years, I’ve written a number of “search and replace in multiple files” scripts. Working on a computer that didn’t have my standard, custom toolset available, I found myself searching for an alternative.

Found this tip over at Dzone Snippets – do it all with a single line of perl.

perl -pi -w -e ’s/search/replace/g;’ *.txt

Of course, you’ll need Perl installed on your system and command line access to make use of it.

Random Posts

Automatically renaming files downloaded with Coast to Coast AM Media Center

eddie — Tue, 08 Apr 2008 02:07:01 +0000

I’m an insomniac.

I have trouble falling asleep at night because thoughts never seem to stop racing through my head – Did I do X yet? What if I did Y? Ooh, I could make it easier to do Y … Hey, I have a cool idea for an awesome invention… and I’m starting to get excited about it … (!)

Listening to radio shows helps me quiet my mind & get to sleep. To that end, I use the Coast to Coast AM Media Center to automatically download shows from the Coast to Coast AM website. (Sure, the content can be really nutty from time to time, but I treat it as entertainment & fiction, not a show to be taken seriously.) One of the limitations of the downloader is that the downloaded mp3s don’t have very useful names – instead of being named something useful, like “03-20 John Titor and Time Travel,” the files instead carry largely useless names such as “Coast to Coast – Mar 20 2008 – Hour 2.mp3.”

I wrote the below perl script to make the downloaded file names more useful. It connects to the Coast to Coast AM website, looks up the file name, creates a directory with that filename, then moves the downloaded mp3s to that directory. To run the script on your Windows machine, you’ll need to have the Cygwin suite installed, along with the cygwin perl module.

#!/usr/bin/perl

$DIR="/cygdrive/c/Documents and Settings/eddie/My Documents/coast to coast am media center";

opendir (DIR, $DIR) or die "cant open dir $DIR";
while (my $file = readdir (DIR)){
   if ($file=~/mp3$/){
      #process mp3 file
      #print "$file \n";
      $file=~/(\w+) ([0-9]+) ([0-9]+)/;
      #$file=~/^.+? (\w) ([0-9]+) ([0-9]+) .+$/;
      #print "mon $1 \n day $2 \n year $3\n";
      $y=$3;
      $month=$1;
      $d=$2;

      print "month is $month\n";
      if ($month eq  "Feb"){
         $m="02";
      }elsif($month eq "Mar"){
         $m="03";
      }elsif($month eq "Apr"){
         $m="04";
      }elsif($month eq "May"){
         $m="05";
      }elsif($month eq "Jun"){
         $m="06";
      }elsif($month eq "Jul"){
         $m="07";
      }elsif($month eq "Aug"){
         $m="08";
      }elsif($month eq "Sep"){
         $m="09";
      }elsif($month eq "Oct"){
         $m="10";
      }elsif($month eq "Nov"){
         $m="11";
      }elsif($month eq "Dec"){
         $m="12";
      }elsif($month eq "Jan"){
         $m="01";
      }else{
         $m="XX";
      }

      $date_string = $m . "-" . $d;

      if ($SHOWS[$y-$m-$d]){
         $title=$SHOWS[$y-$m-$d];
      }else{
         $url = "http://www.coasttocoastam.com/shows/$y/$m/$d.html";
         #print "getting URL $url...\n";
         #Title
         system("wget -q -O /tmp/temp.$$ $url");
         $file_contents = `cat /tmp/temp.$$`;
         $file_contents=~/<\/a>(.+?)<\/h2>/;
         $title = $1;
         $title=~s/\&/and/gis;
         $title=~s/\:/-/gis;
         $title=~s/\!//gis;
         $desc=~s/<.+?>//gis;
         print "TITLE:  $title\n";
         $SHOWS[$y-$m-$d] = $title;
         system("rm  /tmp/temp.$$");
      }

      ## move the file to the proper directory
      $dirname = $date_string . " " . $title;
      if (-e "$DIR/$dirname"){
         #dir already exists
      }else{
         $cmd="mkdir  \"$DIR/$dirname\"";
         system ($cmd);
      }

      $cmd = "mv \"$DIR/$file\" \"$DIR/$dirname\"";
      #print "want to $cmd\n";
      system ($cmd);

   }
}
close DIR;

Random Posts

How to do a search and replace in multiple files using perl and linux

eddie — Tue, 26 Feb 2008 16:36:04 +0000

So you’ve got 100 files in a directory and you want to replace the string “foo” with “bar”. Stop before you waste your time opening up every single file and making the change, you can make all of the changes with a single line of perl.

perl -pi -w -e 's/foo/bar/g' *\.php

On his blog, Liam Delahunty explains what each of the following command line switches means.

-e means execute the following line of code. -i means edit in-place -w write warnings -p loop

There you have it – a nice and easy Perl solution that should save you tons of time.

Eddie On Everything » perl

Rotating Backup Directories using cp -al (hardlinks) to Save Disk Space

Related Posts

Using rsync & ssh to Backup Files Between Linux Machines (Perl)

Related Posts

How to do a search and replace in multiple files using Perl

Random Posts

Automatically renaming files downloaded with Coast to Coast AM Media Center

Title

Random Posts

How to do a search and replace in multiple files using perl and linux

Random Posts