Each of these files is about 100M when decompressed. 

I ran into a big problem when unzipping the files all at once – when I did a simple “unzip \*zip", the server load eventually skyrocketed. 

The load got so high, in fact, that mySQL and httpd couldn’t even keep up with the incoming web requests, and the mySQL database eventually crashed. Well, it didn’t exactly crash, but the requests got so stacked up that it stopped accepting new connections, so it might as well have been crashed.

I assume that this is because the Linux server (Red Hat, in my case) has to do a bunch of “behind the scenes” type things when allocating large chunks of space for new files.  You know, file system record keeping, delayed disk I/O, things like that.

“nice -n19″ wasn’t doing the job, so I tried doing a simple batch file that would sleep 10 seconds before unzipping each file.  This worked, for the most part, but the server load eventually got higher than I like to see. Left alone, the whole thing could have cascaded out of control again, especially given that I wasn’t taking into account the *other* things the server may have been doing at the time. And I wanted to go to lunch, so hanging out and watching it wasn’t an attractive option.

I decided to write a simple script that checked the server load via /proc/loadavg, then only performed the unzip operation if the load was below a certain value.

Here’s my script, written in Perl.

#!/usr/bin/perl

$MAX_LOAD = .9;

open hIN, "<files";

while (<hIN>){
   $filename = $_;
   chomp($filename);

   $cmd = "unzip \"$filename\"";

   print "$cmd\n";
   print `$cmd`;

   # Get the load average
   do {
      $load = `cat /proc/loadavg  | cut -f1 -d " "`;
      chomp($load);

      if ($load > $MAX_LOAD){
         print "Load is $load - Sleeping\n";
         sleep (5);
      }

   }while ($load > $MAX_LOAD);

}
close hIN;

I chose to keep the server load below 0.9, a nice number, IMHO.

Update: I’ve done a little more checking around, and I see that there’s a Unix command that will do a lot of this for you – the “batch” command will only execute a command when the server load drops below a certain threshold. The default is 0.8. Good to know.

The method I outline above is still useful in other instances where “batch” can’t be used. For instance, I’m now using this method of checking the server load in a data loading script. I parse my input files, generate sql & execute it. Then, every couple thousand queries or so I check the server load. If it’s getting too high, the script sleeps until the load falls to an acceptable level.

Related Posts

• The Zen of Taco Bell Programming – Using Unix Tools to Prevent Reinventing the Wheel
• Rotating Backup Directories using cp -al (hardlinks) to Save Disk Space
• Using rsync & ssh to Backup Files Between Linux Machines (Perl)
• “Argument list too long” from rm command. A Perl Script to Get Around the Problem
• Linux .forward file not working? Make sure you have the permissions set correctly
• Caution: filename not matched error when unzipping multiple files workaround
• How to Scan a Computer for Open Ports

Using a single config file in a basic Perl script is pretty straightforward – you simply use a require statement to read the external file, and the variables contained within are automatically imported. (Though if you’re using strict, you’ll need to declare them using the “our” keyword.)

A problem arises when you want to use those global variables in a package file, though – because a Package has its own namespace, variables declared in your external config file are not imported into your module, even if you use the “our” keyword.

Fortunately, I was able to find a solution here, courtesy of Jeff Pang.

Here are a couple of examples.

The simple way to share variables between Perl scripts using “require”

File test.include.pl

#!/usr/bin/perl
$var = ".j.a.p.h.";

File test.pl

#!/usr/bin/perl
use strict;
require "test.include.pl";
our $var;
print "VAR is $var\n";

After creating the two files above and running test.pl, the output is:

eddie@widmers:~/wrk/$ ./test.pl
VAR is .j.a.p.h.


The more advanced way of sharing variables with a Perl Package
This method involves declaring the config file as a package, and then explicitly exporting the variables you’re interested in. Your other scripts and modules will then import those variables, making them available in that scope.

In this case, the config variable package will look something like this:

package SiteConfig;
require Exporter;

our @ISA = qw(Exporter);
our @EXPORT=qw($var $vare);

our $var = ".j.a.p.h.";

1;

While the package you’re looking to use the variables in will look something like this:

package TestTest;
use TestConfig;
print "Var is $var\n";

sub new
{
   my $package = shift;
   print "VAR is $var\n";
  return bless({}, $package);
}
1;

Of course, your particular needs will vary, so you’ll need to use your noodle to modify the above as necessary.

Related Posts

• “Argument list too long” from rm command. A Perl Script to Get Around the Problem
• Managing Server Load when unzipping many large files
• How to Scan a Computer for Open Ports
• The Zen of Taco Bell Programming – Using Unix Tools to Prevent Reinventing the Wheel
• Getting PHP’s print_r Function To Return A String
• Calibre Recipe for Minneapolis Star Tribune (StarTribune.com)
• Rotating Backup Directories using cp -al (hardlinks) to Save Disk Space

This nifty behavior makes cp -al, when combined with rsync, ideal for backup systems. One can use the cp -al command to take a “snapshot” of a given directory tree at a given time, at the expense of very little additional disk space. I use this script in concert with my rsync_backup.pl script to keep 21 days of “snapshot” backups of each of my machines.

#!/usr/bin/perl
use POSIX;

# Rotates backup directories w/ cp -al (hardlinks)
# Deletes directories older than $KEEP_DAYS
# Runs each night ahead of backup process
# (c) 2009 eddie@eddieoneverything.com

$KEEP_DAYS=21;
$LOGFILE = "/var/log/rotate_backups";

@BACKUP_DIRS=(
   '/mnt/backup/hansel',
   '/mnt/backup/tiger',
   '/mnt/backup/june'
);

$ts = get_timestamp();

open hLOG, ">>$LOGFILE";
print hLOG "=" x 80, "\n";
print hLOG "Run START at " . `date` . "\n";
print hLOG "=" x 80, "\n";

## Do the rotation
print hLOG "Do today's rotation\n";
foreach $dir ( @BACKUP_DIRS ){
   print hLOG "\t" , `date`;
   $newfn = $dir ."_" .  $ts;
   $cmd = "cp -al $dir $newfn";
   print hLOG "Execute Command: $cmd\n";
   `$cmd`;
}

## Delete old directories
print hLOG "Delete Old Directories\n";
foreach $dir ( @BACKUP_DIRS ){
   $dir=~m/^(.+)\/(.+?)$/;
   $base= $1;
   $stub=$2;
   #print "dir is $dir\nBASE: $base\nSTUB:$stub\n";

   opendir hDIR, "$base" or die "can't open directory $base";
   @dirlist=grep { /^$stub\_/ && !/^$stub$/  } readdir(hDIR);
   closedir hDIR;

   foreach $d (@dirlist){
      print hLOG "\t" , `date`;
      $d=~/$stub\_([0-9]+)_([0-9]+)_([0-9]+)_.+$/;
      $year= $1;
      $month = $2;
      $day = $3;

      if (dirIsOlder($year, $month, $day)){
         #print "$d\n";
         $remove_dir= $base . '/' . $d;
         $cmd = "rm -Rf \"$remove_dir\"";
         print hLOG "Execute command: $cmd\n";
         `$cmd`;
      }else{
         print hLOG "Keep $d\n";
      }
   }
}
print hLOG "=" x 80, "\n";
print hLOG "Run END at " . `date` . "\n";
print hLOG "=" x 80, "\n";
close hLOG;

# ------------- Subroutines & functions -----------------

sub get_timestamp {
   ($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst) = localtime(time);
   $mon+=1;

   if ($mon < 10) { $mon = "0$mon"; }
   if ($mday < 10) { $mday = "0$mday"; }
   if ($hour < 10) { $hour = "0$hour"; }
   if ($min < 10) { $min = "0$min"; }
   if ($sec < 10) { $sec = "0$sec"; }
   $year=$year+1900;

   return $year . '_' . $mon . '_' . $mday . '__' . $hour . '_' . $min . '_' . $sec;
}

sub dirIsOlder{
   ($fyear, $fmonth, $fday) = @_;
   #print "Check $year - $month - $day \n";

   $now  = mktime(localtime());
   $then =  mktime (0, 0 , 0, $fday-1, $fmonth-1, $fyear-1900 , 0, 0);

   $diff_sec = $now - $then;
   $days_since = $diff_sec / 24 / 60 / 60;
   #print "n: $now. t: $then. Diff ($diff_sec) = $days_since\n";
   #print "$days_since $fyear-$fmonth-$fday\n";

   ## Subtract days from today

   if ($days_since > $KEEP_DAYS){
      return 1;
   }else{
      return 0;
   }
}

Related Posts

• Using rsync & ssh to Backup Files Between Linux Machines (Perl)
• “Argument list too long” from rm command. A Perl Script to Get Around the Problem
• Managing Server Load when unzipping many large files
• How to Scan a Computer for Open Ports
• The Zen of Taco Bell Programming – Using Unix Tools to Prevent Reinventing the Wheel
• Linux .forward file not working? Make sure you have the permissions set correctly
• Caution: filename not matched error when unzipping multiple files workaround

This script runs on the source machine (SOURCE) and connects to the backup machine (BACKUP). (SOURCE->BACKUP). It assumes a user named backup-SOURCE on the backup machine and connects as that user.

One advantage of using SOURCE->BACKUP setup rather than a BACKUP->SOURCE method is that if your BACKUP server is compromised, the attacker will not be able to gain access to all of your production machines.

There are admittedly disadvantages to running a SOURCE->BACKUP setup, rather than a BACKUP->SOURCE setup. One, you’ll need a copy of the script on each of the machines that you want to backup, making maintenance a bit more involved. Second, if your SOURCE machine becomes compromised, the attacker will be able to gain access to your backup server. This second issue can be mitigated by setting the permissions properly on the BACKUP machine. (This is why I use a separate user account for each machine that I backup.)

How to use this script:

1. Edit the configuration options in the script.
2. Add a user called “backup-{$SOURCE} to the BACKUP machine.
3. Set up the appropriate SSH keys for the user to bypass password prompts.
4. Set up a cron job to run the script nightly.
5. If you want Apple-ish “time machine style” backups, set up a script to rotate the backups on the BACKUP machine. I use this rotate_backups.pl script to make backups using cp -al (hardlinks).

#!/usr/bin/perl
# Filename: rsync_backup.pl
# Will back things up from one computer to another over SSH using rsync.
# Use in conjunction with a rotate script on the destination machine for
# time-machine-like incremental backups.
# Note: SSH SSL auth keys must be installed on each end to avoid passwd prompt
# Assumes a user named "backup-$THIS_COMPUTER" on remote machine.
# All output will go to Logfile
#
# (c) 2009 eddie@eddieoneverything.com 

use Data::Dumper;

# This computer's name.  Used to name the backup files.
$THIS_COMPUTER = "yoakam";

# The remote machine's URL or IP address.
$REMOTE_MACHINE = "127.0.0.1";

# The backup directory on the remote machine
$REMOTE_BACKUP_LOCATION="/mnt/backup";

# Log things
$LOGFILE = "/var/log/rsync_backup";

# Directories to backup.
@BACKUP_DIRS=(
   '/home',
   '/root',
   '/etc',
   '/var/log',
   '/var/lib/mysql',
   '/var/spool/cron'
);

# Directories to ignore.
@IGNORE_DIRS=(
   '/home/lost+found',
   '/home/tmp',
   '/home/virtfs'
);

#####################################################
########### END Configuration options ###############
#####################################################

open hLOG, ">>$LOGFILE";
$now = `date`;
print hLOG "-" x 80, "\n";
print hLOG "START BACKUP $THIS_COMPUTER at $now\n" ;
print hLOG "-" x 80, "\n";
# Get a list of dirs to back up
foreach (@BACKUP_DIRS){
   $dir = $_;
   opendir(DIR, $dir) || alert ("cant opendir $dir: $!");
   while ($file = readdir(DIR)){
      if ($file ne '.' && $file ne '..'){
         $my_file = $dir . "/" .  $file;
         if ( -d $my_file){
            if (grep {/$my_file/} @IGNORE_DIRS){
               print hLOG "Ignore Dir: $my_file\n";
            }else{
               print hLOG "Dir: $my_file\n";
               push @BACKUP_LIST, {'dir' => $dir, 'name'=>$file};
            }
         }else{
            print hLOG "file is $my_file -- \n";
            @arr_temp =  ($dir, $my_file);
            push @BACKUP_LIST, {'dir' => $dir, 'name'=>$file};
         }
      }
   }
   closedir DIR;
}

#print Dumper(@BACKUP_LIST);

#Back up each dir in the backup list
foreach $my_entry (@BACKUP_LIST){
   $dir = $my_entry->{'dir'};
   $b = $my_entry->{'name'};

   $cmd = "rsync -avz --delete-after -e ssh \"$dir/$b\" backup-$THIS_COMPUTER\@$REMOTE_MACHINE:\"/$REMOTE_BACKUP_LOCATION/$THIS_COMPUTER$dir\"";
   $now=`date`;
   print hLOG "\n$now\t$cmd ";
   #print "$cmd\n";
   `$cmd >> $LOGFILE 2>&1`;
}

$now = `date`;
print hLOG "-" x 80, "\n";
print hLOG "END BACKUP of $THIS_COMPUTER at $now\n" ;
print hLOG "-" x 80, "\n";
close hLOG;

###################### Subs #########################
sub alert(){
   $a=shift(@_);
   print hLOG "#############################\n";
   print hLOG "#############################\n";
   print hLOG "#############################\n";
   print hLOG "Cannot backup $a $!\n";
   print hLOG "#############################\n";
   print hLOG "#############################\n";
   print hLOG "#############################\n";
}

Related Posts

• Rotating Backup Directories using cp -al (hardlinks) to Save Disk Space
• Managing Server Load when unzipping many large files
• “Argument list too long” from rm command. A Perl Script to Get Around the Problem
• How to Scan a Computer for Open Ports
• The Zen of Taco Bell Programming – Using Unix Tools to Prevent Reinventing the Wheel
• How to Easily Install a Perl Module In Ubuntu Linux
• Linux .forward file not working? Make sure you have the permissions set correctly

Here’s a simple script that will delete all of the files in a given directory, even if the argument list gets really long. I prefer to keep it in my ~/bin directory named “bigrm”.

My bigrm Perl script:

#!/usr/bin/perl

if (!$ARGV[0]){
   print "Usage:  $0 \n";
   print "Will delete all files in specified directory\n";
   exit(1);
}

print "deleting files from $d\n";
$OUTDIR=$ARGV[0];

opendir(DIR, $OUTDIR) || die ("cant opendir $IMAGE_DIR: $!");
print "OPENED $OUTDIR\n";
while ($file = readdir(DIR)){
   if ($file!~/^\./){
      print "unlink $file\n";
      unlink ("$OUTDIR/$file");
   }
}
closedir(DIR)

This is useful for cleaning out directories of temp files, of snapshot directories, etc.

Related Posts

• How to Scan a Computer for Open Ports
• The Zen of Taco Bell Programming – Using Unix Tools to Prevent Reinventing the Wheel
• Rotating Backup Directories using cp -al (hardlinks) to Save Disk Space
• Managing Server Load when unzipping many large files
• Using rsync & ssh to Backup Files Between Linux Machines (Perl)
• How to Easily Install a Perl Module In Ubuntu Linux
• Linux .forward file not working? Make sure you have the permissions set correctly

I had to install a perl module (WWW::Mechanize) and was stuck in a serious dispute with CPAN over dependencies.  Wow, what a mess.

Fortunately I found this post by David Yin about how to use Debian’s totally awesome apt-get command to install the module.

Steps:

1.  Search the apt-get cache to find the package name of the module.

eddie@yoakam# sudo apt-cache search perl WWW::Mechanize
libwww-mechanize-shell-perl – An interactive shell for WWW::Mechanize

2.   Once found, install the module using apt-get.

eddie@yoakam# sudo apt-get install libwww-mechanize-shell-perl

Presto – this simple trick took just 30 seconds to accomplish what I’d just spent half an hour trying to get CPAN to do.

Related Posts

• “Argument list too long” from rm command. A Perl Script to Get Around the Problem
• Managing Server Load when unzipping many large files
• How to Scan a Computer for Open Ports
• The Zen of Taco Bell Programming – Using Unix Tools to Prevent Reinventing the Wheel
• Rotating Backup Directories using cp -al (hardlinks) to Save Disk Space
• Using rsync & ssh to Backup Files Between Linux Machines (Perl)
• Linux .forward file not working? Make sure you have the permissions set correctly