No way, said my system.

Instead of unzipping the files, I got a bunch of warnings that read: “caution: filename not matched,” followed by a filename.  At the end of the listing, unzip told me “1 archive had fatal errors.”  (And that’s the clue… 1 archive.)

To understand what’s happening here, you have to understand what happens when you use a wildcard in a Unix shell.  When you type “unzip *zip,” your shell (bash, bourne, korn, whatever) doesn’t actually pass that “*” to the program you’re calling – instead, it performs the wildcard match against the specified files, then passes the list of files to the program. 

So when you type “unzip *” in a directory containing files A, B, C, and D, your shell actually calls the unzip program with 4 arguments – A, B, C, and D.  A quick check of the unzip man page shows that unzip accepts only one archive name, and treats subsequent file names as files you want to extract from that archive.  In other words, when you type “unzip *”, you’re really saying “unzip A B C D”, which unzip translates into “Oh, you want me to pull files B, C, and D out of archive A.”

To work around this problem, you need only escape the wildcard operator to prevent the shell from translating it. 

So instead of “unzip *zip” type “unzip \*zip”.

Related Posts

• Linux .forward file not working? Make sure you have the permissions set correctly
• The Zen of Taco Bell Programming – Using Unix Tools to Prevent Reinventing the Wheel
• How To Keep An SSH Connection Alive in Linux
• Managing Server Load when unzipping many large files
• How to Scan a Computer for Open Ports
• Rotating Backup Directories using cp -al (hardlinks) to Save Disk Space
• “Argument list too long” from rm command. A Perl Script to Get Around the Problem

You can easily change this setting on your system by editing the sshd configuration file.

Of course, before doing so, you should consider the security implications of keeping SSH sessions active. Setting your server to keep connections alive may present a security risk, as users may inadvertently leave connections active & walk away from their computer. Worse, they may purposely leave them active, not realizing the security risk.

Since I do all of my computing from steel-reinforced steel bunker buried deep in the ground and secured by armed guards 24×7, I’m not worried about the security implications of leaving sessions active.

To set your Ubuntu, Red Hat, or CentOS server to keep ssh connections alive, you need to edit the /etc/ssh/sshd_config file and edit the TCPKeepAlive and ClientAliveInterval lines. Uncomment these lines if commented out, and make the following changes:


vi /etc/ssh/sshd_config

TCPKeepAlive yes
ClientAliveInterval 300


After doing so, restart the sshd daemon.

/etc/init.d/sshd restart


Related Posts

• The Zen of Taco Bell Programming – Using Unix Tools to Prevent Reinventing the Wheel
• Linux .forward file not working? Make sure you have the permissions set correctly
• Caution: filename not matched error when unzipping multiple files workaround
• Managing Server Load when unzipping many large files
• How to Scan a Computer for Open Ports
• Rotating Backup Directories using cp -al (hardlinks) to Save Disk Space
• “Argument list too long” from rm command. A Perl Script to Get Around the Problem

This nifty behavior makes cp -al, when combined with rsync, ideal for backup systems. One can use the cp -al command to take a “snapshot” of a given directory tree at a given time, at the expense of very little additional disk space. I use this script in concert with my rsync_backup.pl script to keep 21 days of “snapshot” backups of each of my machines.

#!/usr/bin/perl
use POSIX;

# Rotates backup directories w/ cp -al (hardlinks)
# Deletes directories older than $KEEP_DAYS
# Runs each night ahead of backup process
# (c) 2009 eddie@eddieoneverything.com

$KEEP_DAYS=21;
$LOGFILE = "/var/log/rotate_backups";

@BACKUP_DIRS=(
   '/mnt/backup/hansel',
   '/mnt/backup/tiger',
   '/mnt/backup/june'
);

$ts = get_timestamp();

open hLOG, ">>$LOGFILE";
print hLOG "=" x 80, "\n";
print hLOG "Run START at " . `date` . "\n";
print hLOG "=" x 80, "\n";

## Do the rotation
print hLOG "Do today's rotation\n";
foreach $dir ( @BACKUP_DIRS ){
   print hLOG "\t" , `date`;
   $newfn = $dir ."_" .  $ts;
   $cmd = "cp -al $dir $newfn";
   print hLOG "Execute Command: $cmd\n";
   `$cmd`;
}

## Delete old directories
print hLOG "Delete Old Directories\n";
foreach $dir ( @BACKUP_DIRS ){
   $dir=~m/^(.+)\/(.+?)$/;
   $base= $1;
   $stub=$2;
   #print "dir is $dir\nBASE: $base\nSTUB:$stub\n";

   opendir hDIR, "$base" or die "can't open directory $base";
   @dirlist=grep { /^$stub\_/ && !/^$stub$/  } readdir(hDIR);
   closedir hDIR;

   foreach $d (@dirlist){
      print hLOG "\t" , `date`;
      $d=~/$stub\_([0-9]+)_([0-9]+)_([0-9]+)_.+$/;
      $year= $1;
      $month = $2;
      $day = $3;

      if (dirIsOlder($year, $month, $day)){
         #print "$d\n";
         $remove_dir= $base . '/' . $d;
         $cmd = "rm -Rf \"$remove_dir\"";
         print hLOG "Execute command: $cmd\n";
         `$cmd`;
      }else{
         print hLOG "Keep $d\n";
      }
   }
}
print hLOG "=" x 80, "\n";
print hLOG "Run END at " . `date` . "\n";
print hLOG "=" x 80, "\n";
close hLOG;

# ------------- Subroutines & functions -----------------

sub get_timestamp {
   ($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst) = localtime(time);
   $mon+=1;

   if ($mon < 10) { $mon = "0$mon"; }
   if ($mday < 10) { $mday = "0$mday"; }
   if ($hour < 10) { $hour = "0$hour"; }
   if ($min < 10) { $min = "0$min"; }
   if ($sec < 10) { $sec = "0$sec"; }
   $year=$year+1900;

   return $year . '_' . $mon . '_' . $mday . '__' . $hour . '_' . $min . '_' . $sec;
}

sub dirIsOlder{
   ($fyear, $fmonth, $fday) = @_;
   #print "Check $year - $month - $day \n";

   $now  = mktime(localtime());
   $then =  mktime (0, 0 , 0, $fday-1, $fmonth-1, $fyear-1900 , 0, 0);

   $diff_sec = $now - $then;
   $days_since = $diff_sec / 24 / 60 / 60;
   #print "n: $now. t: $then. Diff ($diff_sec) = $days_since\n";
   #print "$days_since $fyear-$fmonth-$fday\n";

   ## Subtract days from today

   if ($days_since > $KEEP_DAYS){
      return 1;
   }else{
      return 0;
   }
}

Related Posts

• Using rsync & ssh to Backup Files Between Linux Machines (Perl)
• “Argument list too long” from rm command. A Perl Script to Get Around the Problem
• Managing Server Load when unzipping many large files
• How to Scan a Computer for Open Ports
• The Zen of Taco Bell Programming – Using Unix Tools to Prevent Reinventing the Wheel
• Linux .forward file not working? Make sure you have the permissions set correctly
• Caution: filename not matched error when unzipping multiple files workaround

This script runs on the source machine (SOURCE) and connects to the backup machine (BACKUP). (SOURCE->BACKUP). It assumes a user named backup-SOURCE on the backup machine and connects as that user.

One advantage of using SOURCE->BACKUP setup rather than a BACKUP->SOURCE method is that if your BACKUP server is compromised, the attacker will not be able to gain access to all of your production machines.

There are admittedly disadvantages to running a SOURCE->BACKUP setup, rather than a BACKUP->SOURCE setup. One, you’ll need a copy of the script on each of the machines that you want to backup, making maintenance a bit more involved. Second, if your SOURCE machine becomes compromised, the attacker will be able to gain access to your backup server. This second issue can be mitigated by setting the permissions properly on the BACKUP machine. (This is why I use a separate user account for each machine that I backup.)

How to use this script:

1. Edit the configuration options in the script.
2. Add a user called “backup-{$SOURCE} to the BACKUP machine.
3. Set up the appropriate SSH keys for the user to bypass password prompts.
4. Set up a cron job to run the script nightly.
5. If you want Apple-ish “time machine style” backups, set up a script to rotate the backups on the BACKUP machine. I use this rotate_backups.pl script to make backups using cp -al (hardlinks).

#!/usr/bin/perl
# Filename: rsync_backup.pl
# Will back things up from one computer to another over SSH using rsync.
# Use in conjunction with a rotate script on the destination machine for
# time-machine-like incremental backups.
# Note: SSH SSL auth keys must be installed on each end to avoid passwd prompt
# Assumes a user named "backup-$THIS_COMPUTER" on remote machine.
# All output will go to Logfile
#
# (c) 2009 eddie@eddieoneverything.com 

use Data::Dumper;

# This computer's name.  Used to name the backup files.
$THIS_COMPUTER = "yoakam";

# The remote machine's URL or IP address.
$REMOTE_MACHINE = "127.0.0.1";

# The backup directory on the remote machine
$REMOTE_BACKUP_LOCATION="/mnt/backup";

# Log things
$LOGFILE = "/var/log/rsync_backup";

# Directories to backup.
@BACKUP_DIRS=(
   '/home',
   '/root',
   '/etc',
   '/var/log',
   '/var/lib/mysql',
   '/var/spool/cron'
);

# Directories to ignore.
@IGNORE_DIRS=(
   '/home/lost+found',
   '/home/tmp',
   '/home/virtfs'
);

#####################################################
########### END Configuration options ###############
#####################################################

open hLOG, ">>$LOGFILE";
$now = `date`;
print hLOG "-" x 80, "\n";
print hLOG "START BACKUP $THIS_COMPUTER at $now\n" ;
print hLOG "-" x 80, "\n";
# Get a list of dirs to back up
foreach (@BACKUP_DIRS){
   $dir = $_;
   opendir(DIR, $dir) || alert ("cant opendir $dir: $!");
   while ($file = readdir(DIR)){
      if ($file ne '.' && $file ne '..'){
         $my_file = $dir . "/" .  $file;
         if ( -d $my_file){
            if (grep {/$my_file/} @IGNORE_DIRS){
               print hLOG "Ignore Dir: $my_file\n";
            }else{
               print hLOG "Dir: $my_file\n";
               push @BACKUP_LIST, {'dir' => $dir, 'name'=>$file};
            }
         }else{
            print hLOG "file is $my_file -- \n";
            @arr_temp =  ($dir, $my_file);
            push @BACKUP_LIST, {'dir' => $dir, 'name'=>$file};
         }
      }
   }
   closedir DIR;
}

#print Dumper(@BACKUP_LIST);

#Back up each dir in the backup list
foreach $my_entry (@BACKUP_LIST){
   $dir = $my_entry->{'dir'};
   $b = $my_entry->{'name'};

   $cmd = "rsync -avz --delete-after -e ssh \"$dir/$b\" backup-$THIS_COMPUTER\@$REMOTE_MACHINE:\"/$REMOTE_BACKUP_LOCATION/$THIS_COMPUTER$dir\"";
   $now=`date`;
   print hLOG "\n$now\t$cmd ";
   #print "$cmd\n";
   `$cmd >> $LOGFILE 2>&1`;
}

$now = `date`;
print hLOG "-" x 80, "\n";
print hLOG "END BACKUP of $THIS_COMPUTER at $now\n" ;
print hLOG "-" x 80, "\n";
close hLOG;

###################### Subs #########################
sub alert(){
   $a=shift(@_);
   print hLOG "#############################\n";
   print hLOG "#############################\n";
   print hLOG "#############################\n";
   print hLOG "Cannot backup $a $!\n";
   print hLOG "#############################\n";
   print hLOG "#############################\n";
   print hLOG "#############################\n";
}

Related Posts

• Rotating Backup Directories using cp -al (hardlinks) to Save Disk Space
• Managing Server Load when unzipping many large files
• “Argument list too long” from rm command. A Perl Script to Get Around the Problem
• How to Scan a Computer for Open Ports
• The Zen of Taco Bell Programming – Using Unix Tools to Prevent Reinventing the Wheel
• How to Easily Install a Perl Module In Ubuntu Linux
• Linux .forward file not working? Make sure you have the permissions set correctly

Here’s a simple script that will delete all of the files in a given directory, even if the argument list gets really long. I prefer to keep it in my ~/bin directory named “bigrm”.

My bigrm Perl script:

#!/usr/bin/perl

if (!$ARGV[0]){
   print "Usage:  $0 \n";
   print "Will delete all files in specified directory\n";
   exit(1);
}

print "deleting files from $d\n";
$OUTDIR=$ARGV[0];

opendir(DIR, $OUTDIR) || die ("cant opendir $IMAGE_DIR: $!");
print "OPENED $OUTDIR\n";
while ($file = readdir(DIR)){
   if ($file!~/^\./){
      print "unlink $file\n";
      unlink ("$OUTDIR/$file");
   }
}
closedir(DIR)

This is useful for cleaning out directories of temp files, of snapshot directories, etc.

Related Posts

• How to Scan a Computer for Open Ports
• The Zen of Taco Bell Programming – Using Unix Tools to Prevent Reinventing the Wheel
• Rotating Backup Directories using cp -al (hardlinks) to Save Disk Space
• Managing Server Load when unzipping many large files
• Using rsync & ssh to Backup Files Between Linux Machines (Perl)
• How to Easily Install a Perl Module In Ubuntu Linux
• Linux .forward file not working? Make sure you have the permissions set correctly

I haven’t noticed a performance decrease, as the job is set to run at a fairly low priority. But since I sit near the server, it’s annoying to listen to the hard drive churn for hours and hours. All I wanted to do was re-schedule the job to run when I’m not around. I figured that the job was being kicked off by root’s cron file, but I figured wrong.

After some investigation, I learned the following:

• The job parameters are listed in /etc/cron.daily/mlocate.

#! /bin/sh

set -e

[ -x /usr/bin/updatedb.mlocate ] || exit 0

# See ionice(1)
if [ -x /usr/bin/ionice ]; then
IONICE=”/usr/bin/ionice -c3″
fi

$IONICE /usr/bin/updatedb.mlocate

• The files in /etc/cron.daily are kicked off by /etc/crontab, not by root’s crontab. For some reason, Ubuntu’s default install runs this process around 6am. Changing this file allowed me to make the job run at 3am.

# /etc/crontab: system-wide crontab
# Unlike any other crontab you don’t have to run the `crontab’
# command to install the new version when you edit this file
# and files in /etc/cron.d. These files also have username fields,
# that none of the other crontabs do.

SHELL=/bin/sh
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin

# m h dom mon dow user command
17 * * * * root cd / && run-parts –report /etc/cron.hourly
25 3 * * * root test -x /usr/sbin/anacron || ( cd / && run-parts –report /etc/cron.daily )
47 3 * * 7 root test -x /usr/sbin/anacron || ( cd / && run-parts –report /etc/cron.weekly )
52 3 1 * * root test -x /usr/sbin/anacron || ( cd / && run-parts –report /etc/cron.monthly )
#

• updatedb can be configured to ignore certain paths – since my machine has a number of drives used for backups, I wanted to exclude those from the updatedb job. I accomplished this by editing /etc/updatedb.conf and adding to the PRUNEPATHS listing. Adding paths to the PRUNEPATHS option instructs updatedb to exclude those paths from indexing.

PRUNE_BIND_MOUNTS=”yes”
PRUNEPATHS=”/tmp /var/spool /media /mnt/backup”
PRUNEFS=”NFS nfs nfs4 afs binfmt_misc proc smbfs autofs iso9660 ncpfs coda devpts ftpfs devfs mfs shfs sysfs cifs lustre_lite tmpfs usbfs udf rpc_pipefs”

Related Posts

• Linux .forward file not working? Make sure you have the permissions set correctly
• Caution: filename not matched error when unzipping multiple files workaround
• How to Scan a Computer for Open Ports
• The Zen of Taco Bell Programming – Using Unix Tools to Prevent Reinventing the Wheel
• How To Keep An SSH Connection Alive in Linux
• Rotating Backup Directories using cp -al (hardlinks) to Save Disk Space
• “Argument list too long” from rm command. A Perl Script to Get Around the Problem

Solve the problem using the options:

:set paste

:set nopaste

Random Posts

• Gene Simmons Tape hits the Internet, causes a torrent of uproar
• MySQL Performance Tip: ON DUPLICATE KEY is faster than INSERT IGNORE, but failing with a duplicate key error is a lot faster
• RockMelt takes privacy seriously, but you can’t use it unless you share all of your personal data from Facebook
• Free up Disk Space on your Mac by Removing unnecessary OSX language files
• List of the World’s 100 Most Populated Metro Areas
• No, Dell, I don’t want a warranty extension. But you already knew that, didn’t you?
• Calculate the Days Since a Given Date (Javascript)

Over the years, I’ve written a number of “search and replace in multiple files” scripts. Working on a computer that didn’t have my standard, custom toolset available, I found myself searching for an alternative.

Found this tip over at Dzone Snippets – do it all with a single line of perl.

perl -pi -w -e ’s/search/replace/g;’ *.txt

Of course, you’ll need Perl installed on your system and command line access to make use of it.

Random Posts

• How to Set the Window Size and Position of a Chrome Application Shortcut
• Stop RedPlum Coupons from Clogging Up Your Mailbox
• What Is The # (Pound) Sign in Front of People’s Names on Facebook
• Tip on Renting Office Space – The Quoted Rate May Be Per-Year
• I.Q. Score Range By Occupation
• How to Properly Thaw or Defrost Chicken in a Few Hours
• How To Reveal Stored Passwords in Firefox, Chrome, Safari, and IE

Got this error when running updatedb from a Red Had Linux command prompt:
fatal error: updatedb: create_db(): chdir: No such file or directory

The problem:
[root@iberia george]# fatal error: updatedb: create_db(): chdir: No such file or directory

As you can see, I’m running as root, so this shouldn’t be a permissions problem. Probably a file lock or corrupt database problem.

The cause: Many people get this error when the slocate database becomes corrupt.

The Fix: First, if you’re running two instances of updatedb at once, quit doing that. :) Second, try removing and re-creating the database, as someone advises in a Gentoo linux forum:

Remove contents of /var/lib/slocate/ using
[root@iberia george]# rm /var/lib/slocate/*

Then re-run updatedb
[root@iberia george]# updatedb -u

This should fix the problem.

Random Posts

• Asco Super Stainless Razor Blade Review
• Judge Kent order denying Motion to Transfer in case Smith v. Colonial Penn
• Getting Windows Live Writer to Show Your Template Properly
• Opera changes third-party iframe cookie handing in latest release
• How to stop VI / VIM from auto-indenting and auto-formatting
• Historical List of Kentucky Derby Winners
• Blimpie Sub Calorie Calculator

Getting this Linux error message?

Resource temporarily unavailable

It’s because your system is limiting the number of files / processes / CPU that your particular user ID can use.

If you use WHM/Cpanel, enabling the “Shell fork bomb protection” can make the “Resource temporarily unavailable” message appear more often.

Don’t disable the shell fork bomb protection – it’s a good thing. After all, do you want any jackass user to be able to write a perl script containing the line “while (fork());” and bring your system to its knees?

Instead, consider raising the limits for individual trusted users using ulimit. (You will need root access to do this.)

Make the following changes in the system shell init files.

(these files)
/etc/profile
/etc/bashrc
/etc/profile.d/limits.sh


if [ "$LIMITUSER" = "root" ]; then
ulimit -n 4096 -u 14335 -m unlimited -d unlimited -s 8192 -c 1000000 -v unlimited 2>/dev/null
elif [ "$LIMITUSER" = "YOUR USER NAME" ]; then
ulimit -n 4096 -u 14335 -m unlimited -d unlimited -s 8192 -c 1000000 -v unlimited 2>/dev/null
else
ulimit -n 100 -u 20 -m 200000 -d 200000 -s 8192 -c 200000 -v 200000 2>/dev/null
fi


Related Posts

• Cpanel problem parking domain using a remote nameserver
• Linux .forward file not working? Make sure you have the permissions set correctly
• Managing Server Load when unzipping many large files
• Caution: filename not matched error when unzipping multiple files workaround
• How to Scan a Computer for Open Ports
• The Zen of Taco Bell Programming – Using Unix Tools to Prevent Reinventing the Wheel
• How To Keep An SSH Connection Alive in Linux