It was a pretty simple solution, and was the result of a minor oversight my part.

Still, it took me about 10 minutes to realize the mistake I’d made.

I just thought I’d post here in case someone else makes the same mistake.

I searched for a while but didn’t find the answer… until it finally dawned on me, “duh, I forgot to include the RewriteEngine on directive!”

So, I had this:

Options -Indexes


RewriteCond %{HTTP_HOST} ^site\.com [NC]
RewriteRule ^(.*)$ http://www.site.com/$1 [R=301,L]

But should have had this:

Options -Indexes


RewriteEngine on
RewriteCond %{HTTP_HOST} ^site\.com [NC]
RewriteRule ^(.*)$ http://www.site.com/$1 [R=301,L]

Forgetting that simple line, “RewriteEngine on”, caused me all kinds of headache. :)

Related Posts

• MySQL Performance Tip: ON DUPLICATE KEY is faster than INSERT IGNORE, but failing with a duplicate key error is a lot faster
• How To Use Global Variables From Another File In A Perl Module or Package
• How to Block a Quote
• Getting PHP’s print_r Function To Return A String
• How To Set a Class Name using Javascript in IE and Other Browsers
• Optimizing the Google Syntax Highlighter Wordpress Plugin Even Further
• Customizing the Calibre Associated Press (AP) Recipe

Each database consists of just three tables, but each of those tables has at least 9 million rows.

Because I need to maintain and update a number of data associations, it was necessary to write a custom script to perform the merge. (I chose to use Perl and DBI.)

At one point in my script, I need to do a bunch of “insert this row from database B into database A if it doesn’t exist in database A” type operations. Doing a separate lookup for each record would cost roughly 9 million queries.

At first, I just went ahead and did the insert, allowing the single insert to fail with a unique key error if the record already existed. This caused a bunch of DBD errors to fill the screen, none of which really concerned me, because it didn’t have any bearing on the result. Still, all of those messages printed to STDERR could bury a more serious error, so I looked into handling it another way.

I found this post that recommends using either INSERT IGNORE or ON DUPLICATE KEY to handle these queries. After some investigation, I found that while ON DUPLICATE KEY is a lot faster than INSERT IGNORE, just allowing the thing to fail is the fastest option.

First, some simplified pseudo-code to give you an idea of what I’m doing:

FOR EACH RECORD IN A.TABLE1
   FOR EACH RECORD IN A.TABLE2
      IF THAT RECORD NOT EXISTS IN B.TABLE2,
      DO AN INSERT INTO B.TABLE2

I wanted to avoid doing a separate SQL query at step 3, as that query would have to run 9 million+ times. Since I have a unique key set on the columns I’m interested in, any duplicate rows will not be inserted, thus maintaining my data integrity.

Here are the speed benchmarks I ran for my particular script with each of the three options I tried. The numbers are in terms of iterations of my particular loop (which does a lot more than the pseudo-code above) so the numbers should only be considered relative to each other.

Option 1: Let the query fail with a Unique Key error.

INSERT INTO A (COL1, COL2) VALUES (val1, val2);

Loop iterations per second: 24.5 (avg)

Option 2: INSERT … IGNORE

INSERT IGNORE INTO A (COL1, COL2) VALUES (val1, val2);

Loop iterations per second: 13.3 (avg)

Option 3: INSERT … ON DUPLICATE KEY

INSERT INTO A (COL1, COL2) VALUES (val1, val2) ON DUPLICATE KEY UPDATE id=id;

Loop iterations per second: 18.2 (avg)

As you can see, the fastest of the three options is letting the query fail, followed by ON DUPLICATE KEY as the second fastest.

I’ll leave it to the MySQL experts to explain why this is the case. I have some guesses, but I can’t say for sure.

But I do know one thing: On my setup, for my purposes, letting those queries fail is the quickest option.

Related Posts

• mod_rewrite in .htaccess to Force www in URL Not Working
• How To Use Global Variables From Another File In A Perl Module or Package
• How to Block a Quote
• Getting PHP’s print_r Function To Return A String
• How To Set a Class Name using Javascript in IE and Other Browsers
• Optimizing the Google Syntax Highlighter Wordpress Plugin Even Further
• Customizing the Calibre Associated Press (AP) Recipe

Using a single config file in a basic Perl script is pretty straightforward – you simply use a require statement to read the external file, and the variables contained within are automatically imported. (Though if you’re using strict, you’ll need to declare them using the “our” keyword.)

A problem arises when you want to use those global variables in a package file, though – because a Package has its own namespace, variables declared in your external config file are not imported into your module, even if you use the “our” keyword.

Fortunately, I was able to find a solution here, courtesy of Jeff Pang.

Here are a couple of examples.

The simple way to share variables between Perl scripts using “require”

File test.include.pl

#!/usr/bin/perl
$var = ".j.a.p.h.";

File test.pl

#!/usr/bin/perl
use strict;
require "test.include.pl";
our $var;
print "VAR is $var\n";

After creating the two files above and running test.pl, the output is:

eddie@widmers:~/wrk/$ ./test.pl
VAR is .j.a.p.h.


The more advanced way of sharing variables with a Perl Package
This method involves declaring the config file as a package, and then explicitly exporting the variables you’re interested in. Your other scripts and modules will then import those variables, making them available in that scope.

In this case, the config variable package will look something like this:

package SiteConfig;
require Exporter;

our @ISA = qw(Exporter);
our @EXPORT=qw($var $vare);

our $var = ".j.a.p.h.";

1;

While the package you’re looking to use the variables in will look something like this:

package TestTest;
use TestConfig;
print "Var is $var\n";

sub new
{
   my $package = shift;
   print "VAR is $var\n";
  return bless({}, $package);
}
1;

Of course, your particular needs will vary, so you’ll need to use your noodle to modify the above as necessary.

Related Posts

• “Argument list too long” from rm command. A Perl Script to Get Around the Problem
• Getting PHP’s print_r Function To Return A String
• Calibre Recipe for Minneapolis Star Tribune (StarTribune.com)
• Rotating Backup Directories using cp -al (hardlinks) to Save Disk Space
• Using rsync & ssh to Backup Files Between Linux Machines (Perl)
• Calculate the Days Since a Given Date (Javascript)
• How to Easily Install a Perl Module In Ubuntu Linux

A blockquote lets your reader know that you are quoting another source, and it pays proper attribution to your quoted source.

HTML supports the <blockquote> tag, which does exactly what one would suspect.

To block a quote in HTML, simply use this tag.

Example:

<blockquote> This is quoted text </blockquote>

The above HTML produces the following result:

This is quoted text.

You can control the styling of your blockquote using CSS. For instance, the blockquotes on this site are styled using this CSS:

blockquote {
  margin: 5px 15px;
  padding: 5px 5px;
  background: #8CB5D8;
  border: 1px solid #3A7CAC;
  clear:both;
  }

You can edit the CSS to fit the styling of your site. Or, if you’d prefer, simply keep the blockquote undefined in your CSS – depending on your stylesheet, the default setting may work fine for you.

Related Posts

• How To Set a Class Name using Javascript in IE and Other Browsers
• mod_rewrite in .htaccess to Force www in URL Not Working
• MySQL Performance Tip: ON DUPLICATE KEY is faster than INSERT IGNORE, but failing with a duplicate key error is a lot faster
• How To Use Global Variables From Another File In A Perl Module or Package
• Getting PHP’s print_r Function To Return A String
• Optimizing the Google Syntax Highlighter Wordpress Plugin Even Further
• Customizing the Calibre Associated Press (AP) Recipe

It’s one of the most useful debugging features I’ve seen in any language.

I like it so much that I’ve actually written print_r mimic functions for other languages.

By default, print_r literally prints a variable to the screen. Sometimes it’s useful to have that info not printed though. For instance, you might want to include the print_r result in an email, or an error log, or in a special debugging frame.

Fortunately, print_r has this functionality built in – the function accepts an optional second argument that specifies whether you want the data printed or returned as a string.

To have print_r return the data as a string, include the second argument “true”:

$message = print_r($myObject,true);

If you’re doing anything programmatic with the output, you may want to consider using var_export instead. This function is largely identical to print_r, except that its output is machine parsable rather than human readable. The usage is identical to print_r.

$message = var_export($myObject,true);

All without using ob_start. Nice!

Related Posts

• PHP Programming: U.S. State list functions / State abbreviation to State name function
• How to Make phpbb3 Forum Links Automatically rel=”nofollow”
• How To Use Global Variables From Another File In A Perl Module or Package
• Optimizing the Google Syntax Highlighter Wordpress Plugin Even Further
• Calibre Recipe for Minneapolis Star Tribune (StarTribune.com)
• “Argument list too long” from rm command. A Perl Script to Get Around the Problem
• Calculate the Days Since a Given Date (Javascript)

As anyone who’s worked with Javascript knows, Microsoft’s Internet Explorer can be a real pain the butt.

Things that work in Chrome, Firefox, and Safari don’t always work in IE – so you always have to come up with workarounds to make IE, well, work.

One of those things that has to be done differently is setting a class name.

For whatever reason, IE does not support using “setAttribute” to set class names. Instead, in IE, you have to use className = newClass.

Here is a little bit of code that should do the trick:

if (navigator.appName == "Microsoft Internet Explorer") {
     document.getElementById("id").className="new_class";
}else{
     document.getElementById("id").setAttribute("class", "new_class");
}

Related Posts

• How to Block a Quote
• How To Reveal Stored Passwords in Firefox, Chrome, Safari, and IE
• Calculate the Days Since a Given Date (Javascript)
• mod_rewrite in .htaccess to Force www in URL Not Working
• Adding a Confirm Dialog To Instapaper’s “Read Later” Bookmarklet
• MySQL Performance Tip: ON DUPLICATE KEY is faster than INSERT IGNORE, but failing with a duplicate key error is a lot faster
• How To Use Global Variables From Another File In A Perl Module or Package

The problem is, the thing isn’t very robust – it causes every single page on your blog to load up about 10 separate javascript files, regardless of whether the plugin is actually doing anything on that page or not.

I set out to optimize the code, but did a quick search first and found the folks at Fire Studios had already beaten me to it.

They have a couple of good tips on their page – they’ve combined the separate brush files into a single javascript file, and they’ve altered the code so that the files are only loaded on single posts and single pages.

That’s a great fix, but it wasn’t ideal for my site. On this site, most of my single posts and pages do not contain any code at all. So for like 95% of my pages, I don’t need to load the Google Syntax highlighter at all.

I expanded upon their customization with a further restriction – in my version, the plugin only loads if the page or post contains certain tags – namely “code”, “programming”, “perl”, “php”, “javascript”, or “python”, the tags I normally use when sharing a piece of code.

My google_syntax_highlighter.php file

function insert_header() {
   $current_path = get_option('siteurl') .'/wp-content/plugins/' . basename(dirname(__FILE__)) .'/';

    if(is_single() && has_tag(array('code','javascript','perl','python','php','programming'))){
   ?>

   <?php
   }
}

function insert_footer(){
   $current_path = get_option('siteurl') .'/wp-content/plugins/' . basename(dirname(__FILE__)) .'/';
    if(is_single() && has_tag(array('code','javascript','perl','python','php','programming'))){
   ?>



<?php
   }
}
add_action('wp_head','insert_header');
add_action('wp_footer','insert_footer');

Of course, you may want to use different tags, depending on your setup.

Related Posts

• Getting PHP’s print_r Function To Return A String
• PHP Programming: U.S. State list functions / State abbreviation to State name function
• mod_rewrite in .htaccess to Force www in URL Not Working
• How to Make phpbb3 Forum Links Automatically rel=”nofollow”
• MySQL Performance Tip: ON DUPLICATE KEY is faster than INSERT IGNORE, but failing with a duplicate key error is a lot faster
• How To Use Global Variables From Another File In A Perl Module or Package
• How to Block a Quote

Calibre comes stocked with a number of “recipes” for popular news sites.  One of those is the Associated Press (AP).  Since the AP creates a good deal of the news in many popular dailies,  I’ve found the AP recipe most useful.

I noticed today, though, that the AP recipe that comes with Calibre is incomplete!  It omits at least three sections, Sports, Business, and Entertainment.  I don’t know if the creator, Kovid Goyal,  just doesn’t like these sections, or if they weren’t available when he wrote the original recipe.

(When I wrote my custom recipe for the Minneapolis Star-Tribune, I omitted the sections that weren’t of interest to me.)

In any event, I’ve customized his recipe script to include the omitted sections.  It was a very easy customization – I only had to add three lines to the original python script.  The only tricky part was picking a local “news source” for the AP’s RSS feed – otherwise, the AP site serves up a random source, which can lead to problems with processing the page.

import re
from calibre.web.feeds.news import BasicNewsRecipe

class AssociatedPress(BasicNewsRecipe):

    title = u'Associated Press'
    description = 'Global news'
    __author__ = 'Kovid Goyal'
    use_embedded_content   = False
    language = 'en'

    max_articles_per_feed = 15
    html2lrf_options = ['--force-page-break-before-tag="chapter"']

    preprocess_regexps = [ (re.compile(i[0], re.IGNORECASE | re.DOTALL), i[1]) for i in
[
        (r'.*?' , lambda match : ''),
        (r'.*?', lambda match : ''),
        (r'.*?', lambda match : ''),
        (r'.*?', lambda match : ''),
        (r'.*?', lambda match : ''),
        (r'


.*?
', lambda match : '
'),
        (r'
', lambda match : '

'),
        (r'Learn more about our Privacy Policy.*?', lambda match : ''),
    ]
    ]   

    feeds = [ ('AP Headlines', 'http://hosted.ap.org/lineups/TOPHEADS-rss_2.0.xml?SITE=ORAST&SECTION=HOME'),
                  ('AP US News', 'http://hosted.ap.org/lineups/USHEADS-rss_2.0.xml?SITE=CAVIC&SECTION=HOME'),
                   ('AP World News', 'http://hosted.ap.org/lineups/WORLDHEADS-rss_2.0.xml?SITE=SCAND&SECTION=HOME'),
                   ('AP Political News', 'http://hosted.ap.org/lineups/POLITICSHEADS-rss_2.0.xml?SITE=ORMED&SECTION=HOME'),
                   ('AP Business News', 'http://hosted.ap.org/lineups/BUSINESSHEADS-rss_2.0.xml?SITE=RANDOM&SECTION=HOME'),
                   ('AP Technology News', 'http://hosted.ap.org/lineups/TECHHEADS-rss_2.0.xml?SITE=CTNHR&SECTION=HOME'),
                   ('AP Sports News', 'http://hosted.ap.org/lineups/SPORTSHEADS-rss_2.0.xml?SITE=CAVIC&SECTION=HOME'),
                   ('AP Business News', 'http://hosted.ap.org/lineups/BUSINESSHEADS-rss_2.0.xml?SITE=CAVIC&SECTION=HOME'),
                   ('AP Entertainment News', 'http://hosted.ap.org/lineups/ENTERTAINMENT-rss_2.0.xml?SITE=CAVIC&SECTION=HOME'),
                   ('AP Science News', 'http://hosted.ap.org/lineups/SCIENCEHEADS-rss_2.0.xml?SITE=OHCIN&SECTION=HOME'),
                   ('AP Strange News', 'http://hosted.ap.org/lineups/STRANGEHEADS-rss_2.0.xml?SITE=WCNC&SECTION=HOME'),
        ]

Related Posts

• Calibre Recipe for Minneapolis Star Tribune (StarTribune.com)
• Getting a List of All Books in Your Calibre Library
• How To Tell If Someone Is Lying About Their iPad Experience
• Adding a Confirm Dialog To Instapaper’s “Read Later” Bookmarklet
• Viewing the Clock on the Kindle 2
• Recently Read Documents Appearing at End of List on Kindle
• Use Custom Background Images as Screen Savers on the Kindle

Calibre comes stocked with “recipes” for downloading news from hundreds of websites. Unfortunately, the Minneapolis Star Tribune is not one of them. So I wrote my own.

Here is my very simple recipe for the Star Tribune. It omits many of the newspaper sections that I am not interested in, so you may want to edit my list of sources using this list of RSS feeds as your guide. It’s written in Python, so spacing is important.

Calibre recipe for the Minneapolis Star Tribune

class AdvancedUserRecipe1266267637(BasicNewsRecipe):
    title          = u'Star Tribune'
    oldest_article = 7
    max_articles_per_feed = 100

    feeds          = [(u'Main', u'http://www.startribune.com/rss/?sf=1&s=/'), (u'Local', u'http://www.startribune.com/local/index.rss2'), (u'Minneapolis', u'http://www.startribune.com/local/minneapolis/index.rss2'), (u'St. Paul', u'http://www.startribune.com/local/stpaul/index.rss2'), (u'North Metro', u'http://www.startribune.com/local/north/index.rss2'), (u'East Metro', u'http://www.startribune.com/local/east/index.rss2'), (u'South Metro', u'http://www.startribune.com/local/south/index.rss2'), (u'West Metro', u'http://www.startribune.com/local/west/index.rss2'), (u'Business', u'http://www.startribune.com/business/index.rss2'), (u'Science & Technology', u'http://www.startribune.com/science/index.rss2'), (u'Sports', u'http://www.startribune.com/sports/index.rss2'), (u'Lifestyle', u'http://www.startribune.com/lifestyle/index.rss2'), (u'Travel', u'http://www.startribune.com/lifestyle/travel/index.rss2'), (u'Books', u'http://www.startribune.com/entertainment/books/index.rss2'), (u'Entertainment', u'http://www.startribune.com/entertainment/index.rss2'), (u'Commentary', u'http://www.startribune.com/opinion/commentary/index.rss2'), (u'Editorials', u'http://www.startribune.com/opinion/editorials/index.rss2')]

    def print_version(self, url):
        parts=url.split('/')
        id = parts[4].split('.')
        if not id[0].isdigit():
            id = parts[5].split('.')

        return url.replace(url, 'http://www.startribune.com/templates/Print_This_Story?sid=' + id[0])

I’ve also contributed this code to the wiki repository for user-created recipes.

There is a small bug – I’ve noticed that some stories in the Sports section use some sort of Redirect to steer users to a blog entry. Because of the way the Star Tribune people are doing this redirect, the recipe doesn’t work on those stories. It’s only a very few stories, though, and I’m fine with it, so I’m not going to spend time tweaking the recipe.

Related Posts

• Customizing the Calibre Associated Press (AP) Recipe
• Getting a List of All Books in Your Calibre Library
• How To Tell If Someone Is Lying About Their iPad Experience
• Adding a Confirm Dialog To Instapaper’s “Read Later” Bookmarklet
• How To Use Global Variables From Another File In A Perl Module or Package
• Getting PHP’s print_r Function To Return A String
• Viewing the Clock on the Kindle 2

This nifty behavior makes cp -al, when combined with rsync, ideal for backup systems. One can use the cp -al command to take a “snapshot” of a given directory tree at a given time, at the expense of very little additional disk space. I use this script in concert with my rsync_backup.pl script to keep 21 days of “snapshot” backups of each of my machines.

#!/usr/bin/perl
use POSIX;

# Rotates backup directories w/ cp -al (hardlinks)
# Deletes directories older than $KEEP_DAYS
# Runs each night ahead of backup process
# (c) 2009 eddie@eddieoneverything.com

$KEEP_DAYS=21;
$LOGFILE = "/var/log/rotate_backups";

@BACKUP_DIRS=(
   '/mnt/backup/hansel',
   '/mnt/backup/tiger',
   '/mnt/backup/june'
);

$ts = get_timestamp();

open hLOG, ">>$LOGFILE";
print hLOG "=" x 80, "\n";
print hLOG "Run START at " . `date` . "\n";
print hLOG "=" x 80, "\n";

## Do the rotation
print hLOG "Do today's rotation\n";
foreach $dir ( @BACKUP_DIRS ){
   print hLOG "\t" , `date`;
   $newfn = $dir ."_" .  $ts;
   $cmd = "cp -al $dir $newfn";
   print hLOG "Execute Command: $cmd\n";
   `$cmd`;
}

## Delete old directories
print hLOG "Delete Old Directories\n";
foreach $dir ( @BACKUP_DIRS ){
   $dir=~m/^(.+)\/(.+?)$/;
   $base= $1;
   $stub=$2;
   #print "dir is $dir\nBASE: $base\nSTUB:$stub\n";

   opendir hDIR, "$base" or die "can't open directory $base";
   @dirlist=grep { /^$stub\_/ && !/^$stub$/  } readdir(hDIR);
   closedir hDIR;

   foreach $d (@dirlist){
      print hLOG "\t" , `date`;
      $d=~/$stub\_([0-9]+)_([0-9]+)_([0-9]+)_.+$/;
      $year= $1;
      $month = $2;
      $day = $3;

      if (dirIsOlder($year, $month, $day)){
         #print "$d\n";
         $remove_dir= $base . '/' . $d;
         $cmd = "rm -Rf \"$remove_dir\"";
         print hLOG "Execute command: $cmd\n";
         `$cmd`;
      }else{
         print hLOG "Keep $d\n";
      }
   }
}
print hLOG "=" x 80, "\n";
print hLOG "Run END at " . `date` . "\n";
print hLOG "=" x 80, "\n";
close hLOG;

# ------------- Subroutines & functions -----------------

sub get_timestamp {
   ($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst) = localtime(time);
   $mon+=1;

   if ($mon < 10) { $mon = "0$mon"; }
   if ($mday < 10) { $mday = "0$mday"; }
   if ($hour < 10) { $hour = "0$hour"; }
   if ($min < 10) { $min = "0$min"; }
   if ($sec < 10) { $sec = "0$sec"; }
   $year=$year+1900;

   return $year . '_' . $mon . '_' . $mday . '__' . $hour . '_' . $min . '_' . $sec;
}

sub dirIsOlder{
   ($fyear, $fmonth, $fday) = @_;
   #print "Check $year - $month - $day \n";

   $now  = mktime(localtime());
   $then =  mktime (0, 0 , 0, $fday-1, $fmonth-1, $fyear-1900 , 0, 0);

   $diff_sec = $now - $then;
   $days_since = $diff_sec / 24 / 60 / 60;
   #print "n: $now. t: $then. Diff ($diff_sec) = $days_since\n";
   #print "$days_since $fyear-$fmonth-$fday\n";

   ## Subtract days from today

   if ($days_since > $KEEP_DAYS){
      return 1;
   }else{
      return 0;
   }
}

Related Posts

• Using rsync & ssh to Backup Files Between Linux Machines (Perl)
• “Argument list too long” from rm command. A Perl Script to Get Around the Problem
• How To Keep An SSH Connection Alive in Linux
• How to Easily Install a Perl Module In Ubuntu Linux
• Linux Question: When does updatedb.mlocat / cron.daily run?
• How To Use Global Variables From Another File In A Perl Module or Package
• mod_rewrite in .htaccess to Force www in URL Not Working