Eddie On Everything » code

Managing Server Load when unzipping many large files

eddie — Fri, 22 Apr 2011 17:32:37 +0000

So, I have about 500 zipped up files that I needed to unzip.

Each of these files is about 100M when decompressed.

I ran into a big problem when unzipping the files all at once – when I did a simple “unzip \*zip", the server load eventually skyrocketed.

The load got so high, in fact, that mySQL and httpd couldn’t even keep up with the incoming web requests, and the mySQL database eventually crashed. Well, it didn’t exactly crash, but the requests got so stacked up that it stopped accepting new connections, so it might as well have been crashed.

I assume that this is because the Linux server (Red Hat, in my case) has to do a bunch of “behind the scenes” type things when allocating large chunks of space for new files. You know, file system record keeping, delayed disk I/O, things like that.

“nice -n19″ wasn’t doing the job, so I tried doing a simple batch file that would sleep 10 seconds before unzipping each file. This worked, for the most part, but the server load eventually got higher than I like to see. Left alone, the whole thing could have cascaded out of control again, especially given that I wasn’t taking into account the *other* things the server may have been doing at the time. And I wanted to go to lunch, so hanging out and watching it wasn’t an attractive option.

I decided to write a simple script that checked the server load via /proc/loadavg, then only performed the unzip operation if the load was below a certain value.

Here’s my script, written in Perl.

#!/usr/bin/perl

$MAX_LOAD = .9;

open hIN, "){
   $filename = $_;
   chomp($filename);

   $cmd = "unzip \"$filename\"";

   print "$cmd\n";
   print `$cmd`;

   # Get the load average
   do {
      $load = `cat /proc/loadavg  | cut -f1 -d " "`;
      chomp($load);

      if ($load > $MAX_LOAD){
         print "Load is $load - Sleeping\n";
         sleep (5);
      }

   }while ($load > $MAX_LOAD);

}
close hIN;

I chose to keep the server load below 0.9, a nice number, IMHO.

Update: I’ve done a little more checking around, and I see that there’s a Unix command that will do a lot of this for you – the “batch” command will only execute a command when the server load drops below a certain threshold. The default is 0.8. Good to know.

The method I outline above is still useful in other instances where “batch” can’t be used. For instance, I’m now using this method of checking the server load in a data loading script. I parse my input files, generate sql & execute it. Then, every couple thousand queries or so I check the server load. If it’s getting too high, the script sleeps until the load falls to an acceptable level.

How to Scan a Computer for Open Ports

eddie — Fri, 08 Apr 2011 21:27:13 +0000

There are a bunch of GUI utilities for conducting port scans, but I prefer to work from a command line shell.

In the past, I’ve used a custom script for scanning a machine for open ports from the command line, but I recently needed to get a little more information from a scan than my meager little script provides.

I did some research and found this useful tidbit on a Ubuntu forums site.

If you have any flavor of Linux, you should be able to use nmap to produce a map of open ports on a remote machine.

The command:

nmap -sS -sV –O

Where the flags mean:

-sS: TCP SYN (Half-open) scan

-sV: Probe open ports to determine service/version info

-O: Enable OS detection

And example of the output:

root@frolicanddetour:/home/eddie/bin# nmap -sS -sV -O localhost

Starting Nmap 5.00 ( http://nmap.org ) at 2011-04-08 16:40 CDT Interesting ports on localhost (127.0.0.1): Not shown: 991 closed ports PORT STATE SERVICE VERSION 22/tcp open ssh OpenSSH 5.3p1 Debian 3ubuntu4 (protocol 2.0) 25/tcp open smtp Postfix smtpd 80/tcp open http Apache httpd 2.2.14 ((Ubuntu)) 139/tcp open netbios-ssn Samba smbd 3.X (workgroup: GINGERALE) 443/tcp open ssl/http Apache httpd 2.2.14 ((Ubuntu)) 445/tcp open netbios-ssn Samba smbd 3.X (workgroup: GINGERALE) 631/tcp open ipp CUPS 1.4 3306/tcp open mysql MySQL 5.1.41-3ubuntu12.6 5901/tcp open tcpwrapped

You will have to be logged in as root to run this scan. I like to set up an alias on my standard user account that employs sudo, like so:

alias portscan="sudo nmap -sS -sV -O $1"

Disabling TextArea Scrolling in Flash AS3

eddie — Tue, 23 Nov 2010 05:48:49 +0000

Had a problem while working on a new Flash App today.

I have a number of dynamic text fields (TextAreas), and they scroll in-place when the user highlights the text and when the user uses the mouse wheel while hovering over them.

I searched for a way to stop this scrolling for quite a while and found a few solutions that weren’t quite up to par. For instance, one solution had me disabling the highlight-ability of the text and also disabling the scroll wheel on the text. This wouldn’t be ideal for my app, as I want users to be able to highlight text. I just don’t want it to scroll!

Code that isn’t ideal:

myText.mouseEnabled = false;

myText.mouseWheelEnabled = false;

Also, this code didn’t exactly “work” as expected, either – I still ran in to occasional scrolling issues.

I discovered that the problem is due to Action Script 3’s (AS3) TextFieldAutoSize property. When a text area is set to “auto size,” it’s just a *little bit* small by default, and the Flash player will let a user scroll that text if he tries hard enough.

I found a thread on kirupa that pointed me in the right direction. Though it didn’t work out of the box, I was able to create my own AS3 function that effectively disables scrolling on text areas in AS3.

Here is the function I wrote, along with an example of how you can use it:

function autoSizeTF(tf:TextField){
     var tempHeight:Number;
     var padding:Number = 2;
     tf.autoSize = TextFieldAutoSize.LEFT;
     tempHeight = tf.height;

     tf.autoSize = TextFieldAutoSize.NONE;
     tf.height = tempHeight + padding;
}

autoSizeTF(ca_mc.question_txt);

What this function does is first convert the field to an Auto Size field, get the height, then set it back to a non-Auto Sized text area with the proper height, plus some padding.

This function has disabled mouse wheel scrolling on the text fields, and it has disabled highlighted text scrolling as well.

It took me about 2 hours to find/come up with this solution. I hope it helps you.

The Zen of Taco Bell Programming – Using Unix Tools to Prevent Reinventing the Wheel

eddie — Fri, 22 Oct 2010 17:05:29 +0000

Saw this great article by Ted Dziuba that parallels my programming philosophy.

( In case you’re interested, here’s a reddit thread on the article. )

In short, the article states that much of what you want to accomplish with any project is possible with the standard, tried-and-true Unix tools. When you shun these tools in favor of the latest-and-greatest buzzword tool or library, you’re not only introducing a lot of unnecessary complexity at the get-go, but you’re taking on the responsibility of maintaining that code. You’re also introducing a lot of uncertainty by shunning a near-bulletproof method for a new-kid-on-the-block library that hasn’t had the benefit of being around the block a few billion times over the last 40 years.

I can’t tell you how many times others have looked at my code and called it “messy” because I tend to make use of standard Unix utilities rather than the newfangled libraries. Hey, I tell them, I just wanted to write a quick script that accomplishes a simple task in as little time as possible. I want it to be reliable, and I want it to “just work.” If I had done it your way, it would have taken me 6 hours instead of just 20 minutes. And it probably would have broken in a year or two. Unix’s find, xargs, cat, sort, wget, grep, awk, et al, all work splendidly. Why reinvent the wheel?

Dziuba provides an excellent example:

Here’s a concrete example: suppose you have millions of web pages that you want to download and save to disk for later processing. How do you do it? The cool-kids answer is to write a distributed crawler in Clojure and run it on EC2, handing out jobs with a message queue like SQS or ZeroMQ.

The Taco Bell answer? xargs and wget. In the rare case that you saturate the network connection, add some split and rsync. A “distributed crawler” is really only like 10 lines of shell script.

Moving on, once you have these millions of pages (or even tens of millions), how do you process them? Surely, Hadoop MapReduce is necessary, after all, that’s what Google uses to parse the web, right?

Pfft, fuck that noise:

find crawl_dir/ -type f -print0 | xargs -n1 -0 -P32 ./process

32 concurrent parallel parsing processes and zero bullshit to manage. Requirement satisfied.

Simple, yet brilliant.

Making Yahoo! Sports Fantasy Pro Football Pickem Sortable

eddie — Fri, 15 Oct 2010 00:53:21 +0000

I play Yahoo! Fantasy Sports Pro Football Pickem.

In it, you pick the winner of each NFL game every week.

I play a version that requires you to not only pick the winner, but to assign “confidence points” to each pick as well – that is, you assign a score to each pick indicating how confident you are about that pick.

The Yahoo! interface kinda sucks, though. In order to rank your teams, you have to use drop-down menus instead of a snazzy drag-and-drop interface. It’s kinda annoying.

I was inspired to create a little bit of Javascript code that allows you to arrange your picks using drag-and-drop.

To use my code as a bookmarket, simple drag this link to to your bookmarks bar. (Or if you’re using IE, right click on the link and choose “Add to favorites.” You may get a message that says “this link might not be safe.” Ignore that. It’s perfectly safe – IE is only popping up the message because the link you’re adding contains some javascript.)

Drag or Save this Link to Favorites: y! ff sort

Now navigate to your weekly picks page at Yahoo! and click the bookmark. You should get a dialog confirming that you may now make your picks by dragging and dropping them. Magic! (Be sure to hit “save” when you’re finished.)

mod_rewrite in .htaccess to Force www in URL Not Working

eddie — Mon, 14 Jun 2010 13:44:49 +0000

Setting up a new site today, I ran into a problem with a .htaccess not working.

It was a pretty simple solution, and was the result of a minor oversight my part.

Still, it took me about 10 minutes to realize the mistake I’d made.

I just thought I’d post here in case someone else makes the same mistake.

I searched for a while but didn’t find the answer… until it finally dawned on me, “duh, I forgot to include the RewriteEngine on directive!”

So, I had this:

Options -Indexes


RewriteCond %{HTTP_HOST} ^site\.com [NC]
RewriteRule ^(.*)$ http://www.site.com/$1 [R=301,L]

But should have had this:

Options -Indexes


RewriteEngine on
RewriteCond %{HTTP_HOST} ^site\.com [NC]
RewriteRule ^(.*)$ http://www.site.com/$1 [R=301,L]

Forgetting that simple line, “RewriteEngine on”, caused me all kinds of headache. :)

MySQL Performance Tip: ON DUPLICATE KEY is faster than INSERT IGNORE, but failing with a duplicate key error is a lot faster

eddie — Tue, 09 Mar 2010 15:02:52 +0000

I’m working on merging a couple of pretty big MySQL databases.

Each database consists of just three tables, but each of those tables has at least 9 million rows.

Because I need to maintain and update a number of data associations, it was necessary to write a custom script to perform the merge. (I chose to use Perl and DBI.)

At one point in my script, I need to do a bunch of “insert this row from database B into database A if it doesn’t exist in database A” type operations. Doing a separate lookup for each record would cost roughly 9 million queries.

At first, I just went ahead and did the insert, allowing the single insert to fail with a unique key error if the record already existed. This caused a bunch of DBD errors to fill the screen, none of which really concerned me, because it didn’t have any bearing on the result. Still, all of those messages printed to STDERR could bury a more serious error, so I looked into handling it another way.

I found this post that recommends using either INSERT IGNORE or ON DUPLICATE KEY to handle these queries. After some investigation, I found that while ON DUPLICATE KEY is a lot faster than INSERT IGNORE, just allowing the thing to fail is the fastest option.

First, some simplified pseudo-code to give you an idea of what I’m doing:

FOR EACH RECORD IN A.TABLE1
   FOR EACH RECORD IN A.TABLE2
      IF THAT RECORD NOT EXISTS IN B.TABLE2,
      DO AN INSERT INTO B.TABLE2

I wanted to avoid doing a separate SQL query at step 3, as that query would have to run 9 million+ times. Since I have a unique key set on the columns I’m interested in, any duplicate rows will not be inserted, thus maintaining my data integrity.

Here are the speed benchmarks I ran for my particular script with each of the three options I tried. The numbers are in terms of iterations of my particular loop (which does a lot more than the pseudo-code above) so the numbers should only be considered relative to each other.

Option 1: Let the query fail with a Unique Key error.

INSERT INTO A (COL1, COL2) VALUES (val1, val2);

Loop iterations per second: 24.5 (avg)

Option 2: INSERT … IGNORE

INSERT IGNORE INTO A (COL1, COL2) VALUES (val1, val2);

Loop iterations per second: 13.3 (avg)

Option 3: INSERT … ON DUPLICATE KEY

INSERT INTO A (COL1, COL2) VALUES (val1, val2) ON DUPLICATE KEY UPDATE id=id;

Loop iterations per second: 18.2 (avg)

As you can see, the fastest of the three options is letting the query fail, followed by ON DUPLICATE KEY as the second fastest.

I’ll leave it to the MySQL experts to explain why this is the case. I have some guesses, but I can’t say for sure.

But I do know one thing: On my setup, for my purposes, letting those queries fail is the quickest option.

How To Use Global Variables From Another File In A Perl Module or Package

eddie — Sun, 07 Mar 2010 22:42:33 +0000

I’ve been doing a lot of Perl development for my latest project, and I wanted to use a single config file for many scripts.

Using a single config file in a basic Perl script is pretty straightforward – you simply use a require statement to read the external file, and the variables contained within are automatically imported. (Though if you’re using strict, you’ll need to declare them using the “our” keyword.)

A problem arises when you want to use those global variables in a package file, though – because a Package has its own namespace, variables declared in your external config file are not imported into your module, even if you use the “our” keyword.

Fortunately, I was able to find a solution here, courtesy of Jeff Pang.

Here are a couple of examples.

The simple way to share variables between Perl scripts using “require”

File test.include.pl

#!/usr/bin/perl
$var = ".j.a.p.h.";

File test.pl

#!/usr/bin/perl
use strict;
require "test.include.pl";
our $var;
print "VAR is $var\n";

After creating the two files above and running test.pl, the output is:
eddie@widmers:~/wrk/$ ./test.pl VAR is .j.a.p.h.

The more advanced way of sharing variables with a Perl Package
This method involves declaring the config file as a package, and then explicitly exporting the variables you’re interested in. Your other scripts and modules will then import those variables, making them available in that scope.

In this case, the config variable package will look something like this:

package SiteConfig;
require Exporter;

our @ISA = qw(Exporter);
our @EXPORT=qw($var $vare);

our $var = ".j.a.p.h.";

1;

While the package you’re looking to use the variables in will look something like this:

package TestTest;
use TestConfig;
print "Var is $var\n";

sub new
{
   my $package = shift;
   print "VAR is $var\n";
  return bless({}, $package);
}
1;

Of course, your particular needs will vary, so you’ll need to use your noodle to modify the above as necessary.

How to Block a Quote

eddie — Tue, 02 Mar 2010 17:44:36 +0000

Blockquotes are a great way to reference others’ material.

A blockquote lets your reader know that you are quoting another source, and it pays proper attribution to your quoted source.

HTML supports the

tag, which does exactly what one would suspect.

To block a quote in HTML, simply use this tag.

Example:
 This is quoted text 
The above HTML produces the following result:

This is quoted text.

You can control the styling of your blockquote using CSS. For instance, the blockquotes on this site are styled using this CSS:
blockquote {
  margin: 5px 15px;
  padding: 5px 5px;
  background: #8CB5D8;
  border: 1px solid #3A7CAC;
  clear:both;
  }
You can edit the CSS to fit the styling of your site. Or, if you’d prefer, simply keep the blockquote undefined in your CSS – depending on your stylesheet, the default setting may work fine for you.

Related Posts
How To Set a Class Name using Javascript in IE and Other Browsers
How to Set the Window Size and Position of a Chrome Application Shortcut
Managing Server Load when unzipping many large files
How to Scan a Computer for Open Ports
Disabling TextArea Scrolling in Flash AS3
The Zen of Taco Bell Programming – Using Unix Tools to Prevent Reinventing the Wheel
Making Yahoo! Sports Fantasy Pro Football Pickem Sortable

Getting PHP’s print_r Function To Return A String

eddie — Thu, 25 Feb 2010 22:41:42 +0000

PHP’s print_r function is invaluable. It prints a human-readable string representation of a variable.

It’s one of the most useful debugging features I’ve seen in any language.

I like it so much that I’ve actually written print_r mimic functions for other languages.

By default, print_r literally prints a variable to the screen. Sometimes it’s useful to have that info not printed though. For instance, you might want to include the print_r result in an email, or an error log, or in a special debugging frame.

Fortunately, print_r has this functionality built in – the function accepts an optional second argument that specifies whether you want the data printed or returned as a string.

To have print_r return the data as a string, include the second argument “true”:

$message = print_r($myObject,true);

If you’re doing anything programmatic with the output, you may want to consider using var_export instead. This function is largely identical to print_r, except that its output is machine parsable rather than human readable. The usage is identical to print_r.

$message = var_export($myObject,true);

All without using ob_start. Nice!