Mission Data Blog
Louisville-based Web Development & Software Engineering

Steven pointed out that someone was looking for a way to stream to S3 using PHP and said I should figure out how to get it going. Since he made a patch to let you stream data with Ruby using S3 I figured I should do one for PHP. There may be better ways of doing this but since I’ve already done it with C using curl I figured that would be the fastest way.
(more…)

I think by now most sysadmin types know about rrdtool and the nice graphs it makes. I recently wanted to create some graphs by hand using PHP so I turned to the php-rrdtool extension. I found that it takes a little work to get it to compile but that could be because I’m not constantly recompiling PHP and just don’t know better. You can get this module as an rpm for fedora (php-rrdtool) but I like to compile php by hand so I couldn’t use it. I’m going to assume that you know how to compile PHP normally with whatever other items you want to include and that you have the rrdtool development libraries installed or have compiled and installed rrdtool from source.

Step 1. Get the PHP rrdtool source

Go to the contrib directory on the rrdtool distribution site:
http://people.ee.ethz.ch/~oetiker/webtools/rrdtool/pub/contrib/

There are a number of files in this directory that mention rrd. You want the one named: php_rrdtool.tgz

(more…)

At a customer site, the test and production linux servers for some intranet applications were slowly running out of disk space. The apps themselves were running fine, it was the logfiles generated by the apps that were the issue; logging that came in useful just in case something went wrong, but quickly aged. In one particular case, a catalina.out file was several years old and was 11 gigabytes; 90% of it wasn’t really relevant any longer.

The solution to the problem of wrangling logfiles is probably already installed on your server: it’s logrotate. Chances are logrotate is already configured in your system crontab as a daily task, and chances are it is configured to obey any configuration files found in the /etc/logrotate.d directory. If you find that directory, you are probably good to go.

(more…)

I recently had a need to adapt a script that recrawls a site with nutch. One of my design goals was to use the same command line options as the Fetchtool (one of the steps I had to take to recrawl a site).

It became apparent fairly quickly that bash’s built-in ‘getopts’ didn’t support long command line arguments, so I had to fall back on getopt.

Here is the portion of the script that parses the command line arguments:

set -- `getopt -n$0 -u -a --longoptions="depth: adddays: topN:" "h" "$@"` || usage
[ $# -eq 0 ] && usage

while [ $# -gt 0 ]
do
    case "$1" in
       --depth)   depth=$2;shift;;
       --adddays) adddays=$2;shift;;
       --topN)    topN=$2;shift;;
       -h)        usage;;
       --)        shift;break;;
       -*)        usage;;
       *)         break;;            #better be the crawl directory
    esac
    shift
done

Deconstructing this bit by bit:

(more…)

For alot of people, CVS is history; they’ve switched to SVN and are loving life. For those of us continuing the use this venerable tool, I present the following cobbled together bit of utility.

It’s a bash function. and a nice way to keep an eye on what’s happening on a cvs repository. I work with a team of around 7 guys, and I like to keep an eye on what’s changed and who changed it.

It works like this, summarizing what I consider to be the highlights from the cvs history command, and defaults to showing checkins for the current date:

$ cvshist
M 2006-02-22 10:34 EDT jsmith  1.3  InternalOperation.java         APRJ7A/src/com/ppc/oepica/model/internal
M 2006-02-22 10:53 EDT jsmith  1.37 FooBarCopier.java              APRJ7A/src/com/ppc/oepica/model/external
M 2006-02-22 11:26 EDT darrend 1.23 CommonJobBase.java             APRJ7A/src/com/ppc/oepica/model/common

I use the date command, which gives me some flexibility when specifying a date

$ cvshist yesterday
$ cvshist 2 days ago
$ cvshist january 10
$ cvshist 2006-01-11

Here’s the source. I’ve used this on linux and cygwin; you’ll need bash, gnu date, and cvs.

$ declare -f
cvshist ()
{
  export CVSROOT=:pserver:darrend@shlnxtest1:/home/cvs;
  TARGET_DATE=$(date -d "$*" +%Y-%m-%d);
  cvs history -a -z EDT -c -D $TARGET_DATE | sed "s/\\s*==.*$//" | grep "$TARGET_DATE"
}

Here is a quick rundown on how to create a cryptfs partition that mounts during boot with Fedora Core 4.

Configure the kernel (this may already be done for you):

Device Drivers --->
  Multiple devices driver support (RAID and LVM) --->
    [*] Multiple devices driver support (RAID and LVM)
      < >   RAID support
      <*>   Device mapper support
      <*>     Crypt target support

Cryptographic options --->
  <M>   MD5 digest algorithm
  <M>   SHA1 digest algorithm
  <M>   AES cipher algorithms
   .... 

Install the userland tools:

yum install cryptsetup

Create the filesystem and format it:

cryptsetup -c blowfish -s 64 create fs_name /dev/sda2
mkfs.ext3 /dev/mapper/fs_name

Create /etc/init.d/crytptinit:

if [ -b /dev/mapper/fs_name ]; then
  /sbin/cryptsetup remove fs_name
fi
/sbin/cryptsetup -c blowfish -s 64  create fs_name /dev/sda2

Run it at the runlevels you want:

cd /etc/rc3.d
ln -s ../init.d/cryptinit S08cryptinit

Create the mount point:

mkdir /mount_point

Edit /etc/fstab:

/dev/mapper/fs_name /mount_point               ext3   defaults 0 0

Reboot. You will be asked for your passphrase when the machine boots.

Information in this post was gleaned from several places. Here is one
that matches closely.