while true; do kill -USR1 `pidof dd`; sleep 2; done

(NOTE: if `pidof dd` doesn’t work for you, just use the process id directly)

Switch back to the terminal where dd is running and you should see:

9902751744 bytes (9.9 GB) copied, 732.883 s, 13.5 MB/s
9469+0 records in
9468+0 records out
9927917568 bytes (9.9 GB) copied, 734.914 s, 13.5 MB/s
9496+0 records in
9495+0 records out
9956229120 bytes (10 GB) copied, 736.941 s, 13.5 MB/s

updating every couple of seconds.

First you will want to get the S3 PHP library from Amazon. Then you add the following to the file:

/**
 * putObjectStream -- Streams data to a bucket.
 *
 * Takes ($bucket, $key, $streamFunction, $contentType, $contentLength [,$acl, $metadataArray, $md5])
 *
 *
 * - [str] $bucket: the bucket into which file will be written
 * - [str] $key: key of written file
 * - [str] $streamFunction: function to call for data to stream
 * - [str] $contentType: file content type
 * - [str] $contentLength: file content length
 * - [str] $acl: access control policy of file (OPTIONAL: defaults to 'private')
 * - [str] $metadataArray: associative array containing user-defined metadata (name=>value) (OPTIONAL)
 * - [bool] $md5: the MD5 hash of the object (OPTIONAL)
*/
function putObjectStream($bucket, $key, $streamFunction, $contentType, $contentLength, $acl, $metadataArray, $md5){
        sort($metadataArray);
        $resource = "$bucket/$key";
        $resource = urlencode($resource);
        $httpDate = gmdate("D, d M Y G:i:s T");

        $curl_inst = curl_init();

        curl_setopt ($curl_inst, CURLOPT_CONNECTTIMEOUT, 30);
        curl_setopt ($curl_inst, CURLOPT_LOW_SPEED_LIMIT, 1);
        curl_setopt ($curl_inst, CURLOPT_LOW_SPEED_TIME, 180);
        curl_setopt ($curl_inst, CURLOPT_NOSIGNAL, 1);
        curl_setopt ($curl_inst, CURLOPT_READFUNCTION, $streamFunction);
        curl_setopt ($curl_inst, CURLOPT_URL, $this->serviceUrl . $resource);
        curl_setopt ($curl_inst, CURLOPT_UPLOAD, true);
        curl_setopt ($curl_inst, CURLINFO_CONTENT_LENGTH_UPLOAD, $contentLength);

        $header[] = "Date: $httpDate";
        $header[] = "Content-Type: $contentType";
        $header[] = "Content-Length: $contentLength";
        $header[] = "Expect: ";
        $header[] = "Transfer-Encoding: ";
        $header[] = "x-amz-acl: $acl";

        $MD5 = "";
        if($md5){
                $MD5 = $this->hex2b64(md5_file($filePath));
                $header[] = "Content-MD5: $MD5";
        }

        $stringToSign="PUT\n$MD5\n$contentType\n$httpDate\nx-amz-acl:$acl\n";
        foreach($metadataArray as $current){
                if($current!=""){
                        $stringToSign.="x-amz-meta-$current\n";
                        $header = substr($current,0,strpos($current,':'));
                        $meta = substr($current,strpos($current,':')+1,strlen($current));
                        $header[] = "x-amz-meta-$header: $meta";
                }
        }

        $stringToSign.="/$resource";

        $signature = $this->constructSig($stringToSign);

        $header[] = "Authorization: AWS $this->accessKeyId:$signature";

        curl_setopt($curl_inst, CURLOPT_HTTPHEADER, $header);
        curl_setopt($curl_inst, CURLOPT_RETURNTRANSFER, 1);

        $result = curl_exec ($curl_inst);

        $this->responseString = $result;
        $this->responseCode = curl_getinfo($curl_inst, CURLINFO_HTTP_CODE);

        curl_close($curl_inst);
}

Now you have an updated s3.php you are ready to start streaming. Be aware that the content length value needs to be correct. If it is not the correct size your upload will be either truncated (too short) or it will hang (too long). If it hangs the S3 service will timeout after a small amount of time and give you an error.

Here are two examples of how to use it. First with a file:

<?php

include 's3.php';

class MyClass
{
  var $data;

  function stream_function($handle, $fd, $length)
  {
    return fread($this->data, $length);
  }
}

$my_class_inst = new MyClass();

$fsize = filesize("/tmp/largefile.tar.gz");

$my_class_inst->data = fopen("/tmp/largefile.tar.gz", "r");

$s3_inst = new s3("access key", "private key");
$s3_inst->putObjectStream("abucket", "largefile.tar.gz", array($my_class_inst, "stream_function"), "application/x-gzip", $fsize, "public-read", array(), null);

print "Response String: " . $s3_inst->responseString . "\n";
print "Response Code: " . $s3_inst->responseCode . "\n";
print "Parsed XML: " . $s3_inst->parsed_xml . "\n";

fclose($my_class_inst->data);

?>

Another example of streaming some random text:

<?php

include 's3.php';

class MyClass
{
  var $data;

  function stream_function($handle, $fd, $length)
  {
    return $this->data;
  }
}

$my_class_inst = new MyClass();

$my_class_inst->data = "A test string";
$size = strlen($my_class_inst->data);

$s3_inst = new s3("access key", "private key");
$s3_inst->putObjectStream("bucketname", "test.txt", array($my_class_inst, "stream_function"), "text/html", $size, "public-read", array(), null);

print "Response String: " . $s3_inst->responseString . "\n";
print "Response Code: " . $s3_inst->responseCode . "\n";
print "Parsed XML: " . $s3_inst->parsed_xml . "\n";

?>

Step 1. Get the PHP rrdtool source

Go to the contrib directory on the rrdtool distribution site:
http://people.ee.ethz.ch/~oetiker/webtools/rrdtool/pub/contrib/

There are a number of files in this directory that mention rrd. You want the one named: php_rrdtool.tgz

Step 2. Untar into the correct place

Now that you have the source go into your php source directory and then into the ext directory. So you will be somewhere like this:

/usr/local/src/php-5.1.3/ext/

Now untar the source into this directory.

Step 3. Recreate the php configuration file

There is a warning that you will get if you do not have autoconf 2.13 installed on your system when you try to do this. It is easy enough to get this version if you have fedora so that is what I did.

One tricky part to this is that I had to remove the old configuration file first before the new one could be created.

1. Change directory to your PHP source, if you are still in the ext directory just cd ..
2. Remove the existing configuration file
3. If you are using autoconf 2.13 run the following command:
   PHP_AUTOCONF=autoconf-2.13 ./buildconf –force
   If you are using whatever other autoconf you have installed just run:
   ./buildconf –force
4. You should now have a new configuration file that can be run with the –with-rrdtool option

Step 4. Test

After compiling with rrdtool you should be able to use the phpinfo() function to list the installed extensions. If everything went right you should see rrdtool listed.

The solution to the problem of wrangling logfiles is probably already installed on your server: it’s logrotate. Chances are logrotate is already configured in your system crontab as a daily task, and chances are it is configured to obey any configuration files found in the /etc/logrotate.d directory. If you find that directory, you are probably good to go.

In /etc/logrotate.d you’ll find several short configuration files. Each configuration deals with 1 or 2 related logfiles that should be checked and possibly administered on a regular basis. Logrotate can do alot of things with a logfile: clear the target log, create a copy of it, compress the copy, keep N backup copies, and it can be configured to rotate the target logfile on a daily, weekly, or monthly basis.

For example, here is how my client handles a logfile generated by a log4j-enabled java rmi server:

 /appdir/log/java_rmi_log.txt {
    daily
    rotate 10
    copytruncate
    compress
    notifempty
    missingok
 }

This configuration means

On a daily basis rotate, compress, and retain up to 10 days worth of logs using the copytruncate method. But notifempty; if the log is empty don’t do anything. Finally, missingok, so no need to log an error anywhere if the target logfile isn’t found.

This is nigh on a domain specific language for logfile rotation, and like most DSLs I’ve come across it’s much easier to read than it is to remember when writing. Don’t worry, the man page for logrotate lists all the details.

It became apparent fairly quickly that bash’s built-in ‘getopts’ didn’t support long command line arguments, so I had to fall back on getopt.

Here is the portion of the script that parses the command line arguments:

set -- `getopt -n$0 -u -a --longoptions="depth: adddays: topN:" "h" "$@"` || usage
[ $# -eq 0 ] && usage

while [ $# -gt 0 ]
do
    case "$1" in
       --depth)   depth=$2;shift;;
       --adddays) adddays=$2;shift;;
       --topN)    topN=$2;shift;;
       -h)        usage;;
       --)        shift;break;;
       -*)        usage;;
       *)         break;;            #better be the crawl directory
    esac
    shift
done

Deconstructing this bit by bit:

  set -- `getopt -n$0 -u -a --longoptions="depth: adddays: topN:" "h" "$@"` || usage
[ $# -eq 0 ] && usage
  

‘set –’ unsets the existing postional parameters and sets them to the result of getopt.
The call to getopts works like this:

• -n$0, sets the nicename to the name of the script (so warnings come back nicely from getopts)
• -a, allows long arguments to start with a singe ‘-’ (they ususally have two (‘–’)
• –longoptions=”depth: adddays: topN:”, sets the format of the long options. In this case I have 3 (depth, adddays, and topN). The
  trailing colon indicates I am expecting an additional argument.

• “h”, the short options (-h)
• “$@”, the arguments passed into the script

The ‘||’ at the end and the second line will call my usage statement if an error comes back from getopt (a non-0 return code). The next line make sure we get at least one argument back.

To help understand what goes on next, lets run that command at the shell:

   $ getopt -nrecrawl.sh -u -a --longoptions="depth: adddays: topN:" "h" -depth 5 -adddays 10 -topN 3 -h -x
recrawl.sh: unrecognized option `-x'
 --depth 5 --adddays 10 --topN 3 -h --
  

A few items of note. The first is the warning message we get because ‘x’ is an unknown option (notice it is prefaced by what we supplied to the -n argument). The second is the result of the getopt operation on my command line parameters.

Now to interpret the results:

  while [ $# -gt 0 ]
do
    case "$1" in
       --depth)   depth=$2;shift;;
...
       -h)        usage;;
       --)        shift;break;;
       -*)        usage;;
       *)         break;;            #better be the crawl directory
    esac
    shift
done
  

The while loop loops through each of the arguments return from getopt. If an argument requires an additional value, I use $2 to snag that value and assign it to a variable. When we are done with an argument, we shift passed it and move on to the next one. A few special cases exist:

• -*) matches any unkown option and prints the usage statement
• *) matchs any other argument (in this case it is our required directory)
• –) is the end marker from getopt

The script then goes on to verify the parameters (like the directory exists) and does the crawl….but that is for another day.

It’s a bash function. and a nice way to keep an eye on what’s happening on a cvs repository. I work with a team of around 7 guys, and I like to keep an eye on what’s changed and who changed it.

It works like this, summarizing what I consider to be the highlights from the cvs history command, and defaults to showing checkins for the current date:

$ cvshist
M 2006-02-22 10:34 EDT jsmith  1.3  InternalOperation.java         APRJ7A/src/com/ppc/oepica/model/internal
M 2006-02-22 10:53 EDT jsmith  1.37 FooBarCopier.java              APRJ7A/src/com/ppc/oepica/model/external
M 2006-02-22 11:26 EDT darrend 1.23 CommonJobBase.java             APRJ7A/src/com/ppc/oepica/model/common

I use the date command, which gives me some flexibility when specifying a date

$ cvshist yesterday
$ cvshist 2 days ago
$ cvshist january 10
$ cvshist 2006-01-11

Here’s the source. I’ve used this on linux and cygwin; you’ll need bash, gnu date, and cvs.

$ declare -f
cvshist ()
{
  export CVSROOT=:pserver:darrend@shlnxtest1:/home/cvs;
  TARGET_DATE=$(date -d "$*" +%Y-%m-%d);
  cvs history -a -z EDT -c -D $TARGET_DATE | sed "s/\s*==.*$//" | grep "$TARGET_DATE"
}

Configure the kernel (this may already be done for you):

Device Drivers --->
  Multiple devices driver support (RAID and LVM) --->
    [*] Multiple devices driver support (RAID and LVM)
      < >   RAID support
      <*>   Device mapper support
      <*>     Crypt target support

Cryptographic options --->
  <M>   MD5 digest algorithm
  <M>   SHA1 digest algorithm
  <M>   AES cipher algorithms
   .... 

Install the userland tools:

yum install cryptsetup

Create the filesystem and format it:

cryptsetup -c blowfish -s 64 create fs_name /dev/sda2
mkfs.ext3 /dev/mapper/fs_name

Create /etc/init.d/crytptinit:

if [ -b /dev/mapper/fs_name ]; then
  /sbin/cryptsetup remove fs_name
fi
/sbin/cryptsetup -c blowfish -s 64  create fs_name /dev/sda2

Run it at the runlevels you want:

cd /etc/rc3.d
ln -s ../init.d/cryptinit S08cryptinit

Create the mount point:

mkdir /mount_point

Edit /etc/fstab:

/dev/mapper/fs_name /mount_point               ext3   defaults 0 0

Reboot. You will be asked for your passphrase when the machine boots.

Information in this post was gleaned from several places. Here is one
that matches closely.