while true; do kill -USR1 `pidof dd`; sleep 2; done

(NOTE: if `pidof dd` doesn’t work for you, just use the process id directly)

Switch back to the terminal where dd is running and you should see:

9902751744 bytes (9.9 GB) copied, 732.883 s, 13.5 MB/s
9469+0 records in
9468+0 records out
9927917568 bytes (9.9 GB) copied, 734.914 s, 13.5 MB/s
9496+0 records in
9495+0 records out
9956229120 bytes (10 GB) copied, 736.941 s, 13.5 MB/s

updating every couple of seconds.

cap --set-before branch=testbranch  deploy

…where ‘testbranch’ is the name of my subversion branch, and ‘deploy’ is the action to take. The ‘–set-before option’ sets a variable before the recipes are loaded (sort of). All you need to do to make this work you just need to set the repository URL correctly based on if the branch variable exists or not. In my case, that is editing the top of deploy.rb in the config section of my rails app:

if variables.include?(:branch)
  set :repository,  "http://svn.example.com/project/branches/#{branch}"
else
  set :repository,  "http://svn.example.com/project/trunk"
end

C lib/jt400_3_3.jar
cvs update: move away lib/jt400_3_3.jar; it is in the way

Very peculiar. That file hadn’t changed on the branch. I googled around and an explanation started to come together.

First, from the always excellent Roedy Green’s Java & Internet Glossary on Mindprod:

CVS is disturbed by the appearance of repository files it did not put there. Your best bet is simply to delete the entire directory containing the offending file by hand, and re checkout or reupdate the directory to build the necessary entries. Then you can add the files safely.

Then, I found this mailing list post:

This means the file that CVS wants to checkout exists on the local machine but CVS never created the file in the past. This it isn’t CVS’s file and it complains rather than blindly overwriting.

The solution everywhere was the same: just blow away the directory and check it out again. Worked for me. Now I need to puzzle out how it happened in the first place…

At first I decided my test would just respond with the yamlized request (just like the mongrel sample did). This worked fine but there was a significant size difference in the two documents (WEBrick’s was much bigger). In the end, I just decided to respond with a few bytes of text.

The Code

mongrel_service.rb:

require 'rubygems'
require 'mongrel'
require 'yaml'

class SimpleHandler < Mongrel::HttpHandler
  def process(request, response)
    response.start do |head,out|
      head["Content-Type"] = "text/html"
      out << "I am mongrel, hear me roar"
    end
  end
end

config = Mongrel::Configurator.new :host => "localhost", :port => 4000 do
  listener {uri "/", :handler => SimpleHandler.new}
  trap("INT") { stop }
  run
end

config.join

webrick_service.rb:

require 'webrick'
require 'yaml'
include WEBrick

class SimpleServlet < HTTPServlet::AbstractServlet
  def do_GET(request, response)
    response["Content-Type"] = "text/html"
    response.body = "I am WEBrick, hear me roar"
  end
end

s = HTTPServer.new( :Port => 4001 )
s.mount("/", SimpleServlet)
trap('INT'){ s.shutdown }
s.start

The Test

There are serveral good benchmarking tools available, but I had apache bench (ab) handy, so that is what I used. I just used the stock gem install of mongrel and the WEBrick that comes with ruby 1.8.4 (the one non-stock change I made was to comment out access logging for WEBrick since mongrel doesn’t log by default).

The Results

The Conclusion

Looks like mongrel it is….

The second important item we learned from this process is the performance difference between a true SQL index on the iSeries and a keyed logical index. From iSeries V4R2 on, SQL indexes have page sizes of 64k. By comparison, a DDS logical will max out at 32k, and will typically be much smaller. The performance is so much better that our customer is converting all all their DDS logical indexes to SQL, even if only their legacy RPG applications use it. This IBM document has a good discussion of this issue.

[emerg] (45)Deadlock situation detected/avoided: apr_proc_mutex_lock failed. Attempting to shutdown process gracefully.

Not good. After a little digging we found that the default AcceptMutex switched from pthread to fcntl between apache 2.0.49 and 2.0.52.

Adding:

AcceptMutex pthread

to httpd.conf cleared up the errors and we’ve had smooth sailing ever since.

Step 1. Get the PHP rrdtool source

Go to the contrib directory on the rrdtool distribution site:
http://people.ee.ethz.ch/~oetiker/webtools/rrdtool/pub/contrib/

There are a number of files in this directory that mention rrd. You want the one named: php_rrdtool.tgz

Step 2. Untar into the correct place

Now that you have the source go into your php source directory and then into the ext directory. So you will be somewhere like this:

/usr/local/src/php-5.1.3/ext/

Now untar the source into this directory.

Step 3. Recreate the php configuration file

There is a warning that you will get if you do not have autoconf 2.13 installed on your system when you try to do this. It is easy enough to get this version if you have fedora so that is what I did.

One tricky part to this is that I had to remove the old configuration file first before the new one could be created.

1. Change directory to your PHP source, if you are still in the ext directory just cd ..
2. Remove the existing configuration file
3. If you are using autoconf 2.13 run the following command:
   PHP_AUTOCONF=autoconf-2.13 ./buildconf –force
   If you are using whatever other autoconf you have installed just run:
   ./buildconf –force
4. You should now have a new configuration file that can be run with the –with-rrdtool option

Step 4. Test

After compiling with rrdtool you should be able to use the phpinfo() function to list the installed extensions. If everything went right you should see rrdtool listed.

The solution to the problem of wrangling logfiles is probably already installed on your server: it’s logrotate. Chances are logrotate is already configured in your system crontab as a daily task, and chances are it is configured to obey any configuration files found in the /etc/logrotate.d directory. If you find that directory, you are probably good to go.

In /etc/logrotate.d you’ll find several short configuration files. Each configuration deals with 1 or 2 related logfiles that should be checked and possibly administered on a regular basis. Logrotate can do alot of things with a logfile: clear the target log, create a copy of it, compress the copy, keep N backup copies, and it can be configured to rotate the target logfile on a daily, weekly, or monthly basis.

For example, here is how my client handles a logfile generated by a log4j-enabled java rmi server:

 /appdir/log/java_rmi_log.txt {
    daily
    rotate 10
    copytruncate
    compress
    notifempty
    missingok
 }

This configuration means

On a daily basis rotate, compress, and retain up to 10 days worth of logs using the copytruncate method. But notifempty; if the log is empty don’t do anything. Finally, missingok, so no need to log an error anywhere if the target logfile isn’t found.

This is nigh on a domain specific language for logfile rotation, and like most DSLs I’ve come across it’s much easier to read than it is to remember when writing. Don’t worry, the man page for logrotate lists all the details.

Probably the simplest example is connecting to an iSeries system using JDBC to select from some tables, and with the right libraries it is straightforward. It’s also fun to be doing some nifty quick Ruby but with some preexisting and proven Java libraries.

Also, if you use Java and Ruby, definitely put arton’s RubyJavaBridge in your pocket; you’ll end up using it more than once.

What you’ll need:

• Ruby (well yea)
• Java (ok)
• access to an AS400 system (that’s the whole point here…)
  For testing, I’ve been using a free account available from Holger Scherer Software und Beratung.
• jt400.jar jdbc drivers from JTOpen, a great open-source Java library for AS400 access.
• RubyJavaBridge is the keystone for this.

Installing the software is the hardest part of this exercise, but it’s more tedium than anything. Once you’ve got it all downloaded and running the fun can begin.

require 'rjb'
Rjb::load('jtopen_5_0/lib/jt400.jar',['-Djdbc.drivers=com.ibm.as400.access.AS400JDBCDriver'])

DriverManager = Rjb::import('java.sql.DriverManager')

begin
  connection = DriverManager.getConnection('jdbc:as400://as400.holgerscherer.de','YOUR_USERNAME','YOUR_PASSWORD')
  statement = connection.prepareStatement("SELECT count(*) FROM SYSIBM.TABLES")
  result_set = statement.executeQuery

  while(result_set.next())
    puts result_set.getObject(1).toString
  end
ensure
  result_set.close if defined?(result_set) && !statement.nil?
  statement.close if defined?(statement) && !statement.nil?
  connection.close if defined?(connection) && !connection.nil?
end

$ ruby as400_rjb.rb
19138

Very readable, I think. Java and Ruby actually play well together given half the chance and arton’s fantastic bridge.

Compare this with the pure Java implementation.

import java.sql.*;

public class As400Jdbc
{
  public static void main(String args[])
  {
    Connection connection = null;
    try
    {
      connection = DriverManager.getConnection("jdbc:as400://as400.holgerscherer.de","USER","PASS");
      PreparedStatement statement = connection.prepareStatement("SELECT COUNT(*) FROM SYSIBM.TABLES");

      ResultSet resultSet = statement.executeQuery();

      while(resultSet.next())
      {
        System.out.println(resultSet.getObject(1));
      }
    }
    catch(SQLException e)
    {
      throw new RuntimeException(e);
    }
    finally
    {
      try{ if(connection!=null) connection.close(); }
      catch(SQLException e)
      {
        throw new RuntimeException(e);
      }
    }

  }
}

$ javac As400Jdbc.java
$ java -cp jtopen_5_0/lib/jt400.jar:. -Djdbc.drivers=com.ibm.as400.access.AS400JDBCDriver As400Jdbc
19138

The Ruby code and the Java code are nearly one-for-one so far, and that’s fine. Still, let’s see if we can’t pull the Java code more into Ruby’s world. The begin...end stuff gets tiresome; it would be nice if these Java classes could handle the resource allocation and disposal drudgery for us, using blocks.

require 'rjb'
Rjb::load('/home/darren/dload/jtopen_5_0/lib/jt400.jar',['-Djdbc.drivers=com.ibm.as400.access.AS400JDBCDriver'])

DriverManager = Rjb::import('java.sql.DriverManager')
class <<DriverManager
  def with_connection(*args)
    begin
      connection = DriverManager.getConnection(*args)
      yield connection
    ensure
      connection.close
    end
  end
end

DriverManager.with_connection('jdbc:as400://as400.holgerscherer.de','USER_NAME','USER_PASS') do |conn|
  statement = conn.prepareStatement("SELECT count(*) FROM SYSIBM.TABLES")
  result_set = statement.executeQuery()
  result_set.next()
  puts result_set.getObject(1).toString
end

I know I’m a geek because stuff like this makes me giggle. So many worlds are colliding here–Ruby, Java, AS400, DB2, the client system (Linux in this case)–and it all comes together so nicely.

I’m not saying I’d build an app out of this, but it will more than suffice for ad-hoc queries and quick scripts.

It became apparent fairly quickly that bash’s built-in ‘getopts’ didn’t support long command line arguments, so I had to fall back on getopt.

Here is the portion of the script that parses the command line arguments:

set -- `getopt -n$0 -u -a --longoptions="depth: adddays: topN:" "h" "$@"` || usage
[ $# -eq 0 ] && usage

while [ $# -gt 0 ]
do
    case "$1" in
       --depth)   depth=$2;shift;;
       --adddays) adddays=$2;shift;;
       --topN)    topN=$2;shift;;
       -h)        usage;;
       --)        shift;break;;
       -*)        usage;;
       *)         break;;            #better be the crawl directory
    esac
    shift
done

Deconstructing this bit by bit:

  set -- `getopt -n$0 -u -a --longoptions="depth: adddays: topN:" "h" "$@"` || usage
[ $# -eq 0 ] && usage
  

’set –’ unsets the existing postional parameters and sets them to the result of getopt.
The call to getopts works like this:

• -n$0, sets the nicename to the name of the script (so warnings come back nicely from getopts)
• -a, allows long arguments to start with a singe ‘-’ (they ususally have two (’–’)
• –longoptions=”depth: adddays: topN:”, sets the format of the long options. In this case I have 3 (depth, adddays, and topN). The
  trailing colon indicates I am expecting an additional argument.

• “h”, the short options (-h)
• “$@”, the arguments passed into the script

The ‘||’ at the end and the second line will call my usage statement if an error comes back from getopt (a non-0 return code). The next line make sure we get at least one argument back.

To help understand what goes on next, lets run that command at the shell:

   $ getopt -nrecrawl.sh -u -a --longoptions="depth: adddays: topN:" "h" -depth 5 -adddays 10 -topN 3 -h -x
recrawl.sh: unrecognized option `-x'
 --depth 5 --adddays 10 --topN 3 -h --
  

A few items of note. The first is the warning message we get because ‘x’ is an unknown option (notice it is prefaced by what we supplied to the -n argument). The second is the result of the getopt operation on my command line parameters.

Now to interpret the results:

  while [ $# -gt 0 ]
do
    case "$1" in
       --depth)   depth=$2;shift;;
...
       -h)        usage;;
       --)        shift;break;;
       -*)        usage;;
       *)         break;;            #better be the crawl directory
    esac
    shift
done
  

The while loop loops through each of the arguments return from getopt. If an argument requires an additional value, I use $2 to snag that value and assign it to a variable. When we are done with an argument, we shift passed it and move on to the next one. A few special cases exist:

• -*) matches any unkown option and prints the usage statement
• *) matchs any other argument (in this case it is our required directory)
• –) is the end marker from getopt

The script then goes on to verify the parameters (like the directory exists) and does the crawl….but that is for another day.