Treetop appeared to gather all the concepts together into an understandable domain specific language. All of the tokenization and node structure can go into a single file, and the interactive nature of Ruby makes for the perfect sand box. I felt like I could get somewhere if I invested just an hour into this. I was happy to find that my impressions were correct.

After a short time I had caught on enough to start writing my own code. Once over the hump the rest was easy. I was able to write and test a Treetop grammar for parsing CSV files within a few hours. I chose CSV parsing because I was already familiar with the format, and I could compare my implementation to not just one but two existing Ruby libraries.

commaseparatedfile.treetop

grammar CommaSeparatedFile
  rule lines
    line (newline line)*
    {
      def values
        vals = []
        vals << line.values
        more.elements.each { |additional| vals << additional.line.values }
        vals
      end
    }
  end

  rule line
    valueline / emptyline
  end

  rule emptyline
    '' {
      def values
        []
      end
    }
  end

  rule valueline
    leading:(value separator)* trailing:value
    {
      def values
        list = []
        list += leading.elements.collect { |lead| lead.value.text_value } if leading
        list << trailing.text_value if trailing
        list
      end
    }
  end

  rule value
    quotedvalue / nakedvalue
  end

  rule quotedvalue
    '"' wrapped:( !'"' . / '""' )* '"' {
      def text_value
        wrapped.text_value.gsub(/""/,'"')
      end
    }
  end

  rule nakedvalue
    (!(separator / newline ) .)*
  end

  rule separator
    ','
  end

  rule newline
    [\n]
  end
end

One concept that became clear to me is that the tokens that are defined in the grammar are objects in the resulting tree returned after input is parsed. Ruby code embedded in the Treetop’s node definitions are added as methods to the object declared by that node.

In the simple harness below, the lines object returned from the parse call below is the object representation of the rule lines node above, defined as the root of the parse tree. The values method I call below is defined above in the definition of the lines node.

require 'rubygems'
require 'treetop'
require 'commaseparatedfile' 

parser = CommaSeparatedFileParser.new
lines = parser.parse("this,that,these,those")
puts lines.values.inspect #-> [["this", "that", "these", "those"]]

I think it was much simpler for me to write this parsing as a grammar instead of trying to combine regular expressions with Ruby code. The grammar above handles multiple lines and quote-wrapped values, and escaped quotes.

It was actually fun to write as well, and I’m still learning a lot. Notice that my lines node and my valueline node both approach a similar problem (a list of 0 or more items) differently. For the lines the grammar grabs one line followed by 0 or more separators followed by lines. For the valueline I grab 0 ore more values followed by separators and terminate on the last value. Is one approach better than the other? Are they both junk? I can’t wait to figure that out.

My poor parser is left in the dust by both Ruby’s stdlib csv library, and the fastercsv gem. I ran these tests under the 1.8.5 version of Ruby. The dataset I used is 44k in total and has 961 records.

                       user     system      total        real
stdlib_csv:        0.210000   0.000000   0.210000 (  0.217275)
faster_csv:        0.040000   0.000000   0.040000 (  0.060788)
treetop_grammar:   2.910000   0.010000   2.920000 (  3.126020)

I don’t know what sort of speed I should reasonably expect, actually. I’d hoped that my compiler would be competitive with the standard library’s csv parsing. It’s possible that my grammar’s definition could be innefficient. If anyone has any ideas let me know.

After playing around with Treetop I’ll definitely tinker some more. There are many other features I haven’t explored yet, like extending a base grammar to create a new grammar. PEGs have implementations in other languages so this isn’t a Ruby-only facility, either.

On a side note, I used stdlib’s benchmark to generate the numbers above. Pretty swank stuff.

require 'rubygems'
require 'treetop'
require 'commaseparatedfile'
require 'fastercsv'
require 'csv'
require 'benchmark'

def with_stdlib_csv(csv)
  parsed = CSV.parse(csv)
  parsed.size
end

def with_fastercsv(csv)
  rows = FasterCSV.parse(csv)
  rows.size
end

def with_treetop_parser(csv)
  parser = CommaSeparatedFileParser.new
  parsed = parser.parse(csv)
  parsed.values.size
end

csv = File.read('../kentucky.csv')

Benchmark.bmbm("treetop_parser:".length) do |x|
  x.report("stdlib_csv:") { with_stdlib_csv(csv) }
  x.report("faster_csv:") { with_fastercsv(csv) }
  x.report("treetop_grammar:") { with_treetop_parser(csv) }
end

C lib/jt400_3_3.jar
cvs update: move away lib/jt400_3_3.jar; it is in the way

Very peculiar. That file hadn’t changed on the branch. I googled around and an explanation started to come together.

First, from the always excellent Roedy Green’s Java & Internet Glossary on Mindprod:

CVS is disturbed by the appearance of repository files it did not put there. Your best bet is simply to delete the entire directory containing the offending file by hand, and re checkout or reupdate the directory to build the necessary entries. Then you can add the files safely.

Then, I found this mailing list post:

This means the file that CVS wants to checkout exists on the local machine but CVS never created the file in the past. This it isn’t CVS’s file and it complains rather than blindly overwriting.

The solution everywhere was the same: just blow away the directory and check it out again. Worked for me. Now I need to puzzle out how it happened in the first place…

Here is a simple example:

We’ll start with a table named libris_book that contains book entries. The name libris_book won’t really fly with the Rails pluralized table name approach. It’s easy, however, to fix that disconnect in the model:

class Book < ActiveRecord::Base
  set_table_name 'libris_book'
end

Let’s take a look at our data:

~/src/libris$ script/console
Loading development environment.
>> Book.count
=> 1230

Good enough. Now, we’ll be good little coders and write some unit tests using fixtures. We’ll create a fixture file and our test class:

books.yml

test_book:
  title: Life in Lawrenceburg
  author: Darren Day

book_test.rb

require File.dirname(__FILE__) + '/../test_helper'

class BookTest < Test::Unit::TestCase
  fixtures :books

  def test_fixtures
    assert_equal(2,Book.count)
    test_book = books(:test_book)
  end
end

Looks good, but it won’t work!

~/src/libris$ rake test
(in /home/darrend/src/libris)
/usr/local/bin/ruby -Ilib:test "/usr/local/lib/ruby/gems/1.8/gems/rake-0.7.2/lib/rake/rake_test_loader.rb" "test/unit/book_test.rb"
Loaded suite /usr/local/lib/ruby/gems/1.8/gems/rake-0.7.2/lib/rake/rake_test_loader
Started
E
Finished in 0.01133 seconds.

  1) Error:
test_fixtures(BookTest):
ActiveRecord::StatementInvalid: Mysql::Error: Table 'libris_test.books' doesn't exist: DELETE FROM books
...

Ugh. That doesn’t look pretty. After some googling around the following comes together

• Rename books.yml to libris_book.yml. The fixture yaml file has to share the same name as the database table name.
• Change the fixtures declaration to fixtures :libris_book.
• Use set_fixture_class to connect the fixture to the model class.

Here is our new and improved book_test.rb file:

require File.dirname(__FILE__) + '/../test_helper'

class BookTest < Test::Unit::TestCase
  set_fixture_class :libris_book => Book
  fixtures :libris_book

  def test_fixtures
    assert_equal(2,Book.count)
    test_book = libris_book(:test_book)
  end
end

If you don’t respect these two possibilities, then you will get bit in the butt alot, when you least expect it. To be fair, these situations can happen whether you’re using Hibernate or not, but when we first started using the framework we made lots of assumptions.

Here is a completely ridiculous example that violates the above restrictions:

public class User
{
  private boolean isSurnameLast;
  private Name name;  

  private String fullName;
  private String initials;

  public boolean setSurnameLast(boolean surnameLast)
  {
    this.isSurnameLast=surnameLast;
  }

  public void setName(Name name)
  {
    this.name = name;
    String lastName;
    String firstName;
    initials = new StringBuffer(name.getSurName().substring(0,1)).append(".").append(name.getGivenName().substring(0,1)).toString();
    if(isSurnameLast)
    {
      lastName = name.getSurName();
      firstName = name.getGivenName();
    }
    else
    {
      lastName = name.getGivenName();
      firstName = name.getSurName();
    }
    fullName = new StringBuffer(firstName).append(" ").append(lastName).toString();
  }
  // more getters and setters blah blah blah
}

(Ignore the lame “business logic” here…)

So you look in your database and you see that all the Users database entries have names and properly set surnameLast values, and all the Names database entries have a surName and a givenName. Sure the code is sloppy, but this is gonna work fine!

Nope, this setName is loaded with problems.

First, you’ll notice as your app runs that somehow setName is being called and passed a null. What?! Why! Durn that Hibernate, it is trying to trick you! No matter, we’ll catch it with a null check. You’ll shield your eyes as to when and why Hibernate or cglib or whatever is calling setters on an instance that isn’t even in the database, but at least it won’t blow up. This is the first bad smell you’ve blown through, you sloppy panicky codemonkey:

  public void setName(Name name)
  {
    if(name!=null) // hahah
   {
       this.name = name;
      ....

Then you’ll start realizing that no matter what the database says, sometimes you are getting the surname first in the full name! “Smith Joan!”, you’ll cry, but “it’s true for Joan Smith in the database! Why?” Well, sometimes Hibernate is calling setName before it calls setSurnameLast. It doesn’t matter which you have mapped first in the User.hbm.xml file, or which field you have first in the code. You can’t depend on other properties of User being initialized in the Hibernate setters.

Now, you’re also going to get NullPointerExceptions thrown out of this code, because sometimes that Name instance you’re being passed hasn’t had its properties initialized by Hibernate yet. Talk about something that seems sneaky and evil but is totally above board: Hibernate has created the new Name instance and has passed that reference to other instances, but hasn’t fully populated the Name instance yet.

That means this line will blow sky-high sometimes:

initials = new StringBuffer(name.getSurName().substring(0,1)).append(".").append(name.getGivenName().substring(0,1)).toString();

The sane thing to do here?

  public void setName(Name name)
  {
    this.name = name;
  }

So if you’re using Hibernate, go take a look at your classes and mappings! You may very well be doing something dangerous.

It’s been a month now since our last performance, but it all comes back every time I look at the slides.

The solution to the problem of wrangling logfiles is probably already installed on your server: it’s logrotate. Chances are logrotate is already configured in your system crontab as a daily task, and chances are it is configured to obey any configuration files found in the /etc/logrotate.d directory. If you find that directory, you are probably good to go.

In /etc/logrotate.d you’ll find several short configuration files. Each configuration deals with 1 or 2 related logfiles that should be checked and possibly administered on a regular basis. Logrotate can do alot of things with a logfile: clear the target log, create a copy of it, compress the copy, keep N backup copies, and it can be configured to rotate the target logfile on a daily, weekly, or monthly basis.

For example, here is how my client handles a logfile generated by a log4j-enabled java rmi server:

 /appdir/log/java_rmi_log.txt {
    daily
    rotate 10
    copytruncate
    compress
    notifempty
    missingok
 }

This configuration means

On a daily basis rotate, compress, and retain up to 10 days worth of logs using the copytruncate method. But notifempty; if the log is empty don’t do anything. Finally, missingok, so no need to log an error anywhere if the target logfile isn’t found.

This is nigh on a domain specific language for logfile rotation, and like most DSLs I’ve come across it’s much easier to read than it is to remember when writing. Don’t worry, the man page for logrotate lists all the details.

Probably the simplest example is connecting to an iSeries system using JDBC to select from some tables, and with the right libraries it is straightforward. It’s also fun to be doing some nifty quick Ruby but with some preexisting and proven Java libraries.

Also, if you use Java and Ruby, definitely put arton’s RubyJavaBridge in your pocket; you’ll end up using it more than once.

What you’ll need:

• Ruby (well yea)
• Java (ok)
• access to an AS400 system (that’s the whole point here…)
  For testing, I’ve been using a free account available from Holger Scherer Software und Beratung.
• jt400.jar jdbc drivers from JTOpen, a great open-source Java library for AS400 access.
• RubyJavaBridge is the keystone for this.

Installing the software is the hardest part of this exercise, but it’s more tedium than anything. Once you’ve got it all downloaded and running the fun can begin.

require 'rjb'
Rjb::load('jtopen_5_0/lib/jt400.jar',['-Djdbc.drivers=com.ibm.as400.access.AS400JDBCDriver'])

DriverManager = Rjb::import('java.sql.DriverManager')

begin
  connection = DriverManager.getConnection('jdbc:as400://as400.holgerscherer.de','YOUR_USERNAME','YOUR_PASSWORD')
  statement = connection.prepareStatement("SELECT count(*) FROM SYSIBM.TABLES")
  result_set = statement.executeQuery

  while(result_set.next())
    puts result_set.getObject(1).toString
  end
ensure
  result_set.close if defined?(result_set) && !statement.nil?
  statement.close if defined?(statement) && !statement.nil?
  connection.close if defined?(connection) && !connection.nil?
end

$ ruby as400_rjb.rb
19138

Very readable, I think. Java and Ruby actually play well together given half the chance and arton’s fantastic bridge.

Compare this with the pure Java implementation.

import java.sql.*;

public class As400Jdbc
{
  public static void main(String args[])
  {
    Connection connection = null;
    try
    {
      connection = DriverManager.getConnection("jdbc:as400://as400.holgerscherer.de","USER","PASS");
      PreparedStatement statement = connection.prepareStatement("SELECT COUNT(*) FROM SYSIBM.TABLES");

      ResultSet resultSet = statement.executeQuery();

      while(resultSet.next())
      {
        System.out.println(resultSet.getObject(1));
      }
    }
    catch(SQLException e)
    {
      throw new RuntimeException(e);
    }
    finally
    {
      try{ if(connection!=null) connection.close(); }
      catch(SQLException e)
      {
        throw new RuntimeException(e);
      }
    }

  }
}

$ javac As400Jdbc.java
$ java -cp jtopen_5_0/lib/jt400.jar:. -Djdbc.drivers=com.ibm.as400.access.AS400JDBCDriver As400Jdbc
19138

The Ruby code and the Java code are nearly one-for-one so far, and that’s fine. Still, let’s see if we can’t pull the Java code more into Ruby’s world. The begin...end stuff gets tiresome; it would be nice if these Java classes could handle the resource allocation and disposal drudgery for us, using blocks.

require 'rjb'
Rjb::load('/home/darren/dload/jtopen_5_0/lib/jt400.jar',['-Djdbc.drivers=com.ibm.as400.access.AS400JDBCDriver'])

DriverManager = Rjb::import('java.sql.DriverManager')
class <<DriverManager
  def with_connection(*args)
    begin
      connection = DriverManager.getConnection(*args)
      yield connection
    ensure
      connection.close
    end
  end
end

DriverManager.with_connection('jdbc:as400://as400.holgerscherer.de','USER_NAME','USER_PASS') do |conn|
  statement = conn.prepareStatement("SELECT count(*) FROM SYSIBM.TABLES")
  result_set = statement.executeQuery()
  result_set.next()
  puts result_set.getObject(1).toString
end

I know I’m a geek because stuff like this makes me giggle. So many worlds are colliding here–Ruby, Java, AS400, DB2, the client system (Linux in this case)–and it all comes together so nicely.

I’m not saying I’d build an app out of this, but it will more than suffice for ad-hoc queries and quick scripts.

Honestly, Ruby sells itself though. People tend to sit up and notice blocks and open classes.

We’ll be posting the presentation slides very soon to the permanent page. And as we take the show to other venues we’ll be tweaking and refining the content. Initial feedback is to beef up the handout with a cheatsheet of language features; our whirlwind tour of the language goes by quickly and some sort of reference will help keep everything together.

Our audience expressed great interest in Ruby on Rails and AJAX, even a few requests for some Seeing More Ruby classes. We’re debating on whether “You’ll Be Seeing AJAX” or “You’ll Be Seeing Rails” next. :) Let us know which you’d rather see.

Incidentally, we’re also taking this presentation to a company for a private show, and we’d be willing to come talk with you and your colleagues as well. Just contact us!

It’s a bash function. and a nice way to keep an eye on what’s happening on a cvs repository. I work with a team of around 7 guys, and I like to keep an eye on what’s changed and who changed it.

It works like this, summarizing what I consider to be the highlights from the cvs history command, and defaults to showing checkins for the current date:

$ cvshist
M 2006-02-22 10:34 EDT jsmith  1.3  InternalOperation.java         APRJ7A/src/com/ppc/oepica/model/internal
M 2006-02-22 10:53 EDT jsmith  1.37 FooBarCopier.java              APRJ7A/src/com/ppc/oepica/model/external
M 2006-02-22 11:26 EDT darrend 1.23 CommonJobBase.java             APRJ7A/src/com/ppc/oepica/model/common

I use the date command, which gives me some flexibility when specifying a date

$ cvshist yesterday
$ cvshist 2 days ago
$ cvshist january 10
$ cvshist 2006-01-11

Here’s the source. I’ve used this on linux and cygwin; you’ll need bash, gnu date, and cvs.

$ declare -f
cvshist ()
{
  export CVSROOT=:pserver:darrend@shlnxtest1:/home/cvs;
  TARGET_DATE=$(date -d "$*" +%Y-%m-%d);
  cvs history -a -z EDT -c -D $TARGET_DATE | sed "s/\s*==.*$//" | grep "$TARGET_DATE"
}

Sequence
 number   Library
    10    QTEMP            

    20    DDMFLIB2
....

If it’s computable but a one-off, I immediately go to Ruby nowadays. 10 years ago it would have been Perl, 15 or 20 years ago the choice would have been probably a meld of sed|awk.

I always go with __END__ in these situations; Ruby stops processing at the __END__ line, but you are given an enumerable access to the contents that follow with the magical DATA object.

matches = DATA.collect { |line| line.match(/\\s+\\d+\\s+(\\w+)/) }
matches.compact!
puts matches.collect { |match| match[1] }.join(', ')

__END__
Initial library list:                            

Sequence
 number   Library
    10    QTEMP            

    20    DDMFLIB2
    30    APPSDEV
    50    FILES2                                                       

   110    QRPG                                                              

Let’s try it out (running Ruby/Windows in a Cygwin terminal):

dday@xpmachine21 ~
$ ruby liblist.rb
QTEMP, DDMFLIB2, APPSDEV, FILES2, QRPG

Success!