Treetop appeared to gather all the concepts together into an understandable domain specific language. All of the tokenization and node structure can go into a single file, and the interactive nature of Ruby makes for the perfect sand box. I felt like I could get somewhere if I invested just an hour into this. I was happy to find that my impressions were correct.

After a short time I had caught on enough to start writing my own code. Once over the hump the rest was easy. I was able to write and test a Treetop grammar for parsing CSV files within a few hours. I chose CSV parsing because I was already familiar with the format, and I could compare my implementation to not just one but two existing Ruby libraries.

commaseparatedfile.treetop

grammar CommaSeparatedFile
  rule lines
    line (newline line)*
    {
      def values
        vals = []
        vals << line.values
        more.elements.each { |additional| vals << additional.line.values }
        vals
      end
    }
  end

  rule line
    valueline / emptyline
  end

  rule emptyline
    '' {
      def values
        []
      end
    }
  end

  rule valueline
    leading:(value separator)* trailing:value
    {
      def values
        list = []
        list += leading.elements.collect { |lead| lead.value.text_value } if leading
        list << trailing.text_value if trailing
        list
      end
    }
  end

  rule value
    quotedvalue / nakedvalue
  end

  rule quotedvalue
    '"' wrapped:( !'"' . / '""' )* '"' {
      def text_value
        wrapped.text_value.gsub(/""/,'"')
      end
    }
  end

  rule nakedvalue
    (!(separator / newline ) .)*
  end

  rule separator
    ','
  end

  rule newline
    [\n]
  end
end

One concept that became clear to me is that the tokens that are defined in the grammar are objects in the resulting tree returned after input is parsed. Ruby code embedded in the Treetop’s node definitions are added as methods to the object declared by that node.

In the simple harness below, the lines object returned from the parse call below is the object representation of the rule lines node above, defined as the root of the parse tree. The values method I call below is defined above in the definition of the lines node.

require 'rubygems'
require 'treetop'
require 'commaseparatedfile' 

parser = CommaSeparatedFileParser.new
lines = parser.parse("this,that,these,those")
puts lines.values.inspect #-> [["this", "that", "these", "those"]]

I think it was much simpler for me to write this parsing as a grammar instead of trying to combine regular expressions with Ruby code. The grammar above handles multiple lines and quote-wrapped values, and escaped quotes.

It was actually fun to write as well, and I’m still learning a lot. Notice that my lines node and my valueline node both approach a similar problem (a list of 0 or more items) differently. For the lines the grammar grabs one line followed by 0 or more separators followed by lines. For the valueline I grab 0 ore more values followed by separators and terminate on the last value. Is one approach better than the other? Are they both junk? I can’t wait to figure that out.

My poor parser is left in the dust by both Ruby’s stdlib csv library, and the fastercsv gem. I ran these tests under the 1.8.5 version of Ruby. The dataset I used is 44k in total and has 961 records.

                       user     system      total        real
stdlib_csv:        0.210000   0.000000   0.210000 (  0.217275)
faster_csv:        0.040000   0.000000   0.040000 (  0.060788)
treetop_grammar:   2.910000   0.010000   2.920000 (  3.126020)

I don’t know what sort of speed I should reasonably expect, actually. I’d hoped that my compiler would be competitive with the standard library’s csv parsing. It’s possible that my grammar’s definition could be innefficient. If anyone has any ideas let me know.

After playing around with Treetop I’ll definitely tinker some more. There are many other features I haven’t explored yet, like extending a base grammar to create a new grammar. PEGs have implementations in other languages so this isn’t a Ruby-only facility, either.

On a side note, I used stdlib’s benchmark to generate the numbers above. Pretty swank stuff.

require 'rubygems'
require 'treetop'
require 'commaseparatedfile'
require 'fastercsv'
require 'csv'
require 'benchmark'

def with_stdlib_csv(csv)
  parsed = CSV.parse(csv)
  parsed.size
end

def with_fastercsv(csv)
  rows = FasterCSV.parse(csv)
  rows.size
end

def with_treetop_parser(csv)
  parser = CommaSeparatedFileParser.new
  parsed = parser.parse(csv)
  parsed.values.size
end

csv = File.read('../kentucky.csv')

Benchmark.bmbm("treetop_parser:".length) do |x|
  x.report("stdlib_csv:") { with_stdlib_csv(csv) }
  x.report("faster_csv:") { with_fastercsv(csv) }
  x.report("treetop_grammar:") { with_treetop_parser(csv) }
end

Handling session timeouts on non-Ajax requests

The first step to handing session timeouts with AJAX is to handle them for non-Ajax requests first. One way of doing this is trapping the URL the user is attempting to access (but not authenticated to), redirecting them to a login page, and upon success, redirecting them back to their original request.

If they aren’t authenticated, send them to the login page:

if(!loggedIn())
{
  response.sendRedirect("/login.jsp?uri=" + URLEncoder.encode(fullURI, "UTF8"));
}

Hide the original request in the page:

<form action="/account/login" method='POST'>
  <input type="hidden" name="uri" value="<%=request.getParameter("uri")%>">
  <label for="user_login">ID:</label>
  <input type="text" id="user_login" name="user_login" value=""/>
  <label for="user_password">Password:</label>
  <input type="password" id="user_password" name="user_password"/>
  <input type="submit" name="login" value=""/>
</form>

…and after a successful signup, send them back to where they wanted to go:

if(authenticated())
{
  response.sendRedirect(request.getParameter("uri"));
}

Handling Session Timeouts for Ajax Requests

The first thing we need to know is if it is an ajax request or not. Since I am going to use prototype for all my ajax calls it is as straight forward as looking for the special header prototype sends on each request:

private boolean isXMLHttpRequest(HttpServletRequest request)
{
  return request.getHeader("x-requested-with") != null && request.getHeader("x-requested-with").equals("XMLHttpRequest");
}

Once we know the request type, we will send a non-200 HTTP status code if their is an authentication error. From the standards, the most appropriate status code is a 401 (Unauthorized). This code, however, comes with some baggage. On a non-ajax HTTP request all browsers will respond by displaying an authentication dialog. Most browsers don’t do this on ajax calls. Most. Safari decided that it should pop up the authentication dialog on ajax requests. Without going too deeply into how this is probably correct and we should be using HTTP to its full advantage and using HTTP authentication instead of a custom authentication scheme, I don’t want this behavior. I landed on using the much less appropriate 403 (Forbidden).

So, if the user isn’t authenticated (such as with an expired session), send the 403 response back to the browser:

if(isXMLHttpRequest(request))
{
  response.sendError(403);
}

Now we handle the 403 on the browser side of things:

new Ajax.Request('/search',
     {asynchronous:true,
       evalScripts:true,
       onException: function(req,exception) {
         alert('An exception "' + exception + '" was thrown accessing "' + request.url + '"' )
         return true
       },
       on403: function(t) {
         alert('Your session appears to have expired.  You will asked to log in again and returned here.')
         window.location.reload()
         return true
       },
       onFailure: function(request) {
         alert('An unhandled error occured ' + t.status + ': ' + t.statusText);
         return true
       },
       onLoading:function(request){Element.show('loading')},
       onComplete:function(request){Element.hide('loading')},
       onSuccess: function(t) {
         //do something with the results
       },
       parameters:Form.serialize(this)
     })

The interesting bit of this call is the on403 method. Prototype will call onXXX() where XXX is a non-200 HTTP response code. When a 403 is returned, I simply reload the current page. Since we handled the scenario of authenticating and returning to the requested page for the non-Ajax call, this ‘just works’. You may wish to do something else here, including some nifty ajax widget that authenticates in-line and makes the call again.

Now that we have an Ajax request that suits our needs, we will want to use it throughout our project. Instead of repeating the same code over and over again, let’s DRY it up.

Creating a Wrapper Class

Since I am going to use this scheme for Ajax.Request and Ajax.Updater I created two wrapper classes that use a common function for setting the options. This scheme allows you to override any of the default options I use in the wrapper:

Ajax.MyRequest = Class.create(Ajax.Request, {
  initialize: function($super, url, options) {
    $super(url,_options(options));
  }
})

Ajax.MyUpdater = Class.create(Ajax.Updater, {
  initialize: function($super, container, url, options) {
    $super({success:container},url,_options(options));
  }
})

function _options(options) {
  return Object.extend({
     asynchronous:true,
     evalScripts:true,
     onException: function(request,exception) {
       Element.hide('loading')
       alert('An exception "' + exception + '" was thrown accessing "' + request.url + '"' )
       return true
     },
     on403: function(t) {
       Element.hide('loading')
       alert('Your session appears to have expired.  You will asked to log in again and returned here.')
       window.location.reload()
       return true
     },
     onFailure: function(t) {
       Element.hide('loading')
       alert('An unhandled error occured ' + t.status + ': ' + t.statusText);
     },
     onLoading:function(request){Element.show('loading')},
     onComplete:function(request){window.setTimeout('Element.hide("loading");', 100);}
  }, options || {});
}

So not only can you lookup the current volume of Google stock in a spreadsheet (=GoogleFinance(”GOOG”, “volume”)) or find out Roger Clemens’s ERA (=GoogleLookup(”Roger Clemens”,”earned run average”)) you can access those values real time using an open API. I cobbled together a quick ruby program that can retrieve values from a private or public (published) Google spreadsheet.

The public sample should work for everyone since I published that spreadsheet for the world to see. You will need to provide your own credentials and spreadsheet key to use the authenticated example. I will caution that this is just a proof of concept that has limited functionality, weak parsing, and no error checking.

#!/usr/bin/env ruby

require 'net/http'
require 'net/https'
require 'uri'
require 'rubygems'
require 'hpricot'

#
# Make it east to use some of the convenience methods using https
#
module Net
  class HTTPS < HTTP
    def initialize(address, port = nil)
      super(address, port)
      self.use_ssl = true
    end
  end
end

class GoogleSpreadSheet
  def initialize(spreadsheet_key)
    @spreadsheet_key = spreadsheet_key
    @headers = nil
  end

  def authenticate(email, password)
    url = URI.parse('https://www.google.com/accounts/ClientLogin')
    response = Net::HTTPS.post_form(url,
      {'Email'=>email,
       'Passwd'=>password,
       'source'=>"ruby-ss-example-1",
       'service'=>'wise' })
    @headers = {
     'Authorization' => "GoogleLogin auth=#{response.body.split(/=/).last}",
     'Content-Type'  => 'application/atom+xml'
    }
  end

  def evaluate_cell(cell)
    path = "/feeds/cells/#{@spreadsheet_key}/1/#{@headers ? "private" : "public"}/basic/#{cell}"
    doc = Hpricot(request(path))
    result = (doc/"content[@type='text']").inner_html
  end

  private
  def request(path)
    http = Net::HTTP.new('spreadsheets.google.com', 80)
    response,data = http.get(path, @headers)
    data
  end
end

gs = GoogleSpreadSheet.new('p39irgfu5fsFokTPyPCG8Lg')
puts gs.evaluate_cell("A1")
puts gs.evaluate_cell("A2")
puts gs.evaluate_cell("B1")

gs = GoogleSpreadSheet.new(<your spreadsheet key>)
gs.authenticate(<your email>, <your password>)
puts gs.evaluate_cell("E132")

The fun really just starts here. You can edit existing spreadsheets too (although you can ‘t create or delete them yet). As Rich pointed out to me a few days ago, it won’t be too long before we are adding ‘export to Google spreadsheet’ functions to our web applications.

window.XMLHttpRequest == ! (not) IE

…they are in for some strange behavior (here, here, here, and here for example).

The problem occurs when the code is optimized for non-IE browsers and does not recreate the XMLHTTPRequest object on each event. Back when “window.XMLHttpRequest == !IE” was true, this code worked as expected: IE browsers created a new XMLHttpRequest object on each event and non-IE browsers reused the same XMLHttpRequest object. Now IE7 will correctly process the first event but then fails on the second (and subsequent) requests because you still can’t reuse XMLHttpRequest objects.

We chose to just remove the optimization all together and recreate the object on each event for all browsers. Others may wish to come up with some other check that will allow them to keep it.

rake tmp:sessions:clear # Clears all files in tmp/sessions

Anyway it was easy enough to clean up the stale ruby_sess files by going into the rails webapp/session directory and then running the following command:

find -type f -name "ruby_sess*" -exec rm -f {} \;

I’m not sure why the app is creating sessions but it isn’t something that stores state so I didn’t have to worry about killing active sessions here. If you do need to worry about that you will probably want to toss a time on the find command.

After looking a little more I found a post about this that has a ruby way of cleaning up the sessions.

So why not use Rails instead? The server that I am running the application on has limited resources. Rails is reported to need about 20 MB of memory and the node I have is already paging out memory to disk.

To emulate the webserver getting hit repeatedly with connections I created a connect_dbi.rb and and connect_ar.rb that insert the same row into a Tests database table. The ActiveRecord test also has the model file test.rb

The test was run using a PostGreSQL database on Ubuntu 5.10 with Ruby 1.8.4. The table was truncated between tests. I just wanted to get an idea of the performance difference between ActiveRecord and DBI. So this isn’t the strictest performance test but does provide an idea of the magnitude of difference.

The test was executed from the command line using “ruby -e” to call the connect_dbi.rb and connect_ar.rb files 100 times each. This creates a new database connection each time the script is executed. ActiveRecord generates a series of error messages about unavailable database adapters each time it is started so these messages were capatured in the activeError file. DBI doesn’t produce any errors but I wanted to keep the execution as similar as possible so there is also a dbiError file created.

connect_dbi.rb

#! /usr/bin/ruby -w
require 'rubygems'
require 'dbi'

dbh = DBI.connect('dbi:Pg:tmp_test:localhost','ruby_user','ruby_pass')

begin

dbh.transaction do |dbh|
dbh.prepare("INSERT INTO tests(rank, created_at, message) VALUES(?,?,?)") do |sth|
sth.execute(5, Time.now, 'abcdefghijklmnopqrstuvwxyz abcdefghijklmnopqrstuvwxyz abcdefghijklmnopqrstuvwxyz abcdefghijklmnopqrstuvwxyz abcdefghijklmnopqrstuvwxyz ')
end
end

rescue DBI::ProgrammingError => exception
puts "Exception: #{exception.message}"
end

connect_ar.rb

#! /usr/bin/ruby -w

require 'rubygems'
require_gem 'activerecord'

ActiveRecord::Base.establish_connection(
:adapter  => "postgresql",
:host     => "localhost",
:username => "ruby_user",
:password => "ruby_pass",
:database => "tmp_test"
)

test = Test.new("rank" => 5, "created_at" => "#{Time.now}", "message" => 'abcdefghijklmnopqrstuvwxyz abcdefghijklmnopqrstuvwxyz abcdefghijklmnopqrstuvwxyz abcdefghijklmnopqrstuvwxyz abcdefghijklmnopqrstuvwxyz ')
test.save

test.rb

class Test < ActiveRecord::Base

end

The GNU/Linux time command was used to obtain timings.

As the results show DBI appears to be about three times faster than using ActiveRecord. This is most likely because of the initialization ActiveRecord does each time the connection is established. Considering that the test is only using one table and one ActiveRecord model class this is a fairly significant difference.

~/projects/dbi_test $ time ruby -e '100.times { %x{./connect_ar.rb}}' 2> activeError

real    0m25.916s
user    0m21.575s
sys     0m2.807s

~/projects/dbi_test $ time ruby -e '100.times { %x{./connect_dbi.rb}}' 2> dbiError

real    0m8.154s
user    0m6.620s
sys     0m0.733s

All results are in seconds.  Each test was run ten times.
DBI Results



Mean
Standard Deviation


Real
7.7133
0.9448


User
6.2027
0.8530


Sys
6.2027
0.2760


ActiveRecord Results



Mean
Standard Deviation


Real
26.9003
1.8849


User
21.9256
1.4959


Sys
2.8038
0.4610

To understand the way this works I think it is easiest to break it down into parts:

1. A file upload extention that counts bytes as they are uploaded
2. An interface that monitors the progress of something running on the server
3. AJAX to pull the monitoring into the current screen

Counting bytes when files are uploaded

This was taken from the example listed above. It extends and wraps parts of the commons File Upload classes so that you can count the bytes as they are uploaded to the server. You can download the source with build file or the binary. You will also need the commons file upload, commons io and commons logging. If you download the source put the commons jars in the lib directory before building.

The code is fairly simple to follow. MonitoredDiskFileItemFactory replaces DiskFileItemFactory and the construction of a MonitoredDiskFileItemFactory takes a OutputStreamListener that will be passed on down the chain. The new factory creates MonitoredDiskFileItems instead of DiskFileItems for each file uploaded. When the file needs to be written to disk a MonitoredOutputStream is given back instead of a normal OutputStream. The MonitoredOutputStream calls the OutputStreamListener methods as the bytes are written and with that you now have a way to monitor the byte count as the file is created on the server.

Now to test this all out we can just have an OutputStreamListener that writes its progress out to a logfile or something.

public class FileUploadListener implements OutputStreamListener
{
  private long totalFileSize;
  private long currentFileRead;

  public FileUploadListener(long totalFileSize)
  {
    this.totalFileSize = totalFileSize;
    this.currentFileRead = 0;
  }

  public void start()
  {
    log.debug("Upload started. Total file size: " + totalFileSize);
  }

  public void bytesRead(int byteCount)
  {
    log.debug("Read bytes. Currently " + byteCount + " out of " + totalFileSize + " bytes.");
    currentFileRead+=byteCount;
  }

  public void error(String error)
  {
    log.debug("Hit an error: " + error);
  }

  public void done()
  {
    log.debug("Upload done.");
  }

  public long getTotalRead()
  {
    return currentFileRead;
  }

  public long getTotalSize()
  {
    return totalFileSize;
  }
}

Now we try it out. You can put this in a servlet or jsp so I’m only going to list the parts that matter.

  FileUploadListener listener = new FileUploadListener(request.getContentLength());
  session.setAttribute("LISTENER", listener);
  FileItemFactory factory = new MonitoredDiskFileItemFactory(listener);
  ServletFileUpload upload = new ServletFileUpload(factory);
  List items = upload.parseRequest(request);
  for (Iterator i = items.iterator(); i.hasNext();)
  {
    FileItem fileItem = (FileItem) i.next();
    if (!fileItem.isFormField())
    {
       // code here to process the file
     }
   }

I’m going to assume you can find the correct way to do the actual form upload part.

Note: One issue that you will face at some point is where the upload post goes to becuase when you get to the AJAXy part of things you want the post to stay on the same page. You can use a hidden iframe and the form’s “target” parameter to do this (I have an example later). This is one thing the Java examples I found didn’t have but the PHP examples did and I’m not sure exactly how the Java examples work without it.

Monitoring progress on the server

The next step is to monitor the progress of the upload on the server. What you are monitoring on the server doesn’t even need to be the upload. For the work I was doing the upload goes fairly quickly but what happens to the file after the upload takes a little longer. I wanted to monitor both and that is one reason I think it helps to break this up into parts because you aren’t limited to just monitoring file uploads.

The main thing to keep in mind here is that the application server is multithreaded and you can make more than one request to the server at the same time. You probably know that you can open a tab in firefox or another window in ie and use the same session from the current webapp you are using. Knowing that you can create a page that monitors the status of things as they are running on the server.

From the example above you could toss the listener into the users session. Then insead of logging you just add a couple variables to keep track of the number of bytes that have been uploaded. Then create a simple jsp that pulls the Listener out of the session and dumps its data to a page. Open two windows, one to the upload page and another one to the status page. Start the upload and then start refreshing the status. You should see that the values change as the file is uploaded.

<%@page%>
<%
  FileUploadListener listener = (FileUploadListener)session.getAttribute("LISTENER");
%>
Total size: <%=listener.getTotalSize()%><br/>
Read count: <%=listener.getTotalRead()%><br/>

Of course you will probably want more than just the total size and bytes read as well as more formating like a little progress bar or something but I’ll leave that up to you.

AJAX integration with prototype

You have the major parts to the upload progress done and now all you need is the AJAX part. To do this I chose to use prototype because it cuts right to what you want to do. One call is all you need to use: Ajax.PeriodicalUpdater.

The Ajax.PeriodicalUpdater call will update a container (in my case a div) on a set interval. Here is an example of how to have it update a div with an id of “status” every second.

    new Ajax.PeriodicalUpdater(
                                'status',
                                'status.jsp',
                                {asynchronous:true, frequency:1, method:'get'});

The first argument is the id of the div, the second is the jsp that contains the data to stick into the div every second and the 3rd arguement is a set of options. There are more options availabe if you need them.

You would want to kick the update off whenever the form is posted. When the post is complete the iframe used as a place to post to will load with the results of the servlet or jsp that you posted to. If you return some javascript as a result for the iframe you will be able to create a final “finished” message on the page to let the user know the upload has completed and stop the processing of the AJAX updater.

So there you have it. The basics of setting up an upload progress bar using java and AJAX. I have left out a good bit but you should have enough to at least get you started.

By request I have created a simple example that pulls everything together. The source contains everything you need to create a war file including all source and an ant build file.

Example Source