while true; do kill -USR1 `pidof dd`; sleep 2; done

(NOTE: if `pidof dd` doesn’t work for you, just use the process id directly)

Switch back to the terminal where dd is running and you should see:

9902751744 bytes (9.9 GB) copied, 732.883 s, 13.5 MB/s
9469+0 records in
9468+0 records out
9927917568 bytes (9.9 GB) copied, 734.914 s, 13.5 MB/s
9496+0 records in
9495+0 records out
9956229120 bytes (10 GB) copied, 736.941 s, 13.5 MB/s

updating every couple of seconds.

cap --set-before branch=testbranch  deploy

…where ‘testbranch’ is the name of my subversion branch, and ‘deploy’ is the action to take. The ‘–set-before option’ sets a variable before the recipes are loaded (sort of). All you need to do to make this work you just need to set the repository URL correctly based on if the branch variable exists or not. In my case, that is editing the top of deploy.rb in the config section of my rails app:

if variables.include?(:branch)
  set :repository,  "http://svn.example.com/project/branches/#{branch}"
else
  set :repository,  "http://svn.example.com/project/trunk"
end

Handling session timeouts on non-Ajax requests

The first step to handing session timeouts with AJAX is to handle them for non-Ajax requests first. One way of doing this is trapping the URL the user is attempting to access (but not authenticated to), redirecting them to a login page, and upon success, redirecting them back to their original request.

If they aren’t authenticated, send them to the login page:

if(!loggedIn())
{
  response.sendRedirect("/login.jsp?uri=" + URLEncoder.encode(fullURI, "UTF8"));
}

Hide the original request in the page:

">
  ID:

  Password:

…and after a successful signup, send them back to where they wanted to go:

if(authenticated())
{
  response.sendRedirect(request.getParameter("uri"));
}

Handling Session Timeouts for Ajax Requests

The first thing we need to know is if it is an ajax request or not. Since I am going to use prototype for all my ajax calls it is as straight forward as looking for the special header prototype sends on each request:

private boolean isXMLHttpRequest(HttpServletRequest request)
{
  return request.getHeader("x-requested-with") != null && request.getHeader("x-requested-with").equals("XMLHttpRequest");
}

Once we know the request type, we will send a non-200 HTTP status code if their is an authentication error. From the standards, the most appropriate status code is a 401 (Unauthorized). This code, however, comes with some baggage. On a non-ajax HTTP request all browsers will respond by displaying an authentication dialog. Most browsers don’t do this on ajax calls. Most. Safari decided that it should pop up the authentication dialog on ajax requests. Without going too deeply into how this is probably correct and we should be using HTTP to its full advantage and using HTTP authentication instead of a custom authentication scheme, I don’t want this behavior. I landed on using the much less appropriate 403 (Forbidden).

So, if the user isn’t authenticated (such as with an expired session), send the 403 response back to the browser:

if(isXMLHttpRequest(request))
{
  response.sendError(403);
}

Now we handle the 403 on the browser side of things:

new Ajax.Request('/search',
     {asynchronous:true,
       evalScripts:true,
       onException: function(req,exception) {
         alert('An exception "' + exception + '" was thrown accessing "' + request.url + '"' )
         return true
       },
       on403: function(t) {
         alert('Your session appears to have expired.  You will asked to log in again and returned here.')
         window.location.reload()
         return true
       },
       onFailure: function(request) {
         alert('An unhandled error occured ' + t.status + ': ' + t.statusText);
         return true
       },
       onLoading:function(request){Element.show('loading')},
       onComplete:function(request){Element.hide('loading')},
       onSuccess: function(t) {
         //do something with the results
       },
       parameters:Form.serialize(this)
     })

The interesting bit of this call is the on403 method. Prototype will call onXXX() where XXX is a non-200 HTTP response code. When a 403 is returned, I simply reload the current page. Since we handled the scenario of authenticating and returning to the requested page for the non-Ajax call, this ‘just works’. You may wish to do something else here, including some nifty ajax widget that authenticates in-line and makes the call again.

Now that we have an Ajax request that suits our needs, we will want to use it throughout our project. Instead of repeating the same code over and over again, let’s DRY it up.

Creating a Wrapper Class

Since I am going to use this scheme for Ajax.Request and Ajax.Updater I created two wrapper classes that use a common function for setting the options. This scheme allows you to override any of the default options I use in the wrapper:

Ajax.MyRequest = Class.create(Ajax.Request, {
  initialize: function($super, url, options) {
    $super(url,_options(options));
  }
})

Ajax.MyUpdater = Class.create(Ajax.Updater, {
  initialize: function($super, container, url, options) {
    $super({success:container},url,_options(options));
  }
})

function _options(options) {
  return Object.extend({
     asynchronous:true,
     evalScripts:true,
     onException: function(request,exception) {
       Element.hide('loading')
       alert('An exception "' + exception + '" was thrown accessing "' + request.url + '"' )
       return true
     },
     on403: function(t) {
       Element.hide('loading')
       alert('Your session appears to have expired.  You will asked to log in again and returned here.')
       window.location.reload()
       return true
     },
     onFailure: function(t) {
       Element.hide('loading')
       alert('An unhandled error occured ' + t.status + ': ' + t.statusText);
     },
     onLoading:function(request){Element.show('loading')},
     onComplete:function(request){window.setTimeout('Element.hide("loading");', 100);}
  }, options || {});
}

For my example I chose the latter. Quoted-printable has two distinct advantages: 1) It should be supported by all servers and 2) Unlike base64 it will keep characters that are 7bit ascii in ascii and only encode characters it needs to.

CREATE OR REPLACE PROCEDURE MAIL_TEST IS
  lv_server      VARCHAR2(255) := '';
  lv_rcpt        VARCHAR2(255) := 'to@example.com';
  lv_from        VARCHAR2(255) := 'from@example.com';
  lv_subject     VARCHAR2(255) := 'Le chien paresseux a sauté';
  lv_message     VARCHAR2(255) := 'Voix ambiguë d''un coeur qui au zéphyr préfère les jattes de kiwis';
  lv_conn        UTL_SMTP.CONNECTION;
BEGIN
  lv_Conn := UTL_SMTP.Open_Connection(lv_server);
  UTL_SMTP.Helo(lv_conn, lv_server);
  UTL_SMTP.Mail(lv_conn, lv_from);
  UTL_SMTP.Rcpt(lv_conn, lv_rcpt);
  UTL_SMTP.OPEN_DATA(lv_conn);
  UTL_SMTP.WRITE_DATA(lv_conn, 'Subject: =?UTF-8?Q?' ||
                                UTL_RAW.CAST_TO_VARCHAR2(UTL_ENCODE.QUOTED_PRINTABLE_ENCODE(UTL_RAW.CAST_TO_RAW(lv_subject))) ||
                                '?=' || UTL_TCP.CRLF);
  UTL_SMTP.WRITE_DATA(lv_conn, 'MIME-version: 1.0' || UTL_TCP.CRLF);
  UTL_SMTP.WRITE_DATA(lv_conn, 'Content-Type: text/html;charset=utf-8' || UTL_TCP.CRLF);
  UTL_SMTP.WRITE_DATA(lv_conn, 'Content-Transfer-Encoding: quoted-printable '|| UTL_TCP.CRLF);
  UTL_SMTP.WRITE_DATA(lv_conn, 'Date: ' || TO_CHAR(SYSDATE, 'dd Mon yy hh24:mi:ss' ) ||' -0800 (GMT)' || UTL_TCP.CRLF);
  UTL_SMTP.WRITE_DATA(lv_conn, 'From: ' || lv_from || UTL_TCP.CRLF);
  UTL_SMTP.WRITE_DATA(lv_conn, 'To: ' || lv_rcpt || UTL_TCP.CRLF);
  UTL_SMTP.WRITE_DATA(lv_conn,  UTL_TCP.CRLF);
  UTL_SMTP.WRITE_RAW_DATA(lv_conn, UTL_ENCODE.QUOTED_PRINTABLE_ENCODE(UTL_RAW.CAST_TO_RAW(lv_message)));
  UTL_SMTP.WRITE_DATA(lv_conn, UTL_TCP.CRLF);
  UTL_SMTP.CLOSE_DATA(lv_conn);
  UTL_SMTP.QUIT(lv_conn);
END;

If you wanted to use 8bit ascii you would change the data and transfer encoding header to this:

UTL_SMTP.WRITE_DATA(lv_conn, 'Content-Transfer-Encoding: 8bit'|| UTL_TCP.CRLF);
UTL_SMTP.WRITE_RAW_DATA(lv_conn, UTL_RAW.CAST_TO_RAW(UTL_TCP.CRLF || lv_message || UTL_TCP.CRLF));

Finally, you can also use base64 encoding for the subject:

  UTL_SMTP.WRITE_DATA(lv_conn, 'Subject: =?UTF-8?B?' ||
                                UTL_RAW.CAST_TO_VARCHAR2(UTL_ENCODE.BASE64_ENCODE(UTL_RAW.CAST_TO_RAW(lv_subject))) ||
                                '?=' || UTL_TCP.CRLF);

Calling me a PL/SQL developer would be an insult to PL/SQL developers everywhere. I apologize in advance for any perceived lack of style. Most of the variables would be sourced from parameters passed in to the procedure or database values.

At first I decided my test would just respond with the yamlized request (just like the mongrel sample did). This worked fine but there was a significant size difference in the two documents (WEBrick’s was much bigger). In the end, I just decided to respond with a few bytes of text.

The Code

mongrel_service.rb:

require 'rubygems'
require 'mongrel'
require 'yaml'

class SimpleHandler < Mongrel::HttpHandler
  def process(request, response)
    response.start do |head,out|
      head["Content-Type"] = "text/html"
      out << "I am mongrel, hear me roar"
    end
  end
end

config = Mongrel::Configurator.new :host => "localhost", :port => 4000 do
  listener {uri "/", :handler => SimpleHandler.new}
  trap("INT") { stop }
  run
end

config.join

webrick_service.rb:

require 'webrick'
require 'yaml'
include WEBrick

class SimpleServlet < HTTPServlet::AbstractServlet
  def do_GET(request, response)
    response["Content-Type"] = "text/html"
    response.body = "I am WEBrick, hear me roar"
  end
end

s = HTTPServer.new( :Port => 4001 )
s.mount("/", SimpleServlet)
trap('INT'){ s.shutdown }
s.start

The Test

There are serveral good benchmarking tools available, but I had apache bench (ab) handy, so that is what I used. I just used the stock gem install of mongrel and the WEBrick that comes with ruby 1.8.4 (the one non-stock change I made was to comment out access logging for WEBrick since mongrel doesn’t log by default).

The Results

The Conclusion

Looks like mongrel it is….

[emerg] (45)Deadlock situation detected/avoided: apr_proc_mutex_lock failed. Attempting to shutdown process gracefully.

Not good. After a little digging we found that the default AcceptMutex switched from pthread to fcntl between apache 2.0.49 and 2.0.52.

Adding:

AcceptMutex pthread

to httpd.conf cleared up the errors and we’ve had smooth sailing ever since.

So not only can you lookup the current volume of Google stock in a spreadsheet (=GoogleFinance(“GOOG”, “volume”)) or find out Roger Clemens’s ERA (=GoogleLookup(“Roger Clemens”,”earned run average”)) you can access those values real time using an open API. I cobbled together a quick ruby program that can retrieve values from a private or public (published) Google spreadsheet.

The public sample should work for everyone since I published that spreadsheet for the world to see. You will need to provide your own credentials and spreadsheet key to use the authenticated example. I will caution that this is just a proof of concept that has limited functionality, weak parsing, and no error checking.

#!/usr/bin/env ruby

require 'net/http'
require 'net/https'
require 'uri'
require 'rubygems'
require 'hpricot'

#
# Make it east to use some of the convenience methods using https
#
module Net
  class HTTPS < HTTP
    def initialize(address, port = nil)
      super(address, port)
      self.use_ssl = true
    end
  end
end

class GoogleSpreadSheet
  def initialize(spreadsheet_key)
    @spreadsheet_key = spreadsheet_key
    @headers = nil
  end

  def authenticate(email, password)
    url = URI.parse('https://www.google.com/accounts/ClientLogin')
    response = Net::HTTPS.post_form(url,
      {'Email'=>email,
       'Passwd'=>password,
       'source'=>"ruby-ss-example-1",
       'service'=>'wise' })
    @headers = {
     'Authorization' => "GoogleLogin auth=#{response.body.split(/=/).last}",
     'Content-Type'  => 'application/atom+xml'
    }
  end

  def evaluate_cell(cell)
    path = "/feeds/cells/#{@spreadsheet_key}/1/#{@headers ? "private" : "public"}/basic/#{cell}"
    doc = Hpricot(request(path))
    result = (doc/"content[@type='text']").inner_html
  end

  private
  def request(path)
    http = Net::HTTP.new('spreadsheets.google.com', 80)
    response,data = http.get(path, @headers)
    data
  end
end

gs = GoogleSpreadSheet.new('p39irgfu5fsFokTPyPCG8Lg')
puts gs.evaluate_cell("A1")
puts gs.evaluate_cell("A2")
puts gs.evaluate_cell("B1")

gs = GoogleSpreadSheet.new(<your spreadsheet key>)
gs.authenticate(<your email>, <your password>)
puts gs.evaluate_cell("E132")

The fun really just starts here. You can edit existing spreadsheets too (although you can ‘t create or delete them yet). As Rich pointed out to me a few days ago, it won’t be too long before we are adding ‘export to Google spreadsheet’ functions to our web applications.

window.XMLHttpRequest == ! (not) IE

…they are in for some strange behavior (here, here, here, and here for example).

The problem occurs when the code is optimized for non-IE browsers and does not recreate the XMLHTTPRequest object on each event. Back when “window.XMLHttpRequest == !IE” was true, this code worked as expected: IE browsers created a new XMLHttpRequest object on each event and non-IE browsers reused the same XMLHttpRequest object. Now IE7 will correctly process the first event but then fails on the second (and subsequent) requests because you still can’t reuse XMLHttpRequest objects.

We chose to just remove the optimization all together and recreate the object on each event for all browsers. Others may wish to come up with some other check that will allow them to keep it.