I have a confession to make.

You know that really popular design patterns book that most software developers (especially in the Java world) claim to have read? You know the one I’m talking about. The one written by the “Gang of Four”. The one you were supposed to have read during your junior or senior year of college. Yeah, this one.

I’ve never read it.

Sure, I’ve owned it. I have even thumbed through it a few times. But I’ve never sat down with any real intention of absorbing the information within its covers. The problem for me personally was that when I first opened Design Patterns I was just starting my software development career. I was too young and hadn’t seen enough code to really comprehend the authors’ message or understand the need for such a book.

A few years have passed since then and while it’s still early in my career I now have a much better idea of how to develop software and have a much better understanding of Object Oriented Design. Even though I spend a significant amount of my free time developing my skills and reading about software development I have yet to read the seminal Design Patterns book.

Design patterns have left a bad taste in my mouth. Over the years they have been abused and misunderstood. They seem to have become caught up in the all too familiar silver bullet buzzword vernacular of the IT industry (SOA anyone?). So I simply avoided them. Unless I was about to interview for a new job, design patterns never so much as crossed my mind. Repeat after me: Singleton, Factory, MVC. You’re hired!

But things have changed. Over the past year and a half or so I’ve really started to focus on my career as a software developer. I’ve learned new operating systems and languages, created this blog, and started attending local developer meetings on a regular basis. While I still have my favorite technologies and interests, I’ve been doing a much better job of keeping an open mind about trying out new development tools and methodologies.

Enter Russ Olsen. In addition to being a great developer and all around good guy, Russ is someone who I had the honor of working with at the beginning of my career. He was very much a mentor and I credit him with teaching me many of the applied fundamentals of software development. So what does Russ have to do with design patterns? Well it just so happens that a few years after I left the company where Russ and I worked together he wrote a book called Design Patterns in Ruby. Because of my respect for Russ and my fondness of Ruby I decided I’d give the book a shot. While I was certainly expecting to learn some new things, I was surprised (although I shouldn’t have been) by how much great information I was able to extract from this book.

In their most minimal form, design patterns are simply reusable solutions that have been created to solve common coding problems. While this may seem intuitive, it was actually somewhat of a revelation to me – a person who has ignored the direct usage of design patterns for all these years. I see now that given an appropriate situation design patterns can be used to create consistent, maintainable code. The important thing to remember is that each pattern has a specific use and should only be used when applicable.

Although the title of the book is Design Patterns in Ruby, a majority of the principles and patterns apply to many other object oriented languages as well. In fact, while I was reading Russ’s book I was in the middle of working on a small Java project. This project had to keep track of rapid state changes on a particular website. I needed to keep track of a countdown timer, a list of participants, and an aggregate score. Each of these pieces of information was updated independently, but I needed to make decisions in my code based on the combination of this information at any given time. After reading the chapter on the Observer pattern, I realized it would be the perfect fit. Prior to using the pattern my code was a huge pile of tangled spaghetti. Afterwards, it was clean and easy to read.

I really liked the way the chapters were organized in Design Patterns in Ruby. Russ started each chapter by posing a common programming problem. Then he would explain how a specific design pattern could be used to solve that problem. As the chapter progressed Russ would work through the problem using examples that were very easy to understand. Each chapter ended with real world examples of the patterns as they have been implemented in the Ruby core libraries or various open source projects. I found the real world code samples to be especially useful as compliments to the examples given in the book.

Not only did I enjoy reading this book because it was written by Russ, but I genuinely feel like it has helped to make me a better programmer. In addition, it has helped increase my Ruby specific programming skills. Learning to program in Ruby is easy, but learning to program the “Ruby Way” is a much greater challenge. Ruby is full of powerful idioms that are not necessarily obvious without a deep examination of the language. Russ’s book helps enlighten would be Ruby programmers to many of Ruby’s powerful language features.

For example, compare an implementation of the Proxy pattern written in Java and Ruby:

Java Version

MathService.java

public interface MathService {
    public void add(int x, int y);
    public void subtract(int x, int y);
}

MathServiceImpl.java

public class MathServiceImpl implements MathService {
    public void add(int x, int y) {
        System.out.println("Result ("+x+ " + "+y+"): " + (x + y));
    }

    public void subtract(int x, int y) {
        System.out.println("Result ("+x+ " - "+y+"): " + (x - y));
    }
}

MathServiceProxy.java

public class MathServiceProxy implements MathService {
    private MathService delegate;

    public MathServiceProxy(MathService ms){
        this.delegate = ms;
    }

    public void add(int x, int y) {
        System.out.println("Call made to the add method");
        delegate.add(x, y);
    }

    public void subtract(int x, int y) {
        System.out.println("Call made to the subtract method");
        delegate.subtract(x, y);
    }
}

MathServiceDriver.java

public class MathServiceDriver {
    public static void main(String[] args) {
        MathService ms = new MathServiceProxy(new MathServiceImpl());
        ms.add(1, 2);
        ms.subtract(4, 3);
    }
}

Ruby Version

math_service.rb

class MathService
  def add(x, y)
    puts "Result (#{x} + #{y}): #{x + y}"
  end

  def subtract(x, y)
    puts "Result (#{x} - #{y}): #{x - y}"
  end
end

class MathServiceProxy
  def initialize(math_service)
    @delegate = math_service
  end

  def method_missing(name, *args)
    puts "Call made to the #{name} method"
    @delegate.send(name, *args)
  end
end

ms = MathServiceProxy.new(MathService.new)
ms.add(1,2)
ms.subtract(4,3)

Result of Running Either Program

Call made to the add method
Result (1 + 2): 3
Call made to the subtract method
Result (4 - 3): 1

In this example we use the Proxy pattern to print out a message before each method call. If we decided to add any methods to our MathService interface we would need to update our MathServiceImpl and MathServiceProxy Java classes as well. However, by using Ruby’s built in method_missing method we have the ability to wrap each MathService method call dynamically without having to write a new method in our MathServiceProxy class. Ruby removes the tedium of implementing the Proxy pattern.

This is just one small example of the flexibility and power provided by the Ruby language. Before reading Russ’s book I didn’t necessarily “get” many of Ruby’s idioms. Reading Design Patterns in Ruby has certainly helped solidify my understanding and is great for programmers who would like to take their Ruby skills to the next level.

Code Samples

All code samples from this blog entry can be downloaded from GitHub by clicking here

Since most web pages are simply made of HTML it is easy for a computer to parse and store the information contained within these documents. Each programming language commonly has a host of libraries to assist in the screen scraping/parsing process and Ruby is no exception. To create simple screen scrapers in Ruby I have been using a library called scRUBYt!. scRUBYt! provides methods to access a given website and scrape its content. All the programmer needs to do is provide the XPath string to the desired information.

Using the scRUBYt! library has allowed me to write a small screen scraper script to access the FranklyMLS.com website, check for new listings, and then report back with the results. This has saved me a lot of time and effort. Let's dive into some code to see how this is done.

First, we'll need to create a simple class to store the information that we scrape from the FranklyMLS.com website. The Property class will hold various property related information (price, MLS number, square footage, etc.):

class Property
  attr_accessor :mls, :list_price, :dom, :address,
                :city, :zip, :bed, :bath, :sqft, :built

  def initialize(property)
    @mls = (property/:mls).inner_html
    @list_price = (property/:list_price).inner_html
    @dom = (property/:dom).inner_html
    @address = (property/:address).inner_html
    @city = (property/:city).inner_html
    @zip = (property/:zip).inner_html
    @bed = (property/:bed).inner_html
    @bath = (property/:bath).inner_html
    @sqft = (property/:sqft).inner_html
    @built = (property/:built).inner_html
  end
end

Next we'll need to make sure that scRUBYt! is installed. If you don't already have Github set up as one of your gem repositories do so now by executing the following command:

gem sources -a http://gems.github.com

Then install the scRUBYt! gem:

gem install jspradlin-scrubyt

Side note: I've built some functionality into the scRUBYt! library so you will need to grab the gem from my Github repository (i.e. jspradlin-scrubyt). I've spoken with the lead developer on the scRUBYt! project and it looks like my changes might make it into a future version of the official gem.

At this point we need to give scRUBYt! the URL of a website that we wish to scrape. The FranklyMLS.com website has its own special URL query syntax which displays only properties that meet our specific criteria. For example, if we only wanted to find active listings in the following zip codes - 22201, 22202, 22203 - the FranklyMLS.com URL would be:

http://franklymls.com/default.aspx?m=R&s=(22201,22202,22203)+active

We can dynamically generate a URL with our specific housing criteria by including the following code in our script:

# Generate the URL for FranklyMLS.com given
# the following criteria:
zips = [22201,22202,22203,22209]
beds = [2, 3].collect{ |bed| bed.to_s+'bdr'}
min_price = 150 #in thousands
max_price = 350 #in thousands
exclusions = ['JEFFERSON'].collect{ |exclude| "+-#{exclude}"}

fmls_url = "http://franklymls.com/default.aspx?"
fmls_url += "m=R&l=#{min_price}K&h=#{max_price}K"
fmls_url += "&s=(#{zips.join(',')})+active"
fmls_url += "+(#{beds.join(',')})"
fmls_url += exclusions.to_s

Now we're ready to scrape some housing data. Once the FranklyMLS.com property page loads we are presented with a table that contains information about the listings that meet our criteria (image modified to save space):

The HTML that generates this table would appear like this (modified to save space):

<table id="dgRealtorStyle">
  ...
  <tr style="display:visible">
    <td><a>...</a><a>AR6552162</a></td><!-- td[1]-->
    <td>$256,000</td><!-- td[2]-->
    <td>$339,000</td>
    <td>$</td>
    <td> </td>
    <td> </td>
    <td>562</td>
    <td>1931 CLEVELAND #313</td><!-- td[8]-->
    ...
    <td>ARLINGTON</td>
    <td>22201</td>
    <td> </td>
    <td>CLEVLAND HO</td>
    <td>3/1</td>
    <td>860</td>
    <td>1960</td>
    <td>...</td>
    <td>1</td>
    <td>160</td>
    <td><a>x</a></td>
  </tr>
  ...
</table>

Finally, the Ruby code to scrape this data using scRUBYt!:

#Scrape the FranklyMLS.com website using scRUBYt!
property_data = Scrubyt::Extractor.define do
  fetch fmls_url

  properties '//table[@id="dgRealtorStyle"]' do
    property "//tr" do
      mls "/td[1]//a[2]"
      list_price "/td[2]"
      dom "/td[7]"
      address "/td[8]"
      city "/td[9]"
      zip "/td[10]"
      bed "/td[13]",
        :format_output => lambda {|bed_bath| bed_bath.split('/')[0]}
      bath "/td[13]",
        :format_output => lambda {|bed_bath| bed_bath.split('/')[1]}
      sqft "/td[14]"
      built "/td[15]"
    end
  end
end

If you look at the table, the HTML code, and the Ruby code you'll see that I've color coordinated each separate piece of information to illustrate how it is parsed and then stored. The scRUBYt! library will "fetch" the given URL, locate the HTML elements by the given XPath, and then store the data.

Once we have all of the data collected we may want to do something useful with the information such as convert it into an RSS feed. We can accomplish this by using the Hpricot and Builder libraries (which should be installed as dependencies of scRUBYt!). The code for the RSS conversion would look like this:

# Read in the XML generated by scRUBYt! then
# convert the data into Property objects
# and store them in the property_hash.
property_hash = {}

hp = Hpricot.XML(property_data.to_xml)
(hp/:property).each do |property|
  property_hash[(property/:mls).inner_html] = Property.new(property)
end

# Using the Builder library, iterate through
# the property_hash to generate an RSS feed.
xml = Builder::XmlMarkup.new(:target => $stdout, :indent => 2 )
xml.instruct! :xml, :version => "1.0"
xml.rss :version => "2.0" do
  xml.channel do
    xml.title "Property Feed"
    xml.link "<--YOUR URL-->"
    xml.description "This is my property feed"

    property_hash.each do | key, property |
      pub_date = (Time.now - property.dom.to_i*60*60*24)
      pub_date = pub_date.strftime("%a, %d %b %Y %I:%M:%S")

      xml.item do
        xml.title property.address
        xml.link "http://franklymls.com/#{property.mls}"
        xml.pubDate "#{pub_date} EST"
        xml.description "City: #{property.city}
          Address: #{property.address}
          Price: #{property.list_price}
          Bed: #{property.bed}
          Bath: #{property.bath}
          Sqft: #{property.sqft}
          Built: #{property.built}"
      end
    end
  end
end

To make sure I get routine updates, I run this Ruby code on my server every hour using a cron job and pipe its output to an RSS feed. I am subscribed to the generated RSS feed so now I know exactly when a new property becomes available in my area!

Overall scRUBYt! is very easy to use and for simple screen scraping tasks it should work fine. However, I have found that it can run into some problems when the HTML gets complex. In these cases I would recommend using Hpricot for fine-level scraping.

To view the source code for this entry and to view other screen scraping code that I have written check out my Github page.

If you'd like to look at another example of scRUBYt! in action, feel free to read the post I wrote for my company's blog by clicking here.

Opening Keynote - Rich Kilmer

Rich Kilmer's opening keynote was by far the best presentation of the day. Rich is an incredibly experienced and skilled Ruby developer who was one of the earlier adopters of the language. During his keynote, Rich walked through his experiences with Ruby dating all the way back to 2001. He has certainly worked on some interesting projects for impressive government agencies including DARPA and the United State Air Force. Rich also talked about projects he has created for the Ruby community including FreeRIDE - a Ruby IDE and RubyForge - a code repository for Ruby projects.

One particularly interesting thing Rich mentioned was that Ruby is becoming a mainstream language. I personally find this both exciting and scary. It is exciting that the community is growing at that there will likely be more opportunities to work on Ruby projects professionally, but it is a bit worrisome that the community could start to be overpopulated with run-of-the-mill developers transitioning from other dying mainstream languages.

Ruby Puzzlers - Mike Furr

Mike Furr, a doctoral student at UMCP, gave a presentation on edge case scenarios that occur in the Ruby language. Since I'm pretty new to Ruby I struggled to follow along, but Mike did do a good job of explaining how Ruby code reacted under various situations. Even though it can be rare to deal with edge cases in a language, it is nice to know how to avoid them or work your way out of them if they do arise.

Practical JRuby - David Keener

A Java(ish) presentation at a Ruby conference? It may seem a bit strange at first, but it actually makes a lot of sense. Java as a platform has some pretty rock solid features including performance, garbage collection, and concurrency. A lot time and thought has also been invested into Java libraries and corporate infrastructure so in many ways it makes sense to take advantage of what is already available. David Keener hit on all of these points in his presentation and gave an example of how to integrate JRuby with the JFreeChart Java library.

Tools for Your Ruby Toolbox - Dave Bock

Dave Bock is another FGM employee that I had the pleasure of working with a few years back. Dave has since moved on and started his own shop called Codesherpas. In his presentation, Dave talked about 3 lightweight tools that can be used to quickly accomplish tasks in Ruby. Dave first talked about StaticMatic, a framework for developing simple static websites using templates, but without using all of the built in functionality of Rails. Another web framework Dave talked about was Sinatra. Again, this framework allows developers to create simple web applications without the full Rails stack, but it also includes some RESTful functionality. Finally, Dave talked about GServer, a generic server library that allows developers to easily create server applications of their own. During his presentation Dave demoed a quick "Knock Knock" client he created using the GServer library.

Lightning Talks

There were five lightening talks given yesterday, but again Bryan Liles stole the show with his presentation on Project Management. The basic takeaway: meet with your reports regularly, project confidence, and set up processes that are easy to follow. Sounds so simple doesn't it?

Closing Keynote: Bad Ruby - Stuart Halloway

Stuart Holloway closed RubyNation with a talk on Bad Ruby. Playing off of a point made earlier in Rich Kilmer's talk about Ruby becoming mainstream, Stuart noted that now was the time for Ruby developers to begin to establish best practices for the language so that common pitfalls can be avoided when mainstream developers start to migrate to Ruby. (Assuming of course "mainstream" developers actually migrate to Ruby).

Conference Wrap Up

Overall the conference was pretty great. Day two was a little less energetic than day one, but that may not be too surprising seeing as how it was on a Saturday. I was a little disappointed that Bruce Tate wasn't able to make the conference because I was really looking forward to hearing him speak, but I was pleasantly surprised at the quality of the other speakers. I know that a lot of people put a lot of hard work into the conference so I'd personally like to thank everyone responsible for putting on this years conference and I can't wait to attend again next year.

In case you weren't able to make it, here's a quick rundown of today's events:

Opening Keynote: Ceremony vs. Essence - Neal Ford

Neal Ford is a seasoned consultant and speaker that I've seen present on more than one occasion. I've always enjoyed his presentations, but his talk today really hit home with me. Neal's talk basically boiled down to the fact that we developers need to Keep It Simple (KISS) and avoid the "accidental complexity" that is often unnecessarily added to the problems we are trying to solve. I've listen rhetoric like this a 1,000 times, but I've never heard it articulated so elegantly. It was literally like he was looking inside of my head and expressing the way I (and likely a lot of developers) feel about the current state of the industry (especially if you are a Java programmer like me).

DSLs and Ruby: Blurring the Lines Between Programs And Data - Russ Olsen

I used to work with Russ at a FGM so I was really looking forward to hearing him speak. Russ has become quite a leader in the Ruby community especially after publishing his first book: Design Patterns in Ruby. Russ's talk centered around Domain Specific Languages (DSLs) and how relatively easy it is to implement one using the Ruby language. During his presentation Russ put together a simple Internal DSL for example purposes and also discussed use cases for when it is appropriate to create a DSL to solve a problem.

The Culture of Innovation in Ruby - Glenn Vanderburg

Innovation in testing frameworks in Ruby was the topic of Glenn Vanderburg's talk. Glenn discussed some of the interesting testings frameworks available in the Ruby community (Rspec, Mocha, Flex Mock, Shoulda) and compared them to some of the stagnant frameworks in the Java community (JUnit, TestNG). Glenn also pointed out how Ruby's testing frameworks tend to make it much easier to mock objects and create more thorough tests with much less effort than the equivalent tests in the Java world.

Living on the Edge - Yehuda Katz

Yehuda Katz of EngineYard talked about some of the new(er) technologies that are emerging in the Ruby community including Merb, DataMapper, Sake, Thor, YARDoc, and GitHub. DataMapper looks like an especially interesting alternative to the incumbent ORM ActiveRecord. It promises thread safety and increased performance which are obviously two great things. GitHub looks pretty cool too. I'm not quite ready to abandon Subversion just yet, but I will definitely be checking out Git in the near future.

Lightning Talks

Chris Bucchere, David Jones, Simon Kaczor, and Bryan Liles all gave lightening talks today on a variety of topics including social networking, state machines, command line libraries, and testing respectively. All of the talks presented useful information, but Bryan Liles was without a doubt the most entertaining speaker of the day. He made it through about 65 slides in 15 minutes and was still able to convey a very important message about how often developers should be testing their code. Check out his blog at: http://smartic.us/

Archaeopteryx: A Ruby MIDI Generator - Giles Bowkett

I actually wasn't able to attend Giles Bowkett's session because I had a previous engagement to attend, but I did get a chance to view some screencasts that show off his Ruby MIDI Generator. Pretty cool stuff.

Tomorrow

Tomorrow looks to be a pretty exciting day as well. I'm especially looking forward to talks from my former co-worker Dave Bock and author Stuart Holloway.

By day I'm a Java developer, but I've also spent a fair amount of time studying Groovy. Learning Groovy has been my "training wheels" approach to understanding dynamic languages. Studying Groovy has helped me grasp new (new to me at least) concepts like Metaprogramming and Domain Specific Languages all from the comfort of my Java environment.

As a simple exercise I thought it would be interesting to rewrite the code from my last blog entry in Ruby to see how it compares to the Groovy code I originally wrote. So without further ado let's check out the code:

Authenticating with Google

In order to use the Google Finance API we need to first authenticate with Google by sending our Google Account username and password to Google's client login URL: https://www.google.com/accounts/ClientLogin.

def create_auth_token(username, password)
  https = Net::HTTP.new('www.google.com', 443)
  https.use_ssl = true
  path = '/accounts/ClientLogin'

  query_string = "Email=#{username}&Passwd=#{password}
                 &service=finance&source=company-groovyfinance-1.0"

  https.verify_mode = OpenSSL::SSL::VERIFY_NONE # don't display warnings

  response = https.post(path, query_string)

  if response.code == '200'
    return response.body[/Auth=(.*)/, 1] # The authorization token
  end

  return "Error"
end

In the code above we create an http object and give it the correct URI and path to Google's login service. We then send in a query string with our username, password, service, and source (application name). If everything succeeds Google returns an Authentication token which we will set in the headers in subsequent calls to Google's service.

This code isn't much different from the Groovy code in my previous entry, but I do like the way Ruby allows POST calls directly on the http object. This obsoletes the processRequest() method found in my Groovy code.

Creating a Portfolio

We can create a new portfolio in our Google Finance account by executing calls to the following method:

def create_portfolio(authorization_token, portfolio_name)
  headers={}
  headers["Authorization"] = "GoogleLogin auth=#{authorization_token}"
  headers["Content-Type"] = "application/atom+xml"

  http = Net::HTTP.new('finance.google.com')
  path = "/finance/feeds/default/portfolios"

  atom_string = "<?xml version='1.0'?>
  <entry xmlns='http://www.w3.org/2005/Atom'
         xmlns:gf='http://schemas.google.com/finance/2007'
         xmlns:gd='http://schemas.google.com/g/2005'>
         <title type='text'>#{portfolio_name}</title>
         <gf:portfolioData currencyCode='USD'/>
  </entry>"

  response = http.post(path, atom_string, headers)

  if response.code == '201'
    doc = REXML::Document.new(response.body)
    return doc.root.elements["id"].text[/portfolios\/(.*)/, 1] #portfolio id
  end

  return "Error"
end

This code is almost identical to the Groovy code I wrote last time but I do really like the way Ruby allows you to place header attributes into a Hash and send them directly with a call to http.post().

Placing an Order

Just like the example above, placing an order (stock trade) means that we need to create an Atom feed with our transaction attributes (transaction type, number of shares, etc.) and send them to a Google resource end point. This end point in our case is the URL of the specific portfolio in which we wish to place our new asset.

def create_stock_transaction(authorization_token, ticker_symbol,
                             number_of_shares, transaction_type, portfolio_id)

  headers={}
  headers["Authorization"] = "GoogleLogin auth=#{authorization_token}"
  headers["Content-Type"] = "application/atom+xml"

  http = Net::HTTP.new('finance.google.com')
  path = "/finance/feeds/default/portfolios/#{portfolio_id}/positions/" +
         "#{CGI::escape(ticker_symbol)}/transactions"

  transaction_time = Time.now.strftime("%Y-%m-%dT%H:%M:%S.000")

  atom_string = "<?xml version='1.0'?>
  <entry xmlns='http://www.w3.org/2005/Atom'
         xmlns:gf='http://schemas.google.com/finance/2007'
         xmlns:gd='http://schemas.google.com/g/2005'>
         <gf:transactionData date='#{transaction_time}'
             shares='#{number_of_shares}' type='#{transaction_type}' />
  </entry>"

  response = http.post(path, atom_string, headers)

  if response.code == '201'
    return response.body
  end

  return "Error"
end

Once again the Ruby code looks nearly identical to the Groovy code. We set the correct headers, generate a well formed Atom feed, and the post our data to Google. If all goes well, the transaction will be made and an Atom representation of the transaction will be returned to the client.

Groovy vs. Ruby Thoughts

There are probably "Rubyer" ways to write the code samples listed above, but I did find it interesting that the code I wrote in Ruby was so similar to the code that I wrote in Groovy. I'm not sure whether or not the similarities are due to the subjective nature of having the same programmer write both samples or if it's due to the fact that the languages are so similar to begin with.

Either way, exploring a new language (albeit a similar language) did give me some additional perspective. I found myself bouncing back and forth between the Groovy and Ruby code comparing how similar tasks were performed in each language. This type of information is always useful when trying to determine which technology is the right tool to use for a specific job.

In addition, studying Ruby helped me write better Groovy code. In the create_auth_token() method in the Ruby sample a regular expression is used to parse the authentication token from the returned string. I was able to take this regular expression code and apply it to the same method (createAuthToken()) in my Groovy code to improve its readability.

References

Disclaimer

Again... I am in no way endorsing any of the companies mentioned in this blog entry and all company references are for example purposes only.