*scottstuff*

Yesterday was an interesting day–my now-former employer is restructuring “to support changes in direction and a new funding strategy,” so I’m now looking at new modes of employment, as are several former co-workers.

I’m viewing this as more of an opportunity to move on to new things, rather then a setback. I’ve spent quite a bit of time over the past few months thinking about what I wanted to do next, even though I didn’t expect this job to end quite so suddenly.

Here’s what I’ve decided:

1. I don’t want to work on long, slow, multi-year projects. The world doesn’t work that way anymore. Agility is the future, and I’m not going to enjoy working for any company that takes years to bring a product to market.
2. I want to work on things that I enjoy and challenge me, like Ruby and Rails.
3. I want to spend less time commuting.
4. I’d like the freedom to follow interesting trends and opportunities as they arise, for my own profit and the profit of those I work for.

With these goals in mind, I’ve decided to start looking for Rails consulting/contract work, although I’m also willing to consider full-time employment in the right environment. Please forgive a bit of shameless self-promotion:

My background is a combination of Unix system administration and programming; I’m very good at getting things to work and keeping them working. I’ve been programming professionally since 1988, I have 3 years of Ruby experience, over 6 months of Rails experience, and 13 years of Linux experience. I’m a major contributor to Typo, a Rails-based blog engine; I can easily provide code samples for those who are interested. I’ve written extensively about Typo, Rails, and Ruby here on my blog, feel free to look around. I was once the senior system administrator in charge of over 700 production Linux, BSD, and Solaris systems in a 24x7 environment; I can handle any level of system administration task that may be required.

Most importantly, I’m used to working with small groups of people to design effective, scalable, and affordable solutions to their problems.

Over the next few days I’ll post more details on this site, including a resume and some details on previous experiences. For now, if you have questions, please feel free to send me email, and I’ll get back to you as soon as possible.

One of the things that has really bugged me with recent Typo development is the pain of maintaining 3 different database schema files (PostgreSQL, MySQL, SQLite) along with a set of Rails DB migration scripts. Every time we add a new table, we have to edit 4 different files, even though all of the information that we need is available in the migration file. Unfortunately, without the static schemas, new users would be adrift, so we’re stuck having to hand-modify each of the static schema files. This violates the DRY principle, causes errors, and irritates developers.

So I figured I’d fix it by writing some code that can take a set of Rails DB migrations, fold, spindle, and mutilate Rails itself, and then spit out a database-specific schema file showing all of the tables, indexes, and seed data provided by the migration files. This includes handling cases where a table is added in migration #4, two new fields are added in migration #6, and one field is deleted in migration #9. There are some corner cases that just can’t be handled, mostly relating to seed data that needs to be migrated to be correct with more recent schemas, but I think I can come close enough to make Typo happy, and probably a lot of other open-source Rails projects.

This turned out to be easier then I expected. I’ve put about 4 hours into it so far, and I can take this migration:

# This is db/migrate/4_test4.rb
class Test4 < ActiveRecord::Migration
  def self.up
    create_table :sidebars do |t|
      t.column :controller, :string
      t.column :active_position, :integer
      t.column :active_config, :text
      t.column :staged_position, :integer
      t.column :staged_config, :text
    end

    Sidebar.create(:active_position=>0, :controller=>'category')
    Sidebar.create(:active_position=>1, :controller=>'static')
    Sidebar.create(:active_position=>2, :controller=>'xml')
  end

  def self.down
    drop_table :sidebars
  end
end

And then do this:

$ irb
irb(main):001:0> require 'migrate'
=> true
irb(main):002:0> require 'db/migrate/4_test4' # the code above
=> true
irb(main):003:0> Test4.up
=> ...
irb(main):004:0> puts DBMigrator::Database.dump('postgresql')
CREATE TABLE sidebars (id serial primary key, controller character varying(255), active_position integer, active_config text, staged_position integer, staged_config text) ;
BEGIN;
INSERT INTO sidebars ("staged_position", "active_config", "active_position", "controller", "staged_config") VALUES(NULL, NULL, 0, 'category', NULL);
COMMIT;
BEGIN;
INSERT INTO sidebars ("staged_position", "active_config", "active_position", "controller", "staged_config") VALUES(NULL, NULL, 1, 'static', NULL);
COMMIT;
BEGIN;
INSERT INTO sidebars ("staged_position", "active_config", "active_position", "controller", "staged_config") VALUES(NULL, NULL, 2, 'xml', NULL);
COMMIT;

Postgres works now, at least with the 4 or 5 examples that I’ve swiped from Typo’s migrations. SQLite and MySQL are nearly working; I think I just need to fake out a couple classes each and they’ll be up and running. Once that’s done, I’ll bundle this all up into a Rails generator so people can do this:

$ ./script/generate schema postgresql
      create  db/schema.postgresql.sql
$ ./script/generate schema mysql
      create  db/schema.mysql.sql
$ ./script/generate schema sqlite
      create  db/schema.sqlite.sql

Although blogs are inherently HTML-based, HTML isn’t really a great format for writting plain-text documents. If nothing else, manually adding <p> and </p> around paragraphs interrupts the flow of writing. Most people would prefer to write in a more user-friendly manner, either via a GUI editor or a light-weight markup language like Markdown, which is then translated to HTML automatically. Very few people really want to write raw HTML blog postings on a daily basis.

Out of the box, Typo 2.5 supports the Markdown and Textile markup languages and the SmartyPants HTML-post processing filter, which adds typographical quotes and dashes to HTML. Adding additional filters is difficult because the filter setup is hard-coded into Typo. One of the new features that I’ve been working to add to Typo is the ability to easily add new text filters via filter plugins, similar to the sidebar plugins in Typo 2.5. At the same time, I’ve also added several new filtering plugins that extend Typo’s abilities in a number of useful ways.

The goal of all of this is to make it easier to write using Typo. I’ve tried to find things that cause me pain and then fix them. I want to make it easy to do common writing tasks without having to fire up an external tool. Admittedly, my definition of “common writing tasks” is probably different from most people’s, but the easy ability to extend Typo’s filtering system will allow people to adapt Typo to their own needs without having a deep understanding of Typo’s internals.

<typo:flickr img="31366117" size="square" style="float:left"/>

<div style=\"float:left\" class=\"flickrplugin\">
  <a href=\"http://www.flickr.com/photo_zoom.gne?id=31366117&size=sq\">
    <img src=\"http://photos23.flickr.com/31366117_b1a791d68e_s.jpg\" width=\"75\" height=\"75\" alt=\"Matz\" title=\"Matz\"/>
  </a>
  <p class=\"caption\" style=\"width:75px\">
      This is Matz, Ruby's creator
  </p>
</div>

<typo:code lang="ruby">
  class Foo
    def bar
      "abcde"
    end
  end
</ typo:code>

class Plugins::Textfilters::TextileController < TextFilterPlugin::Markup
  def self.display_name
    "Textile"
  end

  def self.description
    'Textile markup language'
  end

  def filtertext
    text = params[:text]
    render :text => RedCloth.new(text).to_html
  end
end

def macrofilter(attrib,params,text="")
  data = text.to_s.split(/\s+/).join(',')

  if(attrib['data'])
    data = attrib.delete('data').to_s.split.join(',')
  end

  url = url_for(
    {:controller => '/textfilter', 
     :action => 'public_action', 
     :filter => 'sparkline',
     :public_action => 'plot', 
     :data => data}.update(attrib))
  "<img src=\"#{url}\"/>"
end

class Plugins::Textfilters::SparklineController < TextFilterPlugin::MacroPost
  plugin_public_action :plot
  def plot
    ...
  end
end

def self.default_config
  {"wiki-link" => {
    :default => "", 
    :description => "Wiki URL to link WikiWords to",
    :help => "The WikiWords plugin links..."}}
end

filter_text('text to be filtered',[:markdown, :macropost, :smartypants])

filter_text_by_name('more text to be filtered','markdown')

Now that tags are working, I’ve started work on adding text-filter plugins for Typo. The current release (Typo 2.5.3) has support for 5 different combinations of Textile, Markdown, and SmartyPants hard-coded into it. The different combinations are actually repeated in 3 different places–the filtering code itself, the drop-down list for the built-in editor, and the Movable Type API code to list filter options.

That’s all gone now, replaced with a plugin system, similar to the sidebar plugins that made it into Typo 2.5. Individual filters get dropped into components/plugins/textfilters/ and the system picks up on them automatically. Then there’s an interface in the admin UI that lets you combine the filters into named filter sets, so you can combine Markdown and SmartyPants into “My Filters” (or “Markdown with SmartyPants”, which Ecto recognizes and performs some magic to get Markdown previews to work right). The UI isn’t really complete yet, but the entire back end is there, and I’ve added two new filters as a demonstration of what we can do. Here’s the current list:

• Markdown. This is my favorite lightweight markup language, and I use it for everything that I write here.
• SmartyPants. A companion to Markdown, it does a typographical cleanup on HTML, turning ASCII single and double quotes into their typographically correct cousins and fixing em-dashes.
• Textile. Another lightweight markup language, like Markdown.
• Amazon. This turns URLs like <a href="amazon:097669400X" ...> into a link to Amazon’s page for ASIN 097669400X, optionally attaching your Amazon affiliate tag. This is mostly a demonstration of what you can do with filters, although I’ll be using it on my blog.
• Flickr. This sticks a picture from Flickr on the page. This is a bit more complex then the Amazon filter, but similar in concept. It turns <flickr img="31366117" ...> into a formated inline image, linked to Flickr’s full-sized image page, optionally with a caption attached. The full HTML produced is something like <div style=""><a><img/></a><p>Caption</p></div>, which saves a lot of typing.

I’m currently working on a Sparklines plugin, using Glyph’s Ruby sparklines code. It’ll be similar to the <flickr> tag, except it’ll spit out an <img> tag that points to a built-in sparkline generator. Turning <sparkline ...> into an image tag is trivial; allowing a text filter to export an action to the world is a bit more work.

There are currently two things that bother me about this code that I’ll need to resolve before releasing it:

1. The <flickr> and <sparkline> tags–should they look like plain XHTML, or is that a mistake? Should I turn them into pseudo-bbcode tags, like [flickr]? I’m currently leaning towards sticking a typo pseudo-namespace on the front of them, and turning them into <typo:flickr .../> and <typo:sparkline ...>. Any objections to that?
2. The admin interface to this is killing me. I’d love to have a nice, simple way of editing each filter set, but it’s turning into a nightmare. I could just copy the sidebar config page (with a few changes–you can only include each filter once, unlike sidebars), but lots of people have had problems with the sidebar editor, and I’d like something a bit cleaner. Except I have no idea what to do.

If all goes well, I’ll post a public patch for comment early next week, and then kick off the Typo 4.0 process by committing this and the tag code later in the week.

One of the biggest improvements in Typo 2.5 is page caching. By using Rails’s built-in page cache, we can get 100x the performance on many benchmarks without doing more then a few lines of work. This lets us serve high-volume weblogs (like weblog.rubyonrails.com) without requiring heroic measures like clustering.

Unfortunately, there are a number of hidden problems with Rail’s 0.13.1’s page cache implementation. We’ve had to work around a number of them in order to get Typo 2.5 out the door.

Basic page cache usage

Enabling Rails’s page cache is amazingly simple–just add caches_page :actionname to the top of your controller class and the :actionname action will spit out page cache files automatically. A couple small tweaks to Apache’s .htaccess file, and Apache will now serve cached files all on its own without involving Rails. If a client asks for http://blog.example.com/articles/2005/08/08/foo, Apache will first check for a articles/2005/08/08/foo.html file in Typo’s public directory. If that file exists, then it’s sent off to the client without touching Rails at all.

Sweeping

That part of caching is easy. It’s the other end that’s hard: sweeping the cache to remove stale cache entries. Rails provides a simple cache sweeper that can remove specified pages, but that’s not really good enough for us. With Typo, there are a number of events that end up touching a huge number of cached files. Adding a comment, for example, touches the cached article page, but it also changes the comment counter on the main index (if the article is still on the front page), the day, month, and year indexes, some number of category indexes, tag indexes, and potentially paginated versions of all of the above. The code to track these all down was trouble-prone and frequently missed one of the pages that needed to be changed; this led to stale caches. Even worse, some actions, like changing themes, need to invalidate all pages. Rails’s page cache doesn’t keep a list of cached pages, so there’s no clean way to sweep them all.

What we ended up doing was adding a page_caches table to the database and adding hooks to insert a new PageCache entry every time a page was cached. We also added a hook to remove entries from the page cache table whenever a page was manually swept, and then added a PageCache.sweep_all method to flush the entire page cache. For now, we’ve simply ripped out all of our old “smart” sweeping code and force a full sweep of the entire cache whenever anything substantial changes. Sooner or later we’ll start adding smart cache sweeping back in, but for now this works surprisingly well.

Query Parameters and Aliasing

Another shortcoming of Rails’s page cache implementation shows up when you start using query strings. Asking for http://blog.example.com/articles?page=2 ends up handing the ?page=2 parameter to the static .html cache page if it exists instead of calling Rails to ask for page 2. Even worse–if this cached page doesn’t exist, then Rails will generate it and store it for future access, even though it’s the second page of the index, not the first.

Finally, and worst of all, in Typo http://blog.example.com/articles is actually equivalent to http://blog.example.com/, because the article index view is the default index page. This means that the cached page for http://blog.example.com/articles?page=2 is actually /index.html, so anyone visiting page 2 of the article index screws up the front page of the blog. There’s no easy way around this with Rails 0.13.1; for now we’ve had to do work to keep ?page= from paginating anything. There’s one point that we could interrupt the page cache process from inside of Typo, but it doesn’t have any way to see the @request object or any of the query strings.

Long-term, we’re going to need to patch Rails to add a cachable property to @request that gets set to false when there’s a query string present, and also tweak Apache’s rewrite rules to skip static files if a query string is present. That assumes that Apache is even able to do that–every time I read the mod_rewrite documentation I end up with a headache. Since Typo officially supports lighttpd as well as Apache, we’ll need to get both of them to do the right thing, which is far from trivial.

Non 7-bit ASCII URLs and Caching

Finally, Rails screws up cached filenames when the URL has non-ASCII characters. So any URL with accented characters or any non-ASCII script is totally uncachable. At least with Apache and Webrick, Rails sees non-ASCII characters in the URL encoded using the usual %XX URL-encoding scheme. Unfortunately, both servers actually look for unencoded filenames. So Rails writes out the cache file for /foö as public/fo%C3%B6.html (assuming UTF-8 encoding), but Apache actually looks for public/fo<C3><B6>.html (where <C3> is a byte with the value of C3 in hex). This is actually not all that hard to fix–just add a URI::Util.decode to the right place inside of Rails–but it’s not clear what the security implications of this are.

Given all of these problems, I’ve been tempted to try using Rail’s action cache instead of the page cache–the action cache doesn’t let Apache serve the cached files directly, so Typo would have a brief chance to block the cache from handling specific files, and we could approach sweeping from the opposite direction. It’s not clear how big of a speedup the action cache would actually give us, though, compared to the massive win that we get from the page cache. We’d really like to keep using the page cache and fix all of its bugs to its usable by other Rails users.

I missed this when it first came out, but Why has a pointer to a WebDAV server plugin for Webrick, Ruby’s native web server. I wonder how easy this would be to port out of Webrick and into a generic FCGI controller, so we could use it with Rails? That would let Rails act as a file server, which would be useful for some sorts of user interaction. I wonder if we could make use of it in Typo?

Version 2.5.0 of Typo was just released. The Typo web site hasn’t been updated yet, but you can download it from RubyForge.

Typo is the software that runs this website. It’s written in Ruby using Rails.

It’s OSCON season, and most of the open-source world has descended on Portland for the week. This includes most of the leaders of the Ruby community, so the Portland Ruby Group held their own “Free OSCON” night at FreeGeek. Four of the people giving Ruby talks at OSCON gave their talks for free at FOSCON, and most of the rest of the Ruby speakers were there lurking in the back of the room, including Dave Thomas and Matz.

I dragged my trusty D60 along and took a few pictures, along with a few notes.

DHH on Rails

The first speaker was David Heinemeier Hansson, the creator of Ruby on Rails and O’Reilly’s “Hacker of the Year” for 2005. He basically gave us his 15-minute OSCON keynote on Rails, which was both a brief introduction and a marketing talk for Rails. His big theme was “flexibility is overrated”–by reducing the number of ways that you can approach web development, Rails makes it enormously easier to actually get things done.

When David’s talk was complete, our host Phil (Tomson, I presume) presented him with a vintage copy of How to Win Friends and Influence People.

Rich Kilmer on ActionStep

This was deeply cool. Rich Kilmer gave a brief presentation on his current project, ActionStep, a port of OS X’s Cocoa API to Flash. Right now, there are free tools that can create Flash .swf files, but Macromedia’s licensing keeps them from legally using any of Flash’s windowing tools. So Rich decided to write his own windowing toolkit, using NextStep/Cocoa API. They’re over halfway done, and expect to release the first complete version before the end of 2005.

I’m not particularly fond of flash, so I wasn’t paying a lot of attention until Rich started showing off his ideas for Rails integration. He’s building a layer that will glue his Flash front end to a Rails back end, and the demo code that he presented made it look even easier then creating HTML forms for user interaction. I’m not sure that it’ll be something that I’ll ever end up using, but it looked deeply cool. Here are a couple screenshots:

Glenn Vanderburg on Metaprogramming

Next up, Glenn Vanderburg gave a talk on metaprogramming in Ruby. He showed how Ruby itself uses metaprogramming to implement things like attr_accessor, and then showed how people have used and extended Ruby over the years. One theme was the continuing development of metaprogramming idioms in Ruby; he showed how things have changed over the years, starting with an X protocol wrapper that someone wrote years ago, through Rich Kilmer’s Java debug protocol system, and up through Rails. Ruby’s metaprogramming ability is one of the things that makes Rails so successful–the ability to extend the language to let you say things like:

class Article < ActiveRecord::Base
  has_and_belongs_to_many :categories
  has_many :comments
  belongs_to :user

  ...
end

This is one of the things that makes Rails so useful and so much fun to program.

Why

Finally, Why The Lucky Stiff came on stage. Why (or sometimes _why) is sort of the rock star of the Ruby world. His real identity is a closely guarded secret. He’s the author of why’s (poignant) guide to Ruby, which is easily the strangest programming book that I’ve ever seen.

His FOSCON talk was sort of a performance-art interpretation of his book.

We knew that interesting things were afoot when he showed up with a backup band, and we had to stop for a break while they set up on stage.

Once that bit was done, they launched into song. Why had a nice little piece that was essentially the Ruby lexer set to music. “A symbol starts with a colon and is followed by lowercase letters and numbers! A constant is composed of capital letters and underscores!”

This sort of thing went on for a while. He alternated between animated segments on the projector, demonstrations of distributed Ruby programming (with audience participation), singing about Ruby, and utter non-sequiturs.

Why’s presentation ended with the immortal words “stop her, she’s stealing our eigenclasses.”

Thanks to Phil and everyone else from the Portland Ruby group, and all of the presenters for giving us a very memorable and enlightening night. I have a few more pictures on Flickr, if anyone’s interested.

Dave Thomas just posted that Agile Web Development with Rails is complete and has been released for printing.

This is the first book on Rails, and it’s been a fascinating process watching them develop it. Dave claims that their beta book process has made this one of the most Agile book development processes ever. I ordered it on the first day that it was available, and I’ve now sorted through three or four different PDFs and reported a couple different errors. I think I posted the very first error on their public errata-tracking system, but it’s hard to get it to sort by time now that there are zillions of entries from different reviewers.

I’m looking forward to receiving my final PDF and (eventually) the final printed copy of the book.

Congratulations on a job well done, guys.

I’m basically finished with my last block of changes to Typo, and most of them have been merged upstream. At this point, I have most of the features that I care about, but there are still a few things left to do:

• Implement the rest of the Movable Type API, including the DB fields behind all of the useful stuff. Basically, everything that’s available in common blog editors should be available in Typo.
• Optionally link article keywords to Technorati tags.
• Look into turning filters and sidebar items into plugins, where all that’s needed to add them is to drop their files into a directory and then enabling/disabling/reordering them from the admin pages. (sidebars are done: #157)
• Add a Flickr text-formating plugin that will let me say something like [[flickr:scottlaird/24727421 "Some guy on a bicycle"]] and have that turned into a clickable image (with size tags) and a caption. I’m not sure what the right syntax will be for this–I know what Markdown uses, so I can avoid running into it, but I’m not all that familiar with Textile. It might be worth looking at Unicode-only brackets, like «» or 「」. Of course, adding untypable brackets sort of cuts down on the utility of shortcuts like this.
• Add an Amazon text-formatting plugin that allows amazon:<ASIN> URLs and transforms them into links, optionally with affiliate ID attached. This is a lot less complex the the Flickr filter, because it can just search for <a href="amazon:...">.
• Add a better way of mapping stacks of text-formatting filters to names. The Admin UI and the MT API both want symbolic names for filters; right now, this is hard-coded. It’d be nice to make this dynamically managed, but it looks like a total pain.

Done

These used to be up above, but they’re done now.

• Link to author’s email address if provided. (done, #156)
• Add GeoURL support, with a tag in the headers for it and a config option for providing your latitude and longitude. (done, #154)
• Add Flickr config parameters so adding Flickr doesn’t require editing the source. (done, #155)
• Fix the Flickr sidebar so it doesn’t do weird things with portrait images–as it is, the border around the images fits landscape images perfectly but leaves a big gap with portrait images. Alternately, just use the square layout that flickr likes. (done, I’m using square images)

Update: Pretty much everything here is complete as of August 26th. See the Typo Wishlist wiki page for details on where we’re going from here.