*scottstuff*

I just released Typo 2.5.7. This is a relatively minor upgrade over 2.5.6, but it adds two important features:

1. Support for Rails 0.14.x. Due to one little change, Typo 2.5.6 worked fine with Postgres and the new Rails, but MySQL failed horribly. This should now be fixed.
2. Support for overriding views from within themes. The theme contest folks really want this, and it’s been available in the trunk for months, so I backported it.

I’m just about to start in on Typo 2.5.7 development. I have two goals for this update to Typo 2.5.6:

1. Make Typo work correctly with Rails 0.14.x.
2. Backport the new theme updates from the Typo trunk to the 2.5 series so Typo Theme Contest developers can use them.

I’d like to have this on RubyForge before I leave for the airport on Wednesday.

As a reminder, the Typo Theme Contest is still running. The deadline for entry is November 28th, so there’s still time to start, but you don’t want to wait much longer. The prize pool has grown again–the top prizes are a new 15” PowerBook, a 12” iBook, an iPod Nano, and a year’s free hosting. Although it pales in comparison to a new PowerBook, I’d love to bundle the top couple themes with future Typo releases, so you can count on thousands of Typo users enjoying your work.

As of this morning, the Typo trunk requires Rails 0.14.2. I ran rails . and then spent a couple hours cleaning up Rakefile and environment.rb, and then updated all of the tests to work without instantiated fixtures. As advertised, this makes tests run quite a bit quicker; since Typo’s test suite had been flirting with the 5 minute mark on my laptop, anything that we can do to speed up tests is welcome.

Now that we’re safely requiring 0.14.x versions of Rails, I’ve started adding session :off all over the place. My sessions table for scottstuff.net has around 70,000 sessions in it right now, and I doubt that more then 15 or 20 of them were ever useful. We use sessions for user authentication, but that really only matters for admin pages plus a few other cases, like comment posting. In essence, any page that can be cached will never need access to a session. So, I made a little change to the Article controller. Instead of

class ArticlesController < ApplicationController
  caches_page :index, :read, :permalink, :category, 
    :find_by_date, :archives, :view_page, :tag

  ...
end

I’m now using

class ArticlesController < ApplicationController
  cached_pages = [:index, :read, :permalink, :category, 
    :find_by_date, :archives, :view_page, :tag]
  caches_page *cached_pages
  session :off, :only => cached_pages

  ...
end

This is a really common pattern, because pages that can use the page cache really shouldn’t depend on the session in any way. If I was doing this in more then one place, I’d probably want to extract it into a nice little plugin, but I’m just not feeling the motivation right now.

This seems to be the season for talking about Rails migrations. A lot of people are finally discovering them and finding that they’re very useful for maintaining your database schema over time. I’m a big fan of Rails migrations; we’ve been using them with Typo since the middle of July, when they were all new and shiny. We’re currently up to 24 migrations in the Typo source tree. We’re even using migrations to create our initial database, via my Schema Generator. I haven’t done a formal survey, but I suspect that Typo is the biggest open-source user of migrations, and may actually be the biggest user overall.

The big problem is that we’ve been using migrations wrong the whole time, and we just realized it.

There are probably a dozen bugs in Typo’s bug tracker that boil down to “I fell behind the trunk and now rake migrate throws exceptions and I can’t upgrade anymore.” The problem is that migrations are designed to run against an earlier version of your database, but they use the current version of your code. The first time that this caused problems was with the migration from Typo 2.0 to 2.5–we’d added two new fields to articles. Migration number 7 added the permalink field and a before_save hook to make sure that all saved articles have permalinks. Then migration number 9 added GUIDs and a second before_save hook to fill the guid field. Both migrations did Articles.find(:all).each { |a| a.save } to update each Article and populate the new fields.

This worked great for developers who frequently upgraded. A few days after the GUID migration went in, though, we started getting weird bug reports–users who tried to do both upgrades at the same time found that migration number 7 was dying. What was happening was that migration number 7 added the new permalink field to articles, but when it went to run the save loop both before_save hooks ran, and Typo tried to add a GUID to each article. However, the guid field didn’t exist yet, so the migration threw a bunch of exceptions and died.

This caused a bunch of grumbling on the Typo IRC channel. We threw around a bunch of possible fixes. Our favorite was separating migrations into two parts–a schema change part and a data change part. First we’d run all of the schema changes, and then update all of the data. As a work-around, we added a hack that checked the current schema version and disabled specific before_save filters for older versions.

We managed to keep this little bandaid working until a couple weeks ago, when a huge set of new migrations went it; they renamed the articles table and merged several other tables into the new contents table using STI. And, again, we found that older migrations broke when users tried to upgrade from Typo 2.5.6 to the current dev tree. Unlink the permalink/guid case, this time there was no simple workaround. We couldn’t just add a couple if statements in a filter and make it all go away.

The fundamental problem is that we were using the wrong mental model for migrations. I saw migrations as a one-dimensional thing–a list of steps for migrating old data into the new format. In this view, the migration for going from schema version 6 to schema version 7 is constant–once it’s been written, the only reason to change it is if a bug turns up in the logic for that migration. Otherwise, the migration code should remain unchanged over time.

And that’s the problem–migrations aren’t one-dimensional. They are (and need to be) two dimensional–the schema version is one dimension and the code version is the other. Individual migrations exist to migrate from a specific old schema version to the current version, using the current code. Each migration should change over time to adapt to the changes in the code. So, the right fix for the permalink migration that caused so many problems wasn’t to add a bunch of logic to before_save. Instead, we should have deleted the entire save loop from the migration, and trusted the GUID migration to update both fields. If that wasn’t good enough, then we should have added a new migration at the end to do permalink cleanup after the GUIDs were added.

Once I came to grips with this, the migration changes needed to allow 2.5.x users to upgrade to the current trunk were pretty simple, and took about 5 minutes to write and test.

Or was I the only person in the Rails universe who thought about migrations this way?

A number of users complained this weekend that Typo was using way too much memory, with reports of 100+ MB per FastCGI dispatcher. Typo usually uses around 20 MB, and even that’s too much; 100 MB is enough to cause big problems with hosting providers like TextDrive.

The first step that I took was to verify that the problem actually exists outside of TextDrive. I set up a test Apache/FastCGI/Typo server, disabled caching, and then pounded on it using curl:

# while true; do curl http://typo1/ > /dev/null; done

I let that run for a few seconds and watched while my dispatch.fcgi processes grew from 22 MB to 80 MB. I then did a bit of experimenting:

• The main index page leaked
• RSS feeds didn’t leak
• Individual article pages leak
• Static pages, like /pages/about leak
• Error pages even leak

Disabling the layout for a leaking page and then re-testing it showed that the leak followed the layout. Turning layouts back on and removing the sidebar block fixed the leak.

Entertainingly enough, disabling the sidebar from inside of the sidebar infrastructure didn’t fix the leak. The mere act of calling render_component to generate the sidebars seemed to be causing the memory leak. Since Typo is one of the very few users of Rails components, this suggests that render_component may have a leak that no one else has noticed, so I created a new test Rails app with only two files. First, app/controllers/foo_controller.rb:

class FooController < ApplicationController
  def bar
    render_component :layout => false, 
      :controller => 'sidebars/sidebar', :action => 'index'
  end
end

Then components/sidebars/sidebar_controller.rb:

module Sidebars
  class SidebarController < ApplicationController
    def index
      render :text => 'test', :layout => false
    end
  end
end

This is about as minimal as a Rails app can get. Then I set up a FastCGI server running this project, and ran curl against /foo/bar, and watched the process size climb. So the leak is part of Rails, not really part of Typo.

Unfortunately, I’m not sure where the leak is coming from. I read component.rb and made a few small changes, but the leak hasn’t stopped. So I’m going to file this as a Rails bug and see if we can get it fixed before 1.0.

Update: Rails bug 2589.

Update: Thanks to Scott Barron, the bug has been fixed. Users with memory problems should probably install the patch, although a bit of testing would obviously be recommended first. The next release of Rails (either 1.0rc3 or 1.0; I’m not sure what they’re planning) should include this fix.

I just uploaded version 0.2.0 of my Rails Schema Generator to Rubyforge. This is a minor update, but it was needed to make the schema generator work with Rails 1.0rc2 (AKA 0.14.1).

The schema generator is sort of the flip side of the new schema code in Rails 1.0. It takes a set of Rails migrations, aggregates them all together, and spits out a SQL file that describes the DB that you’d get if you ran all of the migrations. Or, viewed in a more useful light, it gives you a SQL file that you can use to create a new DB from scratch. The current version actually produces three different schema files, one for PostgreSQL, one for MySQL, and one for SQLite, each with DB-appropriate syntax and types.

This is an outgrowth of Typo; we’re up to 25 migrations now, and we actively support 3 different DBs. It was getting really painful to maintain 3 distinct schema files in addition to the collection of migrations, so I wrote this schema generator. Now we’re back to DRY-land–we create new migrations and let the schema generator do all of the hard work.

Dave Thomas has a nice little announcement, complete with screenshot: Agile Web Development with Rails is the best-selling programming book on Amazon.com. Even better, Programming Ruby has the number two spot. Corgratulations.

As far as I can see, Typo 2.5.6 (the most recently released stable version) should work fine with Rails 1.0. I just did a brief round of testing with 1.0rc2, and all of the tests pass. Er, except for one test that had a stupid typo that somehow still worked with Rails 0.13.1; the bug is in the test itself, though, so it’s not worth releasing Typo 2.5.7 just for that. If we ever release Typo 2.5.7, then I’ll make sure that the fixed test is included.

Also, Typo 2.5.6 should work just fine with Ruby 1.8.3, too, as long as you’re using Rails 1.0. I haven’t actually tested this yet, but I’d be surprised if it doesn’t work perfectly.

Surprisingly enough, the current Typo trunk (r683 or so) doesn’t work with Rails 1.0. All of the filtering code is broken; I’ll fix it shortly and check in the fix. Fortunately, the current Typo trunk is pinned to Rails 0.13.1 for now, so it should be safe to upgrade the version of Rails on the box; Typo will just ignore the new Rails for now.

Update: the Typo trunk r685 or later should be compatible with Rails 1.0rc2. I’ll probably break Rails 0.13.1 compatibility soon, so it’s time to upgrade.

Just for the record, current versions of Typo (either 2.5.6 or the current Subversion trunk) don’t work with Ruby 1.8.3. There are two problems–the Logger bug that keeps Rails 0.13.1 from working with Ruby 1.8.3 (this is easy to fix), and a second bug that I haven’t read about anywhere else–apparently YAML serialization is broken with Ruby 1.8.3 and Rails 0.13.1. This keeps Typo’s sidebar from working properly.

I’m going to see what it’ll take to get the Typo trunk working with Rails 1.0 (rc1 or rc2, if it’s out today), and then see if it works properly with Ruby 1.8.3. Once that’s done, the trunk will probably shift from 0.13.1-only to 1.0-only.

Update: That was quick. ChrisNolan on IRC pointed out that Rails bug #2304 contains a patch to fix this. The patch is already a part of the current Rails trunk, but you’ll need to patch 0.13.1 manually if you want to use it with Ruby 1.8.3.