*scottstuff*

Geoffrey Grosenbach just announced the start of the official Typo theme contest. There are a number of fantastic prizes (including a 4 GB iPod nano) available, so take a look at the rules and fire up your editor now.

One of the big problems with caching in Rails is the way that Rails’s caching systems handles query parameters. Page caching completely screws this up–the page cache will turn /articles/read?id=100 into /articles/read.html, and Apache will then hand all future hits on /articles/read off to that static HTML file, even if the user was looking for /articles/read?id=99. You can mostly get around this by making sure that you always use named parameters via Rails’s routes, but even then a malicious user can do weird things to your cache by feeding query parameters via ?.

The action cache is slightly better, but it’ll still misbehave with the examples above. What we really need is a caching system that pays attention to all parameters, not one that ignores all of them that aren’t part of a route.

Towards that end, I’ve created caches_action_with_params. It’s a minor derivative of caches_action with a different fragment cache key; instead of using the URL (as generated by url_for), it ignores URLs completely and uses ACTION_PARAM/<host>/<controller>/<<action>/<params>. This way caching isn’t dependent on routing, which will help with some of the stranger problems that Typo has seen. On the downside, if your actions explicitly check the URL that the user used, then caches_action_with_params won’t work for you.

Once I’m off the train and sitting somewhere with usable IP, I’ll post some sample code to the Typo bug tracker and generate a couple benchmarks. I expect this to about about 10% as fast as the page cache, but it should still be faster then 100 hits/second, which is my personal definition of “fast enough” this week. Then, if no one has any big complaints, I’ll commit this and switch off the page cache.

Once that is done, it’ll be fairly easy to add a lifespan to cached pages, so we can say “this page is only good for 2 hours” and have it regenerate automatically after that.

I gave a short talk on caching with Rails last night at the Seattle.rb meeting. The short version is “the page cache is going to hurt a lot worse then you’d expect,” but anyone who read my previous article on caching should already know that. I did the slides for the talk with S5, which was new to me–I had planned on using Keynote, but it seems to have died in the year and a half since I last had a use for it. S5 worked well enough, although there were some formatting issues that kept popping up as the browser window size changed. By and large it was easy to use, and it’s nice to have a HTML version of the talk that doesn’t look like a nasty afterthought.

About halfway through preparing for the talk, I realized that I really need to add a new action cache option, something like caches_action_with_params, so we can explicitly say how query strings and other parameters affect the cache. Here’s a bit of sample code:

class ArticlesController < ApplicationController
  caches_action :index
  caches_action_with_params :read, :id
  caches_action_with_params :permalink, :year, :month, :day, :title

  def index
    @pages, @articles = paginate(
      :article, 
      :per_page => config[:limit_article_display], 
      :conditions => 'published != 0', 
      :order_by => "created_at DESC"
  end

  def read  
    @article = Article.find(
      params[:id], 
      :conditions => "published != 0", 
      :include => [:categories])    
  end

  def permalink
    @article = Article.find_by_permalink(
      params[:year], 
      params[:month], 
      params[:day], 
      params[:title])
  end

  ...
end

At least as of Rails 0.13.1, calling /articles/read?id=10 will create a cache entry for /articles/read, which is wrong, and then asking for /articles/read?id=20 will return the cached entry for id=10. Yes, the user is supposed to use routes for this, but explicit query params still work, and there are times when you really need to use them. Fortunately, this is really only 20 lines of code, so it shouldn’t be too hard to write.

One of the small things that we’ve been adding to Typo recently is UUIDs for assorted bits of content. The only real consumer right now is the Atom 1.0 feed code, but they’re stored in the DB and sooner or later we’ll update the RSS feeds to use them too. Every item in a RSS or Atom feed is supposed to have a unique ID that is stable through time and across aggregators. Done properly, that means that readers will be able to recognize when they receive the same content via two different paths–like when you’re subscribed directly to a blog but also receive some of the same articles via *Planet-style aggregators or search feeds.

Unfortunately, Ruby doesn’t come with any code for generating UUIDs. When I started adding UUIDs to Typo, I found an old project on RubyForge that hasn’t been updated in 18 months, but it doesn’t work right on OS X and I didn’t really feel like fixing it. So I cheated and generated a MD5 from a combination of the object state and the clock and then shoved it into the DB. It’s not RFC 4122-compliant, but it works well enough in practice. Since our previous Atom feed code generated utterly non-unique tag: IDs (there’s nothing wrong with tag:, but we weren’t putting nearly enough data into it to actually generate unique IDs), even non-compliant MD5s were a big step in the right direction, and I figured that it’d be easy enough to plug a real UUID library in once once arrived.

It looks like I was right–Bob Aman sent me mail yesterday to tell me that he’s released a new UUID library for Ruby. I spent a couple minutes this morning ripping out my old UUID hack and replacing it with his RFC-compliant implementation, and once I finish running the last round of tests I’ll check it into the tree. Thanks, Bob.

I just released Typo 2.5.6. This is a minor bug-fix release, but it’s an annoying bug. Flickr changed their RSS feeds slightly a couple weeks ago and this broke Typo’s Flickr sidebar. This release contains the 1-line fix required to make it work again.

I’m starting to look at expanding Typo’s comment system. I have a few goals:

1. Give Typo the best comment system of any weblog engine.

2. Make it easier to handle large numbers of comments. This includes some sort of comment threading as well as comment pagination. Yes, those conflict with each other. No, I’m not sure how I’ll resolve it.

3. Make the comment system more resistant to spam. I’m not really sure which approaches I’m going to use for this either, but this is a well-explored problem, and I have a few ideas. Fortunately, Typo has always had a few features that discourage comment spam, and we’d like to add to that a bit.

4. Add support for authenticated comments, ideally using one or more external user identity systems, like TypeKey or OpenID. Personally, I’m really excited about OpenID, and I’d like to have Typo become both an OpenID client and server.

5. Increase the “socialness” of the comment system. Make it easier to conduct discussions in comments. Allow frequent users to track responses to their comments via RSS or email. Allow “community building” via comments. Consider allowing some forum-like features, like allowing users to re-edit their comments after they’re posted. This obviously depends on having a reasonable identity and authentication system.

6. Make it faster while we’re at it. Posting new comments currently invalidates Typo’s entire page cache; there has to be a way around that so we can keep up with Slashdot when they link to a Typo site.

7. Keep it easy to run. One of my basic goals with Typo is to minimize the number of configuration options in the system, following DHH’s mantra: ”flexibility is overrated.” At the same time, different people *do* have different needs; we need to find the right balance between having 50,000 little configuration options and force-feeding the One True Comment System down people’s throats.

My goal is to have something to show for this in the the next couple weeks; that depends on my consulting schedule and a few other things that are hard to predict, but I’m pretty optimistic about this.

Before I dig deeply into design, does anyone have any particularly good comment systems that we should look at? My current favorite is Dunstan Orchard’s on 1976design.com, but I’m open for suggestions. I’d love to hear about needs that I’m overlooking, too.

Someone pointed out today that none of the “convert from your old blog system to Typo” converters in the current Typo development tree were working. They all produce articles without any HTML in them. This was caused by my big filter update from a week or so ago; apparently no one has tried to convert directly to a development version of Typo in the last week or two. The problem is that none of the text filters were running. Unfortunately, there’s no easy way to make them run because they need access to a working Rails controller, and there isn’t one available from inside of the converters.

At the same time, Piers Cawley asked for an easy way to rebuild all of the HTML generated by filters on his site–he was doing filter development and he needed to rebuild everything. Unfortunately, the filter design doesn’t make this easy, either.

These two are basically the same problem–the way that we run text filters is kind of painful in the current Typo tree. In Typo 2.5 and earlier, filters were applied at the model level, and nothing outside of the model really needed to worry about them–the filters were automatically applied every time that the article (or comment, or page) body changed. Due to the changes in the dev tree, this just isn’t possible any more, but I’d tried to hack it together by changing the dozen or so actions that changed Articles. It worked, but it was ugly, and it breaks when something like a converter needs to create a new article, because the converter has no way to run the filter.

So I’ve been making a few changes to Typo.

The basic problem is that we’ve been using a “push” model for updating the HTML version of articles, when we should really be using a “pull” model. That is, instead of updating the HTML when the article changes, we should really be generating the HTML when the article is viewed and then caching the HTML so we don’t have to do it more then once per article.

Fortunately, this change was pretty easy to make–I just had to search for every reference to body_html, extended_html, or full_html and change it to a reference to article_html(article). Then I moved the filter calls into article_html(article), saving the generated HTML back into article.body_html.

Once that was done, I could rip out all of the complicated filtering code that I’d had to put in to make the new filters work right, and everything Just Worked. I had to tweak a few tests that expected the HTML to be available in the database immediately after posting new content, but I already had tests that verified that the content viewed right, so it was just a matter of removing code, not really adding new code.

There’s one more change that I’m debating making. From an architectural standpoint, we shouldn’t really be stuffing things back into body_html–we should be using Rails’ fragment cache. Switching to the fragment cache would be trivial, it would only take a couple extra lines in article_html, and then I could rip a bunch of lines in the editor actions, because I could use a sweeper instead of explicit calls to article.body_html = nil.

Unfortunately, if we do that then we’ll end up killing Typo’s performance when it’s running in development mode, because caching is disabled in dev mode. So it’d be cleaner, but probably too slow to be useful. I’ll probably revisit this again before the next Typo release–there are a bunch of performance tweaks that we need to make before the next release; once those are done, we might be able to stand the performance hit.

We've been making pretty good progress towards the next major release of Typo. Here's a short list of what's went in so far:

• Tags
• File uploads
• Gravatars
• Filter plugins, including:
  • Easy flickr image linking
  • Syntax highlighting
  • Sparklines
  • Auto-generating Amazon affiliate links
• Comment previews
• More powerful themes
• Atom 1.0 support
• Per-category and per-tag RSS and Atom feeds

There are still quite a few features left on our wishlist; some of those will make it into the next release, some won't. I'm starting to feel like we've passed the halfway point on this release cycle, but we don't have any firm plans for the next Typo release yet. Still, if there's anything that people really want to see in the next major Typo release, now would be a great time to speak up.

My Rails Schema Generator is nearly complete. Here’s a sample run:

$ ./script/generate schema
Found 6 migration classes
Starting migration for AddSidebars
Starting migration for AddCacheTable
Starting migration for AddPages
Starting migration for AddPageTitle
Starting migration for AddTags
Starting migration for AddTextfilters
Adding TextFilters table
Migrations complete.
 Tables found:   6
 Indexes found: 1
 Records found:   8
      exists  db
overwrite db/schema.postgresql.sql? [Ynaq] y
       force  db/schema.postgresql.sql
overwrite db/schema.mysql.sql? [Ynaq] y
       force  db/schema.mysql.sql
overwrite db/schema.sqlite.sql? [Ynaq] y
       force  db/schema.sqlite.sql

The migration classes that I’m using are copied straight from Typo without modification. I’ve left out all of the migrations that add features to “legacy” tables–tables like articles–since there isn’t a table definition that I can use. That’s my next project–adding a 0_initial_schema migration for Typo. Once that’s complete, I have a bit of code cleanup and then I’ll release my schema generator code to the world. Hopefully that’ll be later today.

One of the things that has really bugged me with recent Typo development is the pain of maintaining 3 different database schema files (PostgreSQL, MySQL, SQLite) along with a set of Rails DB migration scripts. Every time we add a new table, we have to edit 4 different files, even though all of the information that we need is available in the migration file. Unfortunately, without the static schemas, new users would be adrift, so we’re stuck having to hand-modify each of the static schema files. This violates the DRY principle, causes errors, and irritates developers.

So I figured I’d fix it by writing some code that can take a set of Rails DB migrations, fold, spindle, and mutilate Rails itself, and then spit out a database-specific schema file showing all of the tables, indexes, and seed data provided by the migration files. This includes handling cases where a table is added in migration #4, two new fields are added in migration #6, and one field is deleted in migration #9. There are some corner cases that just can’t be handled, mostly relating to seed data that needs to be migrated to be correct with more recent schemas, but I think I can come close enough to make Typo happy, and probably a lot of other open-source Rails projects.

This turned out to be easier then I expected. I’ve put about 4 hours into it so far, and I can take this migration:

# This is db/migrate/4_test4.rb
class Test4 < ActiveRecord::Migration
  def self.up
    create_table :sidebars do |t|
      t.column :controller, :string
      t.column :active_position, :integer
      t.column :active_config, :text
      t.column :staged_position, :integer
      t.column :staged_config, :text
    end

    Sidebar.create(:active_position=>0, :controller=>'category')
    Sidebar.create(:active_position=>1, :controller=>'static')
    Sidebar.create(:active_position=>2, :controller=>'xml')
  end

  def self.down
    drop_table :sidebars
  end
end

And then do this:

$ irb
irb(main):001:0> require 'migrate'
=> true
irb(main):002:0> require 'db/migrate/4_test4' # the code above
=> true
irb(main):003:0> Test4.up
=> ...
irb(main):004:0> puts DBMigrator::Database.dump('postgresql')
CREATE TABLE sidebars (id serial primary key, controller character varying(255), active_position integer, active_config text, staged_position integer, staged_config text) ;
BEGIN;
INSERT INTO sidebars ("staged_position", "active_config", "active_position", "controller", "staged_config") VALUES(NULL, NULL, 0, 'category', NULL);
COMMIT;
BEGIN;
INSERT INTO sidebars ("staged_position", "active_config", "active_position", "controller", "staged_config") VALUES(NULL, NULL, 1, 'static', NULL);
COMMIT;
BEGIN;
INSERT INTO sidebars ("staged_position", "active_config", "active_position", "controller", "staged_config") VALUES(NULL, NULL, 2, 'xml', NULL);
COMMIT;

Postgres works now, at least with the 4 or 5 examples that I’ve swiped from Typo’s migrations. SQLite and MySQL are nearly working; I think I just need to fake out a couple classes each and they’ll be up and running. Once that’s done, I’ll bundle this all up into a Rails generator so people can do this:

$ ./script/generate schema postgresql
      create  db/schema.postgresql.sql
$ ./script/generate schema mysql
      create  db/schema.mysql.sql
$ ./script/generate schema sqlite
      create  db/schema.sqlite.sql