*scottstuff*

I just committed my time-limited Rails cache to the Typo trunk. This is an outgrowth of my improved action cache, and fixes most of the caching problems identified in the Rails caching presentation that I gave a few months back.

I haven’t benchmarked it yet, and I’m sure that it’ll need a bit of tuning, but it solves so many architectural issues for Typo that I went ahead and committed it anyway.

It’s currently implemented as a Rails plugin; I’ll probably bundle it up as a stand-alone plugin once I’m happy with the way it works with Typo.

I’m finally getting back into Typo hacking after too long away. I tried to apply a few patches last weekend, but I was traveling and my network access was too spotty. So, I spent my time adding a bit of new functionality. Since then, I’ve been debating whether to commit it to Typo or not. I decided I’d write about it here and see which way the comments go.

The code in question is a time-limited cache for Rails. I’d like to be able to say “cache this page, but only for three hours. After three hours, re-render the page.” This sort of thing comes up in Typo all the time. The most obvious example is the sidebar–some of the sidebar components display information with a short lifetime, and it’d be dumb to keep pages in the cache for weeks when they include sidebar data that’s only good for hours. This isn’t really a problem on busy sites, because the current cache sweeper usually resorts to sweeping the entire cache every time a new article is posted, but it’s a pain on slower sites.

There are certainly other ways to fix the sidebar problem (AJAX sidebars are the obvious example), but the same basic pattern comes up all over Typo. A few examples:

1. Users keep requesting the ability to create articles with a publication date in the future. The article won’t appear on the site until after the publication date. This is common CMS feature, and apparently other blog engines have it as well, but it really doesn’t mesh with Typo’s current cache, because there’s no way to say “sweep the cache at 7:30 today” short of adding a cron job for every article that’s posted this way.

2. We have a bunch of aggregation classes that suck data off of other sites, like Flickr, Upcoming.org, and so on. These usually end up as sidebars, but we need to cache the back-end data somewhere. An expiring fragment cache would work perfectly for this.

3. On really busy sites, we could use something like this to avoid rebuilding comment pages on every comment–we could drop the sweep-on-new-comment code and swap for expire-after-5-minutes. If you’re getting more then 1 comment every 5 minutes, this would be a win. If you’re getting a comment every few seconds (think Slashdot or Curt Hibb’s “hammer my comments” post), this would be a major win.

To accomplish this, I added two new features. First, I added a set of “meta-fragment cache” methods, building on Rails’ existing fragment cache. The fragment cache stores (key, value) pairs, while the meta-fragment code stores (key, value, metadata_hash) triples. This is simply implemented as two fragment cache entries, one for the data and one for the serialized metadata hash.

Then, on top of that, I re-implemented my caches_action_with_params code. This is a variant of Rails’ native action cache with a number of cleanups and bugfixes.

When all is said and done, you’re left with a controller that looks something like this:

class ArticleController < ApplicationController
    caches_action_with_params :read

    def read
      response.lifetime = 3600 # 1 hour
      ...
    end
end

That’s it–the read action will now be cached with a 1 hour lifespan. After an hour, the cached version will expire. If response.lifetime isn’t set, then the page won’t expire on its own, and it’ll need to be swept as usual.

So here’s the big question–should this go into Typo? I can see good arguments on each side.

Pro:

• It solves a lot of cache-with-parameter problems that we’ve had.
• Switching to some variant of the action cache means that switching between production and development mode doesn’t leave cache problem. This is a major cause of bug reports from new users.
• It’ll let us implement future posting easily.
• It’ll make it easy for sidebars to stay current.
• It’ll let us move the aggregation backends behind the sidebars to a more reasonable architecture. For example, we’ll be able to use the Flickr class for the Flickr sidebar instead of (mis-)parsing their RSS feed.
• It’ll make us less dependent on web server configuration and weird rewrite rules.

Con:

• It’s slower then the page cache. I haven’t benchmarked my new code yet, but the last time I checked, on my box I could handle almost 2400 page cache requests per second, while the action cache was good for *10* hits per second. That exposed a couple major Typo performance bugs; I suspect that retesting with the new code would give us 100-200 hits/second, which is pretty busy for a blog. Still, this may be an issue for shared hosting providers.
• The action cache serves cached pages via Rails, while the page cache serves the same pages directly from the webserver without invoking Rails at all. Because of this, I suspect that a lot of sites will want to increase the number of FastCGI Typo processes that they run. With the page cache, running with one FCGI process was usually okay; with the action cache, it might be better to use a second process.

Those are the only two major problems that I see. Basically, if we switch to using the action cache (in any form), we’re going to be harder on big hosting companies like TextDrive and Planet Argon, and they’ve been very supportive of Typo in the past.

Does anyone feel strongly about this one way or another?

I just released a new version of my Rails Schema Generator;. There are two changes in this release:

• Migrations should finally work correctly using either old-style (1_migrate_to_foo) or new-style (001_migrate_to_foo) numbering.
• SQL Server support has been added.

There’s also some support for Oracle hiding in the tree, but it’s disabled as I can’t see a way to get it to work without having all of the Oracle OCI libraries installed. So, if you’re using Oracle with Rails, and you care, try uncommenting the Oracle line in schema_generator.rb and then leave a comment telling me what happens.

It’s taken forever, but I finally have a version of my schema generator that works with Rails 1.0. The actual fix was fairly minor (7 lines), but getting to a point where I could test it has been amazingly difficult–my new PowerBook had a bit of disk corruption and Ruby stopped working. Rebuilding the entire Ruby distribution from DarwinPorts wasn’t enough to fix the problem, somehow, but installing a new copy of REXML did the trick for reasons that are too obscure for understanding.

So, go and enjoy.

Update: I just bumped it to 1.0.1 to fix a MySQL dependancy bug that I’d forgotten about for 1.0.0. Have fun.

I just released a new version of my Rails Schema Generator on Rubyforge. The schema generator takes a collection of Rails database migration scripts and assembles a complete set of SQL schema files using only the information from the migrations. This release supports MySQL, PostgreSQL, and SQLite; it can generate schemas for all three DB types even if the databases aren’t installed on the system.

We’ve found that the schema generator drastically lowers the work needed to keep Typo working correctly with multiple database types.

This release fixes a number of bugs and should finally work correctly with all common migration operations, including field renaming. This has only been tested with Rails 0.14.4, and at least one API has changed recently, so this may not work with earlier 0.14.x releases.

To use the schema generator, run gem install schema_generator, and then (from your Rails project directory) ./script/generate schema. This will create several schema files in the db/ directory, prompting you before overwriting existing files.

I just committed a big patch to Typo that adds the first part of a notification framework. This piece adds the ability to send email or Jabber messages whenever certain events happen. The events currently supported are new article creation and new comment posts.

This is still a work-in-progress, but it should be safe for a few brave people to use. You’ll have to edit your user settings in Typo to configure your notification settings and then edit the general settings to configure which email address Typo should use. After that, it should just work.

I just committed a minor new feature for Typo–there is now a route-of-last-resort that can generate HTTP redirects out of a database table. This is something that people have asked for for months, because it’s of the easier ways to keep old URLs working when moving to Typo.

At the moment, none of the converters know anything about it, so if you want to use it, then you’re going to have to do a bit of coding. I suspect that we’ll see patches within a few days that will help people make the jump. The basic code for adding a redirect is pretty simple:

  Redirect.create(
    :from_path => 'archives/000001.html', 
    :to => '/articles/2005/11/17/first-post')

This would be a bit cleaner if from wasn’t a reserved word in SQL. The from_path bit should be relative to Typo’s root, and shouldn’t start with a slash.

Unfortunately, none of Typo’s URL helpers are available inside of converters, but it shouldn’t be too hard to code this up. Send me mail if you need help.

Typo 2.6.0 is out. Tobi suggested that we bump the version number from 2.5.9 to 2.6.0; I probably should have done this for 2.5.7, but better late then never.

This is largely a bug-fix release–it fixes one brown-paper-bag bug in 2.5.7/2.5.8 (renderfix.rb was in the wrong directory), as well as adding a couple additional cleanups and 0.14.x compatibility fixes.

There are two fixes in here that theme contest users need to pay attention to. First, as part of the live search fix for 0.14.x, I moved the observer for the search field from the layout (default.rhtml) into the search partial (_search.rhtml). This matches the current trunk, so it’ll make themes more portable, but if your theme’s default.rhtml still has the search observer in it, then weird and wonderful things will happen when users try to search. So either nuke the observer in default.rhtml or override _search.rhtml to make it match. I also back-ported a second change from the trunk–each sidebar item is now wrapped in <div class="sidebar-node">. I’d be amazed if this managed to break anyone’s CSS, but theme builders might wish to use it to style the sidebar.

At this point, I have no plans for any further releases between now and November 28th, the last day of the theme contest. If any major bugs pop up, I’ll put out a new release to fix them, but I’m not planning on adding any additional features to 2.6.x, *or* releasing the trunk (as Typo 4.0–it’s a long story) before the end of the theme contest. Good luck.

Well, that was fun. Typo 2.5.7 lasted 80 minutes, only to be replaced by 2.5.8.

This is a security release, fixing a bug that has been in Typo since at least 2.5.0. All Typo users should upgrade to either 2.5.8 or the current Subversion trunk.

Typo has had theme support since before Typo 2.5.0 was released, but the first version of Typo’s theme engine didn’t include the ability for themes to change any of the HTML generated by Typo’s views. Themes could change CSS, graphics, and the top-level layout that generated the HTML for the site, but individual HTML blocks, like comments or article bodies, couldn’t easily be changed.

The latest release, Typo 2.5.7, fixes this shortcoming. It’s now possible for themes to completely override any bit of HTML or XML that Typo generates. Here’s how it works:

Like all Rails apps, Typo stores all of its HTML views in app/views/<controller>/<viewname>. For example, the view that generates individual article pages is app/views/articles/read.rhtml. The theme system in Typo 2.5.7 and newer lets themes override views by putting a replacement file into themes/<themename>/views/<controller>/<viewname>. So if I created a new theme called scottstuff and wanted to override the read.rhtml view from above, I’d put the new view into themes/scottstuff/views/articles/read.rhtml. Typo would then read this view from the theme instead of app/views.

It’s also possible to replace the HTML generated by sidebar plugins; just put the replacement view into themes/<themename>/views/plugins/sidebars/<sidebarname>/content.rhtml.

Typo should pick up on the new views immediately, even in production mode, but you’ll have to flush the page cache before the views really take effect.

module ActionView
  class Base
    alias_method :__render_file, :render_file

    def render_file(template_path, use_full_path = true, local_assigns = {})
      search_path = [
        "../themes/#{config[:theme]}/views",     # for components
        "../../themes/#{config[:theme]}/views",  # for normal views
        "."                                      # fallback
      ]

      if use_full_path
        search_path.each do |prefix|
          theme_path = prefix+'/'+template_path
          begin
            template_extension = pick_template_extension(theme_path)
          rescue ActionView::ActionViewError => err
            next
          end
          return __render_file(theme_path, use_full_path, local_assigns)
        end
      else
        __render_file(template_path, use_full_path, local_assigns)
      end
    end
  end
end