*scottstuff*

I spent an hour or so this morning making sure that everything for this blog is checked into SVK and then created a couple new trees–one as a testing/staging area for the blog, and another for actual production use.

That gives me 4 branches that I use on a daily basis:

1. //typo/trunk–a copy of Typo’s public Subversion tree.
2. //typo/local–where I do development.
3. //typo/staging–where I stage changes to scottstuff.net.
4. //typo/production–the actual production website.

There are several site-specific changes that live in //typo/staging. This includes changes to the CSS for this site, local graphics, Adsense blocks, links, and eventually things like Flickr integration. They aren’t appropriate for //typo/local, because everything in the local tree is eventually supposed to be merged into the main Typo tree. The production tree has a couple additional changes that only make sense in the production tree–there are a couple small changes to Rails’s public/.htaccess file that need to be made every time I push a change from staging to production. Now that it’s all in SVK, I don’t need to worry about merging this sort of thing anymore–I just run svk smerge //typo/local //typo/staging to push local diffs to the staging area, and then svk smerge //typo/staging //typo/production once I’ve tested things in staging.

As mentioned yesterday, I’ve been looking for a good way to do revision-control my changes to Typo. Typo is maintained in a publicly-visible Subversion tree, so it’s easy to see when changes occur, but I don’t have the ability to write to the Typo tree, so I can’t use it to track my own changes. Since I currently have 4 or 5 different patches for Subversion floating around, plus a pile of local modifications (like adding Adsense banners and changing the ‘Link’ section in the sidebar), I really need some form of revision control.

I briefly tried to use Submaster yesterday, but it hasn’t been updated for Subversion 1.2, and I was unable to commit any changes. So I tried using SVK. Unlike Submaster, SVK is available in Debian’s sid tree, so it was trivial to install. After running apt-get install svk, I followed the tutorial and was up and running in a few minutes. It took a few minutes to import all 272 revisions from the Typo Subversion tree, but once that was complete I was able to make my own branch and start making changes within a couple minutes.

Here are the important steps to using SVK:

Initialize your local repository

The most recent SVK docs suggest using a single repository for everything, instead of managing a repository per project, so I’ll document that path. The command needed is:

$ svk depotmap --init

This will create a new depot in ~/.svk/local. This only needs to happen once, when you first install SVK.

Import your external Subversion repository

Since I want to track Typo’s SVN tree, I needed to tell SVK about it. That’s pretty easy, too:

$ svk mirror svn://leetsoft.com/typo/trunk //typo/trunk
$ svk sync --all

Once those were done, I had a full copy of the Typo SVN tree in my SVK repository’s //typo/trunk directory.

Build a local branch

It’s a lot easier to keep track of changes if you make a local branch and then apply all of your development directly to the branch. This works just like it does in Subversion:

$ svk cp //typo/trunk //typo/local

Check out the local branch

A local working branch isn’t much good if you can’t edit it, so check it out somewhere:

$ svk checkout //typo/local local-typo-changes

This will produce a directory called local-typo-changes in your current working directory, and will fill it will all of the files from the local Typo branch. You can now edit files in this tree and use all of the usual version control commands (svk commit, svk add, svk diff, etc–all of the usual Subversion or CVS commands will work with SVK as well).

Producing patches for external consumption

So, let’s say that you’ve hacked a couple nice new features into your local SVK tree. In my case, I added several different features, committing the changes after each change, but I sometimes I found bugs in features after I committed changes to other features. For instance, my threaded-comment patch was modified in revisions 229, 232, and 246. I wanted to roll up all three of those changes into a single testable patch, but not include any of the other changes that were in my tree, like per-article RSS comment feeds.

This turns out to be very easy. First, I created a subdirectory for patchsets within SVK. This isn’t necessary, but I felt like a bit of organization would be useful:

$ svk mkdir //typo/patchsets

Then I created a new threaded-comments branch, starting with the official Typo SVN tree:

$ svk cp //typo/trunk //typo/patchsets/commentthread

Next, I merged the comment thread changesets from my local tree into the commentthread tree:

$ svk merge -c 229 //typo/local //typo/patchsets/commentthread -l
$ svk merge -c 232 //typo/local //typo/patchsets/commentthread -l
$ svk merge -c 246 //typo/local //typo/patchsets/commentthread -l

The -l uses the previous commit message as part of the new merge commit message.

Finally, I checked out the commentthread tree and tested it:

$ svk checkout //typo/patchsets/commentthread commentthread
$ cd commentthread
$ ruby script/server

This way, I had a completely clean tree, using only the comment thread changes, and I was able to verify that everything worked correctly. As it turned out, I’d missed a couple small changes that had snuck into my main tree as part of other, non-commentthread commits, so if I’d submitted this patch without testing, it wouldn’t have worked for people. So, I made the changes, tested things, checked the changes in, and then produced a patch:

$ xemacs <files>
$ ruby script/server
$ svk commit -m 'Fixed stuff'
$ svk diff //typo/trunk //typo/patchsets/commentthread

Now I can go back to working on my main branch. When I fix bugs in the comment thread code in my local branch, I just need to merge the new commit into the commentthread branch and re-diff. Assuming that revision 270 is comment-thread related:

$ cd commentthread
$ svk merge -c 270 -m 'Merge commit 270 from local tree'
$ svk update
$ ruby script/server
$ svk diff //typo/trunk //typo/patchsets/commentthread

One new patch, tested and ready for submission.

Merging with upstream changes

So, what happens when the upstream SVN tree changes? Well, that’s easy enough. First, re-sync your local copy of the trunk:

$ svk sync //typo/trunk
$ svk smerge //typo/trunk //typo/patchsets/articlerss
<follow the directions>
$ cd <working directory>
$ svk update
<verify and test>
$ svk diff //typo/trunk //typo/patchsets/articlerss

I my case, one of my patches was accepted upstream with minor changes. Once I accepted the merge into my patchset tree, I wanted to promote the changes into my main working tree. I did it this way because it should be easier to merge in this order. Probably. Anyway:

$ svk smerge //typo/patchsets/articlerss //typo/local
<follow the directions>
$ ruby script/server
<test>

At this point, my working tree is synced back up with the mainline.

All in all, SVK isn’t much harder to use then plain Subversion, and the ability to import and modify complete trees is fantastic. My next couple goals are:

1. Figure out how to export my local changes to a publically-visible SVN server, like the one at svn.scottstuff.net.

2. Figure out how to manage a SVK share on my development web server and on my laptop. This shouldn’t be hard, but I’m still not fully up to speed with SVK.

I can’t believe I missed that: Typo doesn’t support extended blog entries, where the main index page only shows a subset of the whole article. Yet another patch to write. Sigh.

Of course, a side-effect of this is that the 5 blog posts that I’ve written with extended entries didn’t transfer correctly from Movable Type. Ugh.

I just upgraded this blog. It was running the latest released version of Typo (version 2.0.6). It’s now running a development version of Typo from Typo’s Subversion tree, plus the patches that I posted here earlier.

One upshot of this is that threaded comments should now work. Feel free to play with them here. If you have any problems with them, then please send me mail. Thanks.

One other change is that comments are now formatted with Markdown, so it’s easy to use bold, italics, lists, and links. See the Markdown page for details; eventually I’ll add a blurb to the comment form explaining the simple stuff.

My recent spate of typo patches has left me with a bit of a dilemma. Typo is managed using the Subversion version control system. That’s great–Subversion is quite nice–except I don’t have (and don’t really want) commit privileges to the master Typo SVN repository. I can check things out and edit them, and use Subversion to make patches, but once my patches start piling up, I’m left without any form of local version control. I can’t tell my patches apart, and there’s no easy way for me to merge my patches with other changes without totally losing control of which change goes with which patch.

This has to be a fairly common problem–a lot of open-source software is developed using Subversion these days, and CVS has similar problems. I could just set up my own Subversion repository for Typo changes, but syncing it with the upstream tree will be a pain.

Does anyone have any “best practices” suggestions?

The only thing that I see online that might be helpful is Submaster, but it’s full of comments like this:

WARNING: This is just a proof-of-concept implementation. Don’t use it in production environments! You have been warned!

NOTE: This version of SubMaster is written for Subversion 0.35.0. It might not work with any other version of subversion as it is now.

That doesn’t instill confidence.

Update: I gave Submaster a brief spin this morning. I was able to create a Submaster repository, but I was unable to check changes into the local repository without getting errors from svn. I suspect that it has problems with Subversion 1.2.

Since that doesn’t look like it’ll work, I’m going to give SVK a try. The short tutorial on the SVK wiki makes it look pretty simple. It’s currently cloning 272 revisions out of the Typo SVN tree; hopefully it’ll actually work right once it’s done.

Update 2: This is continued in a new article.

I work up early today, so I made a few more changes to Typo. In addition to the earlier threaded comments patch, I’ve added:

• Sort category lists by name. By default, Typo displays category lists in a more or less random order. This patch causes them to be sorted by name.
• Enhanced text filtering. This adds support for SmartyPants-style filtering with RubyPants, fixes a cross-site-scripting vulnerability, adds the ability to stack filters, and splits comment formating away from article formatting, so you can use different defaults for each.

All three patches are against revision 262 from the Typo SVN repository. For convenience, I’ve generated a combined patch with all three sets of changes, plus a couple other minor fixes.

I finally finished my patches to add threaded comments to Typo. I haven’t updated this site yet, but my test server is up and running with the threaded code. The same code also adds subject lines for comments and keeps track of commenters email addresses (if they choose to submit them).

The Ruby side of threaded comments was really easy, but getting all of the Javascript to allow threaded comment editing completely in-place using Ajax was a royal pain in the neck. It’s been years since I last touched Javascript, so I was forced to do a lot of googling. It would have been a lot easier if I’d just created a new page for comment editing and then bounced the user between that and the comment display page, but it wouldn’t have been as cool.

Next up, I’m going to add support for RubyPants, so I can get smart quotes and dashes working correctly again. Typo’s current code for text filtering assumes that you only want to apply a single filter to each item; I’m not sure if I’ll just create a ‘markdown+rubypants’ filter or add support for stacking filters. I can think of a couple other filters that would be useful, but I don’t know that I really want to create a management interface for filter stacking.

After that, I’m going to add the first bit of comment authentication. I’d like to have my comments automatically tagged so I can apply visual effects to the comments–see Mike Davidson’s blog for a nice example of the effect that I’d like to achieve. In the short run, my goal is to add an optional user_id field to the comment field and then link that to the users table. Then I’ll add a css_class field to the users table and use it when generating comments.

Once that’s done, I’ll start in on modifying the CSS files that come with Typo. Right now, this blog looks identical to every other Typo blog out there, and I really don’t care for that.

As of about 5 minutes ago, the Typo version of this blog is now live, and the old Movable Type version is disabled. All old URLs should still work–I spent quite a bit of time generating a list of redirects to send each old page to the correct URL in Typo.

I still have a lot of template work to do–this site looks really plain to me, and I’ve been meaning to redesign the site for a while now, anyway. That should happen over the next week or so. I’d also like to implement threaded comments in Typo. The display end of threaded comments looks trivial to implement, but I’m not so sure about the other end of things.

Update–Wow, Typo publishes new articles way faster then MT did. From clicking the ‘Publish’ button in Ecto to seeing the new article appear in NetNewsWire is only a couple seconds. MT 2.6 took around a minute before the article was ready. On the downside, categories in Ecto seem to be broken.

If anyone sees problems with the new site, either leave a comment here or send me mail. Thanks.

I’ve spent most of the evening playing with Typo, and I believe I’ve fixed all of the problems that I was seeing this morning. Most of the problems that I saw were caused by the import script; adding a bunch of ORDER BY clauses suddenly made it import items in the same order that they were originally created.

At this point, I’m nearly ready to make it live. I just have three issues left:

1. Typo doesn’t ask for email addresses for comments, even though there’s a spot in the DB for them. I’ve found that it’s a lot easier to respond to people when the can provide me with an email address, even if it’s never published.

2. Similarly, there’s no way for it to display *my* email address. This should just be a quick template hack.

3. Finally, I really dislike Typo’s default date format. Instead of providing an exact date, it says things like “Posted by Scott 672 days ago.” It looks cool for recent entries, but it’s pointless for anything more then a couple weeks old.

That’s pretty much it. Once those are done, I’ll make Typo my primary blog and start turning Movable Type off. If anyone’s interested, they can see my work-in-progress Typo blog, although I don’t guarantee that it’ll always be running.

My test installation of Typo is up and running. First impressions:

1. It’s easy to install, once you have Rails working. I’d never actually installed Rails on my home web server, so I had to do the whole install gems, install rails, hunt down dependancies bit. I had an early version of Active Record installed, and it was being loaded instead of the current version that comes with rails, so I had to hunt it down and kill it.

2. The MT-to-typo conversion script worked okay, once I fixed the MySQL-isms in it. Specifically, it assumed that it could access the MT database by using databasename.tablename; that syntax gets you into different namespaces in Postgres, not different databases. I worked around it by copying the MT tables into the Typo database and then removing the databasename. part of the queries. It probably would have been easier to just pass public as the database name, though.

3. The script also assumes that all articles are formatted as HTML. Mine are (almost) all Markdown. To fix this, I had to install BlueCloth (gem install bluecloth) and then rebuild all of the cached versions of each of my articles. There’s no obvious way to do this in the Typo UI, but Typo’s internals makes it easy. All I had to do was run the Rails script/console command and then run Article.find_all.each { |article| article.text_filter='markdown'; article.save }. For the Ruby-phobic, this iterates through each Article in Typo’s DB, changing the formatting type and then saving the results back to the DB. Typo’s Article model hooks the save method to verify that text format conversions are handled correctly, so that’s all I really had to do. Nice and simple.

4. It’s kind of slow. I’m running Typo in debugging mode right now, and Rails is always slow in Debugging mode. I suspect that Typo will still be a bit pokey, even in production mode, though. I need to look into how it handles caching.

5. For some reason, not everything is sorted correctly. The ‘latest comments’ sidebar seems to be sorted randomly. This is probably an artifact of the conversion script–it looks like entity IDs aren’t in the same order as they were in MT, and that screws stuff up. That’s just a one-line fix to the conversion script.

6. Comments are listed newest-first. I’d rather have them appear in the order that they were written. This should be a one-line change.

7. The admin interface is sort of sparse. Rails makes it really easy to make calls directly into Typo’s API from the command-line, so I’m not too concerned with the lack of things to click on.

8. Somehow, some of the Article-Category associations were lost during the conversion. Most of them seem okay, but not all of them. This might take a bit of work to fix.

9. The categories list on the sidebar seems to be in ID order; it should be sorted by category name. This is easy to fix.

Nothing here looks fatally broken. I need to spend 15 minutes updating the conversion script and another few minutes fixing little issues with Typo, but the system seems to work well enough, at least on initial inspection. I’ll re-import the MT data again later today using the fixed conversion script, and then see what it takes to get it working with Apache in production mode. If that works out okay, then I’ll spend some time tweaking the CSS to make it look a bit more like my current site and then make it live.

Oh–one more thing–I need to have the conversion script save the old URLs from each entry to a file so I can write a bit redirect list that will point each MT article’s URL to the new Typo URL. I get too much traffic from Google to lose all of the links.