IFDB Download Adviser and Meta Installer

Mike Roberts
August, 2009
Original version: September, 2007

Executive Summary

I've designed a new web-based system to provide users with customized installation instructions for the interpreter software required to play virtually any IF game. The custom instruction generator is called the Download Adviser. The system is designed to work with any IF interpreter system that follows the typical VM/interpreter design (namely, a format for distributing game files, and an interpreter program to play them), as well as with games packaged as native applications.

The goal is to make IFDB a one-stop shopping site for IF downloads. That is, IFDB should link directly to everything you need to play a given game - the game file, the interpreter engine, and any ZIP-type extractor tool. We don't merely give users links to other sites where they can find these tools; we'll actually give them direct download links to every bit of software they need, along with instructions to install it all, all on a single page.

The system also supports an even more automatic system called the Meta Installer. This is an optional client/server component, optional in two senses: it's optional for the OS point person to provide it at all for their OS, and it's optional for users to use, because it requires a download to the client machine. This currently exists on Windows in exactly the form described in this document; it's a browser plug-in that provides a button on each IFDB game page that completely automates the entire download and install process - it's basically a "Click to Play" button. Similar functionality is available on the Mac in the form of Andrew Hunter's Zoom multi-format IF player. Zoom has a feature for browsing IFDB pages, and has a "Click to Play" button that lets you download and run any game you're looking at through its IFDB browser.

The Meta Installer architecture is designed to allow the client component to be implemented on any OS, and supports a variety of configurations, including browser plug-ins (such as we use on Windows) and custom IFDB browser apps (such as Zoom on Mac). This document describes the generic architecture of the client piece, plus its specific implementation on Windows. This is all meant as an implementation guide for platform developers who want to create similar functionality for other operating systems, or who want to integrate similar functionality into their custom apps.

Call for Platform Experts

Are you an expert in one of the popular IF game engines? Are you an expert on an operating system? If so, we'd like to ask you to consider becoming an official IFDB Download Adviser expert.

A key element of the Download Adviser design is that the work of entering and maintaining platform-specific instructions can be divided up among a group of experts in the different operating systems and game engine format experts. This gives the people closest to the various systems direct access to update their sections, so they don't have to wait for a central administrator to notice changes or enter updates.

We currently have participants covering the major operating systems and game engine formats, but some of the smaller systems are not yet covered. If you're interested in taking ownership for the IFDB Download Adviser for a game engine and/or operating system, please contact me.

Here's what you'd be asked to do as an engine or OS Download Adviser expert:

Background

I've been working on a new IF catalog and recommendation site that I'm calling IFDB (for Interactive Fiction DataBase). The main thrust of the site is to try some new approaches to community recommendations for IF, but I felt it was important for the sake of user experience to provide integrated catalog features as well. Part of that integration is, of course, direct download links to the cataloged games.

A perennial problem for IF newbies is figuring out how to play a particular IF game once they've found it and downloaded it. The whole idea that you'd need separate "interpreter" software to play a game is rather surprising to the uninitiated - it's just not something you run into with other computer game formats.

It's not hard for new players to understand the idea once someone explains it to them - everyone's accustomed to the analogous requirements for playing MP3 tracks or watching DVDs. But even once you understand the idea in principle, there are still all the practical details: knowing which interpreter you need, finding it, installing it, and keeping it up to date.

Those of us who've been playing or writing IF for a long time take a lot of background knowledge for granted - we know who Hugo is, what a "Frotz" is, how Inform, Z Machine, and Frotz are related, how the directories in the IF Archive are laid out - heck, we know there's such a thing as the IF Archive in the first place.

If you step back and think about it, there are an awful lot of little details you have to correlate to get this stuff working. I've actually been stepping back and thinking about it, because I've been creating sample "recipes" for setting up to run particular IF formats on particular operating systems. What I've found is that all the grousing you hear from newbies on the newsgroups is a lot more justified than I used to think. The main problems are

I've been working on an approach that I think will solve these problems and make it easy for newbies to get accurate, up-to-date information about a particular game they want to play.

My approach is to create a matrix of recipes - one for each combination of format and operating system - and integrate this into the IFDB catalog. This might sound like just another FAQ mechanism, but I think it's a little different. Here's how it solves the problems identified above:

IFDB actually has two mechanisms for helping newbies with installation. The first is the "Download Adviser," which works basically as described above: when the user picks a game, IFDB cross-references the format (as specified in the game listing) with the user's operating system (obtained from the browser ID string, or selected by the user from a drop-down list), looks up that combination in the matrix, and displays the human-readable instructions stored there. The goal is to provide precise and complete instructions, with direct hyperlinks to any files that need to be downloaded, and without any conditionals ("if you're using this version of Windows...") or buck-passing to FAQs or other third-party sites. The instructions should be completely linear and self-contained, so that a user can just run down the list and have the game actually up and running when done.

The second mechanism is the IFDB Meta Installer. This is essentially an automated version of the Download Adviser. Like the DLA, the Meta Installer uses a server-side program that correlates a game's file format information with the user's operating system to produce system-specific download instructions. Unlike the DLA, however, these download instructions are machine-readable rather than human-readable, because they're designed for consumption by a separate client-side portion of the Meta Installer. The client-side piece executes the instructions to actually perform the installation directly, so that the user doesn't have to do any manual installation work.

Download Adviser Algorithm

Note: To see an example of the DLA in action, go to ifdb.tads.org, search for Deep Space Drifter (or any other TADS 2 game), click "Show Me How" in the Download box, and make sure that you have a Windows OS version selected in the OS list. I haven't loaded instructions for most of the other format/OS combinations yet, but this will give you an idea of how the instructions are displayed.

The principal behind the DLA is straightforward. It correlates several pieces of information in a database to find the correct set of instructions. We start out with the following information:

The DLA's goal is to produce step-by-step instructions that start with a "bare" machine of the user's selected OS (i.e., with only the normal set of factory pre-installed software that we can count on being available on any machine running this OS), and finish with the selected game up and running. The algorithm takes the information above and uses it as follows to produce the instruction list:

Handling intepreter version incompatibilities

The basic versioning assumption we make for a game format is that a newer interpreter can play an older game. That is, if a game is compatible with version N of the interpreter, we assume that it's also compatible with interpreter version N+1, N+2, etc.

Whenever this assumption holds, the DLA only needs to know about the latest version of the format's interpreter for a given OS. As long as a user has the latest interpreter, they can be assured that they can play any game created for the format.

However, this assumption is not always true. Some formats have been known to break compatibility at certain version updates. When compatibility is broken, the solution for the DLA is to treat the two versions as separate formats.

For example, TADS 2 and TADS 3 are separate formats in the DLA because they require separate interpreters. (On most platforms, this detail is hidden from users because the TADS interpreters are bundled with both VM versions, and can automatically detect which one to run for a given game. But this isn't necessarily true everywhere, so it's important to treat the two formats as incompatible in the DLA records to allow specifying different interpreters for operating systems that require them.)

Sample DLA Entry

Here's the DLA entry for TADS 2 on Windows:

Instructions for downloading and installing the Interpreter/Player/Viewer application for this format:

If you don't already have a TADS Interpreter on your system, install
it as follows:

<ul>
<li><run>http://www.ifarchive.org/if-archive/programming/tads2/executables/htads_playkit_HT12.exe</run>

<li>Follow the Setup program's prompts to install the Interpreter.
</ul>
   

Instructions for running the Interpreter/Player/Viewer application to run/play/view a local file called {fname}:


To run the game:
<ul>
<li>From the Start menu, select TADS > HTML TADS Interpreter
<li>In the Interpreter window, open the <b>File</b> menu
and select <b>Open New Game</b>.  This will display a file
selector dialog - select {fname} to start the game.
</ul>
   

To see how these entries look once formatted, go to IFDB, search for Deep Space Drifter, click through to the game page, click the "Show Me How" button near the top right; if necessary, select a Windows version from the OS list and click Go.


If your main interest is the human-readable Download Adviser instructions, you can stop here - the rest of this document is about the Meta Installer design.

Meta Installer Operation

The Meta Installer works on the same principle as the Download Adviser to generate the install plan, and uses the same algorithm that the DLA uses to generate instructions. However, rather than assembling the human-readable form of the instructions, the Meta Installer version of the algorithm assembles a machine-readable installation plan. The source of the pieces of the machine-readable plan is right alongside the human-readable instructions - the Meta Install plan fragments are stored in the Instructions table in columns parallel to the human-readable instruction fragments.

Every operating system has different installer needs, so there's no point in designing a universal specification for the detailed installation procedure. Instead, we define only the overall flow of information, and we leave it up to the individual plug-ins to define the details of their installation instructions.

Note that we refer to a "plug-in" as the client-side component that carries out the installation tasks. We use this term generically; this piece will have no user interface presence on the game listing page itself, so it can be implemented using almost any sort of browser extensibility mechanism - or even by creating a non-browser client. Some examples:

In order to allow maximum flexibility for the client piece, we've tried to keep the interaction between the client, server, and game listing page as simple as possible. The plug-in (or whatever it is) has no user interface presence on the game listing page - the only thing it has to do is provide a simple scripting interface, which the "Play Now" button invokes via Javascript when the user clicks on the button. In fact, scripting isn't even necessarily required: the invocation could alternatively be accomplished by hyperlinking the "Play Now" button to a special HREF (using a custom scheme, for example), in which case the client software would simply need a way to intercept outgoing HREF requests, and would trigger the installer when the special HREF appears. Remember, we control the server: we can work with each plug-in developer to create whatever custom HTML plumbing you need to turn a user click on the "Play Now" button into a plug-in invocation.

The overall control flow for the plug-in is as follows:

IFDB uses a limited subset of XML functionality in the returned XML. This is intentional, so that the client can use a very lightweight parser implementation. The XML returned will be limited as follows:

Note that these restrictions apply to the generic XML that IFDB generates. The additional XML that's specific to each operating system can ignore these restrictions if desired. Since the OS-specific part is up to the plug-in developers for each OS to define, this means that you can decide when designing your plug-in whether or not you want any additional XML functionality in your OS-specific part, and weight that against the cost of parsing it.

The XML reply has this overall structure:

  <?xml version="1.0"?>
  <autoinstall xmlns="http://ifdb.tads.org/autoinstall" version="1">
     contents
  </autoinstall>

Note: the result XML won't necessarily be formatted as shown. In particular, we show the results with a nice newline and indenting structure for readability, but the placement of newlines and start-of-line indenting might vary in practice. Clients must parse the data according to the normal XML rules for treatment of whitespace - don't count on indenting or newlines in actual results as a guide to the tag nesting structure.

On any error, the contents will have this structure:

  <error>
    <id>error code</id>
    <message>human-readable error message</message>
  </error>

The possible codes are:

On success, the contents will be as follows:

  <clientversions>
     version info
  </clientversions>
  <ifarchivemirror>mirror URL</ifarchivemirror>
  <tuid>tuid</tuid>
  <ifids>
    <ifid>ifid1</ifid>
    <ifid>ifid2</ifid>
    ...
  </ifids>
  <version>version</version>
  <title>game title</title>
  <author>game author</author>
  <download>
    <game>
      <href>game download URL</href>
      <compression>
        <id>format ID</id>
        <name>format name</name>
        <primaryfile>filename</primaryfile>
      </compression>
      <type>file type</type>
      <format>
        <id>format ID</id>
        <name>descriptive name</name>
        <interpreter>
          items
        </interpreter>
        <interpreter>
          items 2
        </interpreter>
        ...
      </format>
    </game>
    <extra>
      <href>extra file URL</href>
      <title>extra file title</title>
      <desc>extra file description</desc>
      <formatid>format ID</formatid>
      <compression>
        <id>format ID</id>
        <primaryfile>filename</primaryfile>
      </compression>
    </extra>
    ... more extras ...
  </download>

The <clientversions> element contains information about the current versions of the plug-in software. This is to facilitate automatic updates of the client piece, and is discussed in detail below.

The <ifarchivemirror> tag gives the base URL to the user's preferred IF Archive mirror site. This will be supplied only if the user is logged in. The server will already have remapped any IF Archive URLs it actually knows about within the XML; so, for example, the download/game/href and download/extras/extra/href values will already be mapped to the correct mirror URL when you receive the XML. However, the server won't map URLs in the <interpreter> sections, since the contents of those sections are opaque to the server - the plug-in is the only piece that can interpret those sections, so it's where any URL remapping must take place. The mirror URL is supplied in the XML information for this reason, and also so the plug-in can remap any IF Archive URLs encoded directly into the plug-in itself (for example, a plug-in might be designed to check for updates to its own code by looking at a particular IF Archive URL). To use the mirror URL information, simply look for the prefix "http://www.ifarchive.org/if-archive/" in any URL you'll be using for a download, and substitute the contents of the <ifarchivemirror> tag for that prefix when you find it.

The name and title can be used to ask the user for permission and confirmation, to make sure the user intended to install this particular game.

The TUID, IFID, and version number are provided mostly in case the Meta Installer wants to keep track of which games have previously been downloaded in its own internal (client-side) database. The TUID is a stable, unique identifier for the game [6], so it can be used to determine if the same game is already installed - if it is, the plug-in can simply run the existing copy rather than downloading it again. The version is provided so that the plug-in can notify the user that an updated version is available. (Note that the formatting of the version number is arbitrary and might not be suitable for any sort of general-purpose greater-than/less-than comparison algorithm - it could be something like "7", "1.2.29", or even "Third Edition"; it's probably best to assume that the server version is newer if it doesn't match a previous copy's version string byte-for-byte.)

If you do want to cross-reference new requests with past downloads, it's up to you to maintain a record of the past downloads, and to make sure that the user hasn't deleted the game, etc. Maintaining a database of past downloads is nice because it turns IFDB into a sort of universal IF jukebox; from the user's perspective, it doesn't matter whether or not a game has been downloaded before, since clicking "Play It Now" will have the same effect either way, modulo the download time of new downloads. (In the future, we could even think about integrating local saved games with the IFDB listing pages, with some help from the plug-in.)

The IFIDs are provided for the same reason as the TUID. The client component can use the IFIDs to identify the game instead of the TUID, if desired. IFIDs are part of the Treaty of Babel, so they're more likely to be interoperable; this might be important to some client implementations. (But see note [6] on why the TUID might be easier to use.)

The <download> section gives the download information for one game program file, plus any number of extra accompanying files.

The game download information is given as a URL plus file type information, plus possible compression information. The URL is a link to the game's story file, or to a compressed archive containing the story file. This is a direct link to the file resource itself, not to an HTML page containing a link to the file resource, so downloading the game is just a matter of doing an HTTP GET (or equivalent for other protocols, if the scheme in the URL is something other than "http:") on this URL and saving the returned content to the local hard disk.

If the compression information is present, it gives the ID and name of the compression format used for the file. The ID comes from a published list on the IFDB site, and includes entries for ZIP, .tar.gz, .tar.Z, etc. [9] These ID values are permanent public ID values that will not change over time (although new ones might be added). The compression format name is a human-readable name for the format; this isn't guaranteed to be stable over time for a given format, and could even be affected by things like the user's language preferences. The compression format name is supplied mostly for UI purposes; for example, if the client software doesn't know how to handle a format, it can display the format name in an error message.

The "primaryfile" part of the compression entry tells you the name of the game itself within the compressed archive - this is the name of the file that you'll send to the interpreter to run the game after you've extracted the contents of the archive. Note that this information isn't guaranteed to be provided, since it depends on someone having entered it for the download link in the game's listing; if it's not present, the Meta Installer might be able to guess anyway (by scanning the unpacked files and looking for one with the right filename extension, file signature, OS-specific type metadata, etc) - to the extent this is possible for a given format on a given OS, it will have to be encoded in the system-specific <interpreter> section.

The <type> gives the type of the file. This is orthogonal to the format: it tells you what you do with the file rather than how. There are three type codes:

Note that the IFDB server prioritizes file matches in the order shown - so if a story file version is available, that will be the file specified in the XML, even if the other types are also available. Story files are almost always smaller and safer (in terms of malware) than executables, so IFDB returns them in preference to executable formats whenever possible.

The <format> section gives information on the file's format: its human-readable name ("TADS 2", "Z-Machine", etc), and its ID code. The ID code is the published identifier that IFDB uses to identify the format. The Meta Installer might want to store this information its internal database of installed formats, since it might be useful to use in future transactions with the IFDB server. For example, the Meta Installer could pass this to IFDB to request an update for the format separately from any particular game.

The special format "executable" indicates that the downloaded file (or the primary file, if it's a compressed archive) is a native executable application for your operating system. When this type of file is downloaded, there will be no <interpreter> section, for obvious reasons; there will also be no descriptive name (the <name> section) for the format.

Each <interpreter> section provides information on how to download and install an interpreter for this game's format. The contents of this section are entirely up to the plug-in developers for each OS to design. The IFDB instructions matrix is keyed on OS - it doesn't have separate slots for multiple plug-ins on one OS - so all of the plug-ins for a given OS will have to share the same instruction format. However, to the extent an OS has multiple plug-ins with different needs, the plug-ins can still easily differentiate their instructions simply by splitting them up into separate XML subsections. The OS/format expert would simply be required to enter the combination of <interpreter> sub-items to cover all of the different plug-ins for that OS.

The <interpreter> section can appear more than once, because some formats have multiple interpreters available for the same systems. For example, there are several Z-Machine interpreters for Windows. By listing all available interpreters, we allow the Meta Installer to detect if the user has any interpreter compatible with the format. Some users prefer one interpreter over another, so we don't want to bother a user who's already installed one of the alternative terps. If none of the alternatives are found on the user's system, it's up to each OS plug-in to determine how to proceed; we recommend treating the first <interpreter> section as the "preferred" interpreter, and automatically installing that one. (A plug-in could alternatively offer the user the whole list and let the user decide, but this seems at odds with one of the key goals of the project, namely making the game configuration process as seamless and automatic as possible.)

In general IFDB will generate the <interpreter> sections as follows:

The <extra> file sections give the download information for any supplemental files (documentation, feelies, walkthroughs, source code, etc) associated with the game. The title and description information are as entered in the IFDB listing for the file. The compression information works the same way it does for the game file; for extras, the "primary file" is usually irrelevant, but the server includes the information if it's in the database, in case it's ever of interest. For an extra file, the format information is provided only as a format ID (which comes from the same published list as the format ID for a game file). For extra files, we don't provide any information on a viewer application, since it's well beyond the scope of this project to provide automatic installation procedures for arbitrary document format viewers.

Versioning and automatic update support

The client plug-ins will inevitably be updated over time, to add features and fix bugs. Some client implementations might want to provide integrated automatic updates.

This is particularly important for clients that are implemented as browser plug-ins of one kind or another, since these types of programs tend to be fairly invisible to the end user - they usually just sort of blend into the browser, and don't tend to have a separate user interface presence of their own. This makes it important for the client to be as self-maintaining as possible.

To facilitate automatic updates, the XML sent from the server includes the <clientversions> section. This section contains elements that give the version information for the current client programs available for download from the server. This section's direct children are sub-tags for the individual client implementations. Currently, the contents are as follows:

<windows-plugin>
  version information for the Windows browser plug-ins
</windows-plugin>

The contents of a given section are up to the corresponding plug-in developer to define. In general, the plug-in developer will send the IFDB administrators the exact contents of their relevant section with each release update, and the IFDB administrators will copy this text verbatim into the main version file on the server; the server will in turn copy this text verbatim into the XML sent to the client.

As new plug-ins are developed, we will add new tags to this section corresponding to the new plug-ins.

The IFDB administrators will be responsible for maintaining the IFDB server with the current client software download sets and the corresponding version information. Plug-in developers will send each new release's download set to the IFDB administrators along with current version numbers; the administrators will copy the release set to the server and update the version file. It's up to each plug-in developer to determine the format of an update set; on Windows, for example, this is simply a ZIP file containing the full current client binary release.

The <interpreter> section for Windows

Now we come to the detail specific to the Windows plug-ins. This section applies only to Windows; other systems that have plug-ins will presumably need similar functionality, but how they accomplish it will depend on the local OS conventions and facilities; and, of course, systems that don't have plug-ins won't need this information at all.

Basic installation process

On Windows, there are several ways that freeware-type applications are commonly distributed:

To encode these possibilities, we need three pieces of information: the URL of the file to be downloaded; the compression format, if any; and the installation action.

If a compression format is specified, we'll unpack the downloaded file into a temporary directory, and execute the remaining steps on the unpacked contents of the archive; we'll delete the temporary folder when done.

(On Windows, ZIP is by far the dominant compression format. Even so, we'll probably support a couple of other formats in the Meta Installer for the sake of games, since there are a few games on the Archive in other formats - particularly .tar.gz and .tar.Z. Once the support is there for unpacking games, there's no cost to using it for unpacking format downloads as well.)

The action code can be one of these:

We'll encode the information for these various installation procedures as follows:

  <interpreter>
    <href>url</href>
    <action>action</action>
    <compression>
      <id>format ID</id>
      <name>format name</name>
      <primaryfile>filename</primaryfile>
    </compression>
  </interpreter>

The "href" gives the URL of the file to download. The Meta Installer starts by downloading this file and saving it in a temporary directory; the temporary directory and its contents will be deleted at the end of the installation process.

If the <compression> section is present, it specifies the compression information: the format ID, human-readable format name, and the primary file. The ID is a published identifier that can be found on the IFDB site; it's guaranteed to be permanent for a given format. The compression format name is a human-readable name for the format, supplied for use in the UI. The primary file in this case is the SETUP program to launch, if applicable. If compression is specified, the installer extracts the contents of the downloaded file into another temporary directory before proceeding.

The action is either "save" or "run":

Past installations and updates

Before we embark on the installation process above, it would be courteous to make sure that we actually need to do all that work: if the user already has a current copy of the target program installed, we should skip the installation process.

Note that we need a current copy of the target program. This means that if the program has been updated since the user's version was installed, we should install the new version (or at least tell the user that there's a new version, and see if they want to upgrade). This means that we need to encode version information in our download instructions. The version numbers should be encoded in such a way that two version IDs can be compared to see if one is newer than the other. Since almost everyone in the software business uses version numbers of the form "x.y.z...", we'll adopt this as our standard format.

There are two cases to consider in detecting past installations:

The first case - where the software was previously installed by the Meta Installer itself - is easy to handle. The Meta Installer can record each installation it carries out in its own internal database. When confronted with a new install request, the Meta Installer can simply look at its database to determine if it previously installed the same software, and can compare versions to make sure the installed version is current.

The second case is harder. The problem is that there's no standard way on Windows to detect if a particular application has been installed - we can't rely on folder locations, because most software lets the user control that; we can't rely on well-known registry keys, since there's no requirement that applications use registry keys at all; we can't even rely on file associations, since not all apps use these, and even apps that do use them often make them optional, and in any case they're notoriously unreliable because of contention among unrelated apps for ownership of common extensions. As I see it, there are three routes we could go here:

  1. Look for a way to detect a pre-installed version of each of the existing IF interpreters, based on their existing installation properties - registry keys, file locations, or whatever else. [7]
  2. Design a new technical standard for detectability that interpreters must implement in order to participate in the Meta Installer system.
  3. Ask the user. [8]

Option 1 would clearly ideal, when it can be made to work. It doesn't require any new work by interpreter developers (apart from the work of documenting their particular detection procedures); it'll work for users who have pre-IFDB interpreter versions installed (the new scheme won't, because the Meta Installer won't be able to detect older versions that predate the new detectability mechanism); and it avoids the question of how we'd get the new standard added to legacy systems that are no longer maintained.

Unfortunately, Option 1 isn't universally possible, because of the lack of Windows standards for application identification.

So, we use a combination of (2) and (3). (2) is reliable; its only drawback is its lack of backward compatibility. To address that, (3) seems easier to implement and more reliable than (1). While (3) has the drawback that it creates work for the user, the work is (a) basically one-time-only (once the user tells us where a particular interpreter is, we won't have to ask again), and (b) won't happen at all for newbies who haven't already installed older versions.

IFDB registry keys system

Option 2 is technically straightforward. In essence, the goal is to advertise the presence of an installed service for consumption by unrelated applications, and the way to do this in Windows is to add keys to the registry. So, our technical standard is to define a set of registry keys that each interpreter sets as part of its native installation process - that is, it sets these keys for every install, whether initiated by the IFDB Meta Installer or manually by the user.

The Meta Installer needs to know (1) whether or not the application is installed at all, (2) the version, and (3) how to invoke the interpreter program to run a given game. It's also useful to specify (4) how to invoke the interpreter to run a given game and immediately restore a given saved-game file, so that the plug-in can be enhanced in the future to provide a saved-game browsing and launch feature.

The IFDB registry keys are under a common root: \HKEY_CURRENT_USER\Software\IFDB.tads.org\MetaInstaller\Interpreters. Each interpreter (or its installer) creates a sub-key of this key, with a name chosen by the interpreter developer. I recommend following the Java style of domain-style naming so that we don't have to worry too much about coordinating to avoid collisions. For example, for HTML TADS I'd use, say, htmltads.tads.org as my sub-key, so my full key would be \HKEY_CURRENT_USER\Software\IFDB.tads.org\MetaInstaller\Interpreters\htmltads.tads.org.

Under this interpreter-specific key, the interpreter installer sets the following values:

Each interpreter developer is responsible for these items:

Tracking saved games

It might be useful at some point for the Meta Installer to be able to find saved position files. The installer could use this to create a more integrated "jukebox" experience for the player, by offering a list of saved games when starting the game.

We'll assume that saved games for each format can be identified by their file extension, so we'll make this information available to the Meta Installer, via an additional tag in the <interpreter> section:

  <savedGameExtension>.extension</savedGameExtension>

Note that the "." at the start of the extension must be included.

This element is optional - if the intepreter doesn't use a fixed extension for saved games, this element should be omitted, as extensions won't be useful as a way to identify these files. This element can appear more than once if the interpreter uses multiple extensions for saved game files. Note that only the extensions relevant to the current format should be included for a multi-format interpreter. (For example, if an interpreter handles Z-Machine and Hugo games, and it uses ".zsave" for Z-machine saved games and ".hsave" for Hugo saved games, the Hugo-specific <interpreter> record should mention only the ".hsave" extension.)

Windows <interpreter> section summary

Putting all of the pieces together, here's what the full Windows XML fragment for an <interpreter> section looks like:

  <interpreter>
    <name>name</name>
    <version>version</version>
    <registry>subkey name</registry>
    <legacyinstall>
      <filetypekey>HKCR filetype key name</filetypekey>
      <exename>terp filename.EXE</exename>
      <exeMD5>MD5 hash value for terp .EXE file</exeMD5>
    </legacyinstall>
    <href>url</href>
    <compression>
      <id>format ID</id>
      <name>format name</name>
      <primaryfile>filename</primaryfile>
    </compression>
    <action>action</action>
    <RunGame>run-game command line</RunGame>
    <RunGameRestore>run-and-restore command line</RunGameRestore>
    <savedGameExtension>.extension</savedGameExtension>
  </interpreter>

The name is a human-readable name, including the human-readable version string; this is so that the UI can tell the user what needs to be installed.

The version is the machine-readable version string, in the format "x.y.z..." (a minimum of one decimal integer optionally followed by additional decimal numbers separated by periods; elements are in decreasing order of significance).

The "registry" value is the name of the interpreter's IFDB registry sub-key, from the IFDB registry keys section. This is not the full HKEY_CURRENT_USER\Software... string, but just the final subkey name. This is a domain-style name identifying the interpreter and vendor, as in "htmltads.tads.org".

The "href" gives the URL to the downloadable interpreter installer. This can be an .EXE or .MSI installer, a .ZIP containing same, the application itself as a raw .EXE file, or a .ZIP containing the application .EXE and possibly other support files. This URL must point to the direct file download - it shouldn't point to an HTML page that offers the file for download, for example. The point is that the Meta Installer will be able to do an HTTP GET (or equivalent for other protocols) on this URL and save the returned content to obtain the installer/ZIP/etc.

The "compression" section is optional; it's only present if the download target is compressed. The format ID is one of the published compression format IDs on IFDB: for ZIP files, this is "zip". The format name is a human-readable name describing the compression format, for use in the client UI as needed. The "primaryfile" value gives the filename (using the relative path as stored in the ZIP file) of the primary file within the ZIP. In the case of an installer program packaged within a ZIP, this is the name of the installer program executable; in the case of an interpreter program that doesn't have a separate installer, this is the name of the interpreter .EXE file.

The action code can be one of these:

The RunGame and RunGameRestore values give the values of the IFDB registry keys that the interpreter normally sets up. These values are the Windows command lines, respectively, to invoke the interpreter to run a game, and to invoke the interpreter to run a game with immediate restoration of a saved game file. In these values, %0 stands in for the name of the interpreter (it's replaced with the full name, including the full directory path), %1 stands in for the name of the game file to be executed, and %2 stands in for the saved game file to be restored. RunGameRestore is only specified if the interpreter supports the run-and-restore operation.

The RunGame, and RunGameRestore values allow the Meta Installer to set the IFDB registry keys in lieu of the interpreter's installer doing so. Normally, we'd leave it to the interpreter's installer to do this work. However, there are three cases where the Meta Installer has to set these keys itself:

Syntax rules for RunGame and RunGameRestore: These are essentially DOS-style command lines. Tokens are separated with spaces; tokens that include spaces can be constructed by enclosing the whole token in double quotes. A double quote can be used within a quoted token either by putting two quotes in a row, or by escaping the quote with a backslash; other backslashes have no special meaning. Take careful note that the executable name often contains spaces because of the default "c:\Program Files" location on US-localized systems, so these should usually be enclosed in double quotes to ensure proper parsing.

Detecting "legacy" interpreter versions

For our purposes here, a "legacy" interpreter is a version that pre-dates the IFDB registry keys. The new keys obviously weren't implemented by the versions of the various interpreters that were already in distribution when the Meta Installer came along, so if the Meta Installer is going to be able to detect that an older version of one of these terps is installed, it needs some means other than the IFDB registry keys.

Fortunately, there's an approach that happens to work for most of the existing interpreters. Most of the interpreters write keys to the Windows registry to create associations between particular filename suffixes (for example, ".gam" for a TADS 2 story file) and the corresponding interpreter executables. These associations let the Windows shell launch the appropriate interpreter application when the user double-clicks on a story file in Windows Explorer. These file-type association keys are stored in a standard way, so we can exploit them to find interpreter executables.

The <legacyinstall> element tells the Meta Installer how to use one or more file-type associations to find a legacy version of the interpreter.

The <legacyinstall> element is optional. The Meta Installer uses it as a fallback, only when the key specified in the <registry> element is not found. If you don't specify this element, the Meta Installer will simply assume that the interpreter isn't installed if it can't be found using the IFDB registry keys.

The <filetypekey> element gives the name of one file-type association key that the interpreter creates under HKEY_CLASSES_ROOT. This value gives the key name of the abstract type key for the story file type that this interpreter handles. Each Windows file-type association has two elements. The first is the concrete extension key: this is a key of the form "HKCR\.xyz", where .xyz is a file suffix; the (Default) value of such as a key is a string giving the file type name. For example, TADS sets the key "HKCR\.GAM", with (Default) value "TADS.Game". The second key is the abstract file type key; it's called "HKCR\filetype", where filetype is the type name referenced in the extension key, so in the case of TADS, this is "HKCR\TADS.Game".

The <filetypekey> value is just the key name. Don't include the HKCR\ prefix. For example, the TADS 2 file type key is in the registry as HKCR\TADS.Game, so the tag is written as <filetypekey>TADS.Game</filetypekey>.

The <filetypekey> element may appear any number of times, since some interpreters handle more than one story file type.

The <exename> element gives the base filename of the interpreter executable. By "base" filename, we mean the filename without any path information: so for HTML TADS, we'd specify <exename>htmltads.exe</exename>. (The value isn't case-sensitive, since the Windows file system isn't.) The executable name needs to be specified to differentiate among interpreters that use the same file type keys - it helps ensure that the Meta Installer is really detecting the right interpreter.

The <exeMD5> gives the MD5 hash code for the current release version of the interpreter executable. If the Meta Installer finds a candidate interpreter based on the file type association key and executable filename, it computes the MD5 of the actual installed file, and compares it with the value given here; if the values match, the Meta Installer knows that the user has the current release installed. If the values don't match, the Installer can offer to install the latest version. The <exeMD5> value is optional, but if it's not provided, the Meta Installer will have no way of knowing if the installer version is up to date. The MD5 value is given in the canonical printable format: 32 hex digits, with no embedded spaces or hyphens. Case is not significant.

Here's an example of how the scheme is used. The TADS 2 definition looks like this:

 <legacyinstall>
   <filetypekey>TADS.gam</filetypekey>
   <exename>htmltads.exe</exename>
   <exeMD5>4C874F7E5AD303412D20D02FFF0CCEA9</exeMD5>
 </legacyinstall>

The Meta Installer looks at HKCR\TADS.Game\shell\open\command, and sees that this key exists and has default value "C:\Program Files\TADS\htmltads.exe" "%1". The installer parses the value as a command line and extracts the first token, which is C:\Program Files\TADS\htmltads.exe. The root filename from this token is htmltads.exe, which matches the <exename> value. The interpreter checks to see if the file exists, and if so computes the MD5 hash of the file's entire binary contents, and compares the result to the <exeMD5> value. If the MD5 values match, the installer knows that HTML TADS is installed and up to date. If the file is present but the MD5 values don't match, the installer offers to install an update. If the file or the registry keys are missing, the installer assumes that HTML TADS is not installed.

Meta Installer maintenance

We probably need a maintenance UI for the Windows Meta Installer - a screen the user can bring up that lists the interpreters that we've installed and lets them uninstall and reinstall them.

Meta Installer Security

The nature of the Meta Installer makes it immediately suspicious as a malware risk. The program's purpose is to download and install executable software. Executables are by far the easiest way for malicious developers to distribute malware, since they're essentially unrestricted in accessing and modifying a computer's disk drives and other resources.

Unlike most web sites that use plug-ins, IFDB will not attempt to install any plug-in automatically. IFDB will only generate HTML that displays the plug-in after the user has explicitly asked to install the plug-in - this means that you'll never see a browser security warning pop up from a game listing page about the page needing a plug-in. You'll either have the plug-in already, or the page won't try to use it at all. This makes the plug-in a purely "opt-in" feature - users who aren't interested won't ever be bugged about installing it. For users who haven't installed the plug-in and are using an OS/browser combination where a plug-in is available, IFDB will display a link advertising the plug-in; the user can follow the link to go to a separate page that explains what the plug-in does and how to install it.

Browser plug-ins are also notorious security risks. Plug-ins are risky because they allow web pages to invoke executable code on the client. This is problematic because users are accustomed to being able to trust the browser to "sandbox" web pages enough that there's little risk in just visiting an arbitrary web site. Plug-ins essentially bypass this sandboxing by giving web pages a way to invoke client-side executable code outside of the browser.

The main risk with plug-ins isn't that the whole idea is bad, or that the plug-in architecture itself is risky. Rather, the problem is that a plug-in is just native executable code, so the full burden of responsibility to make the plug-in secure is on the plug-in's developer. In contrast, scripts and applets and the like are restricted by the browser, so a careless applet or script developer can make all kinds of mistakes and still be saved by the security measures in the browser. A plug-in developer is completely on her own to make the plug-in secure. Any mistakes or omissions can compromise system security because the plug-in isn't restricted by the browser's security measures.

There are several things we do in the Meta Installer design to reduce the security risks. Some of this is in the overall architecture of the system, and some is in the individual plug-ins.

Architectural Security Features

The following elements of the system architecture are designed to enhance security of the Meta Installer. These are features of the whole system, so they apply to every plug-in, regardless of its internal implementation details.

Plug-In Security Measures

Some security measures must be implemented individually in each plug-in:

DLA Instructions Table Maintenance

One of the key design elements of IFDB is that the game entries are open to public editing, along the lines of Wikis. This design is an attempt to immunize IFDB against a chronic problem among IF reference sites (and the web in general, really): our reference sites tend to slip out of date because they get only sporadic attention from their owners. By opening up the site to allow users to make updates, the site will be as up to date as its users want it to be - if a user sees something that's out of date, she can simply fix it herself rather than waiting for the site's owner to get to it.

It's even more important to keep the DLA instructions up to date. If they ever become obsolete - especially if the download URLs they contain are broken - the Meta Installer will cease to function, and the DLA instructions will be misleading.

There are too many operating systems and interpreter in use today for any one person to be an expert in all of them. That, coupled with the basic risk of sporadic attention, makes it clear that we don't want the site's owner to be a bottleneck for maintaining the DLA tables. Clearly, we need to distribute the responsibility.

On the other hand, it would be a bad idea to make the DLA tables open to the public for editing, the way game entries are. The problem is that the DLA entries contain instructions for downloading and installing executable programs. If users are going to trust the DLA and the Meta Installer, it's vitally important that the sources of the download instructions be protected. If anyone could edit the tables, a bad guy could easily set a download link for an interpreter to point to malware.

To balance these needs - distributing the maintenance responsibilities vs. protecting the tables against malicious updates - IFDB is set up so that access to the DLA tables is restricted to authorized users. The system makes it possible to grant DLA access to specific users on the basis of their format or OS expertise.

So our plan is that the DLA entries will be maintained by a small group of trusted individuals. We'll seek to cover all of the formats and operating systems this way. Ideally, we'll be able to enlist the chief developer for each of the main format interpreters, which should cover most of the bases. Enlisting interpreter developers is the best way to ensure timely updates - ideally, updating IFDB's DLA entries would become a routine part of each interpreter's release process.


Footnotes

[1] In terms of the real-world entities involved, it would arguably be better to treat each operating system's executables as separate formats: we'd have Windows Executable, Macintosh Executable, etc. That's a better mapping to actual file formats, since "Windows executable" and "Macintosh executable" are file formats in exactly the same sense that JPEG, GIF, MP3, and "Word Document" are: "Windows executable" is a particular way of laying out bytes in a file to represent a composite data structure, in this case a program that executes on Windows.

We chose not to go that route for a couple of practical reasons. First, we need to distinguish not only operating systems, but OS versions, since many OSes have multiple releases in circulation, with varying degrees of compatibility. Including all of the OS versions in the file format list would have made the list too long for comfort in the UI. Second, we need a separate list of operating systems anyway, since we need a way for the user to identify which OS they're using. If we had separate "Exectuable for X" types, we'd have to correlate those with the entries in our OS list. The data model is better normalized if we treat the OS and file type as orthogonal entities.

[2] For example, a Word .doc file can be viewed on any platform with software that reads the .doc format. In the case of Word, actually, there are numerous incompatible sub-formats: Word 97, Word 2000, etc. If it were necessary to use different viewer apps with the different sub-formats, we'd handle this by defining separate file types for the sub-formats.

[3] We look for records tied to OS versions less than or equal to the target OS version because we assume backward compatibility for new versions of the operating system. We don't assume upward compatibility for applications. That is, we assume that an app designed for version N of a given OS will run on the OS versions N+1, N+2, etc. However, we don't assume that an app designed for version N of the OS will run on version N-1, since the app might depend upon new OS features that were introduced in version N. This type of backward compatibility isn't guaranteed, but it's usually the case. For particular applications that don't migrate to newer versions beyond a given point, we assume that the app developer will create a new version of the app for the new OS, at which point we'd simply introduce a separate Instructions record for that OS version pointing to the new app version: our algorithm ensures that users with the older OS will still pick up the older app version, and users with the newer OS will get the newer app version. When backward compatibility is significantly broken across the board by a new OS version, we handle this by branching off the new OS version as a whole separate OS; we do this with Mac OS X vs Mac OS 7/8/9.

[4] As an example of how this works, let's look at the Windows/Internet Explorer case. The plug-in for this target browser is implemented as an ActiveX control, because that's IE's primary plug-in mechanism. When IFDB detects the Internet Explorer browser on Windows, it inserts some scripting code into the generated page that uses vbscript techniques to detect whether the user has the IFDB ActiveX plug-in installed. If the scripting code's test determines that the plug-in is indeed present, the script injects an <OBJECT> tag into the page. The <OBJECT> tag is the IE mechanism for instantiating an ActiveX object, so this fires up the plug-in. The <OBJECT> tag contains a a <PARAM> tag that passes the TUID of the game being viewed to the plug-in. The script code further injects the "Play Now" button as an <IMG> tag with an "onscript" javascript event handler that invokes the plug-in's main entrypoint, which is what initiates the installation process.

[5] For example, the Windows/IE ActiveX plug-in uses a <PARAM> tag to pass this value to the plug-in when instantiating the ActiveX object.

[6] Note that IFIDs were also designed to have the same properties (stability, global uniqueness), so we could use the IFID list for this purpose instead. However, IFIDs aren't guaranteed to be single-valued (a particular game can have several, and a future version will have a new IFID if the author doesn't know about the Babel architecture or doesn't bother to use it); this multi-valued property makes IFIDs a bit more work for the client component to manage. In addition, IFIDs could conceivably be entered into the database incorrectly and then later revised, whereas TUIDs are assigned automatically by IFDB and so aren't subject to revision. Finally, IFDB doesn't currently require game listings to include IFIDs, so some games in the database might not include IFID values at all.

[7] For HTML TADS on Windows, and probably for most apps that incorporate Uninstall programs, you can tell that the application is installed by checking for the presence of the standard Windows uninstall registry key - in HTML TADS's case, this is HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Uninstall\htmltads.exe\UninstallString. This tells us (a) that the application is installed, and (b) with some work, where to find the interpreter program. The trick with (b) is that we have to parse the value of the key to pull out the command-line argument, which tells us the directory where the uninstall data file is located, which also happens to be the base install directory. We can then form the full path to the interpreter by combining the directory path from this uninstall string with the name of the interpreter .EXE file. With this in hand, we have or can synthesize nearly all of the information we're after - the one thing we're missing is the version information. For that, I don't think there's any good solution; probably the only thing we could do would be to calculate an MD5 hash of the .EXE file, and assume that a mismatch with the server version's MD5 hash means that we have an old version.

Assuming we wanted to use an approach like this, we'd have to come up with a way to encode it in the XML instructions, to tell the Meta Installer what to look for. Something like this:

<existingversion>
  <directory>
    <registry>
      <key>
        HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Uninstall\htmltads.exe
      </key>
      <item>
        UninstallString
      </item>
      <parse>
        *\TADSUINS.EXE $\UnInst*.inf
      </parse>
    </registry>
  </directory>
  <file>
    htmltads.exe
  </file>
  <version>
    <md5>A5C729910177F7AE97B00F28</md5>
  </version>
</existingversion>
    

The idea is that the registry string is parsed by matching the "parse=" template, such that "*" matches arbitrary text, "$" matches arbitrary text that becomes the result of the parse, and everything else is matched literally. The result of the parse is then used as though it were the contents of the <directory> tag, so this is combined with the <file> tag to form the name of the executable. The "md5" value means that we can compare versions by computing the MD5 hash of the interpreter executable and checking to see if it matches the value given.

This is all awfully hacky; the question is whether something like this would generalize to the other interpreters. We obviously can't have a completely separate approach for each system; we need something that we can parameterize so that it's readily extensible to additional systems without requiring updates to the Meta Installer itself.

[8] What I have in mind is a UI something like this:

[9] Which compression formats is a Meta Installer expected to support?

Since most story files are widely portable, and since the compilers for most formats are also widely ported, it's possible in principle for a story file to use any compression format - including formats that aren't commonly used on the Meta Installer's native platform. For example, a Windows user might encounter a game packaged with StuffIt, since the game's author might have been working on a Macintosh.

In practice, though, ZIP is by far the most common format on the IF Archive, so a Meta Installer that supports ZIP will handle most of the games in the Archive. Virtually the only other formats that appear among games written in the last ten years are .tar.gz and .tar.Z. If you handle ZIP, .tar.gz, and .tar.Z, you'll handle virtually everything in the Archive. There are other files in formats on the Archive, but they're hardly ever used for portable files. There are a lot of old native Amiga games in .LHA format; and Mac-native games are usually packaged in StuffIt format. None of these are usable on other OSes, though, so a Meta Installer for another platform should never encounter them.

[10] Note that there's a problem with game installers on Windows: the Meta Installer will have no idea of how to execute the game itself, since it will only know how to execute the installer. The Windows Meta Installer might simply disallow this type of download.