Adam Kewley

Jobson 1.0.0

After many weekends and evenings of fixing little bugs, cleaning up the codebase, and polishing the build, I’ve finally managed to publish v1.0.0 of jobson.

I open-sourced jobson late November 2017. The version I demoed here was already close to release-grade in terms of implementation (the server had >200 tests, was used in prod, etc.). However, the deployment, installation, documentation, and maintenance needed work.

For the open-source release, I wanted to make sure that jobson was OSS-grade before putting a 1.0.0 badge on it. The main changes over the last year are:

  • Stabilized all user-facing APIs (CLI, configuration, HTTP). No known breaking changes since Feb 2018.
  • Added more systemtests to ensure the above
  • Reimplemented UI in Typescript
  • Refactored and cleaned up server code
  • Fixed various bugs (race conditions, etc.)
  • Added various features into the specs (better output collection, templating, etc.)
  • Added more datatypes (float, int, long, etc.)
  • Added a lot more documentation, including API documentation
  • Significantly improved the build, which now builds the full stack into Debian packages, Docker images, etc.

I plan on patching 1.0.0 slightly with some little annoyances I spotted (immediately after deploying, of course), followed by another round of YouTube videos and other media. After that, it’s time to start slowly chipping away at 1.1.0.

Side Project: Live Demos

After several days of faffing around with Docker and build systems, I’ve finally managed to launch a demo’s page here. I’ll eventually integrate these into my about page, but they’re at least a useful resource for showing some of the technologies I’ve worked with.

One useful side-product of this work is that Jobson now has a basic docker image, which enables users to boot a pre-integrated Jobson UI + Jobson stack.

The Demos:

Side Project: Rust: fo2dat

tl;dr: I used Rust to make this CLI utility for extracting Fallout 1+2 DAT files.

I love the occasional playthrough of Fallout 1 and 2. They were some of the the first “serious” games I played. Sure, DOOM/Quake/Command & Conquer were also “mature”—I played them around the age of ~10, which marked me as doomed (heh) by the kind of adults that would also think eating sweets is a surefire path to heroin addition or something—but F1+2 included prostitutes, drugs, slavery, and infanticide: irresistibly entertaining topics for a teenager.

You might think that, with Bethesda buying the rights to Fallout over 10 years ago, F1+2 would’ve had a commercial re-release by now, much like what happened with Baldur’s Gate 1/2, or that such a popular franchise with so many fans would create enough momentum to get an open-source engine ported, but those things haven’t really happened. Probably because of various historical hitches.

F1+2 were originally developed by Interplay, which had a meteoric rise in the 90s followed by a precipitous fall in the early 00s (details). Bethesda excavated Fallout from Interplay’s remains in 2007. However, Interplay’s zombie, adopting a 90s zombie movie strategy of having a character bitten right before the credits roll, decided to give away F1+2+Tactics for free just before the IP passed over. As a consequence, Bethesda probably sees a F1+2 reboot as unprofitable. This assumes that the source code is even available to Bethesda. Interplay may have only handed them the sales rights + binaries, which would be a big shame.

I’ve always wanted to make a F1+2 source port, but it’s an incredibly difficult task for several reasons: the literature for open-source RPG game engines is thin, which means an engine will need to be built from first-principles; F1+2 uses a bunch of bespoke file formats, which means deserializers will need to be coded for each of them; and the game logic—what characters do, actions, etc.—is held as a binary opcode language, which requires a small virtual machine to handle.

The Falltergeist developers are the closest to surmounting the F1+2 Everest. They’ve created something that is close to a complete engine, which is amazing. I decided to chip away at the smaller problem of F1+2’s file formats. The result was fo2dat, a CLI utility for unpacking F2 DAT files, which might help any developers wanting to view the game’s assets in a file explorer.

The micro-project was also a chance to try Rust, the new hot thing in systems programming languages. I enjoyed the experience quite a bit: it feels like the Rust developers really understood the strengths of languages like C (non-OO, simple procedures handling data arguments), C++ (powerful compilers), Scala (pattern matching and abstractions), and Lisp (code generation). They combined those ideas with an excellent build system and package manager, which has resulted in a very nice platform.

Cover Design: Core–Shell Crystals of Porous Organic Cages

A cover I designed for the Cooper Group’s latest breakthrough. Modelled in plain Blender 2.79b. Post processing done in GIMP 2.8.18.

[raw high-res render] , [cover official link], [journal article]


Core–shell crystals of porous materials allow multiple functionalities to be used synergistically, for example choosing the core material for capacity, and the shell for selectivity, or surface properties. The few previous examples have all been based on frameworks, but in this Communication (DOI: 10.1002/anie.201803244), T. Hasell, A. I. Cooper et al. present the first report of core–shell crystals formed from porous molecules, in this case porous organic cages—which are directed by chiral recognition. Depicted are octahedral crystals with a fluorescent cage as the core, and a non‐fluorescent cage shell layer growing on the surface.

Integrating Software

tl;dr: If you find you’re spending a lot of time integrating various pieces of software across multiple computers and are currently using a mixture of scripts, build systems, and manual methods to do that, look into configuration managers. They’re easy to pick up and automate the most common tasks. I’m using ansible, because it’s standard, simple, and written in python.

Research software typically requires integrating clusters, high-performance numerical libraries, 30-year-old Fortran applications by geniuses, and 30-minute-old python scripts written by PhD students.

A consistent thorn in my side is downloading, building, installing, and deploying all of that stuff. For example, on a recent project, I needed to:

  • Checkout a Java (Maven) project from svn
  • Build it with a particular build profile
  • Unzip the built binaries
  • Install the binaries at a specific location on the client machine
  • Install the binaries at specific location on a cluster
  • Reconfigure Luigi to run the application with the correct arguments
  • Copy some other binaries onto the cluster’s HDFS
  • (Sometimes) rebuild all the binaries from source, if the source was monkey-patched due to a runtime bug
  • (Sometimes) Nuke all of the above and start fresh

Each step is simple enough, but designing a clean architecture around doing slightly different permutations of those steps is a struggle between doing something the easy way (e.g. a directory containing scripts, hard-coded arguments in the Luigi task) and doing something the correct way.

The correct way (or so I thought) to handle these kinds of problems is to use a build system. However, there is no agreed-upon “one way” to download, build, and install software, which is why build systems are either extremely powerful/flexible (e.g. make, where anything is possible) and rigid/declarative (e.g. maven).

Because there’s so much choice out there, I concluded that researching each would obviously (ahem) be a poor use of my valuable time. So, over the years, I’ve been writing a set of scripts which have been gradually mutating:

  • Initially they were bash scripts
  • Then they were ruby scripts that mostly doing the same as the bash scripts
  • Then they were ruby scripts that integrated some build parts (e.g. pulling version numbers out of pom.xml files), but were mostly doing the same as the bash scripts
  • Then they were a mixture of structured YAML files containing some of the build steps and ruby filling in the gaps
  • Then they were a mixture of YAML files containing metadata (description strings, version numbers), YAML files containing build steps, and Python filling in the gaps because Python’s easier to integrate with the existing researcher/developer’s work

After many months of this, I decided “this sucks, I’ll develop a new, better, way of doing this”. So I spent an entire evening going through the weird, wonderful, and standard build systems out there, justifying why my solution would be better for this problem.

Well, it turns out this problem isn’t suitable for a build system, despite it having similar requirements (check inputs, run something, check outputs, transform files, etc.). Although my searches yielded a menagerie of weird software, what I actually needed was a configuration manager. Ansible being a particularly straightforward one.

This rollercoaster of “there probably isn’t a good solution already available to this problem”, “I’ll hack my own solution!”, “My hacks are a mess, I should build an actual system”, “oh, the system already exists” must be common among software developers. Maybe it’s because the problem isn’t actually about developing a solution: it’s about understanding the problem well enough. If the problem’s truly understood, it will be easier to identify which libraries/algorithms to use to solve it, which will make developing the solution a lot easier. Otherwise, you’ll end up like me: Keeper of the Mutant Scripts.