Adam Kewley

Jobson: Webify CLI Applications

This is a post about Jobson, an application I developed and recently got permission to open-source along with its UI.

Do any of these problems sound familiar to you?:

I’d like my application to have a UI.

I want to trace my application’s usage.

I want to share my application.

They’re very common problems to have, and are almost always solved by building a UI in a framework (e.g. Qt, WPF), sprinkling on usage metrics, and packaging everything into a fat executable.

That development approach is becoming challenging to do well nowadays because clients are more likely to use a mixture of OSs and 3rd-party resources. Therefore, new projects tend to need to choose between several options:

When scoped in this (biased) way it’s clear that webapps are ideal vehicles for delivering the full “product” but CLI applications are ideal for ease-of development and flexibility.

It’s clear that developing a method for turning CLI applications into webapps would be valuable. It would enable developers to rapidly develop and roll out applications. That’s what I explored with Jobson: a web server that turns CLI applications into webapps.

Jobson’s Approach

Jobson has now been working in a production environment for a few months now and has proven to be an effective vehicle for delivering new platforms. However it cannot turn any application into a webapp. That would be tough: the potential “inputs” (applications) and “outputs” (webapps) are far too varied.

Jobson’s primary limitation is that the only applications it handles are batch applications that: a) start, b) take typical inputs, c) write typical outputs, and d) end. Most applications follow this model (e.g.echo, gcc, nmap, and ls).

With that limitation in place, Jobson could then be designed with simple principles in mind:

Overall, this design means that Jobson can webify almost any application very quickly. I could webify a major CLI tool in less time than it took to write this post, resulting in a web API and UI for that tool.

Why It’s Useful

Here are some typical development problems Jobson could help out with:

Problem: You’ve got a cool application idea you want to share

Without Jobson:

With Jobson:

Problem: You’ve got a toolchain (e.g. a pentesting toolchain) that has hard-to-remember commands. You want to streamline that toolchain such that you can run and queue multiple commands.

Without Jobson:

With Jobson:

Problem: You want to automate data requests at work, but there’s a lot of flexibility in the requests (FYI: this is why Jobson was made).

Without Jobson:

With Jobson:

Overall, I believe this approach to developing a job system is extremely flexible and much easier to work with. Jobson abstracts all the typical job system faff (auth, APIs, queues, etc.) away from what’s actually important (an application that does something), resulting in a much cleaner system.

The next steps with Jobson are to streamline installation (by implementing installers), add a landing+documentation page, and start making tutorial videos for it. Watch this space =)

Textadventurer Has Actual Games Now

I’ve been distracted by other things going on, but I finally managed to spend an evening or two adding actual games into textadventuer.

The following games were added:

One thing the game search did for me was find similar platforms. I should’ve looked before hacking away at textadventuer because some of those platforms (e.g. are very well made. Some even use clever tricks like emulating DosBox in the browser. However, I’m yet to come across a platform that can run any application via a browser (textadventuer’s “USP”, if you will), so hopefully other developers will find the server and UI source helpful if they have an idea along those lines.

Text Adventurer

One of the first things people learn when they start programming is how to write a text prompt. It’s a decades-old exercise that teaches new programmers input-output.

In order to inject a little excitement, learners are normally encouraged to write interactive games using standard IO. This helps them learn programming by interactively - they will need to learn conditional logic to handle the “Will you stab the monster with your sword or run away?” prompt.

This creative learning process is great but, unfortunately, it’s hard to show the creations to other people. Any players will have to undergo the nuisance of installing an interpreter, libraries, and the game in order to play. These deployment problems are alleviated on the web: javascript can be distributed and executed remotely in a browser with no effort required from the client.

If people wrote their text adventures in javascript, they could be easily be shared with a URL. However, javascript is not necessarily a good teaching language. Enforcing people to write their text adventures in it for the sake of distribution detracts from the learning experience.

An ideal system would allow text adventures (console applications) to be written in any language but also be distributed on the web. This is what my latest project, textadventurer, tries to achieve (gallery).

textadventurer keeps interaction in the browser while moving execution onto the server. Communication between those two layers is achieved with websockets. This makes it easier for people to play the game. The frontend focuses on presenting games (CLI applications) to the players and provides a basic UI for sending input (STDIN) to the server and recieving output (STDOUT) from the server.

Using process forking server-side affords a lot of flexibility: the server is completely agnostic to the language or framework that the game is written in. This means I can use textadventurer to distribute any standard interactive CLI application. With that in mind, I plan on deploying historic text adventure games to textadventurer when I get the chance so that people can enjoy those games once more without having to faff around with installers, legacy interpreters, etc.

Keep Side Projects Small

I’ve officially thrown a deployment of PlateyPlatey (PP) up onto (galley) along with its server and frontend source code. I previously explained PP in a blog post.

PP is missing many features I would of liked to have. However, a few weeks ago I took a long look in the mirror and decided to just tidy it then throw it up on the net. This let me start sharing it on sites such as reddit to gauge if anyone’s interested. After about a week of PP being up, I concluded that no, no one is interested and yes, I spent way too much time coming to that conclusion.

This is my attempt to articulate general some general rules of thumb I have picked up over the last few side-projects. It was written just after PP, one of my bigger projects, didn’t live up to my expectations. It is not a definitive guide nor something you should take seriously without reading other developers’ thoughts.

I used to approach software side projects as an opportunity to write full-blown applications that I might someday commercialize or turn into a big hit. This, I’ve learnt, is a silly approach. Side projects should be approached as side projects, not as full-blown projects.

If a side project requires a light scripting engine, command controller, custom UI elements, a fair amount of interactivity, servers, etc. then it’s doomed to failure. This isn’t because these features can’t be implemented—I believe any developer can implement any feature if given enough time—but because big projects don’t suit only getting “and hour here” and “an evening there”.

The problem with PP is that I specced my minimum viable product way too ambitiously. The spec included several headline features:

In isolation, each of these features are easy to implement. However, implementing all of them into one system requires more care and, crucially, architecture.

If you write software full-time as a job then you can afford to spend a month or two making prototypes, designing hierarchies, and, ultimately, implementing a system that’s quite large. However, side-projects won’t get more than, say, ~20 % of the time an at-work project will get. Because of that, those few weeks of prototyping and planning will stretch out across months.

In principle, many months isn’t a problem for a side-project with no time limits. However, spending many months without making anything requires a lot of discipline: no one finds planning and prototyping as fun as diving into the heart of an idea. The same goes for boring old architecture documents, automated testing, and design specs.

Those things mostly exist to make working on bigger projects more predictable and to make the maintenance of a working system easier for other developers. A small side project is only going to be worked on by one, maybe two, developers over a relatively short period of time; therefore, it isn’t worth worrying about the theoretical overhead of maintaining and sharing it. If a small side project becomes an overnight sensation then—because it’s small—there is scope for dropping the initial codebase and starting again with all the knowledge and expertise the first runthrough gave you.

Dropping those things for a bigger side project, though, is dangerous. Not having plans, specs, architecture, or tests in place for a bigger project will result in development grinding to a halt under its own technical debt. When that happens, you’ll need to do a big refactor. This happened roughly twice with PP. First, to move all the javascript code into separate hierarchies with a makefile build system. Second, to translate all the javascript into typescript and use webpack to build everything. The time taken by those reorganization steps really sucked some of the momentum out of PP’s development. Especially because the actual functionality of PP didn’t change after each step - they just made working on the codebase easier.

The main take home is that side projects get developed much more slowly than real on-the-job projects because of time constraints. That makes working on bigger side projects especially painful because the boring planning, speccing, and testing steps—which tend to come in handy for bigger projects—last too long to keep the momentum of a side project going. By contrast, smaller projects work well with a “code and fix” development model which gives immediate feedback and a semi-working system very quickly. This helps maintain momentum and interest; however, “code and fix” gets treacherous once projects start growing, so don’t let them grow.

Keep small. Keep agile. Try things out. Get the audience’s opinion. Then grow.

Low Level Programming

I haven’t posted in a while because I have been busy with a few random studies and projects.

Low level programming has always, to a high-level programmer like myself, felt like a mysterious playground where all the real hackers write graphics engines, password crackers, and operating systems. I have always had a fascination with people who can write something hardcore like an emulator. Sure, I can open a window, draw some graphics, and update the UI to reflect state changes. However, an application that does all that and some low-level wizardry is in an entirely different league of complexity.

Many prominent programmers recommend learning something like C because it is a good way of learning how computers really work. Apart from the occasional toe-dip into C, I have almost exclusively coded in higher level languages.

I want to get better at programming. I’m getting bored of the high-level: there’s only so many ways classes, functions, currying, type systems, can be repackaged before you’ve seen it all. I like the idea of mastering the low level. However, deciding on where to start was challenging.

Last weekend, however, I decided to set myself what I thought would be a simple exercise:

Write a websocket server that capitalizes and echoes any text it recieves. You cannot use a websocket library.

I eventually hacked something together that works (github). I took a shortcut and used the nodejs http-parser and a base64 encoder written by Brad Conte. However, the websocket implementation was derived directly from RFC6455.

Even though it’s pathetically basic, is blocking (I don’t fork or do any fancy event queueing on the connections), and uses compile-time allocated buffers for most of the messages (so it can’t stream big websocket frames), this was still a very eye-opening first step into some low-level programming. I somewhat regret writing the process forking for text adventurer in clojure (with the help of websocketd) after seeing how straightforward a lower-level implementation would be.