The Future Is Now

Cargo-dist: System Dependencies Are Hard (So We Made Them Easier)

My latest blog post is over at my employer’s blog post and talks about the work I’ve done to get system dependency management integrated into cargo-dist, an open source release management tool for Rust. The new release lets users specify non-Rust dependencies in Cargo.toml using a Cargo-like syntax and also provides a detailed report on the resulting binary’s dynamic linkage. Here’s a sample of the dependency syntax:

cmake = { targets = ["x86_64-apple-darwin"] }
libcue = { version = "2.2.1", targets = ["x86_64-apple-darwin"] }

cmake = '*'
libcue-dev = { version = "2.2.1-2" }

lftp = '*'
cmake = '3.27.6'

Go read the blog post to find out more!

Untangling Another Lingo Parser Edge Case

I was testing out a new Macromedia Director CD in ScummVM, and I noticed a non-fatal error at startup:

WARNING: ######################  LINGO: syntax error, unexpected tSTRING: expected ')' at line 2 col 70 in ScoreScript id: 2!
WARNING: #   2: set DiskChk = FileIO(mnew,"read"¬"The Source:Put Contents on Hard Drive:Journey to the Source:YZ.DATA")!
WARNING: #                                                                            ^ about here!

It may have been non-fatal, but seeing an error like that makes me uneasy anyway—I’m never sure when it’ll turn out to have ramifications down the line. This comes from the parser for Director’s custom programming language, Lingo, so I opened up the code in question1 to take a look. The whole script turned out to be only three straightforward lines. The part ScummVM complained about came right at the start of the file, and at first glance it looked pretty innocuous.

set DiskChk = FileIO(mnew,"read"¬
"The Source:Put Contents on Hard Drive:Journey to the Source:YZ.DATA")
IF DiskChk = -35 THEN GO TO "No CD"

The symbol at the end of that first line is a continuation marker, which you might remember from a previous blog post where I debugged a different issue with them. The continuation marker is a special kind of escape character with one specific purpose: it escapes newlines to allow statements to extend across more than one line of code, and nothing else.

At first I thought maybe the issue was with the continuation marker itself being misparsed, like in the error I documented in that older blog post; maybe it was failing to be recognized and wasn’t being replaced with whitespace? To figure that out, I started digging around in ScummVM’s Lingo preprocessor. Spoiler: it turned out not to be an issue with the continuation marker, but it pointed me in the right direction anyway.

ScummVM handles the continuation marker in two phases. In a preprocessor phase, it removes the newline after the marker in order to simplify parsing later. Afterwards, in the lexer, it replaces the marker with a space to produce a single normal line of code. The error message above contains a version of the line between those two steps: the preprocessor has combined the two lines of code into one, but the continuation marker hasn’t been replaced with a space yet.

If we do the work of the preprocessor/lexer ourselves, we get this copy of the line:

set DiskChk = FileIO(mnew,"read" "The Source:Put Contents on Hard Drive:Journey to the Source:YZ.DATA")

In this form, the error is a bit more obvious than when it was spread across multiple lines. The problem is with how the arguments are passed to FileIO: the first two arguments are separated by a comma, but the second and third aren’t. The newline between the second and third arguments makes it easy to miss, but as soon as we put it all together it becomes obvious.

In the last case I looked at, described in the previous blog post, this was an ambiguous parse case: the same line of code was valid if you added the missing comma or not, but it was interpreted two totally different ways. This time is different. If you add the missing comma, this is a normal, valid line of code; if you don’t, it’s invalid syntax and you get the error we’re seeing at the top.

As far as I can tell, the original Director runtime actually accepts this without throwing an error even though this isn’t documented as correct syntax. The official Director programming manual tells the user to use commas to separate arguments, but it’s tolerant enough to support when they’re forgotten like they are here2. ScummVM doesn’t get that same luxury. As I mentioned in the previous blog post, later Director versions tightened up these ambiguous parse cases, and supporting the weird case in Director 3 would significantly complicate the parser. Since this is only the second case of this issue, though, it’s not really necessary to support it either. ScummVM has builtin support for patching a specific disc’s Lingo source code, so I was able to simply fix this by patching the code to the properly-formatted version.

The disc in question still doesn’t fully work, but I’m putting some time into it. I’m planning on writing a followup on the other fixes necessary to get it running as expected. And for today’s lesson? Old software is weird. Just like new software.

  1. Before version 4, Director software was interpreted from source code at runtime—so, conveniently, that means that you can peek at the source code to any early Director software.

  2. MacroMind Director Version 3.0: Interactivity Manual. (1991). MacroMind, Inc. Page 64.

How I Dumped an Arcade Game for MAME

I recently had the chance to do something that I’ve wanted to do for years: dump an arcade game to add it to MAME.

Screenshot of The Gladiator's title screen

MAME is a massive emulator that covers thousands of arcade games across the history of gaming. It’s one of those projects I’ve used and loved for decades. I’ve always wanted to give back, but it never felt like I had something to contribute—until recently.

You might remember from one of my other posts that I recently got into collecting arcade games. This year I’ve been collecting games for a Taiwanese system called the PolyGame Master (PGM), a cartridge-based system with interchangeable games sold by International Games System (IGS) between 1997 and 2005. It has a lot of great games that aren’t very well-known, in particular some incredibly well-designed beat-em-ups.

A couple months ago, I bought a copy of The Gladiator, a wuxia-themed beat em up set in the Song dynasty. My specific copy turns out to be an otherwise-undumped game revison. Many arcade games were released in many different versions, including regional variations and bugfix updates, and it can take collectors and MAME developers years to track them all down. In the case of The Gladiator, MAME has the final release, 1.07, and the first few revisions, but it’s missing most of the versions in between. When my copy came here, I booted it up and found out it was one of those versions MAME was missing: 1.04.

Luckily, I already had the hardware on hand to dump it. I own an EPROM burner that I’d bought to write chips so that I could mod games I’d bought, but EPROM burners can read chips as well. I own an adapter that supports several common chips that, luckily, can handle exactly the chips I needed for this game.

Photo of an EPROM burner with a 27C160 chip in it

It’s easy to think of game cartridges as just being a single thing, but arcade game boards typically have a large number of chips. Why’s that? It’s partly technical; specific chips can be connected directly to particular regions of the system’s hardware, like graphics or sound, which means that even though it’s less flexible than an all-in-one ROM, it has some performance advantages too. The two chips I dumped here are program code for two different CPUs: one for the 68000 CPU in the system itself, and one for the ARM7 CPU in the game cartridge.

The other advantage is that using a large number of chips can make it easier to update a game down the line. Making an overseas release? It would be much cheaper to update just a couple of your chips instead of producing new revisions of everything on your board. Releasing a bugfix update? It’s much more quick and painless to update existing games if all your program code is on a single chip.

From looking at MAME, I could tell that every other revision of The Gladiator used a single set of chips for almost everything. Only the two program ROM chips are different between versions, which made my life a lot easier. I was also lucky that these chips were easy to get to. Depending on the kind of game, chips might be attached straight to the board, or they might be in sockets where they can be easily removed and reattached. The Gladiator has two game boards, one of which uses two socketed chips. And, thankfully, those were the only two chips I had to remove and dump.

To remove the chips, I used an EPROM removal tool—basically just a little set of pliers on a spring, with pair of needle noses that get in under the edge of the chip in the socket so you can pull it out. The two chips were both common types that my EPROM burner supports, so once I got them out they weren’t difficult to read. The most important chip, which has the game’s program code, is an EPROM chip known as the 27C160—a 2MB chip in a specific form factor. I already own a reader that supports that and the 4MB version of the same chip, which you can see in the above photo. The second chip is a 27C4096DC which has a much smaller 512KB capacity.

Photo of an open game cartridge showing the boards and ROM chips

Why are there two program ROMs? Many games for the PGM use a fascinating and pretty intense form of copy protection. As I mentioned earlier, the PGM motherboard has a 20MHz 68000 processor, a 16-bit CPU that was very widely used in the 90s. The game cartridge, meanwhile, has a 20MHz ARM7 coprocessor. For early games, that ARM7 was there just for copy protection. Game cartridges would feature an unencrypted 68000 ROM and an encrypted ARM7 ROM; the ARM7 must successfully decrypt and execute code from the encrypted ROM for the main program code to be allowed to boot and run. By the end of the PGM’s life, they’d clearly realized it was silly to be devoting the ARM7 just to copy protection when it was faster than the CPU on the motherboard, so they put it to use for actual game logic. On games like The Gladiator, the unencrypted 68000 ROM and the main CPU are only doing very basic bootstrapping work and then hand off the rest of the work to the ARM7, which runs the game using code on the encrypted ARM7 chip.

I spent awhile fumbling around trying to get the dumped ARM7 ROM to work, but it turns out that’s because I read it as the wrong kind. Oops. My reader has a switch that switches between the 2MB and 4MB versions of the chip… and I had it set to 4MB, even though the chip helpfully told me right on the package it’s a 2MB chip. So, after I spent a half hour fumbling, I realize what I’d done and went back to redump it—and that version worked first try. Phew.

Screenshot of The Gladiator's boot screen with the program ROM versions

Once I dumped it, I was able to figure out that one of the two program ROMs is identical to one that’s already in MAME; only the ARM ROM is unique. That meant adding it to MAME was very easy; I could mostly copy and paste existing code defining the game cartridge, changing just one line with the new ROM and a few lines with some different metadata, and I was good to go. I submitted a pull request and, after some discussion, it was merged. For something I’ve wanted to be able to contribute to for years, feels good and, honestly pretty painless. And now, as of MAME 0.249, The Gladiator 1.04 can finally be emulated!

Do You Speak the Lingo?

I’ve been spending some time lately contributing to ScummVM, an open-source reimplementation of many different game engines that makes it possible to play those games on countless modern platforms. They’ve recently added support for Macromedia Director, an engine used by a ton of 90s computer games and multimedia software that I’m really passionate about, so I wanted to get involved and help out.

One of the first games I tried out is Difficult Book Game (Muzukashii Hon wo Yomu to Nemukunaru, or Reading a Difficult Book Makes You Sleepy), a small puzzle game for the Mac by a one-person indie studio called Itachoco Systems that specialized in strange, interesting art games. Players take on the role of a woman named Miss Linli who, after falling asleep reading a complicated book, finds herself in a strange lucid dream where gnomes are crawling all over her table. Players can entice them to climb on her or scoop them up with her hands. If two gnomes walk into each other, they turn into a strange seed that, in turn, grows into other strange things if it comes into contact with another gnome. Players guide her using what feels like an early ancestor to QWOP, with separate keys controlling the joints on each of Linli’s arms. It’s bizarre, difficult to control, and compelling.

A lot of early Director games play fine in ScummVM without any special work, so I was hoping that would be true here too. Unfortunately, it didn’t turn out to be quite that simple. I ended up taking a dive into ScummVM’s implementation of Director to fix it.

Director uses its own programming language, Lingo, which is inspired by languages like Smalltalk and HyperCard. HyperCard was Apple’s hypermedia development environment, released for Macs in 1987, and was known for its simple, English-like, non-programmer friendly programming language. Smalltalk, meanwhile, is a programming language developed in the 70s and 80s known for its simple syntax and powerful object oriented features, very new at the time; it’s also influenced other modern languages such as Python and Ruby. Lingo uses a HyperCard-style English-like way of programming and Smalltalk-style object oriented features.

Early versions of Director are unusual for having the engine interpret the game logic straight from source code1—which means if you’ve got any copy of the game, you’ve got the source code too. It’s great for debugging and learning how it works, but there’s a downside too. If you’re writing a new interpreter, like ScummVM, it means you have to deal with writing a parser for arbitrary source code. As it turns out, every issue I’d have to deal with to get this game working involved the parser.

I’ll get into the details later, but first some background. To give a simplified view, ScummVM processes Lingo source in a few steps. First, it translates the text from its source encoding to Unicode; since Lingo dates to before Unicode was widely used, each script is stored in a language-specific encoding and needs to be translated in order for modern Unicode-native software to interpret it correctly. Next, there’s a preprocessing stage in which a few transformations are made in order to make the later stages simpler. The output of this stage is still text which carries the same technical meaning, it’s just text that’s easier for the next stages to process. This is followed by the two stages of the actual parser itself: the lexer, in which source code text is converted into a series of tokens, and the parser, which has a definition of the grammar for the language and interprets the tokens from the previous stage in the context of that grammar.

This all sounds complicated, but my changes ended up being pretty small. They did, however, end up getting spread across several of these layers.

1. The fun never ends!

The very first thing I got after launching the game was this parse error:

WARNING: ######################  LINGO: syntax error, unexpected tMETHOD: expected end of file at line 83 col 6 in MovieScript id: 0!

Taking a look at the code in question, there’s nothing that really looks too out of the ordinary:

factory lady
method mNew
    instance rspri,rx,ry,rhenka,rkihoncala,rflag,rhoko,rkasoku
end method
method mInit1 spri
# etc

This is the start of the definition of the game’s first factory. Lingo supports object-oriented features, something that was still pretty new when it was introduced, and allows for user-defined classes called “factories”2. Following the factory lady definition are a number of methods, defined in a block-like format: method NAME, an indented set of one or more lines of method definitions, and an end method line.

That last line, it turns out, was the problem. To my surprise, it turns out those end method blocks are totally optional even though it’s the documented syntax in the official Director manual. Not only can it have any text there instead of method, but it turns out you don’t need any form of end statement at all. If ScummVM didn’t recognize it, it seems that many games must have just skipped it.

Luckily, this was a very easy fix: I added a single line to ScummVM’s Bison-based parser and it was able to handle end statements without breaking support for methods defined without them. I hoped that was all it was going to take for Difficult Book Game to run, but I wasn’t quite so lucky.

2. Language-dependent syntax

Unlike most modern languages, Lingo doesn’t have a general-purpose escape character like \ that can be use to extend a line of code across multiple lines. Instead, it uses a special character called the “continuation marker”, ¬3, which serves that purpose and is used for nothing else in the language4. (Hope you like holding down keys to type special characters!) Here’s an example of how that looks with a couple lines of code from a real application:

global theObjects,retdata1,retdata2,ladytime,selif,daiido,Xzahyo,Yzahyo,StageNum, ¬

Since Lingo was originally written for the Mac, whose default MacRoman character set supported a number of “special” characters and accents outside the normal ASCII range, they were able to get away with characters that might not be safe in other programming languages. But there’s a problem there, and not just that it was annoying to type: what happens if you’re programming in a language that doesn’t use MacRoman? This is before Unicode, so each language was using a different encoding, and there’s no guarantee that a given language would have ¬ in its character set.

Which takes me back to Difficult Book Game. I tried running it again after the fix above, only to run into a new parse error. After checking the lines of code it was talking about, I ran into something that looks almost like the code above… almost.

global theObjects,retdata1,retdata2,ladytime,selif,daiido,Xzahyo,Yzahyo,StageNum, ツ

Spot the difference? In the place where the continuation marker should be, there’s something else: , or the halfwidth katakana character “tsu”. As it turns out, that’s not random. In MacRoman, ¬ takes up the character position 0xC2, and is at the same location in MacJapanese. That, it turns out, seems to be the answer of how the continuation marker is handled in different languages. It’s not really ¬, it’s whatever character happens to be at 0xC2 in a given text encoding.

Complicating things a bit, ScummVM handles lexing Lingo after translating the code from its source encoding to UTF-8. If it lexed the raw bytes, it would be one thing: whatever the character is at 0xC2 is the continuation marker, regardless of what character it “means”. Handling it after it’s been turned into Unicode is a lot harder. Since ScummVM already has a Lingo preprocessor, though, it could get fixed up there: just look for instances of followed by a newline, and treat that as though it’s a “real” continuation marker5. A little crude, but it works, and suddenly ScummVM could parse Difficult Book Game’s code6. Or, almost…

3. What’s in a space?

Now that I could finally get in-game, I could start messing around with the controls and see how it ran. Characters were moving, controls were responding—it was looking good! At least until I pressed a certain key…

Her arms detached—that doesn’t look comfortable. In the console, ScummVM flagged an error that looked relevant:

Incorrect number of arguments for handler mLHizikaraHand (1, expected 3 to 3). Adding extra 2 voids!

This sounded relevant, since “hiji” means elbow. I figured it was probably the handler called when rotating her arm around her elbow, which is exactly what visually broke. I took a look at where mLHizikaraHand and the similar handlers were being called, and noticed something weird. In some places, it looks like this:

locaobject(mLHizikaraHand,(rhenka + 1),dotti)

And in other places, it looked slightly different:

locaobject(mLHizikaraHand (rhenka + 1),dotti)

Can you find the difference? It’s the character immediately after the handler name: instead of a comma, it’s followed by a space. Now that I looked at it, the ScummVM error actually sounded right. It does look like it’s calling mLHizikaraHand with a single argument (rhenka + 1). After talking it over with ScummVM dev djsrv, it sounds like this is just a Lingo parsing oddity. Lingo was designed to be a user-friendly language, and there are plenty of cases where its permissive parser accepts things that most languages would reject. This seems to be one of them.

Unfortunately, this parse case also seems to be different between Lingo versions. Fixing how it interprets it might have knock-on effects for parsing things created for later Director releases. Time to get hacky instead. The good news is that ScummVM has a mechanism for exactly this: it bundles patches for various games, making it possible to fix up weird and ambiguous syntax that its parser can’t handle yet. I added patches to change the ambiguous cases to the syntax used elsewhere, and suddenly Miss Linli’s posture is looking a lot healthier.

This whole thing ended up being much more of a journey than I expected. So much for having it just run! In the end, though, I learned quite a bit—and I was able to get a cool game to run on modern OSs. I’m continuing to work on ScummVM’s Director support and should have more to write about later.

Thanks to ScummVM developers djsrv and sev for their help working on this.

  1. Later versions switched to using a bytecode format, similar to Java or C#. This makes processing a lot easier, since bytecode produced by Director’s own compiler is far more standardized than human-written source code.

  2. Despite the name, it isn’t really implementing the factory pattern.

  3. The mathematical negation operator.

  4. It’s a bit of a weird choice, but Lingo didn’t do it first. It showed up first in Apple’s HyperCard and AppleScript languages.

  5. Tempting as it is to refactor the lexer, I had other things to do, and I really wasn’t familiar enough with its innards to take that on.

  6. As it turns out, this wasn’t the only game with the same issue. Fixing this also fixed several other Japanese games, including The Seven Colors: Legend of Psy・S City and Eriko Tamura’s Oz.

Exploring JVS

Everyone had their weird pandemic hobby, right? Well, mine is that I bought an arcade video game. Not the whole cabinet - just a board, to hook up to a TV and a game controller. If I can’t go to the arcade, at least I can bring a favourite game home, right?

As you might imagine, an arcade board isn’t just plug and play; you can’t just plug in a USB gamepad and call it a day. Finding out how to actually use it took some research. I looked up the different methods, and found I had more or less two options: a more expensive one involving repurposed arcade hardware, and an open-source one using off-the-shelf parts. While I was figuring out if the more expensive options were worth it, I decided I’d try out the open source one, OpenJVS. I started out as just a user, but ended up getting involved in development. Over the past few months, I’ve contributed a bunch of new features and bugfixes. In the process I learned a lot about the specification… and quashed my fair share of weird bugs.

So what is JVS anyway?

Imagine this: you’re an arcade machine manufacturer, and you need to make it easy for a machine operator to swap a new board into one of their cabinets. How do you make sure they don’t have to run dozens of wires to get the controls hooked up? Video is easy; you can just use VGA, DVI, or HDMI. Power is easy too. What else is left? Well, there’s many different kinds of input so players can actually play the game; coin inputs and money bookkeeping; smart cards to communicate information to a game; in other words, all kinds of input from the player.

The solution? JAMMA Video Standard, or JVS, the standard used by most arcade games since the late 90s.1 It’s a serial protocol that makes arcade hardware close to plug and play: connect a single cable from your cabinet’s I/O board to your new arcade board, and you get all of your controls and coin slots connected at once.

If you want to use a JVS game at home, you could always put something together using an official I/O board, but - as it happens, the JVS spec is available2. There’s nothing stopping you from making your own I/O board, and it turns out a few different people have. There are a few different open source options intended to work in a few different ways. I use OpenJVS, written by Bobby Dilley, which transforms a Raspberry Pi running Linux into a JVS I/O board.

How does JVS work?

So that’s JVS, but how does it work? I won’t go into deep detail, but here’s the 10,000 meter view.

JVS defines communication between an I/O board and a video game board. It uses the RS-485 serial standard to communicate data, meaning that it’s possible to use a standard USB RS-485 device to send and receive commands.3 The I/O board handles tracking the state of all of the cabinet’s inputs, and it’s also responsible for all the coin management: it has its own count of how much money players have spent, and the game just reads that number instead of keeping track of it itself.

The game board communicates with the I/O board by sending commands, then receiving packets of information in return. The game board tells the I/O board how many bytes of data to expect, then sends one or more messages as a set of raw bytes. The I/O board reads those bytes, interprets them, then sends a set of responses for each command.

Open source JVS implementations don’t just emulate the things a player would normally touch in the arcade, either. They also simulate features like the service and test buttons, which are tucked away inside a cabinet where only arcade employees can normally use them. The service button allows access to the operator menu, where employees can change hardware and software settings, while the test button does exactly what it sounds like. Arcade players will never get to use these, but for home players it’s useful to be able to do things like change how many coins a play costs, turn on free play, or change the game language. OpenJVS supports both of these, and they seemed to work when I tested them. The test button did cause OpenJVS to log a message about an “unsupported command”, which seemed suspicious, but Bobby and I suspected this was just the board playing fast and loose with the spec so we ignored it. (This will come up again later.)

OpenJVS was already basically complete and implemented all of the common parts of the protocol, so I started out as just a user. As I ran into a few bugs, I started contributing bugfixes and then implementations of more parts of the protocol.

Let’s talk money

I mentioned coin management earlier. Obviously, in arcades, games are pay-to-play. Most game boards have a free play option so you can play as much as you want without paying, and a lot of home players will just switch their boards into the free play mode. There’s nothing stopping you from emulating a coin slot though. OpenJVS lets you map a button on your controller to inserting a coin for something closer to the authentic arcade feeling. It seems like this might not be a common usecase, but the option is always there.

Before long, I noticed a really strange bug. I’d be playing games, and then notice that somehow I had over 200 credits in the machine. I definitely wasn’t mashing the coin button that much, so I knew something had to be wrong. After some experimentation, I eventually figured out it only happened if you did a few specific things in exactly the right order:

  • Insert one or more coins
  • Enter the service menu and then the input test menu
  • Exit the service menu

At this point, I realized something suspicious was happening. I was always getting 200+ coins… but it was more specific than that. I was ending up with exactly 256 minus the number of coins I had when entering the service menu. For example, if I started with 3 coins, I’d always have 253 coins after leaving the service menu. That was a pretty good sign I was seeing something in particular: integer underflow.

Experienced programmers, feel free to skip this paragraph. But for those who aren’t familiar: languages like C, which OpenJVS is written in, feature something called integer overflow and underflow. Number types have a maximum size which affects how many numbers it can represent. A 16-bit (or 2-byte) integer that can store only positive numbers, for example, can only represent numbers between 0 and 65535. Picture, for a moment, what happens if you ask a program to subtract 1 from a 16-bit integer that’s already at 0, or add 1 to a number that’s already at 65535. In C, and many other languages, it will underflow or underflow: subtract 1 from 0, and you get 65535; add 1 to 65535, and you get 0.

Having figured out that I was probably seeing underflow, I took a look at the protocol. Since the JVS I/O board keeps track of the money balance, the protocol provides commands for dealing with that and it seemed like the most likely place I’d find the bug. When I dug into OpenJVS’s implementation of the “decrease number of coins” command, it wasn’t too hard to find the culprit. It was right here:

/* Prevent underflow of coins */
if (coin_decrement > jvsIO->state.coinCount[0])
    coin_decrement = jvsIO->state.coinCount[0];

for (int i = 0; i < jvsIO->capabilities.coins; i++)
    jvsIO->state.coinCount[i] -= coin_decrement;

When OpenJVS received the command to decrement the number of coins by a certain amount, it tried to prevent underflows. Unfortunately, the underflow protection itself was buggy: it checked the number of coins in the first coin slot, but then decremented the number of coins in every slot. If slot 2 has fewer coins than slot 1, then slot 2 will end up underflowing by whatever the difference is. I still wasn’t sure why it was trying to remove 256 coins, which seemed weird. I figured it must just be trying to clear the slot of all coins and trusting the I/O board to prevent underflows, and moved on.

With that bug fixed, I decided to keep working at improving coin support. While I was working on that command, I noticed that OpenJVS was ignoring a similar command. While the board usually only needs to send commands to reduce the number of coins in the balance, like when the player starts a game, it can also send a command to increase the number of coins. I’d noticed that my game board was trying to send that command, but OpenJVS had only implemented a placeholder that logged the request and then moved on without doing anything. The quickest way to figure out what was going on was just to implement the command myself. The actual command in the spec is pretty simple:

Purpose Sample
Command code (always 35) 0x35
Coin slot index 0x01
Amount (first half) 0x00
Amount (second half) 0x01

Easy enough: you can identify that it’s the command to increase coins by the first byte, 35, and then it tells you which coin slot to act on and how many coins to add to it. But when I was replacing the old placeholder command, I noticed something funny:

    debug(1, "CMD_WRITE_COINS\n");
    size = 3;[outputPacket.length++] = REPORT_SUCCESS;

The placeholder command just reported success without doing anything, then jumped ahead in the stream by the command’s length. But it jumped ahead 3 bytes, and according to the spec this command should be 4 bytes long. I tested OpenJVS with the command fully implemented, and noticed two things: pressing the test button now inserted 256 coins into the second coin slot, instead of doing nothing; and the “unsupported command” error I used to see went away. Why? When the buggy placeholder skipped ahead by three bytes, that left one byte in the buffer for OpenJVS to find and mistake for being a command. That byte, 0x01, was actually the last byte of the “insert coin” command that was being sent when the test button was activated. I hadn’t even set out to fix the “unsupported command” bug, but I fixed it anyway.

At this point I’d fixed all the bugs I set out to, but I was still seeing something that just didn’t feel right. When exiting the service menu, the game board now withdrew coins from the balance; when pressing the test button, the board now added coins. But the number of coins looked wrong: it was happening in increments of 256, instead of 1, and even for test commands that seemed unlikely. So I took another look at the spec, and realized the answer had been staring me in the face the entire time. Let’s take another look at the last part of that table from earlier:

Purpose Sample
Amount (first half) 0x00
Amount (second half) 0x01

The number of coins to add or remove is a two-byte, or 16-bit, value. Since JVS is communicating through single bytes, any multi-byte values have to be split up into single bytes for transmission. The spec helpfully tells us how to decode that: the numbers are stored in the big-endian, or most-significant byte first, format. But taking a look at OpenJVS’s code shows that anywhere it was decoding these values, it was decoding them in little-endian format - in other words, it was reading the bytes backwards. What’s 256 in little-endian format? It’s the bytes “0” and “1”, in that order. What’s 1 in big-endian format? It’s those same two bytes in that same order. In other words, the game board hadn’t been trying to add or subtract 256 coins all this time: it was just trying to add and remove single coins.

Putting it all together, what exactly was happening when I saw coins being added and removed? It actually turns out to be pretty simple. Pressing the test button inserts a coin in the second coin slot. When the operator menu is activated, it can be used to test the coin counter. When the operator leaves the menu, the unit sends commands to remove all of the coins that were added by the test button; this should leave it with the same number of coins as it had when they started. The strange behaviour was a combination of all the bugs working together: the test button didn’t do anything because the command wasn’t implemented, and the buggy bounds checking and incomplete “remove coins” command meant it could underflow and leave the player with hundreds of coins.

I originally set out to fix some pretty simple bugs, but every new bug I uncovered revealed a new one. The experience was quite a fun one. I can’t say I ever thought I’d be writing software for arcade machines, but not only did I fix my own problems, but I had the chance to learn more about how things work in a domain I might never otherwise have gotten the chance to touch.

  1. This was actually the second JAMMA standard, following a simpler standard that was used internationally between 1985 and the late 90s.

  2. Including an excellent English translation by Alex Marshall!

  3. Mostly. JVS actually introduces an extra wire, known as the sync line, but I’m going to ignore that here to keep the explanation simple.

Bundle Install With Homebrew Magic Using Brew Bundle Exec

Has this ever happened to you? You’re writing a project in Ruby, JavaScript, Go, etc., and you have to build a dependency that uses a system library. So you bundle install and then, a few minutes later, your terminal spits up an ugly set of C compiler errors you don’t know how to deal with. After dealing with this enough times I decided to do something about it.

Homebrew already has a great tool in its arsenal for dealing with these problems. Homebrew needs to be able to build software reliably and robustly, after all - even if the user’s system has weird software installed on it or strange misconfigurations. The “superenv” build environment features intelligent automatic setup of build-related environment variables and PATHs based on just the requested dependencies, which filters out unrequested software and prevents a lot of common build failures that come from interfering software. It also uses shims for many common build tools to enforce just the right arguments passing through to the real tools.

So I thought to myself - we solved that problem for Homebrew builds already, right? Wouldn’t it be nice if I could just reuse that work for other things? So that’s what I did. Homebrew already provides the Brewfile dependency declaration format and the brew bundle tool to library dependencies with Homebrew, and as a result there’s already a great way to get the dependency information we’d need to produce a reliable build environment. Since brew bundle is a Homebrew plugin, it has access to Homebrew’s core code - including build environment setup. Putting these together, I wrote a feature called brew bundle exec. It takes the software you specify in your Brewfile and builds a dependency tree out of that, then sets up just the right build flags to let anything you want use them.

For example, say I want to gem install mysql2. Often, you get something like this:

$ gem install mysql2
Building native extensions. This could take a while...
# several dozen lines later...
linking shared-object mysql2/mysql2.bundle
ld: library not found for -lssl
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make: *** [mysql2.bundle] Error 1

Ew, right? Let’s make that better.

By creating a Brewfile with the line brew "mysql", we can specify that we want to build against a Homebrew-installed MySQL and all of its dependencies. Just by running our command prefixed with brew bundle exec --, for example, brew bundle exec -- gem install mysql2, we can run that command in a build environment that knows exactly how to use its dependencies. Suddenly, everything works—no messing around with flags, no special options passed to gem install, and no fragile bundle config trickery.

$ brew bundle exec -- gem install mysql2
Building native extensions. This could take a while...
Successfully installed mysql2-0.5.3
Parsing documentation for mysql2-0.5.3
Installing ri documentation for mysql2-0.5.3
Done installing documentation for mysql2 after 0 seconds
1 gem installed

What exactly does brew bundle exec set? There’s a variety of flags set which are useful for a variety of different compilers and buildsystems.

  • CC and CXX, the compiler specification flags, point to Homebrew’s compiler shims which help ensure that the right flags are passed to the real compiler being used.
  • CFLAGS, CXXFLAGS, and CPPFLAGS ensure that C and C++ compilers know about the header and library lookup paths for all of the Brewfile dependencies.
  • PATH ensures that all of the executables installed by Brewfile dependencies will be found first, before any tools of the same name that may be installed elsewhere on your system.
  • PKG_CONFIG_LIBDIR and PKG_CONFIG_LIBDIR ensure that the pkg-config tool finds Brewfile dependencies.
  • Buildsystem-specific flags, such as CMAKE_PREFIX_PATH, ensure that buildsystems can make use of the Brewfile dependencies.

So the next time you’re bashing your head against build failures in your project, give brew bundle exec a try. It might just solve your problems for you!

The Hunt for the Lost Phoenix Games

Among trashgame fans, Phoenix Games are one of the most infamous game publishers of all time. They’re famous for making games of the lowest possible quality for the lowest possible price; their most infamous “games” were just poorly-animated movies on a PS2 disc. They’re the kind of fascinating bad you just can’t help but love for their own sake.

In their last year of life, Phoenix started publishing Nintendo DS games. They had an aggressive release schedule; in 2008 they announced plans to release 20 games inside of a year, most of which never came out. Which games did or didn’t come out is still hard to figure out. Many online release lists take Phoenix Games’s estimated release dates at face value, including many games that probably never came out. I’ve put together a list of what I’ve been able to confirm in the hopes that anyone with more information can help out.

Out of the 20 games Phoenix announced, only five definitely came out under Phoenix themselves. There are confirmed physical copies of these games and ROM dumps on the internet. Three more were licensed to other publishers after Phoenix’s bankruptcy. The other twelve are less certain; I’ve divided these into three categories with explanations below. Phoenix declared bankruptcy in early 2009, and I haven’t been able to confirm that any games were released after the end of 2008. If you have any information, please contact me via email or on Twitter!

Definitely released:

  • 12 - NorthPole Studio, 2008
  • Adventures of Pinocchio - Unknown, 2008
  • Peter Pan’s Playground - Unknown, 2008
  • Polar Rampage - CyberPlanet, 2008
  • Valentines Day - NorthPole Studio, 2008

Released by other developers:

  • Coral Savior - NorthPole Studio; released as Bermuda Triangle: Saving the Coral by Storm City Games, in North America, 2010; and as Bermuda Triangle by Funbox Media, in Europe, 2011
  • Hoppie II - CyberPlanet; released as Hoppie, by Maximum Family Games, in North America, 2011
  • Veggy World - CyberPlanet; released by Maximum Family Games, in North America, 2011

Possibly released:

These games had finalized cover art and announced release dates. The games in this category have been found in some online store catalogues, suggesting they may have either been released or at least had already been offered to stores.

  • Jungle Gang: Perfect Plans - Listed in an MCV article as having been released October 17, 2008. Listed in a few online store catalogues as sold out.
  • Love Heart - Reviewed by Polish site, who gave it a 2008 release date.

Probably not released:

Included in release lists, but with no store pages to indicate they were ever actually released. Some of these games were given multiple release dates, probably because they were delayed and solicited multiple times.

  • Dalmatians 4 - Listed in a PlayStation Collecting forums catalogue as having been released on February 13, 2009.
  • Lion and the King 3 - Listed in a PlayStation Collecting forums catalogue as having been released on both October 24, 2008 and February 13, 2009.
  • Monster Egg II - Listed in a PlayStation Collecting forums catalogue as having been released on both November 28, 2008 and March 13, 2009.
  • Rat-A-Box - Listed in an MCV article as having been released October 17, 2008, but no store catalogue entries.

Definitely not released:

  • Cinderella’s Fairy Tale - Final cover art, but never included in any release lists.
  • Greatest Flood - No cover art or homepage.
  • Iron Chef II - Final cover art, but never included in any release lists.
  • Monster Dessert - No cover art or homepage.
  • Princess Snow White - No cover art or homepage.
  • War in Heaven - Final cover art, but never included in any release lists.

Recover From FileVault Doom

Just upgraded to Mojave? Getting the 🚫 of doom when you try to boot your Mac? Worried you’ve lost all your data? Don’t panic! Your Mac is recoverable, and this guide will help you get your drive back in shape.

This bug is caused by a new limitation introduced for FileVault encryption. As mentioned in the release notes, any APFS volume encrypted using Mojave’s implementation of FileVault becomes invisible to earlier versions of macOS until the encryption or decryption process completes. By itself, this is mostly harmless. However, this is rendered much worse by a bug1 that causes the Mojave installer to incorrectly trigger reencryption of an already-encrypted FileVault volume during the install process. Because full drive encryption cannot complete in a timely manner, the Mojave installer times out before encryption completes. The result is a macOS 10.13 (or older) install with a macOS 10.14 FileVault volume that hasn’t yet finished encrypting. This volume can’t be read by the startup manager, and so becomes unbootable.

In order to fix this issue, we’ll need to let your drive finish encrypting. This guide will walk you through the process. Before we get started, make sure you have the right supplies on hand. You’ll need a USB key with a capacity of at least 8GB and access to a working Mac.

1. Create a USB Mojave installer

Download the Mojave installer application on your working Mac using the App Store. Don’t worry, you won’t be using that to install - just to create a USB installer for your broken Mac. Then follow these instructions from Apple. This will erase your USB key and turn it into a bootable USB install drive.

2. Boot into the USB installer

Turn off the Mac that you’re upgrading to Mojave and insert the USB drive you just created. Turn the Mac back on, and immediately hold the option key until the Startup Manager appears. Select the drive you created.

3. Enter Recovery Mode

The installer should provide an option to enter Recovery Mode. This is a different version of Recovery Mode than the one that’s builtin to your Mac, and it will be able to see your hard drive.

4. Unlock and mount your hard drive

From the Utilities menu bar at the top of the screen, select Terminal. From the terminal prompt, type diskutil apfs list. In the output, you should see one volume whose FileVault status is listed as FileVault: in progress (or something similar). If it’s already listed as “unlocked”, then you’re good - proceed to step 5.

Unlock this drive by running diskutil apfs unlockVolume DISK, where DISK is the identifier listed after APFS Volume Disk for the volume you identified in the previous paragraph. For example, you might type diskutil apfs unlockVolume disk1s1. This command will ask you for a password; enter either your personal log in password, or a FileVault recovery key.

Once you’ve completed this step, exit Terminal. You should be returned to the Recovery Mode main menu.

5. Create a new APFS volume

Open Disk Utility from the Recovery Mode main menu. Select your main hard drive, right-click it, and choose “Add APFS Volume”. Don’t worry, this isn’t repartitioning your drive - this bootable volume will live on the same volume as your main volume, and you will be able to delete it later. Name it “Temporary”, then click “Add”.

Once you’ve completed this step, exit Disk Utility. You should be returned to the Recovery Mode main menu. Return to the main Mojave installer menu.

6. Install Mojave on the new volume

Choose to install Mojave on the volume named “Temporary” that you created in the previous step. This will be a temporary fresh install, so don’t worry too much about it - go ahead and keep all the options at their defaults.

7. Allow drive encryption to finish

Once your install has finished and you’ve logged in, open Disk Utility. Unlock your normal hard drive by clicking on it and clicking the “mount” button in the toolbar. You will be asked for a password; like in the previous step in the terminal, you will need to provide either your login password (from your old Mac install), or your recovery key.

At this point, drive encryption will start back up again. You will need to wait for that to finish; this could take a few hours. If you want to check on the progress, you can open a terminal and look at the output of diskutil apfs list; encryption is finished when you run that command and see the status read FileVault: Yes.

8. Upgrade your main drive to Mojave

Now you’re finally (finally!) ready to let the Mojave install complete. Open up your USB drive in the Finder and run the macOS installer app. This time, instead of picking the temporary volume, pick your original Mac volume. Let the installer complete. This time, your Mac should boot up like normal. You’re saved!

9. Clean up

Now that your main Mac volume is bootable again, you can get rid of the temporary volume we created in the previous step. To do that, open Disk Utility and right-click the volume named “Temporary”. Click “Delete APFS Volume”, then confirm by clicking “Delete” again.

You’re done!

With any luck, you should be all set at this point. Feel free to reach out to me if you’re still having trouble! Big thanks to @mikeymikey for the suggestion of using a second APFS volume to install the temporary macOS 10.14. Thanks also to my manager, who allowed me to use my internal blog post as the basis for this post.

  1. I wish I could provide more detail about this bug, but as far as I know no details have been published. I’ve filed a radar but haven’t received a response. All I know is that this occurs, seemingly at random, to a small percentage of people who try to upgrade from macOS 10.13.6 to macOS 10.14 when FileVault is already enabled.

Review: Undertale Vinyl Soundtrack

I got my copy of the Undertale vinyl soundtrack in the mail the other day. While I’ve seen a lot of excitement for it, I haven’t seen many people talk about what it’s like, so I decided to write up my impressions.

First, to start with the good news: the art design is beautiful, and gorgeously printed. The red-on-black front cover is stylish, and the mostly-new pixel artwork is very well-done. The gatefold opens to a house party scene with a ton of the game’s NPCs, which is adorable and fits the mood of the game really well too. Most of the printing is matte, with a few details in glossy ink that really stands out—the Undertale logo on the cover, and the coloured lights in the party scene.

Unfortunately the quality of the sleeve itself is subpar. The cardboard feels thin and flimsy compared to other double-albums I own; it lacks weight. The gatefold hinge is also poorly-folded, and doesn’t stay closed on its own when the records are in; it flops open awkwardly. The records also don’t slide comfortably into the sleeve when the gatefold is closed, which makes putting records back after a play more awkward than it has to be. (See right: the record is inserted as far as it gets when the gatefold is closed. The black thing poking out is the inner record sleeve.) If you open the sleeve to insert the records all the way, then they can’t be removed while the gatefold is closed, which is even more awkward. The overall feeling is surprisingly cheap for a $35 album.

The records themselves are also mixed quality. The transparent red and blue coloured vinyl is classy, and the simple label design is attractive as well. It’s a minor detail, but the inner record sleeves are very attractive and easy to get the discs in and out of.

Unfortunately, my copy shipped with large scratches on sides A and B. (See the photo on the left—the scratches are very visible at full size.) Both discs were also covered in dust fresh out of the package. Everything I’ve seen suggests this isn’t an isolated issue—I know other people whose iam8bit records shipped with scratches before the first play, and from scanning their twitter feed, it looks like this is a common complaint with other customers. Needless to say, this is a huge issue, and I’m shocked it’s as common a problem as it is with them.

Given those defects, I was surprised when the discs sounded great. The mastering quality is very good, and leaves very little to complain about if you’re lucky enough to get undamaged discs. I did notice a mastering error that clips off the first note of Snowy on side one, but that was the only discernible error.

Given just how huge the Undertale soundtrack is, the entire thing was not going to fit on two discs. The vinyl release curates a selection of tracks to fit the available running time. They generally made a good set of calls, hitting all the major notes and popular moments, though by necessity your favourite deep cut is probably missing. Given the time constraints, I don’t see much to complain about in the selection. Pushing the soundtrack to three discs would probably have been a good call, even if that would have pushed up the price.

Despite the good sound quality, it doesn’t feel like iam8bit actually has the experience it takes to release a premium package like Undertale was meant to be. It’s pretty clear they have big ambitions, and I hope they learn from their mistakes so they get to the point that their quality lives up, but they’re not there yet. The stuff they’re good at—art and graphic design—is offset by poor physical design and unreliable QC. I can’t really recommend this as-is, especially with the high chance of getting a dud.

Aside from the main album, preorders came with a special dogsong single. This one-sided 7" single has an extended version of dogsong, and is pressed on transparent vinyl with an image of the annoying dog on the other side. It’s very cute, and for what it is it’s well-produced and pretty-looking.


A few months ago, I wrote decoders for the PCM formats used by the Sega CD and Sega Saturn versions of the game Lunar: Eternal Blue1. I wanted to write up a few notes on the format of the Sega CD version’s uncompressed PCM format, and some interesting lessons I learned from it.

All files in this format run at a sample rate of 16282Hz. This unusual number is based on the base clock of the PCM chip. Similarly, the reason that samples are 8-bit signed instead of 16-bit is because this is the Sega CD PCM chip’s native format.

Each file is divided into a set of fixed-size frames; the header constitutes the first frame, and every group of samples constitutes the subsequent frames. These frames are set to a size of 2048 bytes—chosen because this is the exact size of a Mode 1 CD-ROM sector, and hence the most convenient chunk of data which can be read from a disc when streaming. This is also why the header takes up 2048 bytes, when the actual data stored within it uses fewer than 16 bytes.

Each frame of content contains 1024 samples. Because each sample takes up one byte in the original format, this means only half of a frame is used for content. The actual content is interleaved with 0 bytes. Despite seeming a pointless, inefficient use of space, this does serve a purpose by allowing for oversampling. The PCM chip’s datasheet lists a maximum frequency of 19800Hz, but this trick allows for pseudo-32564Hz playback.

The format used in Lunar: Eternal Blue supports stereo playback, though only two songs are actually encoded in stereo2. Songs encoded in stereo interleave their content not every sample, as in most PCM formats, but every frame; one 2048-byte frame will contain 1024 left samples, the next frame will contain the matching 1024 right samples, and so forth. This was likely chosen because the PCM chip expects samples for only one output channel at a time, and doesn’t itself implement stereo support.

The loop format used is highly idiosyncratic. Loop starts are measured in bytes from the beginning of the header; that is, a position to seek to from the start of actual content. The loop end time, however, uses a different unit; loop end is measured in terms of the sample at which the loop ends, not the byte. Because one sample is stored for every two bytes, and because left/right samples in a stereo pair are counted numerically as the same sample, this means that this count differs from the byte position at the end of the song by a factor of either 2 or 4. This is a quirk of how the PCM playback routine works; it’s more efficient to keep track of the number of samples played instead of the bytes played, and therefore storing the data in that format means that no extra math has to be performed to determine if the end of a loop has been reached. Similarly, the treatment of left/right samples as being the same sample is likely an artifact of what was the simplest way for the PCM playback code to handle this condition.

Looking at these PCM files gave me a new appreciation for design, and helped me appreciate more how important it is to understand the reason behind design decisions. In a lot of ways this format feels like it should be “bad” design: it’s strange, it’s idiosyncratic, it’s internally inconsistent. But every single detail is carefully chosen; everything serves a purpose. Each of these idiosyncrasies was carefully chosen to solve a particular problem, or to ensure peak performance in a particular bottleneck. Given the restrictions of a 16-bit game console, all of these choices were necessary to be able to support constantly streaming audio like this in the first place. Aligning every single detail of a format or API with the job which needs to be done is its own kind of elegance—a kind of elegance I understand a little better now.

  1. I referenced a couple of open-source decoders for the Sega CD version’s format when writing my own (foo_lunar2 by kode54, and vgmstream), along with some notes given to me by kode54.

  2. The Pentagulia theme, and the sunken tower.