From the span of two evenings of doing some Pico-8 I went from hacking everything in one cart to having a workflow based on compression, serialization and outputing the result from many carts to a single one, increasing the reliability and speed of the process. And I’m still kinda surprised by how relatively easy it was. Let’s talk a bit about it.

A table describing how long can be the process of optimizing a routine to make it worth it. Source: XKCD, licence CC-BY-NC 2.5

I added a few days ago another project down the never-ending backlog of mine and this one was a Pico-8 project a bit bigger than the usual coming from me. After a few dozen of bite-sized sketches and half-done projects, I wanted to finish another half-done project. I’ll talk more in details about what it is if I get it in a more developped state than the previous attempt at devlogging anything. At least, this time, I’m learning a few tricks gameplay-wise, same for Pico-8-based project organization.

Pico-8 saw itself getting a few facilities to build games at a bigger scale, like loading map/GFX/music data from other carts, importing/exporting spritesheets from pictures, passing around execution among multiple carts or exporting a game comprised of multiple carts into a single binary. One of the latest versions saw also the addition of an #include directive which does roughly what a C programmer would expect: blitting a cart or file’s code directly where the include directive at runtime or at export time. Not only it can help with sharing code among multiple projects but also with some trickery involving printh, I got a nifty workflow to create levels for my current project I’ll describe in a few (small I hope) parts.

But first let’s talk about…

Parallel Serialization. Serialization in gamedev is the process of converting an object into a blob of data representing enough of the object’s internal state so unserializing said blob would create a copy of the object. Think of it like saving your game progress: you’re saving everything you need to recreate the game, its progress, your character and inventory to run at the exact (or similar) situation where you left it before. This system bears a few names in some other programming spheres, the ones that comes to my mind is “pickling” or “beaning” (?). One of the first games I programmed on a computer, a roguelike, used Java’s serializing feature to save the game’s state.

The compression detour

As I was drawing bits and bobs on the current level, I was thinking I could quickly run out of space, I wanted to be able to load levels so I could expand the game’s scope. I took a few wrong turn and was first interested in compressing data into the “Working RAM” (WRAM) of Pico-8 memory, so I took Zep and Felice’s PX9 compression library and tried compress the map’s data directly in the WRAM. Sadly, after a certain amount of time where I couldn’t extract a cart’s WRAM to another, the realization slapped me in the face: the WRAM is, as its name implies, just RAM. Unlike the WRAM employed in Game Boy cartridges, it’s not a persistent memory section and trying to write it to the cartridge file will do nothing pertinent. So I ended up moving to storing data in strings, they have the advantage to be light in tokens and if done well, you can pack quite a lot in a single cart thanks to them.

For a simple example, imagine you want to store a series of digits. Having them in a list will cost you roughly two tokens per number as you have to include the comma, plus the curly braces. if you store them in a string, you only have to pay for the parser, space and a bit of CPU but then you get a solution that gets linearly more profitable as your list gets bigger. It’s funny to see that in the wave of using more storage and memory to spare CPU time, Pico-8’s own limitations push you to do the opposite.

Packing the bits and the bobs

Pico-8’s 0.2.0 version brought us a very pleasant blessing for everyone who needed to pack data in strings: ord and chr. Those two functions are processing numbers and characters. Ord will convert a string character into its decimal value (between 0 and 255 inclusive) where chr does the opposite. No more need for comparing against a table or sticking to base32/base64 and wasting time and space, we can finally use the whole string’s data easily. I did sure used those quite a bit in my serializer.

My serialization system itself is inspired from MessagePack, if with less features. Staying in the line of keeping stuff small and well, I implemented the same data packing than MessagePack: the header’s high nibble contains the code or part of the stored value and the lower nibble can provide an additional data info or part of the stored value.

Let’s describe a few of the stored types:

  • The small integer, flagged with the highest bit on, can store an integer number between 0 and 127 (inclusive) by packing it in the rest of the header.
  • I have a 2 and 4 bytes number types but I guess at least the 2-bytes could use the header’s lower nibble to store a few more bits and/or a negative flag.
  • I have two string types : small and normal. The small string stores the length in the lower nibble and the “normal”-sized uses both the lower nibble and a whole byte to describe the string’s length, raising the size up to 2^12 characters, so 4096. I think I’ll hardly get a situation where I’ll need more, but if I do so, I can eventually use 2 bytes instead to describe a string as big as Pico-8’s character size limitation.
  • I also have an array type, terminated with a zero character. This type is recursive. The actual item you’re getting by unserializing a string is a list with your items in it.

On 4 bits, I could fit 16 types. I decided to make the small integer use 7 bits instead, so I’m only able to store 9 types ids, (0 to 8). With some trickery I could git all the types and size variations in the routine, but I feel like the current configuration is rather fitting my needs for now. Once the game stalls or is released, I’ll look into making the un/serializer a library, it might get you a quick way to squeeze more data without worrying too much about the size wasted by declaring the data’s structure or typing.

Coupling this with PX9 to store the map, I can easily store the content of a level, map and objects data in a couple hundrers bytes and roll with it effortlessly. I could even* do fancy stuff like storing spritesheets but I don’t really think that’d work it out that well, I don’t have a lot of spritesheet space and as the last page is shared with the map data, that’d require a little bit of planning.

For an example, here’s how the sandbox level’s code looks like for now in its cart:

room {0,0,16,16,{
	{torch, 4,11},
	{torch, 10,11},
	{torch, 5,3},
	{book, 5, 4, "light"}
}}
room {16,0,16,16,{
	{torch,24,5}
}}
room {32,0,16,8,{}}
room {8,16,16,16,{}}
room {24,16,20,16,{
		{wtrfall, 33,18,5,5}
}}
do_it(0,0,128,64, "l0")

My games splits the map area into rooms. Here, room is just a function adding the argument in the room list. It’s more a convenience helper and syntaxic sugar than anything. It’s the same code than room({blah, blah, blah}), it’s less tokens and characters and less visaul noise.

Each room has its coordinate and size in the map data and a list of objects (the type identifiers are actually variables storing numbers) and for each object a list of argument for the loader to know what to initialize the object with. See the pattern? It’s similar than the concept of serialization I was mentioning earlier: With a blob of data and the minimum amount of metadata, I can instanciate the game’s objects the way I want them to be. The main difference is here I’m hand-writing the values, I’m not saving them from within the game.

From one cart to many

Now, the question was: how would I properly edit a map without impeding on the game’s code and data? At this point and due to slapping the prototype code inside the game cart, I was nearing 6000 tokens used, which doesn’t leave me lot of space to fool around. Logically, The next step was splitting the map exporting process from the game.

The first version of a level cart could compress, serialize and unserialize the data well, but I sure didn’t know how to pass the data to the game cart! I checked the manual and noted the mention of using printh and @clip to write a string in the host’s clipboard, which worked but not totally due to two issues.

The first one is that pasting something to Pico-8’s editor would make me lose the case, turning for instance all leters into uppercase, i.e. a into A. That doesn’t sounds too bad, but I quickly got level loading errors as it turned the stored values into the wrong value. The first way I found to circuvent that would be to directly paste the data in the P8 file through an external editor, which isn’t that optimal, but heh, it worked fine.

The other issue was non-printable characters. The null character and the line return wouldn’t be properly exported and the resulting pasted string would lack them. Again, that meant losing data and facing loading errors. To fix that, once the data was serialized into a string, I replaced all instances of issue-causing character with their escape code representation (\XX) before exporting to the clipboard.

Now I had a working system with minimal headaches, but, somehow having to deal with editing the P8 externally was a chore bringing lots of “not loaded due to unsaved changes” popups telling me that the changes weren’t applied. Maybe there is a way to include the level data automatically…

Eurêka

There actually is one! The realization slapped me in the face strikingly too: #include, my old C preprocessor frenemy, you’re just full of surprises! The trick is two fold, printh is called to write the compressed data into a temporary file and then I’d include that file’s content with an include invocation. And suddenly, I got an automatically working pipeline, all happening inside Pico-8, instead of games like Gim’a’gift where I tried a workflow based on an external editor and files compiled into a P8.

Doing the extra architecture bit because that’s the running joke with me, I ended up with that kind of structure:

Common level code cart
    PX9 compression code
    Serialization code
    Convert and export to temp file function

Level cart
    #include common_level_code
    rooms object list
    call to the export function
...

The game cart
  #include level_cart1
  #include level_cart2
  ...

Dependency graph of the project.

And to have an helper to make sure that everything’s up to date and to facilitate the export process, I even created a Makefile, because why the hell not. It just builds the carts with a basic dependency system (the levels depend on the common level code cart, the game depends on the levels, the binaries depend on the game) and it just works. I have a vaguely scalable system I can easily extend my game on, once I have the need for, hahah. Plus the Makefile is done well enough so I can run the level exports in parallel, so in a situation where the serialization could change, the process time is minimal.

Well, let’s hope now that I have a decent workflow where I don’t always have to leave Pico-8 I can finish that project! Stay tuned! See you later!