From the span of two evenings of doing some Pico-8 I went from hacking everything in one cart to having a workflow based on compression, serialization and outputing the result from many carts to a single one, increasing the reliability and speed of the process. And I’m still kinda surprised by how relatively easy it was. Let’s talk a bit about it.

I added a few days ago another project down the never-ending backlog of mine and this one was a Pico-8 project a bit bigger than the usual coming from me. After a few dozen of bite-sized sketches and half-done projects, I wanted to finish another half-done project. I’ll talk more in details about what it is if I get it in a more developped state than the previous attempt at devlogging anything. At least, this time, I’m learning a few tricks gameplay-wise, same for Pico-8-based project organization.

Pico-8 saw itself getting a few facilities to build games at a bigger scale, like loading map/GFX/music data from other carts, importing/exporting spritesheets from pictures, passing around execution among multiple carts or exporting a game comprised of multiple carts into a single binary. One of the latest versions saw also the addition of an #include directive which does roughly what a C programmer would expect: blitting a cart or file’s code directly where the include directive at runtime or at export time. Not only it can help with sharing code among multiple projects but also with some trickery involving printh, I got a nifty workflow to create levels for my current project I’ll describe in a few (small I hope) parts.

But first let’s talk about…

Parallel universes Serialization. Serialization in gamedev is the process of converting an object into a blob of data one can store representing enough of the object’s internal state so unserializing said blob would create a copy of the object. Think of it like saving your game progress: you’re saving everything you need to recreate the game, its progress, your character and inventory to run at the exact (or close enough) situation where you left it before.

This system bears a few names in some other programming spheres, the ones that comes to my mind is “pickling” or “beaning” (?). One of the first games I programmed on a computer, a roguelike, used Java’s serializing feature to save the game’s state.

The compression detour

As I was drawing bits and bobs on the current level, I was thinking I could quickly run out of space, I wanted to be able to load levels so I could expand the game’s scope. I took a few wrong turn and was first interested in compressing data into the “Working RAM” (WRAM) of Pico-8 memory, so I took Zep and Felice’s PX9 compression library and tried compress the map’s data directly in the WRAM. Sadly, after a certain amount of time where I failed to extract a cart’s WRAM to another, the realization slapped me in the face: the WRAM is, as its name implies, just RAM. Unlike the SRAM (Save RAM) employed in Game Boy cartridges, it’s not a persistent memory section and trying to write it to the cartridge file will do nothing pertinent. So I ended up moving to storing data in strings, they have the advantage to be light in tokens and if done well, you can pack quite a lot in a single cart with their help.

Let’s use a simple example. Imagine you want to store a series of digits. Having them in a list will cost you roughly two tokens per number as you have to include the comma, plus the curly braces. if you store them in a string, you only have to pay for the parser, space and a bit of CPU but then you get a solution that gets linearly more profitable as your list gets bigger. It’s funny to see that in the wave of using more storage and memory to spare CPU time, Pico-8’s own limitations push you to do the opposite.

my_list = {1, 2, 3, 4, 5, 6} -- 11 tokens
my_list = read_list("1,2,3,4,5,6") -- 3 tokens
-- read_list is left as an exercise to the reader.
-- tip : it involves split(str, ",")

Packing the bits and the bobs

Pico-8’s 0.2.0 version brought us a very pleasant blessing for everyone who needed to pack data in strings: ord and chr. Those two functions are processing numbers and characters. Ord will convert a string character into its decimal value (between 0 and 255 inclusive) where chr does the opposite. No more need for comparing against a table or sticking to base32/base64 and wasting time and space, we can finally use the whole string’s data easily. I did sure used those quite a bit in my serializer.

My serialization system itself is inspired from MessagePack, if with less features. Staying in the line of keeping stuff small and well, I implemented the same data packing than MessagePack: the header’s high nibble contains the code or part of the stored value and the lower nibble can provide an additional data info or part of the stored value.

Let’s describe a few of the stored types:

• The small integer, flagged with the highest bit on, can store an integer number between 0 and 127 (inclusive) by packing it in the rest of the header.
• I have 2-bytes and 4-bytes number types but I guess at least the 2-bytes could use the header’s lower nibble to store a few more bits and/or a negative flag.
• I have two string types : small and normal. The small string stores the length in the lower nibble and the “normal”-sized uses both the lower nibble and a whole byte to describe the string’s length, raising the size up to 2^12 characters, so 4096. I think I’ll hardly get a situation where I’ll need more, but if I do so, I can eventually use 2 bytes instead to describe a string as big as Pico-8’s character size limitation.
• I also have an array type, terminated with a zero character. This type is recursive. The actual item you’re getting by unserializing a string is a list with your items in it.

On 4 bits, I could fit 16 types. I decided to make the small integer use 7 bits instead, so I’m only able to store 9 types ids, (0 to 8). With some trickery I could git all the types and size variations in the routine, but I feel like the current configuration is rather fitting my needs for now. Once the game stalls or is released, I’ll look into making the un/serializer a library, it might get you a quick way to squeeze more data without worrying too much about the size wasted by declaring the data’s structure or typing.

Coupling this with PX9 to store the map, I can easily store the content of a level, map and objects data in a couple hundrers bytes and roll with it effortlessly. I could even do fancy stuff like storing spritesheets but I don’t really think that’d work it out that well, I don’t have a lot of spritesheet space and as the last page is shared with the map data, that’d require a little bit of planning I haven’t took the time to have yet.

For an example, here’s how the sandbox level’s code looks like for now in its cart:

room {0,0,16,16,{
	{torch, 4,11},
	{torch, 10,11},
	{torch, 5,3},
	{book, 5, 4, "light"}
}}
room {16,0,16,16,{
	{torch,24,5}
}}
room {32,0,16,8,{}}
room {8,16,16,16,{}}
room {24,16,20,16,{
		{wtrfall, 33,18,5,5}
}}
do_it(0,0,128,64, "l0")

My games splits the map area into rooms. Here, room is just a function adding the argument in the room list. It’s more a convenience helper and syntaxic sugar than anything. It’s the same code than room({blah, blah, blah}), it’s less tokens and characters and less visaul noise.

Each room has its coordinate and size in the map data and a list of objects (the type identifiers are actually variables storing numbers) and for each object a list of argument for the loader to know what to initialize the object with. See the pattern? It’s similar than the concept of serialization I was mentioning earlier: With a blob of data and the minimum amount of metadata, I can instanciate the game’s objects the way I want them to be. The main difference is here I’m hand-writing the values, I’m not saving them from within the game.

From one cart to many

Now, the question was: how would I properly edit a map without impeding on the game’s code and data? At this point and due to slapping the prototype code inside the game cart, I was nearing 6000 tokens used, which doesn’t leave me lot of space to fool around. Logically, The next step was splitting the map exporting process from the game.

So I ended up with this workflow: to each level its cart where I could freely edit the map, input its object data and run it to generate the data I could append to the game cart. It allowed me to easily have multiple map editors in the same screen while all of them would share the same export mechanism.

The first version of a level cart could compress, serialize and unserialize the data well, but I sure didn’t know how to pass the data to the game cart! I checked the manual and noted the mention of using printh and @clip to write a string in the host’s clipboard, which worked but not totally due to two issues.

The first issue is that pasting something to Pico-8’s editor would make me lose the case, turning for instance all leters into uppercase, i.e. a into A. That might sound not too bad, but as it changes the letter composing the data, it also changes their value, causing level loading errors. The first way I found to circuvent that would be to directly paste the data in the P8 file through an external editor, which isn’t that optimal, but heh, it worked fine.

The other issue was non-printable characters. The null character and the line return wouldn’t be properly exported and the resulting pasted string would lack them. Again, that meant losing data and facing loading errors. To fix that, after serialization I added a pass in which I replaced all instances of problematic characters with their escape code representation (\XX) before exporting to the clipboard. A good reference for this conversion is this excellent post from Zep. Here’s his code for reference.

function escape_binary_str(s)
 local out=""
 for i=1,#s do
  local c  = sub(s,i,i)
  local nc = ord(s,i+1)
  local pr = (nc and nc>=48 and nc<=57) and "00" or ""
  local v=c
  if(c=="\"") v="\\\""
  if(c=="\\") v="\\\\"
  if(ord(c)==0) v="\\"..pr.."0"
  if(ord(c)==10) v="\\n"
  if(ord(c)==13) v="\\r"
  out..= v
 end
 return out
end

Now I had a working system with minimal headaches, but, somehow having to deal with editing the P8 externally became a chore bringing lots of “not loaded due to unsaved changes” popups telling me I forgot to save and reload or didn’t follow the steps perfectly. Maybe there was a way to include the level data automatically…

Addendum

Since the first version of this blog post, the case loss issue caused by copy-pasting data directly in Pico-8 was eventually fixed with the introduction of puny mode (ctrl-p) which allows typing lowercase and pasting without losing case.

Also, since the time I first wrote this post other features were added to Pico-8 to ease loading data from binary strings, like poke accepting more than one value and chr and ord working on mutiple values too. My serializer was designed without those in mind and I might have designed it around those if I were to redo it from scratch.

Eurêka

There actually is a solution! #include, my old C preprocessor frenemy, you’re just full of surprises! The trick is two-fold:

1. printh is called to write the compressed data into a temporary file
2. I include that temporary file’s content with an include invocation.

Suddenly, I got an automatically working pipeline, all happening inside Pico-8, rather than previosu game projects like Gim’a’gift where I tried a workflow based on an external editor and files compiled into a P8.

Doing the extra architecture bit because that’s the running joke with me, I ended up with that kind of structure:

Common level code cart
    PX9 compression code
    Serialization code
    Convert and export to temp file function

Level cart
    #include common_level_code
    Rooms object list
    Call to the export function
    Generation of a text file containing the generated level data
...

The game cart
  The rest of the owl
  Inclusion of generated level data files
    #include level_cart1.txt
    #include level_cart2.txt
    ...

And to have an helper to make sure that everything’s up to date and to facilitate the export process, I even created a Makefile, because why the hell not. GNU Make is usually easy to install on a *NIX OS and I knew how to quickly write what I needed.

The process just builds the carts with a basic dependency system :

• The game needs the level data
• The level data depends on the level carts
• The level carts depend on the exporter cart.

And it just works. I have a scalable system in which can easily extend my game on, once I have the need for, hahah. Plus the Makefile is done well enough so I can run the level exports in parallel (make -j and a number of parallel jobs), so in a situation where the serialization could change, the process time is minimal.

Fresh from the repository

Here is the Makefile file for reference with extra annotations in an attempt to provide explanations on what happens when I want to apply one or more level changes into the game.

# Just a note: GNU Make only supports tabulations as the indent character.
PICO8 := pico8

# lbase.p8 is the common cart used by the level carts to export their data

# Here we're collecting all filenames starting with l and ending with .p8,
# namely the level carts
LEVELS := $(wildcard l*.p8)
# Let's skip lbase
LEVELS := $(filter-out lbase.p8, $(LEVELS))
# Generate the filenames of the generated level files from the carts' filenames.
# lPantoufle.p8 => lPantoufle_out.txt
LEVEL_DATA = $(patsubst %.p8, %_out.txt, ${LEVELS})

all: amp.p8

# Call Pico-8 to generate the binaries out of the game cart. amp.bin "requires"
# amp.p8. If amp.p8 is more recent than the generated files, the job will run
# again.
amp.bin: amp.p8
	${PICO8} -export amp.bin amp.p8

# To have access to the game cart, we need first to generate the level data.
amp.p8: ${LEVEL_DATA}

# To generate the level data, we need to have the up-to-date common level cart
# and the level cart. Call Pico-8 to regenerate the level data if needed.
l%_out.txt: l%.p8 lbase.p8
	${PICO8} -x $<

# "Interface"

# To clean the intermediate files or the release ones.
clean:
	rm -r amp.bin
	rm ${LEVEL_DATA}

release: amp.bin

# Let's have a command to pop a Pico-8 with the cart loaded for quick debug.
run: amp.p8
	${PICO8} -run amp.p8

# .PHONY targets are not dependant on files.
.PHONY: run release all

Here what does look like in action:

# The "INFO:" lines are output by the export script.
$ make
pico8 -x l0.p8
INFO: -exporting-
INFO: > compressing map
INFO: > serializing level
INFO: -done-
INFO: string legnth = 534
pico8 -x l1.p8
INFO: -exporting-
INFO: > compressing map
INFO: > serializing level
INFO: -done-
INFO: string legnth = 362
pico8 -x l2.p8
INFO: -exporting-
INFO: > compressing map
INFO: > serializing level
INFO: -done-
INFO: string legnth = 854
pico8 -x l3.p8
INFO: -exporting-
INFO: > compressing map
INFO: > serializing level
INFO: -done-
INFO: string legnth = 684

# Faking an update of l3 by using the touch command to change the file's
# timestamp.
$ touch l3.p8

$ make
pico8 -x l3.p8
INFO: -exporting-
INFO: > compressing map
INFO: > serializing level
INFO: -done-
INFO: string legnth = 684

The dependency tracking does its job: on the second call, only the job to update l3.p8’s output file is done as the other files were untouched. Magic!

Let’s recap it!

So, the project works like this:

• I’ve got a cart containing the level generation script.
• I’ve got as many carts as I have levels in the game. They include the level generation cart and executes its script to generate a file with the level data.
• I’ve got the game’s cart containing all the game’s logic and includes the generated level data to uncompress and load when needed.
• Having one cart per level allows me to generate level data in parallel with the help of a Makefile script, which allows me too to easily export the game when needed..

Well, let’s hope now that I have a decent workflow where I don’t always have to leave Pico-8 I can finish that project! Stay tuned! See you later!

Update log

• 2022-01-19 - Thanks to merwok for reporting the issues in my post.
  • Rewrote parts of the “From one cart to many” section to improve readability.
  • Fix the description of the level data generation to precise the separation between the level cart and the generated file.
  • Improved the dependency graph.
  • Added the project’s Makefile and explanation around its usage.
  • Rewrote parts of the “Compression detour” paragraph and added a simple example.
  • Added a small recap.