Tech Stuff

As I’m in the middle of a bunch of changes, I don’t have any new and pretty screenshots. So I apologize in advanced to those who aren’t programmers or aren’t interested in the way the engine works….

Back when I started the game engine for Banished, I didn’t want to write an editor. This was mostly for development speed. Writing nice tools can take an immense amount of time. As a result of this, all the data is human readable text, other than inherently binary assets like textures and audio.

My goal was to build a resource system that could serialize a C++ class from a text definition that looks like the class itself, but also support reads and writes to a binary format for speed and size. I also wanted the ability to one day support a tool that could edit the data, and write it back out to text format without extra code being added to the classes.

So here’s an example of what I came up with. In the user interface, a sprite is defined by this (simplified) class.

class ImageDescription : public ElementDescription
	virtual void Serialize(IO::Stream& stream);
	ExternalPtr<SpriteSheet> _spriteSheet;
	String _spriteName;
	int _width;
	int _height;

A ‘SpriteSheet’ is a object that contains many sprites on a single texture – in this case it’s wrapped by a pointer to the external resource. On disk an instance of the image sprite looks like this.

ImageDescription imageNewGame
	Alignment _alignment = MidCenter;

	int _bottomPad = 4;
	int _leftPad = 8;
	int _rightPad = 8;
	int _topPad = 4;

	SpriteSheet _spriteSheet = "SpriteSheet.rsc";
	String _spriteName = "NewGame"

This nicely fills in all the fields. You’ll note that _width and _height are missing, and there are additional fields not listed in the class. Any missing fields are default constructed, and the others are in the base class. _alignment is an enum, and the system allows string definitions to define values for enums.

To serialize the class to any format, the code looks like this.

void ImageDescription::Serialize(IO::Stream& stream)
	IO::Serializer s(stream);


The data stream that is read in or written out by the Serialize() method can be many things. It can be a text parser that reads in text (ascii or wide), a text writer that writes to text, binary read, binary write, memory read, memory write, or any binary form in a compressed format. While I don’t actively use it, it can also be a stream that reads the types and addresses of each field so that an automatic property page can be built to edit the data.

The _Field() macro is used just to pull the name from the variable. _InternalField() can also be used to serialize data that isn’t meant to be written to text form, such as binary image data. Smart pointers can be read and written. All collections (arrays, sets, maps, etc), all integral types and custom data types can be serialized as well. The Serializer class handles headers and footers on the data to ensure data integrity, as well as supporting versioning the data – old data can be skipped, or new data can be written out.

The nicest thing about this system is that the text reader supports data derivation. If I want to make another image that is very similar to the one above, I can use inheritance, similar to C++ inheritance.

ImageDescription imageQuitGame : "imageNewGame"
	String _spriteName = "QuitGame"

The imageQuitGame inherits all the properties of imageNewGame, but overrides the actual sprite name. This cuts down on typing time, copy paste mistakes, and also allows changes to a single object to effect everything similar. The entire UI visual style can be changed by just editing a few base objects.

When the text data changes, the game will parse the new text and then save out a cached binary version. This cached version is then used until the text resource or some dependent file changes. At the same time, binary assets are transformed into a state ready for hardware to use – shader programs are compiled, textures are converted to DXT formats, WAV files are compressed. This gives great loading times as costly parsing, data conversion, and error checking is totally avoided once everything is cached.

While the underlying system to make this work is fairly complex, adding new objects is fast and hardly takes more time than writing the class definition and constructor. And I have a great editor – it’s the same program that I code in. It also makes pseudo data reflection available in C++.


    April 1, 2013 11:02 am

    Whoa! I know kung-fu! 😉

    Seriously though – that was a very interesting read from those of us who have always been on the periphery of code-monkeys :)

    April 1, 2013 11:09 am

    I’ll be honest… I have no clue what any of this really meant but I still appreciate the updates. Pretty screenshots are important, but making a game that runs well and can be easily updated is the base for it all. Nice work!

    April 1, 2013 12:05 pm

    Great post :) good for us as Jerry said “code-monkeys” hehe. Look forward to playing this game and you have done one hell of a great job so far with it.

    April 1, 2013 12:17 pm

    You may want to pick some other convention than prefix your members with underscores:

    April 1, 2013 2:48 pm

    Great post. Looking forward to more programming updates

    April 1, 2013 4:56 pm

    Love the coding post.

    Curious if the serialise method code is exact, or if you include some versioning data too?

    April 1, 2013 6:06 pm

    I’ve never had a problem with leading underscores followed by a lower case letter for field names. Should I ever have a linker issue I’ll fix it then.

    The serialize method is exact code – if it had versioning in it you’d see a number in the Serializer constructor for the version and then something like ‘if (s.Version() > 2) s.Serialize(_Field(_newField));’ If the change is massive (more than a simple add or remove) I can check the version and write custom code to deal with the change.

    April 1, 2013 7:40 pm

    thanks for the response, really excited about this game – amazing you’re doing everything yourself, massive!

    April 1, 2013 11:04 pm

    I can not believe that the development of your entire resource system didn’t take longer than coding a proper editor/tool.

    @ Jean: Please tell us why you liked this post! Imagine I would write: “Bad Post! I’ll never look at this devlog again for no reason!”

    April 2, 2013 1:50 am

    so you cache almost all your data in the game into memory untill it needs to be altered. How much memory are we talking?

    April 2, 2013 7:44 am

    I actually cache the resources to disk as well. The binary cached version tends to be close to identical to the data that the game uses directly so it’s very fast to load.

    In terms of memory, the game currently takes between 250-300M when running.

    April 2, 2013 8:35 am

    Nice post. :)

    Fast loading games are nice things to have. Are there some limitations which would prevent the game from working on older laptops?

    April 2, 2013 8:43 am

    The game requires a video card that supports shader model 2.0 and 256-512M ram. An Intel integrated card probably won’t be fast enough, but a laptop with hardware acceleration will work.

    April 2, 2013 2:22 pm

    It may not have been as interesting regarding seeing the game itself but any update makes me happy because it means I get a bit more to read about the game. Also near instant loading is amaze-balls

    April 2, 2013 4:59 pm

    Thanks for the update. I can’t wait for a beta and hope I’ll have an opportunity to participate. Keep up the great work.

    April 3, 2013 10:54 am

    It would be amazing if you decide to allow anyone who pre-buys the game to play development builds (like what Wolfire and Mojang did). Even right now, it looks like a ton of fun.

    Also, +1 for a Linux build, but it sounds like it has low enough system requirements that it could run in Wine on a reasonable

    General Funky Feel
    April 3, 2013 4:51 pm

    Honestly, my eyes glazed over. But apparently some people did understand better, and I do appreciate any and all updates, so it’s all good.

    Still super stoked about this. Looking forward to other updates.

    April 3, 2013 5:05 pm

    dafuq did i just read lol. anyway continue whatever you are doing, i want to throw my money at you 😀

    April 4, 2013 5:47 am

    in english please?

    April 25, 2013 6:59 pm

    What a great post, thanks for sharing this.