On Creating Tools

When I started coding my own engine and game, one of my constraints was that I wasn’t going to write an editor for whatever game I made. This was mostly a time saving constraint. Good tools take time, as I certainly learned while making console games. Often the editor tools to go with a new system took as much time to write and debug as the system itself.

However, as I work on a new game, I find that I need an editor for some things. I need to be able to place objects, make paths, and edit properties. Which is potentially a huge project unto itself. So I’ve been looking for ways to write such a tool with minimal code. More immediately, I’ve been working on some procedural generation code for terrain, but I want to be able to edit the generation properties and see results immediately, instead of changing parameters, running the code to see a result, quitting the program, and repeating over and over.

Also important, is the need to not waste time on user interface for code that is under-going development. If I add a parameter to the terrain generation code, then decide I don’t need it, I shouldn’t be spending time adding a new button or number edit field on some UI. This means some sort of automatic user interface.

So what I need is an automatic property page style dialog that uses type reflection to determine field names, types, and values.

Type Reflection

The idea of reflection is that the code that’s running knows about the properties of itself as it’s running. Given some code that defines some data and the functions that operate on that data, the code can find out and use names, types, and all sorts of other useful information about itself. Strangely, the engine sort of has this functionality, but it wasn’t built for this purpose. It just needed some extra glue to make it work.


What the engine does have is a very flexible serialization system. A single function is created to read and write data to and from disc. It can operate in binary mode or text mode. Text mode is a bit fancy. I wrote it to be able to edit data (since I had no editor) in a human readable way. It looks somewhat like JSON, but I built it to look more like the C++ class that is used in code. For example, here’s an ImageBuffer (that’s called a texture in most engines…)

ImageBuffer editorSheet
	String _imageName = "Build/EditorSheet.png";
	Type _type = Rectangle;
	Usage _usage = Texture;
	Format _format = RGBA8;
	Flags _flags = Mipmaps;
	AddressMode _addressU = Clamp;
	AddressMode _addressV = Clamp;
	AddressMode _addressW = Clamp;
	FilterMode _filterMode = Linear;

The C++ that defines the ImageBuffer looks very similar.

class ImageBuffer 
	int32 _width;
	int32 _height;
	ImageType _type;
	Usage _usage;
	Format _format;
	Flags _flags;
	AddressMode _addressU;
	AddressMode _addressV;
	AddressMode _addressW;
	FilterMode _filterMode;
	System::String _imageName;

And serializing the image to and from disc looks like this:

bool ImageBuffer::Serialize(IO::Stream& stream)
	IO::Serializer s(stream);


	return true;

There’s some magic here. The stream of data could be text or binary, (or something else) and when the serializer is instantiated for that stream, an appropriate reader or writer is created. The _Field and _InternalField macros define what’s available to the text reader, and what isn’t. For example width and height come from the image, and aren’t user editable.

Nothing says that the serializer has to come from or go to disc. It could be to memory, or some other crazy system, like following object pointers for garbage collection. So by writing a new back end for the serializer, I get exactly what I need for reflection. The field names, types, values, and addresses are available. I even know which fields I can present visibly and which ones not to.

So with a tiny bit of extra code, I get object reflection without modifying any of my engine code. Amazing! Did I luck out or what? While the ImageBuffer example doesn’t show it, this system also works for user defined types, enums, flags, pointers, arrays, colors, type ids, etc.

The User Interface

Again, I’m lucky. In making Banished I wrote a UI system that has a lot of control types. So again I don’t have to write a ton of code or bring in a separate UI library.

The UI in the engine has edit boxes, number edits and spinners, combo boxes, list boxes, check boxes, scroll bars, windows, layout tools, sprites, text, etc. Another game may not have needed so many different controls. I’m not sure what I’d be doing for making this user interface otherwise.

Putting it Together

PropertyEditorSo all I really needed to do was write some code that would create a control for each specific type. Number edits for integers, a set of check boxes for flags, combo boxes for enums, etc. When the user changes the values in the controls, I’d just need to push the new value into the back end of my serializer and send the data back to the object using the Serialize function.

And since the serialization system is already built to do things like compile a PNG into a block compressed format ready for GPU use, when the values are changed in the propery editor, the same compilation occurs as if it were being loaded from disc.


I say that it was easy, but it’s more that the new functionality wasn’t invasive to the existing engine. It was still a fairly sizeable bit of code and took a while to write properly.

The Property Editor

PropertyEditor2So without too much fuss, and zero engine modifications (just additional code), I have a property editor for any object in my engine. I can edit textures, sounds, or any new game object I add to the engine.

I have a few more editors to write. For example I don’t have a color picker. Right now I just type in a hex value like 0xFFCO80AB. But that shouldn’t be too bad to create in a day or so. And I probably need some resource editor that allows me to browse everything on disc.

I could also modify the serialization functions to provide some meta-data for the editor. For example minimum/maximum values for numbers, increments for spinners, read only fields, field groupings, etc. But this will require modifying every existing serialization function. All in good time.

This is a long way from a full featured editor, but it’s a start and it has a lot of possibilities. First is that I can edit all objects that I serialize without any additional code. It can also be used to do strange things, like editing the user interface properties of the property editor. And because it uses my own UI, in the future I could use it in game to select objects, see their properties, and change them for debugging purposes.



1.0.7 Beta

Here’s a new Banished beta for the modding community to test out. It adds 10 mod-usable resource flags and resource limits. It also adds the ability to add these limits to the game windows that use them, such as the status bar, limits, and town hall production/graph. It also has a new memory allocator that doesn’t have a hard limit on system memory used.

If you use these, please let me know how they work out. Thoughts, bugs, crashes, etc. Email support @ shiningrocksoftware . com


Without further ado, here’s the beta mod kit:

Be sure to check out the documentation. There’s a new section on using the resource limits and a new example mod that enables them all.

Here’s a patch for 1.0.6 to Beta

Note that you need to apply the patch to version 1.0.6. Previous versions of the game won’t work with this patch. Once downloaded, just unzip the archive into the folder where you have Banished installed. This is usually C:\Program Files\Shining Rock Software\Banished\.

As always the Beta is up on Steam. If you are using Steam, go into your game library and right click on Banished. Select properties, and then in the windows that opens, select the BETAs tab. Select the drop down and pick Beta Test for 1.0.7.

With either the patched version or steam beta, the base game won’t change (except for a few minor bug fixes), but they’ll let you run mods generated with the new mod kit.

Again, let me know how this build works out.


Mega Update

Way back in June, I was working on getting the game code running on Mac and Linux. And while the game runs on those platforms, there are still a bunch of details to implement for a final product, plus testing. But I’ve been itching to work on new games, so I’ve been working on new things more than working on Mac and Linux versions of Banished.

Here’s a rundown of the past few months.

Terrain Rendering

A lot of the new games I have in mind require a larger terrain than is in Banished, and unlike Banished, they would require a free camera that can look up to the horizon. So I went ahead and coded a new terrain system.

I started with a paper called CDLOD (Continuous Distance-Dependant Level of Detail Rendering) for height maps. My implementation started out true to the paper, but now I use a slightly different tessellation of the terrain with alternating diagonals, rather than all diagonals in the same direction. So my LOD is slightly different. And at the moment I still wanted to support hardware that doesn’t support vertex texture fetch (or does but it’s slow), so I cache chunks of the terrain on the CPU instead of reading the height data with the GPU. Having the vertex data available on the CPU is required anyway for collision detection.

This gives me a terrain that can render all the way to the horizon with no real size limits and nice LODs that fade gently with distance. A 128km x 128km terrain is easily rendered on low end hardware, and the level of detail can be tweaked to reduce triangle count, or increase detail depending on hardware. Of course a terrain that big has lots of procedural detail, since storage would be limiting.

Here’s a view of the underlying triangles that would render when flying far above the terrain. I’m not trying to win any art awards here, this terrain and texture is purely programmer art for testing.


Beyond just getting the triangles to render, I also need to add visual detail that fills in the gaps between triangles with something that looks like much higher detail. So I added a system that can handle unique texturing across the entire terrain and use higher frequency data than the triangles used to render.

With some quickly generated normal maps, the view above turns into this.


The terrain also does simple occlusion culling so that large mountains block whats behind them and it won’t render things far away.

I still have a bunch of work to do to turn this into something usable in a game, such as good color textures and terrain generation that supports gameplay instead of simple fractal noise.

Cross platform toolset

Since I had terrain working, I wanted to make sure it worked on all Linux and Mac as well as Windows, since I’d prefer to be able to launch my next game cross platform all at once instead of doing it after the fact.

But I had a bad workflow. On Windows, I can edit resource files and the game reloads them on the fly, or the resources compile when needed as they get loaded from disc.

On Mac and Linux, I only ported the game code, and none of the toolset. So the workflow goes:

  • – Make sure PC build works.
  • – Compile all resource and generate a pack file.
  • – Put the pack file somewhere accessible by the Mac and Linux machines.
  • – Make sure Mac and Linux compile.
  • – Run Mac or Linux build.

This is fine until I need to change data to debug a problem. If I’m working under Linux and want to do something simple like view the terrain in wireframe mode, this requires a data change (yes, all my render state is data), which requires a pack file recompile on Windows, etc, etc. Additionally when I travel, I prefer a light and small laptop, such as my MacBook, but if I can’t edit data on it, then there’s a limit to the kind of things I can work on.

And so I was motivated to make the entire toolset work cross platform. Yikes. All that Windows specific code had to go!! So I ended up having to rework the engine by using cross platform code for fonts, image loading, dxt block compression, audio decoding, and a few other things. Thankfully the FBX format works on OSX and Linux, so I didn’t have to change model formats.

A bunch of changes later and the OSX/Linux version can now build the project from raw resources and no longer requires a Windows machine to build the data. Woot. I wish it had actually been as simple as writing that sentence…

The only downside to this is that Linux/OSX can’t compile DirectX shaders into binary form. It generates the text though. So I can’t use data created on Linux directly on a Windows machine at the moment, but that’s only a problem building releasable products. For day to day work I can now work on any platform. However 3DSMax forces me to use Windows, but I don’t do artwork every day, especially during early development.

Major Refactoring

Next I decided that if I was going to start a new game, I was going to clean things up properly. Just implementing the new terrain made me wish parts of my game engine were different.

Back when I made console games, as soon as a game shipped, the code would be branched and the next project was lined up and work started immediately. Sometimes this happened even before the previous product shipped. This required just adding new code on top of the existing game engine to support the features of the new project. After several games, this starts to add up, and makes making changes fairly difficult. Occasionally entire systems would be rewritten, but it was rare.

So while I knew there wasn’t a whole lot of change to what my game engine could do after being refactored, I decided to clean up the things that were implemented quickly or not as well as they could have been so that future projects aren’t impeded by code pileup.

I felt two ways about this. On one side, I was longing to be prototyping a new game, testing new ideas. I’ve been thinking about new game ideas since early on in Banished development. On the other side I knew that my changes would make creating prototypes easier. If I happened to keep the prototype (which might happen), I wouldn’t have to refactor that code or the data that went with it, if and when I decide to change the way things work.

The Changes

The first refactor I did was to merge the Windows OpenGL code with the OSX/Linux OpenGL code. They were almost identical, but the Windows version was different because it’s in a dynamically loadable library so that multiple rendering back ends can be supported. I may end up with dynamic libraries on OSX and Linux as well, as Metal and Vulcan may be implemented one day in addition to OpenGL. Either way, having just one OpenGL path to maintain is much better.

I then removed support for Shader Model 2.0 under DirectX. I can’t do instancing in a memory efficient way with SM2, and I can’t do efficient deferred rendering effects with it either, so I’m not going to support it. This removed a lot of code that was used to make instancing work and really cleaned up the rendering path across all back ends. So the next game will at minimum require Shader Model 3.0, but I may even drop this. Time and GPU surveys will decide.

Often times I want to test some new code outside of the current game project, which is quite hard once the game is in place. Where do you test? I would end up commenting out the startup code of the game and calling some other temporary code. So instead I made the ‘game’ code a dynamically loadable library. With this setup I can maintain multiple test projects and the test code can stay around and doesn’t interfere with the game. A command line switch specifies which game library to load and that’s it. This change also allowed me to have multiple data directories so I can keep test data separate from the main game.

I also removed all serialization versioning code. Between the beta release of Banished and now, a bunch of data formats have changed, but the game still has to support the old versions. This is really just code bloat, and a new game doesn’t have to load Banished resources, so it’s pretty nice to clean them all up and have very readable serialization functions.

I changed the object model that the game uses as well. Previously I had resources fully constant – the game wouldn’t let you change them once loaded from disc. So if you have an object, say a character, it has one C++ class for the runtime data (where it is, what animation is playing), and a second class for the data that comes from disc (what model to use, what textures, etc). Now I can have the two classes be one in the same (or separate if the shared data is large). This cuts down on code classes and maintenance and it also allows flexibility. So now the constant data object is cloned when I need a runtime version, and can be changed as needed. If I need to change models or textures at runtime, it’s easy, instead of adding more overhead to both the data class and runtime class.

Next I changed the way I store meshes and load them from FBX files. In Banished, every model I exported from 3DSMax had the same format – position, normal, and 3 texture coordinates. (with skinning data if it was animated). This was a little wasteful, as only trees used all 3 texture coordinates. Some models only used 1. To fix this, I changed mesh formats to pair models with the vertex format that the shader programs use. This allows a custom format per model if desired, saves memory since only what’s used is stored, and allows internal verification that materials applied to a model will actually be able to render properly.

And then a ton of other small changes. Moving files around, renaming classes, removing hard text strings and making them configurable, making the engine more generic, moving game level code that can be reused down into the engine, the list of small things goes on and on.

All this should make the development of the next game smoother. Coding these changes feels a little bit like spinning my wheels, or going down a rabbit hole that doesn’t have a bottom. But when the change is made, it’s nice to see the code get smaller and have better organization.

Ok, it was time to start prototyping!

Steam Dev Days

But then I went to Seattle for Valve’s second developer conference. I met a lot of other awesome indie developers, talked to old friends, made new friends, and saw the new Valve VR controllers. I had a great time. The conference was fairly VR centric, but was also about where Steam is heading in the future.


Talking to other programmers

Back from the conference, there I was, done with refactoring, done with getting the engine ready for a new game! I was writing gameplay code! New camera controls and behavior! A new method for placing objects and paths on terrain without a grid! Hooray!

Then one day I had lunch with a programmer that I used to work with. Talking with other programmers, especially when working solo, is really good to go over ideas and design. And sometimes it spawns ideas that you wouldn’t have thought of yourself.

Somewhere in the conversation, my friend said “In my game engine, I’ve been thinking about having the hierarchy of objects be its own system.” That thought stuck with me, and my friend and I talked about it over the next few days.

My code didn’t do this, and against my better judgement, I decided to take a stab at implementing it and see if I liked the change.

And it really cleaned up my game engine. Positional and rotation control, animation, and attachment is its own system, and the graphics, collision, and audio systems simply reference transformations from the hierarchy system. It also really cleaned up the other systems that used to store the scene hierarchy in duplicate. It made rendering just about rendering, animation about animation, and collision just about collision. It also removed a lot of high level management that was used to keep things in sync that probably shouldn’t have been at game level anyway.

It’s a good change. However making this change was pretty large. Several hundred source files were affected, and I don’t think I could have done it with a game in place due to the number of things that would have broken. Making the change felt a bit like being in a pit of despair until it was done. I just kept getting deeper and deeper into changes to the lowest levels of the rendering, collision, and audio code. Several times I considered just reverting all my changes because it was such a large modification.

So while it’s a good change, it made me decide I’m done with major tech changes and code refactors until I really have a need for them. I don’t want to just make my code better and better forever. I need to be adding features I need or working on game code.

Things not done

One interesting thing I found while doing all this refactoring are the things I didn’t do. I had maintained a list of things I wanted to change about the code from before Banished shipped, but were too massive to do without majorly breaking the game.

Once I got to working on them, I realized about half of them didn’t really need to be done, or that they were implemented a certain way to handle edge cases I had forgotten about. Or the code wasn’t actually bad enough to warrant modification.

Not that cutting half my planned refactors saved any time – as refactoring progressed, I ran into things that needed to be changed or were implemented poorly that I hadn’t thought of ahead of time.

Back to Banished!

Most recently I’ve been back to working on a new prototype, but I took some time out to work on an update for Banished that will only effects mods. Aside from a few bug fixes, the base game won’t change but the game will allow modders to use configurable resource flags and limits, and hopefully remove the limit on memory allocations. More on this to come, but for now, here’s a screenshot.



Passing values…

A few months ago I had some interesting performance problems with OpenGL on OSX. I identified the problem and made some work arounds for development to continue. This week I’ve properly fixed the issue, and I want to record it here for myself and others to avoid this mistake.

So here’s a scene, rendering on OSX, at an abysmal frame rate of 14 on a MacBook Pro. That’s right. 14. I’ve got the game paused so there isn’t any time spent on updates, this is just drawing.


If I move the camera to a different location, the frame rate is 126. Thats a difference of 63 or so milliseconds. Ouch.


So after much debugging I determined that rendering animated models was causing the slow down. The image of just trees doesn’t have any deer or people moving around. And if I remove the people from my original test scene, the frame rate is over 100.


Since rendering houses and trees really only has minor differences with animated models I disabled the shader code that animates the models and the frame rate went back up to normal. This looks funny, and runs fast.


So here’s the basic code that handles animation in GLSL. It looks pretty standard and is simple code. This isn’t the entire shader, just enough to get an idea of how the animation part works.

struct BoneConstants
    mat4x4 transforms[64];

uniform BoneConstants bc;

in vec3 inputPosition;
in vec4 inputWeight;
in ivec4 inputIndex;

vec3 SkinPosition(vec3 position, ivec4 index, vec4 weight, BoneConstants bones)
		((bones.transforms[index.x] * vec4(position, 1.0)) * weight.x + 
		 (bones.transforms[index.y] * vec4(position, 1.0)) * weight.y + 
		 (bones.transforms[index.z] * vec4(position, 1.0)) * weight.z + 
		 (bones.transforms[index.w] * vec4(position, 1.0)) * weight.w)).xyz;

void main()
    vec3 position = SkinPosition(inputPosition, inputIndex, inputIndex, bc);
    gl_Position = (gc.worldToProjection * (tc.transform * vec4(position, 1.0)));

What this code does is transform the position of a vertex by up to four bones in the models structure. It then weights them by how much influence each bone has on the vertex.

I stared at this code for a while (more than a while actually), and after messing about a bit, it finally dawned on me what’s wrong with it. Face Palm.

To fix it, instead of calling a function to animate the models, I manually inlined the code. And my frame rate returned to normal, with animated characters.

void main()
   vec4  position = 
   		((bc.transforms[inputIndex.x] * vec4(inputPosition, 1.0)) * inputWeight.x + 
		 (bc.transforms[inputIndex.y] * vec4(inputPosition, 1.0)) * inputWeight.y + 
		 (bc.transforms[inputIndex.z] * vec4(inputPosition, 1.0)) * inputWeight.z + 
		 (bc.transforms[inputIndex.w] * vec4(inputPosition, 1.0)) * inputWeight.w)).xyz;
    gl_Position = (gc.worldToProjection * (tc.transforms[gl_InstanceID] * vec4(position, 1.0)));

Wow. So whats going on there?

There’s two ways to pass parameters to a function. Either by value, or by reference.

When you pass a parameter by value, a copy of the variable is made so that any changes to the variable in the function don’t effect its value in the calling function.

When you pass a parameter by reference any modifications to the variable change it directly. No copy is made.

In my case with animation, the entire array of bone transformations is being copied, because it’s being passed by value. My suspicion is that the program running on the GPU doesn’t have enough registers to make this copy, so the GLSL compiler is generating code – copying the array bit by bit, and then is running the code over and over to evaluate the final result. What’s just a few matrix multiples, scaling, and adding becomes many many copies and conditionals. This possibly results in different execution paths per GPU thread, causing even more slowdown.

My first attempt before manually inlining this code was actually to pass the array by reference, but the OpenGL compiler yelled at me that you can’t pass a uniform by reference.

On Windows and Linux, I suspect the compiler is smart enough to see that the function doesn’t modify the array, and optimizes the copy away. (Or my GTX 980 and 290X are just too fast for me to notice the slowdown…)

Most people directly reference the global list of uniform bone transformations directly and never run into this issue. But since my custom shader language that generates GLSL doesn’t have a concept of globals, everything is passed to functions if it’s needed. Arghghghg.

So what’s the real fix?

I don’t want to have to manually repeat code in shaders, that’s just bad programming practice. Luckily, I control the compiler for my own shading language, so I can get it to generate different code.

So I just recently added an ‘inline’ keyword for functions. The code gets inlined automatically and any value passed by reference isn’t copied when the GLSL is generated.

Previously my skinning function looked (in SRSL, not GLSL) like this:

inline float3 SkinPosition(float3 position, int4 index, float4 weight, BoneConstants bc) {...}

And now it looks like this

inline float3 SkinPosition(float3 position, inout int4 index, inout float4 weight, 
	inout BoneConstants bc) {...}

No more repeated skinning code everywhere.

Getting my compiler to inline the code is pretty easy. However, as most shader languages don’t feature a goto or label statement to jump over remaining code, it’s hard (if not impossible) to inline a certain class of functions. So my inline feature doesn’t handle inlining when returning from complex flow control. This really isn’t an issue for shaders, as the programs tend to be straight forward and not have many loops or conditionals.

So long story short, don’t pass uniform arrays and large structs to a function by value in GLSL.


Quick Fixes

I just uploaded some quick bug fixes introduced with the last build. 1.0.6 is live on Steam. If you need to redownload it from Humble you can log in and grab it, or you can use this tool: https://www.humblebundle.com/resender. GOG.com should have an updated build shortly.

There’s a new modkit, available here: BanishedKit_1.0.6.160521.zip, though there shouldn’t be any changes to it from 1.0.5.

Changes in this build:

  • – Fixed a crash that occurred when clicking on the town hall if a translation mod was in use that was built with 1.0.4. Missing text data will now be blank.
  • – Fixed a bug that caused orchards and pastures to not drop items inside their boundaries as was intended.