Mega Update

Way back in June, I was working on getting the game code running on Mac and Linux. And while the game runs on those platforms, there are still a bunch of details to implement for a final product, plus testing. But I’ve been itching to work on new games, so I’ve been working on new things more than working on Mac and Linux versions of Banished.

Here’s a rundown of the past few months.

Terrain Rendering

A lot of the new games I have in mind require a larger terrain than is in Banished, and unlike Banished, they would require a free camera that can look up to the horizon. So I went ahead and coded a new terrain system.

I started with a paper called CDLOD (Continuous Distance-Dependant Level of Detail Rendering) for height maps. My implementation started out true to the paper, but now I use a slightly different tessellation of the terrain with alternating diagonals, rather than all diagonals in the same direction. So my LOD is slightly different. And at the moment I still wanted to support hardware that doesn’t support vertex texture fetch (or does but it’s slow), so I cache chunks of the terrain on the CPU instead of reading the height data with the GPU. Having the vertex data available on the CPU is required anyway for collision detection.

This gives me a terrain that can render all the way to the horizon with no real size limits and nice LODs that fade gently with distance. A 128km x 128km terrain is easily rendered on low end hardware, and the level of detail can be tweaked to reduce triangle count, or increase detail depending on hardware. Of course a terrain that big has lots of procedural detail, since storage would be limiting.

Here’s a view of the underlying triangles that would render when flying far above the terrain. I’m not trying to win any art awards here, this terrain and texture is purely programmer art for testing.

terrainwire

Beyond just getting the triangles to render, I also need to add visual detail that fills in the gaps between triangles with something that looks like much higher detail. So I added a system that can handle unique texturing across the entire terrain and use higher frequency data than the triangles used to render.

With some quickly generated normal maps, the view above turns into this.

terrainsolid

The terrain also does simple occlusion culling so that large mountains block whats behind them and it won’t render things far away.

I still have a bunch of work to do to turn this into something usable in a game, such as good color textures and terrain generation that supports gameplay instead of simple fractal noise.

Cross platform toolset

Since I had terrain working, I wanted to make sure it worked on all Linux and Mac as well as Windows, since I’d prefer to be able to launch my next game cross platform all at once instead of doing it after the fact.

But I had a bad workflow. On Windows, I can edit resource files and the game reloads them on the fly, or the resources compile when needed as they get loaded from disc.

On Mac and Linux, I only ported the game code, and none of the toolset. So the workflow goes:

  • – Make sure PC build works.
  • – Compile all resource and generate a pack file.
  • – Put the pack file somewhere accessible by the Mac and Linux machines.
  • – Make sure Mac and Linux compile.
  • – Run Mac or Linux build.

This is fine until I need to change data to debug a problem. If I’m working under Linux and want to do something simple like view the terrain in wireframe mode, this requires a data change (yes, all my render state is data), which requires a pack file recompile on Windows, etc, etc. Additionally when I travel, I prefer a light and small laptop, such as my MacBook, but if I can’t edit data on it, then there’s a limit to the kind of things I can work on.

And so I was motivated to make the entire toolset work cross platform. Yikes. All that Windows specific code had to go!! So I ended up having to rework the engine by using cross platform code for fonts, image loading, dxt block compression, audio decoding, and a few other things. Thankfully the FBX format works on OSX and Linux, so I didn’t have to change model formats.

A bunch of changes later and the OSX/Linux version can now build the project from raw resources and no longer requires a Windows machine to build the data. Woot. I wish it had actually been as simple as writing that sentence…

The only downside to this is that Linux/OSX can’t compile DirectX shaders into binary form. It generates the text though. So I can’t use data created on Linux directly on a Windows machine at the moment, but that’s only a problem building releasable products. For day to day work I can now work on any platform. However 3DSMax forces me to use Windows, but I don’t do artwork every day, especially during early development.

Major Refactoring

Next I decided that if I was going to start a new game, I was going to clean things up properly. Just implementing the new terrain made me wish parts of my game engine were different.

Back when I made console games, as soon as a game shipped, the code would be branched and the next project was lined up and work started immediately. Sometimes this happened even before the previous product shipped. This required just adding new code on top of the existing game engine to support the features of the new project. After several games, this starts to add up, and makes making changes fairly difficult. Occasionally entire systems would be rewritten, but it was rare.

So while I knew there wasn’t a whole lot of change to what my game engine could do after being refactored, I decided to clean up the things that were implemented quickly or not as well as they could have been so that future projects aren’t impeded by code pileup.

I felt two ways about this. On one side, I was longing to be prototyping a new game, testing new ideas. I’ve been thinking about new game ideas since early on in Banished development. On the other side I knew that my changes would make creating prototypes easier. If I happened to keep the prototype (which might happen), I wouldn’t have to refactor that code or the data that went with it, if and when I decide to change the way things work.

The Changes

The first refactor I did was to merge the Windows OpenGL code with the OSX/Linux OpenGL code. They were almost identical, but the Windows version was different because it’s in a dynamically loadable library so that multiple rendering back ends can be supported. I may end up with dynamic libraries on OSX and Linux as well, as Metal and Vulcan may be implemented one day in addition to OpenGL. Either way, having just one OpenGL path to maintain is much better.

I then removed support for Shader Model 2.0 under DirectX. I can’t do instancing in a memory efficient way with SM2, and I can’t do efficient deferred rendering effects with it either, so I’m not going to support it. This removed a lot of code that was used to make instancing work and really cleaned up the rendering path across all back ends. So the next game will at minimum require Shader Model 3.0, but I may even drop this. Time and GPU surveys will decide.

Often times I want to test some new code outside of the current game project, which is quite hard once the game is in place. Where do you test? I would end up commenting out the startup code of the game and calling some other temporary code. So instead I made the ‘game’ code a dynamically loadable library. With this setup I can maintain multiple test projects and the test code can stay around and doesn’t interfere with the game. A command line switch specifies which game library to load and that’s it. This change also allowed me to have multiple data directories so I can keep test data separate from the main game.

I also removed all serialization versioning code. Between the beta release of Banished and now, a bunch of data formats have changed, but the game still has to support the old versions. This is really just code bloat, and a new game doesn’t have to load Banished resources, so it’s pretty nice to clean them all up and have very readable serialization functions.

I changed the object model that the game uses as well. Previously I had resources fully constant – the game wouldn’t let you change them once loaded from disc. So if you have an object, say a character, it has one C++ class for the runtime data (where it is, what animation is playing), and a second class for the data that comes from disc (what model to use, what textures, etc). Now I can have the two classes be one in the same (or separate if the shared data is large). This cuts down on code classes and maintenance and it also allows flexibility. So now the constant data object is cloned when I need a runtime version, and can be changed as needed. If I need to change models or textures at runtime, it’s easy, instead of adding more overhead to both the data class and runtime class.

Next I changed the way I store meshes and load them from FBX files. In Banished, every model I exported from 3DSMax had the same format – position, normal, and 3 texture coordinates. (with skinning data if it was animated). This was a little wasteful, as only trees used all 3 texture coordinates. Some models only used 1. To fix this, I changed mesh formats to pair models with the vertex format that the shader programs use. This allows a custom format per model if desired, saves memory since only what’s used is stored, and allows internal verification that materials applied to a model will actually be able to render properly.

And then a ton of other small changes. Moving files around, renaming classes, removing hard text strings and making them configurable, making the engine more generic, moving game level code that can be reused down into the engine, the list of small things goes on and on.

All this should make the development of the next game smoother. Coding these changes feels a little bit like spinning my wheels, or going down a rabbit hole that doesn’t have a bottom. But when the change is made, it’s nice to see the code get smaller and have better organization.

Ok, it was time to start prototyping!

Steam Dev Days

But then I went to Seattle for Valve’s second developer conference. I met a lot of other awesome indie developers, talked to old friends, made new friends, and saw the new Valve VR controllers. I had a great time. The conference was fairly VR centric, but was also about where Steam is heading in the future.

devdays

Talking to other programmers

Back from the conference, there I was, done with refactoring, done with getting the engine ready for a new game! I was writing gameplay code! New camera controls and behavior! A new method for placing objects and paths on terrain without a grid! Hooray!

Then one day I had lunch with a programmer that I used to work with. Talking with other programmers, especially when working solo, is really good to go over ideas and design. And sometimes it spawns ideas that you wouldn’t have thought of yourself.

Somewhere in the conversation, my friend said “In my game engine, I’ve been thinking about having the hierarchy of objects be its own system.” That thought stuck with me, and my friend and I talked about it over the next few days.

My code didn’t do this, and against my better judgement, I decided to take a stab at implementing it and see if I liked the change.

And it really cleaned up my game engine. Positional and rotation control, animation, and attachment is its own system, and the graphics, collision, and audio systems simply reference transformations from the hierarchy system. It also really cleaned up the other systems that used to store the scene hierarchy in duplicate. It made rendering just about rendering, animation about animation, and collision just about collision. It also removed a lot of high level management that was used to keep things in sync that probably shouldn’t have been at game level anyway.

It’s a good change. However making this change was pretty large. Several hundred source files were affected, and I don’t think I could have done it with a game in place due to the number of things that would have broken. Making the change felt a bit like being in a pit of despair until it was done. I just kept getting deeper and deeper into changes to the lowest levels of the rendering, collision, and audio code. Several times I considered just reverting all my changes because it was such a large modification.

So while it’s a good change, it made me decide I’m done with major tech changes and code refactors until I really have a need for them. I don’t want to just make my code better and better forever. I need to be adding features I need or working on game code.

Things not done

One interesting thing I found while doing all this refactoring are the things I didn’t do. I had maintained a list of things I wanted to change about the code from before Banished shipped, but were too massive to do without majorly breaking the game.

Once I got to working on them, I realized about half of them didn’t really need to be done, or that they were implemented a certain way to handle edge cases I had forgotten about. Or the code wasn’t actually bad enough to warrant modification.

Not that cutting half my planned refactors saved any time – as refactoring progressed, I ran into things that needed to be changed or were implemented poorly that I hadn’t thought of ahead of time.

Back to Banished!

Most recently I’ve been back to working on a new prototype, but I took some time out to work on an update for Banished that will only effects mods. Aside from a few bug fixes, the base game won’t change but the game will allow modders to use configurable resource flags and limits, and hopefully remove the limit on memory allocations. More on this to come, but for now, here’s a screenshot.

limits

34 Comments

    Aaron
    February 6, 2017 9:06 pm

    Thanks for the update, and sounds like you’ve been crazy busy.

    But by the end of it, I couldn’t figure out whether there’s going to be much of a change to Banished or not? It’s been quite a while since I played it, but I still have a savegame I’m itching to continue but the lag has been killing me.

    Yr 212
    1411/426/435

    Please let me know if any of these optimisations are going to make it to Banished!

    Ben
    February 6, 2017 9:11 pm

    I always enjoy your posts, I have no clue when it comes to programming but reading your posts always gives me a deeper understanding of what goes on behind the scenes of a game. It’s great. Thanks for writing them!

    Scott
    February 6, 2017 9:22 pm

    I’m with Ben, I only know the very basics of programming, but your posts are so interesting to read. They make me really appreciate the amount of work that goes into creating a game!

    Loved Banished. Looking forward to whatever else you make in the future.

    Demortes
    February 6, 2017 9:35 pm

    The level of programmer pron you provide is amazing. I greatly appreciate your updates, Banished related or not. I’m hoping to accomplish something like Banished before I die, but first I think I’m going to start with Unreal Engine. πŸ™‚

    Jim
    February 6, 2017 9:42 pm

    In his comment Ben pretty much summed up my post. I’m not a programmer either but really enjoy and appreciate your updates and your conversational style is more like we were having lunch or a coffee. A few months ago I switched to Linux but still play on Windows occasionally. Thanks for keeping us informed and glad you are still motivated.

    Jason
    February 6, 2017 9:54 pm

    Although I’ve been trying to avoid game development because of the way the industry treats its devs, its awesome to see the thought process you go through to build something as big as Banished.

    Its also good to hear that you’ve refined your development process to reduce the overhead for making future games.

    Heather Clark
    February 7, 2017 12:27 am

    I think what we all really want to know is – have you any immediate plans to launch a new game?

    It’s great to hear from you and we eagerly await more news.

    Ron
    February 7, 2017 12:48 am

    These posts are always a nice read. I have been an avid follower of Banished but I don’t think I have thanked you for it. So thanks and more power to you and your endeavors!

    Klabautermann
    February 7, 2017 1:22 am

    If i have that knowledge of programming, i would spend the rest of my Life to make a Anno 1503 reloaded. Banished have something from Anno that never exists into that genre, and it would be great to implement it to an Anno.

    sometimes dreams come be true…and hope dies last πŸ™‚

    regard
    Charles

    Vibhu
    February 7, 2017 4:13 am

    Really nice to get a nice technical post after so long. Waiting for the next game.

    Marek
    February 7, 2017 4:45 am

    Very nice read, I always loved your blog posts and also Banished. Also it’s inspiring for me as I study programming and would like to make a game sometime in future. So thank you very much for Banished and your great posts here. Also can’t wait to see your next project!

    Geoff
    February 7, 2017 5:48 am

    Hooray on the resource flags and limits! This should really improve the functionality of a number of different mods.

    Paeng
    February 7, 2017 6:37 am

    As much as I’m looking forward to your next game – I’m absolutely thrilled by the fact you’ve spent some time to improve a few things in Banished for our modding community! Awesome, thank you so much πŸ™‚

    AC
    February 7, 2017 9:18 am

    Welcome back! Very interesting read. Can’t wait to finally have a native linux build.. thanks so much for putting so much effort into your cross platform ports.

    Ianmoone
    February 7, 2017 10:44 am

    Banished is a great game, I’m playing it since its premiere, and I have still hope that I will able to play it with my friends via multiplayer or even some server

    brads3
    February 7, 2017 11:56 am

    can the toolbars be reformatted ? set the toolbar to organize buildings by materials needed so when u click on houses it shows woodhouses,then stone+wood,then stone+wood+iron,etc.also is there a way to create slots so all mods from a modder go to his/her own slot?
    looking forward to the map and graphic control changes. wish for larger maps and varrying size lakes. will it be possable to have an ocean on 1 side? varrying climates,desert to south hills and pines to north,mixed trees and valleys in the middle??? also a better and easier way to mod animated animals.

    Salamanderrake
    February 7, 2017 2:05 pm

    One major changes you should make is switching to use SDL2 for handling input, to make things easier for you and others on linux/mac.

    gpdev
    February 7, 2017 3:46 pm

    Yay!
    Still waiting for macOS Banished release

    Jon
    February 8, 2017 2:23 am

    It’s very interesting to read all this, as it always is.

    And awesome to read about the resource flag modification! :O πŸ˜€

    I whish you happy times with developing a new game Luke! I loved reading all development blogs for Banished, will you do this too for your next project?

    Daniel Toebe
    February 8, 2017 7:27 am

    I currently have Banished from gog and play it regularly on wine. If you want Linux testers I’ll be more than happy too, also if needed I’ll buy it on steam as well if you want testing there too.

    John
    February 8, 2017 7:38 am

    Thanks! Been looking forward to these updates. Banished is a great game and I want to play it for the long haul, just like I do Age of Empires and Q3:Arena.

    banishme
    February 8, 2017 10:50 pm

    Awesome! I randomly stumbled upon some Banished videos the other day on YouTube. I have a MacBook Air, so wasn’t sure what to do. Broke down and bought Banished an hour ago and am running it very sluggishly using Parallels Desktop and Windows 10. Excited to hear there will be a native MacOS version coming someday!

    Tom
    February 9, 2017 4:06 pm

    Aww no Mac version yet. But it is nice to see you have been so busy :)!

    Good luck on your paper!

    HΓ₯kan
    February 10, 2017 4:43 pm

    If you need mac testers, I work on three different macs and would love to help! πŸ™‚ Good luck with everything, looking forward to your next game. Banished is one of my all time favorite games.

    Ashantin
    February 10, 2017 5:28 pm

    Thank you for the very interesting update. I have a Mac and really want to play Banished. Please keep working on getting your game to us.

    osx port
    February 11, 2017 3:38 am

    good stuff!

    No windows here and I pre-purchased the game over a year ago to support your mac porting efforts. No doubt R&D and prototyping is a lot more fun than the porting grind but don’t give up! Was just watching a Banished stream and I’m really hoping I’ll get to play it sometime this year.

    Cheers

    Torp
    February 11, 2017 5:44 am

    I bought the game because it was interesting, but while I do have a Windows install, I just can’t be bothered to reboot to it any more to play a game. Not when I have a huge backlog of games that do run on OS X.
    This means that I bought Banished but I only played it once, on some Saturday that I rebooted. Looking forward to actually finishing a campaign if you ever put out those Mac/Linux versions.
    Btw, my backlog is so huge now that I don’t buy Windows only games any more. I know I won’t get to playing them.

    Martijn
    February 11, 2017 9:34 am

    Please please please release the Mac port if you’ve got it working. It would be so great to be able to play this game on my Mac. It has been sitting in my Steam library for ages just waiting for the port to be released.

    Jon
    February 11, 2017 2:16 pm

    I’m still amazed that you’re still working on this Luke. I thought about donating some money to you for all the hard work, but how?

    And today I noticed the game is on sale on GOG (-75%), I bought the game on the release day through Humble Bundle so why not buy it again? πŸ˜€
    You can see my second payment for this game as a gift to buy a drink or two. πŸ˜€

    Arifin
    February 12, 2017 10:33 pm

    I’ve tried a lot of city building games, but Banished is the most enjoyable city building games I’ve ever played, really. The mechanics is more interesting than City Skylines, Simcity, or any other games. And especially with colonial charter mod and megamod that makes this games upgraded to the whole new level. I think you should develop this game more, maybe upgrade the game to have more better graphics and add more interesting idea. I just imagine if this game has graphics like City Skylines, this games would be a killer app.

    Flamekebab
    February 14, 2017 6:41 pm

    I’m in the same boat as gpdev, Tom, Ashantin, osx port, and Martijn. Bought the game ages ago when it looked like a Mac port was coming.

    Still waiting.

    Malban
    February 15, 2017 8:07 am

    Waiting for now – years – for MAC support, and all you say is the same as half a year ago:
    “yeah its working but unfinished – you don’t get it.”

    Why don’t you just be honest and cancle it – at least that way I don’t wait for it!

    sebapatr
    February 15, 2017 10:05 am

    I’m super glad that you take your time to write such amazing dev articles. These are really valuable informations for starting developers like me. Thank you.