Too many executables…

This contains a lot of technical programmer nonsense. TL;DR? For some reason I had 4 game executables and 2 sets of data. After some refactoring I now have 1 of each.

A while back I did some testing on DirectX 11 vs Directx 9. The takeaway there was that DirectX 11 performed much better on newer systems and it was worth having two versions of the game – though at this point I’m curious how many DirectX 9 only users are out there…

The bad part of this, is that the way I originally wrote the code was to have a fully different version of the game. I treated it as a separate platform – as if I was implementing a port to another OS. This made sense when I did it. It got the code ready for supporting multiple platforms, but for a distribution across windows PCs, it’s not ideal.

There was a DirectX 9 executable, and a DirectX 11 executable. Both versions have their own copies of all the game data as well. All the data is precompiled for fast loading, and DirectX 9 uses some repeated data and index buffer tricks to implement fast instancing. DirectX 11 doesn’t need this. The vertex and pixel shaders are also different between shader model 2 and shader model 4.

To compound the multiple exe problem, throughout development I’ve also compiled 32-bit and 64-bit versions of the game. I think this promotes good programming practices, sets up for easier porting to other platforms, and makes my game engine easily reusable in the situation where it actually uses more than 2G or 3G of RAM.

So that’s 4 exes total:

  • Banished-dx9-x32.exe
  • Banished-dx9-x64.exe
  • Banished-dx11-x32.exe
  • Banished-dx11-x64.exe

While a lot of people that play this game would be tech savvy enough to click on the best choice for their system, I felt like there are a lot of users that just want to click on the game icon and get the best experience automatically for their system. The solution? I can think of two. One is to have a simple 32-bit ‘launcher’ program that determines what the system can handle, and start the right version of the game. The other is to put the rendering interface into DLLs and load the best one.

I didn’t really like the first solution, because while it would be fast to implement, I’d still have two copies of the game data. But I was dreading doing the second. Dreading it like I know it’s going to waste two weeks of time, and introduce 50 annoying bugs. It’s something that needs to be done though, so I started some experiments to see how bad it would be to implement. (I had scheduled 1 week to deal with this issue….)

Luckily it wasn’t so bad and I finished it over a weekend.

Already having separated the rendering code to support DirectX 9 and DirectX 11, it was just a matter of code reorganization – taking the respective functionality out of the engine code, putting them in two different DLLs, and dealing with converting a few static libraries into shared DLLs. Then I wrote an common interface that calls the right functions depending on which renderer is active. At video initialization time, the engine can see which DLLs it can load – if DirectX 11 can’t load then only DirectX 9 is supported. If both can load, then the user can pick which rendering interface to use, but it defaults to DirectX 11.

Sharing the data was also easyish. Most all the game data is the identical, and what’s different just stores both copies of the data in the different formats. The renderer determines which set of data to load. I could have just stored the raw data and converted on load, but I hate to increase loading time.

And since I really just moved code around, there won’t be many bugs. None reported yet, anyway.

On a happy note, since I had already implemented a way to restart the graphics system at runtime, I could use that same code to switch rendering APIs during the game. Anyone that encounters bad stalls and hardware issues with the DirectX 11 renderer can at anytime switch to DirectX 9. You don’t even have to restart the game to switch – just head to the options menu, pick DirectX 9, wait 2 seconds, and the game continues. This is very non-useful for most people, but it made me smile that it worked without too much effort – and it gives me a really easy way to test performance between different graphics rendering interfaces.

Because the rendering calls are really separated now, it should now be easy to implement an OpenGL renderer and test performance of it versus DirectX 9 and DirectX 11. And with the newly announced AMD Mantle, I have a perfect test bed for trying it out and seeing what performance gains it could give. Wait – no no no, I’m done with feature creep at the moment. Maybe later.

So back to my original intention – this work brought me back to two executables:

  • Banished-x32.exe
  • Banished-x64.exe

The obvious choice here is to just ship Banished-x32.exe. The worst I’ve seen the game take memorywise is 320M of ram using DirectX 9. DirectX 11 takes significantly less. Clearly I’m not even close to the limit of a 32-bit memory space. And while sometimes it appears that the 64-bit version runs at something like 0.25% faster than the 32-bit version, I can’t really validate it since it’s not always true.

So in the end, one executable to deal ship with instead of 4.

Now back to working on the in-game tutorial!

41 Comments

    Dara
    September 27, 2013 7:54 am

    Regarding your musing about Directx 9 users, here is Steams stats on that
    http://store.steampowered.com/hwsurvey/videocard/

    However, if you have already done the bulk of the work….

    RMuldoun
    September 27, 2013 8:01 am

    So with all this information I take it we are closing in on the beta stages? Have you decided on a preorder price yet friend?

    Hammer
    September 27, 2013 8:04 am

    I would include the 64x exe just for the hell of it – There’s probably as many people who would enjoy, or prefer, having the 64x exe, that there are people who use directx 9

    SnappedBark
    September 27, 2013 8:07 am

    The more I read the more we cannot wait for the game to come out. Just for interest your game has reached the South African shores. Me being one of the people here that cannot wait for it. Just to see how you code and your mind set on creating this game is mind boggling.

    appel
    September 27, 2013 8:08 am

    I love reading about your development progress and all the things you find on its road. It’s something i have never seen before and it gives such a personal touch to this game that even if this game never sees the light of day, i would still come here and read all about it.

    alex
    September 27, 2013 8:09 am

    Agree with hammer, a x64 exe would be very nice. We are soon 10 years past windows pro 64 bit edition, yet there are very few titles in the 64 bit space. Believe the next “x” game will be the first exclusively 64bit game out there.

    Jeff
    September 27, 2013 8:16 am

    Every post I read I feel this game is getting so much closer to my SSD!! Really looking forward to playing this title.

    I am sure it has been suggested before but can you please consider a multiplayer element in a future expansion. Not this release as there has been enough feature creep!!

    Shane
    September 27, 2013 8:31 am

    Even if you don’t ship with it, please make a 64 bit executable available to download. I’ve got a 64 bit operating system and I want to run 64 bit executables, regardless of how little benefit it may give. I think it’d actually help to sell your game to say it had native 64 bit compatibility as 32 bit just sounds old to anyone with a 64 bit OS.

    Andy D
    September 27, 2013 8:32 am

    In game tutorial sounds like you’re getting close man! Well done! Lets hope things are going your way and ours.

    Burt
    September 27, 2013 8:34 am

    Cheers, can’t wait.

    sleepSoon
    September 27, 2013 8:35 am

    Less than 320mb of ran during full simulation? That’s some impressively efficient code! While most RTS devs are finally getting air to breath with 64 bit memory allocation you come in and bust out those stats. I don’t know if others are interested but I would love to read about how the game logic keeps track of so many objects and stays under 320mb.

    My own experiments got 262144 declared, initialised, but ultimately static objects clocking in at around 60mb. But they were basic objects with only 5 of so variables (mostly byte and short types). And that was without any gamelogic running. But youre running a world, a town, and it’s populous almost down to a micro level.

    Do a blog on memory allocation, teach me to be better!

    Out of interest how large are the saved game files? I assume they are a bitstream?

    Celt
    September 27, 2013 9:00 am

    Informative and interesting post as usual. Thanks!

    gongo
    September 27, 2013 9:03 am

    Can you give an indicative date of release?
    someone can explain me the differnce between direti x 11 and 9, and 32 and 64 bit?
    thx, I’m a little bit ignorant in IT

    Eurofighter1200
    September 27, 2013 9:04 am

    I thing a 64 bit exe would be nice because in maybe 5 or 10 years nobody can use a 32 bit exe i have now sometimes problems with x32 programms in the future it would be harder i think when i read the blog i think i would choose the x64 Dx 11 version of your game. Many Games provide x 32 and x 64 in there store so you can chose what you want or download i think that would be also great here

    Fouxdoux
    September 27, 2013 9:09 am

    So are we getting close to release? Can’t wait to play it!

    SilverWarior
    September 27, 2013 9:13 am

    As a programmer to another programmer: Keep both 32 bit and 64 bit versions of game EXE.
    Why?

    1. As you alrady mentioned most of 64 bit code achieves better performance on 64 bit CPU than 32 bit code does. Of course this performance gain laregly depends on the COU itself.
    Some CPU’s still support native 32 bit instruction sets. On such CPU’s there won’t be much performance difference between 32 bit and 64 bit code.
    But some CPU’s no longer have native 32 bit instruction set support but instead map 32 bit instruction sets to native 64 bit instruction sets, do all cacuzlation usng 64 bit insruction sets and then finally format the end result to 32 bit instruction set. 32 bit code will perform a bit slower on such CPU’s due to that 64 bit to 32 bit conversion of resulted data.

    2. I read somewhere on the internet that Microsoft is planning on scrapping 32 bit support from their Operating System soon. They sa that Windows 10 will no longer have 32 bit support.
    But lately things about this become a bit quiet. P resume that this is due to Microsoft targetting mobile platforms where most of them still use 32 bit processing.
    Anywhay if Microsoft does decides to scrap 32 bit support for any of their OS it means that users of that particular OS won’t be able to play your game.
    So if you wanna make your game so that gamers will be playing it after about 5 years or so from now including 64 bit support is almost a muss.

    3. By leaving 64 bit support you also alow yourself ability to add more memory demanded features in the future (larger maps, more detailed simulation systems, etc.).
    And if you ever intent to make you game modable 64 bit support will be a muss as moders will probably quite quickly made a ton of new content for your game which will result in increased memory requirements.

    4. This SHOULD NOT be an exuse for leving 64 bit support!
    But incase if your game has a bug which is causing a memory leak it will take longer for 64 bit version to come to a crash than 32 bit one.
    And since nowadays most computers already have 64 bit CPU’s it means that such bug will be less of a nuisance for most of your gamers.

    Other than tha great progress on your game.
    As a game developer myself I’m constantly hearing other game developer telling me that it is imposible for a single game developer to make great game. Of course I don’t belive them a word about thet.
    And you are the perfect example of this.
    So when your game comes out I will spread the word about it on any gaming forum/website I will visit.
    And yeas I’m already spreding the word about it already.

    Oscar
    September 27, 2013 9:15 am

    Quick note, can you make it so that it detects the OS version and Direct X version with a BAT file with a script to querry for the dxdiag and from there get the version of Direct X???

    Hope this helps, keep up the good work!

    Fearlessjay
    September 27, 2013 9:17 am

    One good reason to use 64 bit as well is if you intend on letting users use mods then you want there to be as much memory available otherwise there is some arbitrary limit that everyone hates, Also if you continue to release updates and more content then your game may start to take more that what a 32 bit could handle.

    Andy
    September 27, 2013 9:18 am

    Take my money

    MorningSun
    September 27, 2013 9:28 am

    I’d say put into DLLs and load the best one, but I think there might be some problems with some notebook with Optimus. Maybe not. So 32-bit DX11 is the way to go for me. Anyway, good luck!

    Tokanya
    September 27, 2013 9:36 am

    I see you have update a lot more often.

    There’s an another game called Starbound (http://playstarbound.com) That do things really similar to you. They have daily update, explain everything in depth and their standard of work is insanely high, the same goes to you. also they plan to release Beta at the end of this year too.

    My point is if you check on the pre-order page they have gain almost 2 millions dollar. I pre-order it in the first week and it was quite a long time ago. while the game it self not even half done.

    This is just my friendly warning. I feel that you should considered marketing side more seriously at this point.

    Let’s say none of my friends ever heard of this game until I gave them the url. The only reason I was here in the first place because I looking for city builder game in Wikipedia. That’s how bad your marketing scheme at this point. So I think consider a pre-order, kick starter or contact any distributor soon. It’s better safe than sorry.

    I don’t know what’s stop you from doing so, I have paid and play for a lot of indie games. Most of the ‘Early Access’ or ‘Beta’ mostly on Steam or Desura, I feel that they weren’t even half worth their prizes ($10, $15, $20, $50) With your work quality now it should be more than fine for a finish product with some feature add in later (after Beta that is)

    I don’t want this really great game to go under.

    Trondkla
    September 27, 2013 9:43 am

    Awesome post! Nice to see the problems and the decisions taken :)

    Brandon
    September 27, 2013 9:53 am

    SilverWarior has said covered anything I would have said (which would have been very little in comparison).

    So I will go on with this; I really appreciate all the great work and letting us in on your thoughts and progress. It feels more personal, as I watch something grow to fruition.

    Keep up the amazing work; I can’t wait to waste hours of my free-time (I know just by looking at what you have done that the hours will disappear).

    shannonphoenix
    September 27, 2013 10:09 am

    Me stupid. Me no understand this programmer-speak.

    All me hear is, you use best performance for older graphics cards.

    Me dinosaur, me use old graphics card.

    Me approves!

    LOL, Okay, that was terribad. Anyway, thank you for thinking of those of us whose cards aren’t exactly top-of-the-line. While I didn’t really understand most of the programming-speak, I did understand enough to know that I’ll be able to play it.

    I am on Win8 with DX11… but I still appreciate knowing that (unlike some games), the most tired and old part of my system won’t be my undoing with regards to the game I am most looking forward to.

    Thank you, and keep up the great work!

    Lukio
    September 27, 2013 10:20 am

    Why not both? Is it too much trouble to maintain two executables? Have a look at: http://store.steampowered.com/hwsurvey

    TaRan
    September 27, 2013 10:29 am

    While your reasoning is sound, I would still like to see support for a 64-bit version of the game. SilverWarior had some good points as to why to keep it as well.

    vaendryl
    September 27, 2013 10:37 am

    I’m really starting to feel the pain of going with a DX10 card when I did. could’ve had dx11 if I waited only a matter of months x_x

    doesn’t look like I’ll have funds for a new card anytime soon either.

    oh well, at least you still have dx9 support. more and more games have gone dx11 only recently.

    sasukeskapa
    September 27, 2013 10:54 am

    Very interesting post. And I also think that you should ship the 64 bit version too. My reason is that if I like a game I keep it forever and play it after even 10 years too and with a 32 bit version it could be a pain in the *ss to make it run 10 or more years later if my actual PCs will have defects.
    Also SilverWarior had some really convincing arguments too :)

    DrogenShomuro
    September 27, 2013 12:18 pm

    I think it’s funny that the PC version of Saints Row: The Third makes you choose between DX 10 or 11 at startup. Meanwhile, here you are as a solo game developer doing it properly…

    Alin
    September 27, 2013 12:20 pm

    I would like to see the 64 bit version also. I still have a single core athlon from 10 years ago and that is 64 bit…

    Lkas
    September 27, 2013 12:45 pm

    Please also include the 64 bit version; you are doing an excellent job, looking forward to playing the game

    Tom Duhamel
    September 27, 2013 7:34 pm

    @SilverWarior, from one programmer to another programmer, that is from me to you :)

    Please keep in mind that the game primarily targets Windows, and thus the x86 / x86-64 processors. We are not talking any other processor.

    “As you alrady mentioned most of 64 bit code achieves better performance on 64 bit CPU than 32 bit code does.”
    This is incorrect. Very few instructions gain any advantage in 64-bit, as compared to 32-bit, and those that do will gain a very slight advantage. 64-bit offers advantages in other places, just not this one. An improvement of only .25% does not surprise me.

    “But some CPU’s no longer have native 32 bit instruction set support but instead map 32 bit instruction sets to native 64 bit instruction sets…”
    There is no such thing as a “native 32 bit instruction set” or a “native 64 bit instruction set”. x86-64 is not a distinct architecture as is, it is an extension of the existing x86. Though it adds a few new instructions, there is one single instruction set that covers 64-bit, 32-bit and 16-bit “modes”. The instruction set is forward and backward compatible in these three “modes”. The x86-64 protocol requires these three “modes” and is unlikely to drop any of these in the foreseeable future. No 64-bit processor in existence today dropped 32-bit or needs to map anything.
    “I read somewhere on the internet that Microsoft is planning on scrapping 32 bit support from their Operating System soon.”
    This is only partially true. It is true that Microsoft stopped support for 32-bit-only processors with Windows 2008 R2. That is, that version and any subsequent versions of the server edition of Windows cannot run on a non x86-64 processor. However, 32-bit programs still work just fine on it. As for desktop edition, Win8 still has a 32-bit version.

    As of this day, all 32-bit editions of Windows still support 16-bit programs (though I concede that even the word “support” is a bit too big in this case). They only dropped it from the 64-bit editions because the 16-bit subsystem requires virtual mode, which is no more available on x86-64. (Keeping it would have required a major rewrite, which would obviously have been a waste of time.) Based on that, I can’t imagine Microsoft dropping the 32-bit subsystem any time soon, as it is and will remain compatible with the 64-bit processor. I couldn’t find any hint from Microsoft regarding this, neither for or against this.

    “Microsoft targetting mobile platforms where most of them still use 32 bit processing”
    Mobile platforms use a totally different architecture (namely ARM). 32-bit on ARM and 32-bit on x86 are totally unrelated. The requirement to keep a 32-bit edition for ARM has absolutely no effect on whether or not they must keep it for x86.

    “if Microsoft does decides to scrap 32 bit support for any of their OS it means that users of that particular OS won’t be able to play your game”
    That, I must agree. It is a valid argument, though I think is unlikely to occur soon.
    Your third point, about more memory being available for modders, is totally valid. However, given that he mentions the memory footprint of the game being 320 MB, it seems unreasonable to me that a modder could need more than 2 extra GB (though obviously not impossible).

    You disqualify your 4th point by yourself :) He mentioned in a post that he sometimes spent and entire 8-hour shift playing the game. I assume he would have discovered it by now, if there was any major memory leak. Small memory leaks will not crash the game. But it’s not impossible, indeed.

    All of this said, and without any argument in our favor… If given the choice, I would too pick the 64-bit version. If he decides to only release one version, I will not complain because I understand there isn’t much of a difference. For now.

    “it is imposible for a single game developer to make great game. Of course I don’t belive them a word about thet.”
    I must disagree again. It is indeed impossible for a single developer to make a good game. This guy is a genius. He has accomplished a miracle.

    C Wolf
    September 27, 2013 11:08 pm

    Ah, Tom. That was a great read! You’re dead on. And you’re right. It is a miracle.

    Drusek
    September 28, 2013 4:38 am

    Hi, you could ship it like
    Banished.exe as default
    and Banished-x64.exe for those that want to start that one

    mac
    September 28, 2013 6:38 am

    Hi,

    A Windows installer for example like one made with NSIS can easily decide by himself which executable version is the correct one to install. NSIS allows to check if it runs on a 32 or 64 bit OS and also what version of DirectX is installed and therefore can install automatically the best version for the target system.

    I have a lot of experience specifically with NSIS. I created multi language installers for several mod projects, the most complicated beeing the one for xUI for NWN 2. So if you need some help I would be glad to offer you my service. :)

    Have Fun

    SilverWarior
    September 28, 2013 7:14 am

    @Tom Duhamel
    “I must disagree again. It is indeed impossible for a single developer to make a good game. This guy is a genius. He has accomplished a miracle.”

    It IS posible for single developer to make great game.
    Wanna example? Then check out next game:
    http://www.desura.com/games/projekt-w-phase-2

    Daniel North
    September 28, 2013 2:48 pm

    I Am starting to think you have reached some sort of godlike coding level, while you are at it, please work all known forms of cancer into a 3D protein model would you good chap! Shouldn’t take only a few months!

    Daniel North
    September 28, 2013 2:51 pm

    This was your great x32 grandfather in 510 BC, http://www.youtube.com/watch?v=rQUCX4LEufk

    Zulnex
    September 28, 2013 9:52 pm

    Keep up the splendid work! Looking forward to the release very much.

    Dan
    October 1, 2013 12:54 am

    Just do the alpha pre-order deal for 20 or 30 AND TAKE MY MONEY ALREADY!

    Chris
    October 22, 2013 3:25 am

    Not a programmer, but I’m an app packager. It would be simple enough to configure a MSI to just install the 64bit executable based on OS type at install. Hell it would be easy enough to give the installing user a choice via the feature dialog.

    Ensuring the executables are in different features with the feature condition set to VersionNT64 (or NOT) in the feature table would do it simply and invisibly to the user.

    There’s a few freeware msi compilers out there (like Wix Toolset), just attempt to avoid anything that attempts to make use of custom tables and dlls for common tasks(it can make maintenance/patching a hassle). As for actual editing cannot recommend InstedIt enough (though you would want a commercial license).