The trials of using C++

I’ve been using C++ for a long time now. It’s been my programming language of choice since about 1998. Before that I used C and Assembly, and before that I used Pascal and BASIC. Even though I’ve been programming a long time and I’m reasonably familiar with the features of C++, sometimes it still causes me headaches. Here’s an example.

Maps in the game are randomly generated – but the generation is deterministic. This means that given the same starting conditions, such as a seed value, map size, and terrain type, the same terrain can be generated over and over. I use this functionality for the tutorial map. There’s no reason to store the terrain when it can be generated and the outcome is known.

However the other day I found that in only the 32-bit release build of the game, the terrain was coming out differently compared to the other compilations. The tutorial then wanted the player to place a building in the middle of a lake…

This is a hard thing to debug, as when switching back to any debug mode, or release 64-bit, the game worked correctly. I spent a few hours working on the worst kinda of debugging. This consists of logging the state of variables as they undergo thousands of state changes and looking for differences between the correct and incorrect versions. I ensured that the random number generator was always generating the same values, and then after several hours I found this line of code. It looks fairly innocent, but it contains is a non-obvious but critical mistake.

position = Vector3(rng.RandFloat(), rng.RandFloat(), 0.0f);

rng.RandFloat() just generates a random number between 0.0 and 1.0. So what’s wrong here? When the code was first executed, the majority of compilations resulted in Vector3(0.241534, 0.645345, 0.0), but in this case of 32-bit release mode, it was generating Vector3(0.645345, 0.241534, 0.0).

There is no sequence point here. In C++ the order of evaluating parameters to a function call is up to the compiler. So while the random number generator picks the same two random numbers as it’s called, the order that the parameters are evaluated in can be different, and this changes the outcome.

Changing it to the following code fixes the problem because the order of operations is explicit.

float randomY = rng.RandFloat();
float randomX = rng.RandFloat();
position = Vector3(randomX, randomY, 0.0f);

This made me feel like a newb. While I was vaguely aware of this type of issue, I expected the compiler to choose left to right or right to left evaluation and stick with it, and not change order due to optimizations. At least now if I ever implement support for more platforms, I can be sure the terrain generation will come out the same on all systems…

57 Comments

    MattBrox
    November 17, 2013 7:54 am

    As someone learning C++ right now, reading your technical posts such as this one is both interesting and terrifying.

    szunyi
    November 17, 2013 7:55 am

    Thats why php & mysql enough for me :)

    jpinard
    November 17, 2013 7:56 am

    Interesting. Too bad you can’t just ignore 32-bit altogether.

    szunyi
    November 17, 2013 7:57 am

    32bit only is still a big customer base
    Never lose it

    Phillip
    November 17, 2013 7:57 am

    Nice to know!
    I haven’t used C++ in more than a year and am now doing C at uni. Pointers for me are the worst still, coming from Java and C#.

    Shanee
    November 17, 2013 7:57 am

    Nice one :) This is one of those “gotcha” moments!

    Lukio
    November 17, 2013 8:00 am

    Do I conclude correctly, that we will be able to share world seeds, for example more challenging maps or “newbie friendly” maps and the world will always generate the same for everyone?

    BillM
    November 17, 2013 8:03 am

    I’m curious to know why it doesn’t happen in the 64bit version as well?

    BillM
    November 17, 2013 8:04 am

    Let me rephrase that, I want to know why it only happens in the 32bit version and not the 64bit version.

    llnk
    November 17, 2013 8:05 am

    Be sure to use a factory method:

    Vector3 RandVector3() {
    float randomY = rng.RandFloat();
    float randomX = rng.RandFloat();
    return Vector3(randomX, randomY, 0.0f);
    }

    Thomas
    November 17, 2013 8:05 am

    check that i was browsing steam and found banished now i know where to buy and im defintly doing so

    Charles
    November 17, 2013 8:14 am

    Nice catch, I don’t think I’d have catched a similar one myself before checking out 3 times what every other line of code does.

    @BillM: it didn’t exactly “work” either in 64 bit. He had just been using the map generated in 64bit to create his tutorial, so it worked fine as long as things were tested on that platform.
    But once he changed to 32bit compilation, compiler generated the random numbers in Vector3 parameters in a different order than 64bit version, giving a different tutorial map.

    Templar_X
    November 17, 2013 8:21 am

    Nasty bugger. Compiler induced bugs are a PITA.
    Nice catch :)

    Matthew Horne
    November 17, 2013 8:23 am

    32bit systems will become obsolete at some point. Games today are all being developed on 64bit and many of the betas I have played lately are 64bit only and 32/64 for release but it highlights a shift, a long awaited shift towards 64bit adoption. I can’t think of a single reason to use 32bit today.

    I understand some software doesn’t work on 64bit and that it could be expensive for companies to move over but for the average user I can’t see any reason not to use 64bit systems.

    32bit systems can only access around 3.5GB of RAM, Multitasking on a system with just 4GB of RAM installed is a nightmare.

    pfudke
    November 17, 2013 8:33 am

    Well this reminds me of something…
    http://forums.ltheory.com/viewtopic.php?f=12&t=1427

    And if you want to share how you generate the maps, I would be glad :)

    Casey
    November 17, 2013 8:34 am

    This was a very cool post thank you. I also appreciate the fact that you are building primarily 64bit coming from someone with 32gb of RAM I really hate seeing so much 32 bit. Way to go and keep up the good work.

    Daxamar
    November 17, 2013 9:42 am

    One step closer to release! Good reads too. I wish you the best of luck catching them nasty bugs.

    If you have time, can you make a video soon. Just something to hold me over a bit. :)

    Dukus
    November 17, 2013 10:00 am

    The issue here isn’t 32 vs 64 bit. It just happens to be that the 32-bit compiler/optimizer moved things around. This could just as easily happen when upgrading or using different C++ compilers, regardless of target architecture.

    @llnk: I have a Vector3::Rand() but its -1.0 .. 1.0 and in all three components. What I was doing in terrain generation is a bit too specific for a generic reusable function.

    craig
    November 17, 2013 11:24 am

    That is one of those obvious as soon as you find it bugs. Takes ages to find and makes you feel daft as soon as you do. Good find, thanks for sharing!

    SirHumphreyAppleby
    November 17, 2013 12:50 pm

    I agree with Dukus, it’s not a 32 vs 64 bit issues. As a C programmer myself, I appreciate all the wonderful shortcuts C/C++ offer, but one must always be aware that while many of the behaviours are defined, many others are not.

    Is relying on the random number generator here the best solution? Can you guarantee it won’t change if you were to install a service pack for the dev tools? Might this change with a different runtime release? Could this be different in the release and debug versions?

    SomeOtherGeek
    November 17, 2013 12:52 pm

    Awesome catch! It is almost fate even. It is those little things that make a huge impact. It doesn’t matter how long you program, you will always find/learn something new. That is what I love about programming – the learning curve is still there.

    Don Neufeld
    November 17, 2013 1:03 pm

    Wow, I hit the exact same sequence point problem once, except in our case (PlanetSide) it was in some network serialization / deserialization code so it depended on how the code was compiled on both ends of the wire!

    Mr Steam
    November 17, 2013 3:16 pm

    wow, that’s a bit strange. well done.

    Shane
    November 17, 2013 4:14 pm

    As always, your posts make for extremely interesting reading! I’d ask for more of them, but the time you spend writing the posts is time away from getting the game out there for us to play! :D
    Looking forward to the release!

    Postie
    November 17, 2013 4:24 pm

    That’s pretty bizarre. I wonder if that is unique to C++ or if other languages like C# have the same quirk?

    Pies
    November 17, 2013 5:02 pm

    This has nothing to do with 32-bit or C++. The problem is that when writing code we make assumptions as to what the computer will do, which are based on abstractions. Call arguments which are in themselves dynamic calls is a rather deep abstraction, because machine code has nothing like that. Therefore the compiler has to do something very similiar to what was done here, but does it more cleverly (optimizes for memory/speed) and misses our meaning.

    It also stems from random number generators being a special case, because they’re not stateless.

    Jon
    November 17, 2013 5:21 pm

    I find this very interesting to read.
    Good luck with the resting bugs. You deserve a good holiday. :)

    Chronos
    November 17, 2013 5:29 pm

    If maps are randomly generated, do you think it’s at all possible to add a terraforming feature to create our own maps at some point?

    Poolboy
    November 17, 2013 6:52 pm

    I’m getting a bit frantic here. Can you throw us some crumbs, please? A video would be amazing.

    Panny
    November 17, 2013 8:09 pm

    Hope you’ve got a few hundred testers submitting bugs for you :D

    name*
    November 17, 2013 10:55 pm

    If I promise to keep reading your blog, will you release the game faster?

    I’m not saying that’s why it’s taking so long, but if it is, we’re not going anywhere, man.

    YMIHere
    November 17, 2013 11:04 pm

    That’s an interesting bit of knowledge. I wonder how many times I’ve run into this and just not known about it.

    Mahono
    November 18, 2013 12:52 am

    Just as some advice, due to the immense hype of the Steam Box and Steam OS, it would be very wise and profitable to work on releasing this on Linux later. It would also make me very happy. :3

    Luaan
    November 18, 2013 3:41 am

    Matthew Horne, Casey, etc:
    Please, don’t confuse two separate cases of 32 bit vs. 64 bit. 32 bit applications on a 64 bit OS (or at least, on Windows) have no overhead, and in fact in some cases, they can be faster (may be a better 32 bit compiler or something) – and in *all* cases, they take less memory. 32 bit applications can still use all the memory you have, they just can’t gobble it up all for themselves (ie. you can run 10 applications that use 4 GB of memory each, and they will use up 40 GB of your system memory; ti’s just that you can’t have a single 32 bit application use 40 GB of memory for itself).
    This means that as long as you are far enough from the 4 GB limit, 64 bit is no improvement, and can in fact be detriminal (case in point: the authors of MS Visual Studio are still fighting hard against doing a 64 bit version of Visual Studio, because given the pointer-centric nature of such a tool, 64 bit version would take up almost twice as much memory space as the 32 bit version).
    Now, in a game, that ratio would be significantly different (“twice as much” is pretty much the upper limit), but I can still see the 32 bit version using 10% less memory or so.
    64 bit OS – oh yes. 64 bit applications? Only if you can’t squeeze yourself into 32 bit. At least on Windows. I have no idea how the 32 bit on 64 bit OS works on the various POSIXes.

    mrbisonm
    November 18, 2013 7:24 am

    Off this topic a little, but I was wondering for awhile now what the views are ingame. They seems to be horizontally variable (more like CIV IV), but how close (zoom in) and how far (zoom out) can we get in the screenview. Simple 2 example pics would answer this question that I am sooo curious about. ;)

    mrb

    Jon
    November 18, 2013 8:42 am

    Hey Mrbisonm,
    This might be an answer on your question:
    http://www.youtube.com/watch?v=bBGX0dl5Ziw
    http://www.youtube.com/watch?v=6oJW013U638

    We don’t know what’s the max afaik but it gives a nice idea.

    Eliza
    November 18, 2013 12:11 pm

    More elegant than what I’d have done to fix it, I must admit. (I probably would have if-ed it with hard-coded parameters.)

    BillM
    November 18, 2013 12:47 pm

    @Charles

    That makes sense. Thanks for the reply!

    mrbisonm
    November 18, 2013 2:04 pm

    Thanks Jon, I saw these vids before but I guess I didn’t really pay attention to the zooms before. Yes, it gave me a good idea.

    mrb

    Darkclox
    November 18, 2013 3:33 pm

    Is this game is going to be free?

    Darkclox
    November 18, 2013 3:33 pm

    The game is going to be free?

    Yin
    November 18, 2013 6:36 pm

    oh come on, that’s just like being at university… technical problems to tax your brain!

    I spent half my life doing brain dead jobs that a donkey could do!

    Section8
    November 18, 2013 10:00 pm

    Nice one! Debugging release code sucks.

    Alex
    November 18, 2013 10:59 pm

    Tutorial – “Place the building in the lake.”
    Player — “But… the lake?..”
    Tutorial – “Place.. the building.. in the
    lake.”
    Player — “Really? But that doesn’t make
    se…”
    Tutorial – “PLACE THE DAMN BUILDING IN THE
    LAKE!” >..<

    Tortex
    November 19, 2013 3:20 am

    @Darkclox
    One word: No.

    Fr8monkey
    November 19, 2013 5:26 am

    @ Tortex… Don’t feed the trolls. The only get bigger when you pay attention to them.

    Ryan
    November 19, 2013 11:22 am

    After following your progress for months.. How soon can I give you my money so I can play this. Sooner is better than later. Take my money!!

    Dawesome
    November 19, 2013 12:48 pm

    Pies, Dukus, SirHumphreyAppleby…. this actually has everything to do with 32-bit C++ vs 64-bit C++, or at least the Windows compilers C++ on those targets (which is, Dukus, I think your point). Parameter evaluation is determined by calling convention, and most calling conventions on x86 is right-to-left, while x64 is always left-to-right.

    See http://msdn.microsoft.com/en-us/library/zthk2dkh.aspx and http://msdn.microsoft.com/en-us/library/984x0h58.aspx.

    Steve
    November 19, 2013 4:51 pm

    Is this something that is necessarily unique to c++? Couldn’t other languages with more than one compiler have a similar issue?

    SirHumphreyAppleby
    November 19, 2013 5:17 pm

    @Luaan. It is NOT true that 32-bit applications always use less space. When running 32-bit applications on 64-bit editions of Windows, you get a dual stack allocation per thread. This wastes a chunk of of memory per thread, leads to memory fragmentation issues, and lower overall capacity for highly threaded applications running on 64-bit systems with equivalent hardware to their 32-bit counterparts.

    @Dawesome. No, it’s not 32-vs-64 bit. Calling convention relates to how parameters are represented in memory when calling a function. In the case of a nested function calls, only the result of the call is being passed. When multiple function calls are required to assemble the results to pass to another function, this job is entirely up to the parser/compiler.

    Dawesome
    November 19, 2013 8:22 pm

    @SirHumphreyAppleby: Calling conventions also specify the order that the parameters are passed. While the standard is clear that the order of parameter evaluation is unspecified, Microsoft compilers follow the calling order, thus the x86/x64 difference. Try it for yourself and see!

    Calimero100582
    November 20, 2013 7:48 pm

    You should compare in different OSes also

    Voltaire1512
    November 21, 2013 4:04 am

    Since i stumbled over this project a few months ago I am fascinated by the progress you are making with this.

    Of course many of us are in a “Shut up and take my money”-mood but, on the other hand, we all like games with not too many bugs / glitches.

    Long story short: This is incredibly interesting and I am eagerly but also patiently waiting. Keep up the good work.

    Luaan
    November 21, 2013 4:39 pm

    SirHumphreyAppleby: Sorry, you are right, of course. Serves me right for saying things like “all” :D
    The real memory footprint is obviously greatly dependent on the application. If you create a thousand threads, each working with 1 MB of data, the overhead is going to be massive. However, the 32 bit stack that comes “extra”, is still just 50% more memory than the purely 64 bit stack. That’s a huge difference when you’re keeping tight control of your memory (all those 10 MB messenger apps etc.), but in a large application including a game like Banished, it will mean a lot less overhead compared to total memory consumption.

    Thanks for making me find out the real impact of 32 bit on WOW, though. Now I actually have exact numbers, and indeed, they are larger than I expected. I still think that most applications can benefit from 32 bit even today, though, especially in a managed environment, which “hides” a lot of the lost resources and performance.
    I made a huge mistake in combining the impact on different kinds of applications (calculation heavy, versus pointer heavy, versus memory heavy…), and only taking the benefits from each, which is obviously very wrong, and I’m sorry for that.

    MindALot
    December 2, 2013 8:54 pm

    I cannot find a definition of RandFloat that says it will always return the same value given the same seed. Because of this, you may want to hardcode the “random numbers” that you use for your tutorial.

    Chris Fleming
    December 16, 2013 5:39 am

    MindALOt – I’m pretty rusty on this stuff, but I believe that RandFlot is really a psudo-random number generator, but starting at a known seed value the sequence of random numbers will be known.

    MindALot
    December 16, 2013 7:34 am

    Chris : It seems like srand(x) is a valid way to always get the same random sequence – however I haven’t found any place that guarantees it. I’d highly recommend saving the values used for the tutorial, unless creating the tutorial a second time could be done with very little effort if/when needed.