Take the 2-minute tour ×

We've 4 embedded firmware at hand. Two of them are released, are in maintenance phase. Other two are going to be released. The released product uses OKI 411 micro, where as yet to be released products are on OKI 431 micro.

So far we've been compiling the code using OKI's CCU8 3.08 compiler. It supports both 411 and 431 hardware. Recently they've updated the compiler to 3.10 and we started using it for all the products.

Suddenly we've found that 3.10 is producing unexpected behaviour in one 431 based project. It is related to compiler optimization. So the release build exhibited a bug and two whole days were wasted to figure it out that the firmware works fine if compiled with 3.08 version of the compiler. Note that the other 431 based product didn't show any issue.

So we've decided to abandon the latest compiler for all projects and use the previous one.

What steps should be taken when this kind of situation occurs? Also when upgrading compiler, what should be done to ensure that it will not break any build?

share|improve this question
The real question is why do you even need to upgrade the compiler? –  Pemdas Apr 10 '11 at 1:28
This is 2011 and a newer version of compiler is supposed to be smarter. But alas, it doesn't! –  Donotalo Apr 10 '11 at 2:02
Smarter how? And what benefit would the project gain from it. Changing the compiler just because there is new one out is not sufficient justification. If anything it is more risky because there hasn't been as much usage yet. I would always favor a compiler that has been throughly used and rung through dirt over a brand new one, unless there was specific feature that I needed that wasn't in the older version, which I find unlikely in most cases. –  Pemdas Apr 10 '11 at 2:25
What was the vendor response for this bug? Working with them to get this fixed sounds like a good idea. I don't like relying on previous versions of a given compiler to get a software working. I always try to make it work with a current and supported version. I once had a similar problem when I had to downgrade (don't ask me why) gcc for a project and found a optimization related bug too. Since changing the compiler was a customer requirement, we solved this turning off a specific optimization only for a given translation unit. –  Vitor Apr 10 '11 at 3:27
If he is going to talk to the vendor than he is going to need proof that there is an actually a bug in the compiler, which will require a code sample. –  Pemdas Apr 10 '11 at 4:14

6 Answers 6

up vote 5 down vote accepted

Often, "upgrading the compiler introduced a bug" really means "upgrading the compiler revealed an existing bug".

This is especially so for C and C++, where the language spec has lots of areas where the behaviour of such-and-such is not defined. Aggressive optimization can causes changes in behaviour for code that strays outside of the envelope of what is defined.

(OTOH, it could be a real compiler / optimizer bug. However, you need some hard evidence before you start pointing fingers.)

What steps should be taken when this kind of situation occurs?

There are some ideas:

  • Recompile with compiler warnings turned up to the max, and go though and fix your code so that the warnings go away.

  • Run the code through Lint or an equivalent bug checker, and fix every reported problem.

  • Turn down / off optimization. The down side is that your code will run slower.

  • Revert to the old compiler. The downside is that you risk getting stuck.

  • Check out the release notes / forums / issues database for the development suite to see if there are clues to what might have changes.

  • If you are willing to do some difficult detective work, try and isolate the location of the problem to a particular file, then compare the machine code of the working / non-working versions of the code.


There's nothing you can really do. You just have to try it and see what happens. Of course, that means that you need to be able to revert top the old tool-chain (at least as a short-term fix) if something does go wrong.

The other thing is to get out of the mindset that a broken build is a disaster ... or being embarrassed about. It is only a major problem if the broken build has real consequences for production systems ... or your customers (loosely defined).

share|improve this answer
I have come across compiler bugs which broke code and were fixed to restore functionality that had been working correctly prior. Most specifically there was a bug in the binary code generated by the file concatenation version in Turbo Pascal 5.5. Using either the 5.0 or 6.0 compiler it would work identically and correctly, using the 5.5 compiler using that function would crash an application. –  jwenting Apr 12 '11 at 8:11
One point should be added to that list: isolate the problem to a small portion of code, write a professional bug report and send it to the compiler vendor. Wait until a new release of the compiler is published where the bug is fixed, or the vendor tells you a workaround. We had some success with this strategy for compiler and library vendors, even Microsoft gave us a decent reply (though smaller vendors often react quicker). –  Doc Brown Sep 11 at 16:33

Compilers being upgraded is part of the "technical debt" that all applications need to go through, and must be considered on a regular basis.

If it is considered the same as any other library that the program uses, it is clear that it should only be upgraded at a point where it will not hurt if it breaks (like close to a deadline) and that you need to run a full quality check in the same way as you would if you did any other change to your program.

Note, that you may want to to with keeping the old environment in an old virtual machine, but you have then postponed all the upgrading to later, making it much more painful if you are forced to upgrade.

share|improve this answer
I agree, it is normal to upgrade your compiler from time to time, whether because you need a bug fix or new feature (code performance is a feature), because support will soon cease on the current one, ... it just need to be done with care and for good reasons. –  Matthieu M. Apr 10 '11 at 12:06

I think upgrading the compiler used for existing projects is a terrible idea for the very reasons you clearly discovered. Realistically, if you change the compiler, any new release of the existing product needs to go through a complete QA cycle as if it was brand new. You can't assume anything works.

In the specific situation that you described above, I would be willing to bet that there is a variable or a number of variables that should have the volatile qualifier that don't. One way to try and confirm this theory would be to turn off the compiler optimizations. If the build works then there is strong support for this theory.

share|improve this answer
True. Turning off compiler optimization works. –  Donotalo Apr 10 '11 at 2:03
Not upgrading the compiler may not be an option. Imagine for example you find that your compiler has been compromised as described in cm.bell-labs.com/who/ken/trust.html –  user1249 Apr 10 '11 at 14:16
There are cases where upgrading the compiler is necessary or desirable. I won't argue that. In generally the idea should be heavy scrutinized. I have trouble believe that the case you linked to would not have been found in the first version of the product, even the author suggests this. The danger has more to do with upgrading the compiler on an existing product. I believe the author shares my sediment as he blatantly says not to trust third-party code. –  Pemdas Apr 10 '11 at 19:16
the Thompson Turing talk is about compiling your own source with the compiler you compiled yourself and it STILL contains Trojan code. Without inspecting the generated machine code you cannot tell. When was the last time you inspected the output of your compiler? –  user1249 Apr 12 '11 at 15:40
Last week, but I work in the embedded field, so it is not that uncommon to inspect the assembly. –  Pemdas Apr 12 '11 at 18:27

Honestly, I'm a little surprised by all the "why are you upgrading your compiler" feedback. Compilers are software, like any other. They have a life cycle, like any other. At some point, you need to stop using older software and move to the newest version which is supported by the vendor and has support for newer features.

Or do you want to be the guy who ends up on TheDailyWTF for going into the IT Admin office asking for a machine to be set up with Windows 95 so you can compile your software because that's the last version the compiler for your software will install on? You'll be the same as the guy who makes the security admins facepalm in unison when he says his software needs Java 1.4.

Upgrading your compiler (or runtime environment) should be treated like upgrading any third party linked library (because that's essentially what it is): test, test, test. It should be a completely normal part of software maintenance and one that you plan for in the future. Your dev software has a life cycle just like the code you produce. Be aware that it will be end-of-life'd.

share|improve this answer
Microsoft goes through great pains to ensure backwards compatibly. Windows 7 and Vista have what they call compatibly mode where you can run programs in older version of windows as far back as windows 95. –  Pemdas Apr 10 '11 at 3:58
@Pemdas You've missed the point of my post completely by nit picking the analogies and ignoring the argument. The point is you don't want to be in a situation where you're required to use unsupported software which only runs on unsupported systems. You also don't want to have to say "we have to use gcc 3.95" or similar because then you're relying on compiler quirks. This is the definition of poor software maintenance. What if you need to compile your application to support Execute Bit Disable 3.0 for Windows 12 compatibility and your ancient compiler can't handle it? Oops! You're screwed. –  Bacon Bits Apr 10 '11 at 4:46
Execute Bit Disable has nothing to do with the compiler. I know exactly what are you saying. I am just pointing out that compilers don't really expire that often. Compatible with the operating system is almost a mute point. There are not too many cases where you really need to upgrade or should upgrade the compiler on EXISTING projects. Runtime environments are are a completely different story. –  Pemdas Apr 10 '11 at 5:12
the Windows 95 scenario is very close to what Visual Basic 6 maintainers live through. –  user1249 Apr 10 '11 at 14:17

I do not entirely agree with @Pemdas. IMO it depends on two factors:

  • The first factor is project size. For small projects you do not have to upgrade the compiler and you may stick with the current till the project is delivered to the customer and avoid the pain of upgrade. OTOH, if the project is big and you or the organization will spend years in maintenance it will be much more painful for the developer later to both maintain old applications and use old tools. Upgrading in this case will give you more benefits too like new features to give to the customer and improve the product with simple and little more effort which will help promote product in the market.
  • The other factor is the compiler itself. If the compiler provider has the reputation of maintaining good backward compatibility you do not have to invest much effort in testing everything. OTOH, if the compiler provider usually deprecate features and make breaking changes with every version you will know it will be a painful upgrade.

To avoid upgrade pain you should:

  • Read the documentation carefully before you start and you may give it a little time till more people in the community take the step and start blogging about it so your path be more clear.
  • If you are using third party components or libraries you may choose to wait till they are done upgrading or if it's open source and simple you may start with them.
  • Make backup copy of course and perform the upgrade on a different branch of your version control system. If for some reason you do not want to do this AND if all the files are source files (text in nature not binary resources) you may use compiler directives to make the source able to compile on both the old and the new compiler.
  • Pick a good start point like a component or library with few dependencies to be a pilot to know early if it's going to fail or not.

However, I have to say this will not ensure that it will not break the new builds but doing upgrades in organized manner will get you on top of things and help fix issues quickly.

share|improve this answer
I fail to see how the compiler can generate new features for the customer. I also don't really understand what you mean by "improve the product with simple and little more effort" . I do see your point about maintenance though. I also don't think the compiler vender's reputation can justify minimizing the test coverage. –  Pemdas Apr 10 '11 at 3:39
@Pemdas The new compiler comes with new features in the language. These new features may inspire the developer to use them with simple effort to add new features for the customer and improve the product. Also usually the new compiler release is part of whole new version of the development tool that introduces new features in the component library and visual controls and so on. What I am saying is that it's good to make use of all this and give something to the customer as you upgrade. –  M.Sameer Apr 10 '11 at 12:17
@Pemdas Backward compatibility is some kind of commitment of development tool vendors. Some vendors usually provide backward compatibility better than others and some usually make breaking changes with every release. My experience with Delphi for example of over 10 years the breaking changes that caused problem were always minimal so I upgraded the compiler from version 5 all the way to 2007 with minimum testing effort because I know from experience and from documentation that Delphi is committed to backward compatibility. –  M.Sameer Apr 10 '11 at 12:21

We maintain a fairly complete set of automated tests on all of our code so that we can be relatively confident when upgrading compilers and other tools. It can be tricky to test some things in the embedded world, but it's generally worth the effort if you can do it.

share|improve this answer

Your Answer


