Take the 2-minute tour ×

I work at a place that is CVS-crazy and Bugzilla-nuts.

  1. There are so many branches off each release that one cannot count them. Everyone is constantly auto-merging.

  2. There is no fluidity at this job. Everything feels lock-step. It takes 25 steps even for a simple thing. It's not like being on a factory production line: it's like setting up a factory myself every day.

Example situation:

To fix a single bug, first I obtain a clean, new virtual machine. Then I create a branch for that single bug fix, based on another branch described in the Bugzilla report. I install the branch on the machine, set that up. I fix the bug. I check it in, leaving it and the machine for others to test with. Then I have to go into the bug control software and explain what I did and write a test case, with all the steps. Eventually someone else merges it with a release.

No matter how tiny the bug is, I have to do all of these things. Sometimes people combine work on multiple bugs, but as I said there are so many branches that this is hardly possible.

At any other job, I'd just go in and fix the bug. I barely even remember using SCM, although every job I've had has used it: that's because at every other job, they somehow kept it out of the way.

Is there a point at which the process gets in the way and becomes an end unto itself? Is that even engineering?

share|improve this question
Feel bad for you :( Does company where your are working has CMMI 3 and higher ? –  artjom Jul 13 '11 at 11:28
Sounds like the organization has been badly bitten earlier and have grown defenses. Perhaps you should ask about the history of this? –  user1249 Jul 13 '11 at 11:39
And since the supervisors encourage constant distractions, your job must be a real hell... –  vines Jul 13 '11 at 11:48
Is this a question or a blog post? –  Rei Miyasaka Jul 13 '11 at 12:29
If the software was the control system for a nuclear power plant this doesn't seem unreasonable. If for a Facebook fan page it seems extremely excessive. Without the context it is hard to say if this unreasonable or not: there are certainly projects for which it isn't, and others where it certainly is. –  edA-qa mort-ora-y Jul 13 '11 at 12:32
show 10 more comments

16 Answers


Heavy processes are common, unfortunately. Some people - especially management - religiously imagine that processes produce products. So they overdo the processes and forget that it's really a handful of hard-working, smart people who actually create the products. For upper management, it's frightening to even think that their business is in the hands of few geeks, and so the close their eyes from the reality and think of their dear "process" instead, which gives them the illusion of control.

That's why agile startups with a handful of good engineers can beat big, established corporations, whose workers spend 95 % of their energy on process and reporting. Some examples of once small startups that once did beat their competitors and/or created completely new markets:

  • Apple (Apple I was created by 1 engineer; there were 3 men at the company back then).
  • Google (created originally by 2 programmers).
  • Facebook (1-man effort originally).
  • Microsoft (2-man company in 1975).

One could easily say that these are just outliers, extreme exceptions, and to do something serious, you'd better be a big, established corporation. But the list goes on. And on. It's embarrassingly long. Almost every today-major corporation started as a garage shop, which did something unusual. Something weird. They were doing it wrong. Do you think they were doing it according to the process?

share|improve this answer
I'm wondering, is there evidence for any of the claims you present here? Are you a primary source (i.e. executive management)? Have you conducted or read interviews with them? It's very interesting how all sorts of replies saying "YES! RIGHT ON!" appear to be coming from people who have never been on the giving end and thus couldn't possibly vouch for the accuracy. I think it's important for us to distinguish answers which are actually true from those which developers (who are notoriously anti-management) would simply like to believe. –  Aaronaught Jul 15 '11 at 11:27
I really don't think that the onus should be on myself or anybody else to provide "better backed information" when criticizing an answer like this; you've made a very strong, broad, sweeping claim and presented zero evidence - not even anecdotal evidence - to back it up. It's unfortunate that a supposedly professional community is so easily swayed by this kind of populist nonsense. –  Aaronaught Jul 17 '11 at 21:18
What, really, you don't think it's easy to get votes by telling people what they want to hear? Yes, to put it bluntly, I don't have a lot of respect for the mob that upvotes these undeserving answers. I guess I can't blame people like you for doing the absolute minimum when the community rewards that behaviour, but nevertheless, I wish people would at least try to improve their answers when criticized instead of stubbornly pointing to the upvotes as a justification. –  Aaronaught Jul 18 '11 at 16:23
The entire thing? "Some people" - which people? "Especially management" - why assume that they are more likely to believe this? "Religiously imagine" - how are you certain that their beliefs have no basis in facts or logic? "Processes produce products?" - who exactly has made that specific claim and in what context? "Overdo the processes" - what precisely does that mean? "Business is in the hands of few geeks" - to what degree, and how so? "close their eyes / illusion of control" - explain? "agile startups... can beat big, established corporations" - do you assert that these aren't outliers? –  Aaronaught Jul 19 '11 at 11:01
@Aaronaught: This forum is not a scientific paper. Nobody, not you nor me, is going to provide an explanation for every single sentence he writes. You're only asking those for this answer because you don't like it. Apparently it hits the nerve, but how about disagreeing in civilized manner? As for startups beating big corporations, read e.g. just the two first sentences of product description from this: amazon.com/Radical-Innovation-Companies-Outsmart-Upstarts/dp/… –  Joonas Pulakka Jul 19 '11 at 11:16
show 8 more comments

Companies typically suffer from what I'd like to call the Control-Flexibility dilemma. The less rules, structures and bureaucratic overhead the easier and faster it is to accomplish things (it's also more fun). However, it is equally easy to do "bad" things as "good" things. That means you're only fine when you have skilled employees which make few non-critical mistakes.

On the other hand, if you have lot's of low to semi-skilled employees and/or the cost of making mistakes are too high (like the risk of space shuttle debris over the northern hemisphere) companies tend to pile on more and more "rules" and "processes" to try and minimize them.

The only problem is that the cumulative overhead of these processes makes it hard to accomplish anything which will result in the more talented employees leaving the company. This results in the average skill going down even more, requiring even more processes and bureaucracy. So the death spiral continues until something radical happens or the company goes belly up.

If you find yourself at a company who seems to have gone past the point of no return in this aspect you can either resolve yourself to "not caring" about your job (which is what most that have stayed have done) or getting the hell out of there with your soul intact :)

If the company has not gone too far and you have the means you can try to reverse the course through sheer determination and willpower. Beware though that it can take tremendous amount of work and personal energy with no guarantee of success so be very sure that it's worth it. Sometimes it's better to just cut one's loss and count oneself one experience richer.

share|improve this answer
add comment

There is only one valid reason for such style of development: the software developed is absolutely mission-critical and must not, under any circumstances, contain any bugs. Think jet engine fuel injection firmware in passenger planes, or heart pacemaker firmware, or nuclear missile launch system.

In all other situations the cost overhead will kill the business. Time to move on.

share|improve this answer
They claim it's mission critical, but one can say this about any piece of software; it's a matter of how much the customer is accepting of glitches. And if it were mission critical, I'd have to ask why for instance, they seem to really dislike the idea of giving ownership of any project to anyone. Recently I was asked about a complex piece of software I wrote 3 months ago, and I couldn't provide a clue, because they had moved me off that work abruptly once I got it working. These people are idiots. They consider everyone disposable, and anyone's opinions other than theirs are worthless. –  Ponk Jul 13 '11 at 12:16
@Ponk, Everyone is disposable though. When the processes define the work, and the customer already accepts the software and money is rolling in then everything and nothing is mission critical anymore. Why care about innovation at this point? The customer probably just cares that at a moments notice, their vendor has a trained and ready to work software development team that can get the new features out in less than a year. In the meantime it is not important that you and the team accomplish anything substaintial other than the illusion that you are working. –  maple_shaft Jul 13 '11 at 12:53
@maple: One thing is getting made redundant. Another is if people died because of a small typo of yours and on top of losing job you face charges of manslaughter with several years of prison. THIS is what I call mission-critical, and there aren't many pieces of software where you face such risk. –  SF. Jul 13 '11 at 13:34
It isn't only one reason why they doing it in way they doing. And saying that one developing process is better than other is same as saying that orange is better than banana. If they are profitable and can pay salary this process works better than some agile oriented company. From what was written I just can see that this person is in wrong job. There is lot of company who provide more freedom for developers. –  Dainius Jul 13 '11 at 16:48
show 1 more comment

This question really contains two questions, which need to be addressed separately:

Why do some teams have a strict development process?

The simple answer is because if they don't, mistakes happen. Costly mistakes. This is true for development and it is true for the rest of the IT field as well (sysadmins, DBAs, etc.).

This is very hard for a lot of developers and IT workers to understand because most of us have only ever worked at one of the "extremes" - either big, Fortune-style companies with at least a dozen developers and strict processes to follow, or small, micro-ISVs or even freelance work where people just don't screw up badly, or the cost of a screw-up is low.

But if you've ever seen a company in between these phases - even a company with bright, talented IT staff - you'll understand the dangers of having no process or having a half-assed process. You see, communication among staff suffers from a Combinatorial Explosion problem; once you reach the level of around 6-10 developers on a single team, the primary cause of major or critical defects is not a lack of talent or know-how, but rather a lack of communication.

Alice asks around on Monday morning and decides it's OK to do reconstructive surgery in the trunk because nobody else is working on that part. Bob arrives an hour later, back from his vacation and full of energy and decides he's going to implement a new major feature in that exact same area, and why bother with a branch because nobody ever touches that code anyway? So Alice pays off that "technical debt", Bob implements his killer feature that's been on the back-burner for 6 months, and when they finally both check in their code (right before closing time on Friday, of course!), the entire team has to stay behind and try to work through the nightmarish hell of conflicts which continue to live on as bugs and regressions throughout the next couple of weeks.

Alice and Bob Both did a great job on the coding tasks, but they both started off with a bad decision ("what's the worst that could happen?"). The team lead or project manager takes them through a post-mortem, and hammers out a checklist to prevent this from happening again:

  • Check-ins must be daily to minimize the impact of conflicts;
  • Changes that will take significantly more than 1 day must be done on branches;
  • All significant tasks (including non-feature work such as refactoring) must be properly tracked and assigned in the bug tracker.

I'll bet that, to a lot of us, this "process" just seems like common sense. It's old hat. But did you know that a lot of smaller teams don't do this? A two-man team might not even bother with source control at all. Who cares? It's honestly not necessary. The problems only start to happen when the team grows but the process doesn't.

Of course, process optimization is like performance optimization; it follows an inverse exponential curve. The checklist above may eliminate 80% of the defects, but after you implement it, you find that some other thing accounts for the remaining 80% of defects. In our fictitious-but-familiar example it might be build errors due to having different build environments, which is in turn due to the fact that there's no standard hardware and developers are using open-source libraries which get updated every 2 weeks.

So you have three choices: Either (a) standardize the hardware and severely restrict 3rd-party library usage, which is costly and may hurt productivity significantly, or (b) set up a build server, which requires cooperation from the sysadmin group and a full-time developer to maintain it, or (c) let developers do this themselves by distributing a standard virtual machine and telling the developers to build on that. Clearly (b) is the best long-term solution but (c) has a better short-term balance of reliability and expedience.

The cycle goes on and on. Every "policy" you see has generally been instituted to resolve a real problem. As Joel Spolsky wrote all the way back in 2000 (on a completely different topic mind you, but relevant nevertheless):

When you go into a restaurant and you see a sign that says "No Dogs Allowed," you might think that sign is purely proscriptive: Mr. Restaurant doesn't like dogs around, so when he built the restaurant he put up that sign.

If that was all that was going on, there would also be a "No Snakes" sign; after all, nobody likes snakes. And a "No Elephants" sign, because they break the chairs when they sit down.

The real reason that sign is there is historical: it is a historical marker that indicates that people used to try to bring their dogs into the restaurant.

It's the same in most (I won't say all) software teams: Policies such as "You need to add a test case for every bug fix" almost invariably indicate that the team has historically had problems with regressions. Regressions are another one of those problems that are most often due to communication breakdown rather than incompetence. As long as you understand the policy, you may be able to take legitimate shortcuts (e.g. I had to fix 6 small bugs but they were all in the same feature, so I can actually just write one test case for all 9 of them).

That explains why the processes are there, but it's not the whole story. The other half is:

Why is the process so hard to follow?

This is actually the simpler question to answer: It's because the team (or its management) is focused on repeatable results and minimizing defects (as above) but haven't given sufficient attention to optimization and automation of that process.

For example, in the original question, I see several problems:

  • The revision control system (CVS) is legacy by today's standards. For new projects, it has been superseded almost entirely by subversion (SVN), which itself is quickly becoming eclipsed by distributed systems such as Mercurial (Hg). Switching to Hg would make branching and merging far simpler, and even in my hypothetical example above, the daily commit requirement would become much less painful. The code doesn't even have to compile, because the repository is local; - in fact, the lazier developers could even automate this step if they wanted, setting up a logoff script to automatically commit changes to the local repository.

  • No time has been spent automating the Virtual Machine process. The entire process of obtaining, configuring, and downloading source/libraries to a virtual machine could be 100% automated. It could be an unattended process you run on a central server somewhere while you work on the bug fix on your local machine (and only use the VM to assure a clean build).

  • On the other hand, at a certain scale the VM-per-developer solution starts to get silly and you should just have a Continuous Integration server. That's where the real productivity benefits come in, because it (mostly) frees individual developers from having to worry about the builds at all. No need to worry about setting up clean virtual machines because the build server is always clean.

  • The wording of the question ("test case with all the steps") implies that there is some manual testing going on. This, again, may work for small teams with a relatively low work load, but makes no sense at all at a larger scale. Regression tests can and should be automated; there are no "steps", just a class or method added to the unit/integration test suite.

  • Needless to say, moving from Bugzilla to a newer (better) bug tracking system would make that part of the experience far less painful.

Companies aren't necessarily cheap or stupid just because they haven't solved these problems. All they know is that the current process works, and in some cases they are risk-averse and reluctant to change anything about it. But really they just need to be convinced of the benefits.

If developers spent a week tracking their time on all of the non-coding tasks, then you could easily add it up, show the management that (for example) a zero-capital, 100-man-hour investment in an upgrade to Mercurial would eliminate up to 10 man-hours per week on resolving merge conflicts, then that's a 10-week payoff and they'll almost certainly agree to it. Same idea with build servers (CI) or better bug tracking.

To recap: The teams haven't done these things yet because nobody has convinced the management that it is important enough to do today. So take the initiative and turn it into a cost-benefit equation; find out how much time is spent on tasks that could be simplified/automated with minimal risk, and calculate the break-even point and eventual payoff of a new tool or technique. If they still don't listen, then you already know what your remaining options are.

If developers spent a week tracking their time on all of the non-coding tasks, then you could easily add it up, show the management... and turn it into a cost-benefit equation etc...

Above part looks worth further expanding.

I can confirm that it works. Programmers used it few times in one of the projects I worked on and every time it led to desired changes.

My overall impression was that if done right, this trick can er overrule quite heavy amounts of management ignorance and inertia.

I would like to note though that company where we (developers) had to resort to this kind DIY approach was very immature IT-wise. At more seasoned software vendors I seen stuff like that mostly being done by managers themselves. And as a rule they were more productive at that than us programmers. Much more productive.

share|improve this answer
add comment

If you are working in a heavily regulated industry, there might some reason for that cumbersome process : one day you could be audited and you'll have to show all your records to explain who fixed that bug, how, who reviewed it, who tested it, etc...

So it might be a necessary evil.

On the other hand, if there is no justification for that process, other than maybe a lack of trust from the management, you should try to talk to your manager and tell him how you could save time (and thus money) for the company.

No one in its right mind who refuse a quicker and better process if it is correctly explained.

But be ready to defend your argument to justify the change.

share|improve this answer
I interviewed for a job like that once, it was health care related and they portrayed the process as living hell. Nice of them to be honest. –  Ponk Jul 13 '11 at 12:21
Such processes however presuppose that the current implementation is pristine and defect free. Having such a process essentially lock-in a broken product is real danger. –  edA-qa mort-ora-y Jul 13 '11 at 12:36
"No one in its right mind who refuse a quicker and better process if it is correctly explained." --- I can think of plenty of agendas a decision maker could be following where this statement does not hold true. –  user29776 Jul 13 '11 at 19:46
show 2 more comments

Half of the problem is that you're using obsolete tools in a process, that they where not designed for. What you describe is very easy to have in modern DVCSes, without tedious task of creating branch per bug.

Another problem is that you're clearly not in the line of work you enjoy. You work in maintenance, while you'd like development. There is little that can be done about that besides changing the job.

share|improve this answer
add comment

The discipline of software engineering is inherently "all about process" so to wonder if it "has become" that way is kind of missing the point.

While most programmers would rather be bothered with the absolute minimum of process, to the extent that some will advocate for agile methodologies even when they don't solve the problems their organization is facing, it is entirely possible for a company to decide to use "heavy" processes for sound business reasons, such as "the customer demands it." This is common if your customers are Fortune 500 companies, universities, or government agencies. The rewards of selling to these customers can be sufficiently great that it justifies the added process overhead.

Your company is one data point, and it's impossible to generalize your experience into an industry-wide trend toward or away from heavier processes. The real question is, what balance works best for your company, your products, your customers, and you, personally, as a programmer? If you don't like working for that company, instigate change or get another job.

share|improve this answer
show 1 more comment

From the other question I've seen from you today, you seem pretty unhappy with your job. You've not been there for long, you can just tell your supervisor that you think you've made a mistake, and it's time for you to part ways earlier than expected.

If you're good at your job, you really won't have a very hard time finding a new one, and honestly, if there isn't a good reason for this process to exist, I'd probably consider moving too. Using CVS at all would really be a deal-breaker for me, but I always ask the source control question at the interview. Obviously, I can't know your situation, and it may be impossible to leave a job if you have other obligations.

share|improve this answer
ahhh ouch @maple_shaft, that is harsh. Still think of what you can tell the grand kiddies... I worked on powerbuilder apps... :D #life-fail –  Anonymous Type Jul 13 '11 at 22:08
show 2 more comments

I was going to spout about how software engineering is being flooded with very bad programmers, but the situation you describe is awful.

In my personal experience, some of this "process" you describe is accompanied with management and system administration being completely unaware of the inefficiencies of the systems they're imposing on the programmers. Examples include restricting choice of operating system, restricting software used, internet access, personal desktop administrative privileges; all these things are essential to good development.

Furthermore, compatibility requirements between the "magic solutions" employed by different branches of companies. Examples include head offices imposing centralized source control, off-site Outlook servers, and coding guidelines or languages that might not be appropriate for every problem.

Not a lot of fun keeping the wheels of the enterprise juggernauts rolling, but I've found that little bubbles of innovation, creativity and brilliance do exist in some companies.

share|improve this answer
show 1 more comment

You probably work in a process oriented company. I would try to find a result oriented company instead, where it matters what you do not how you do it.

In my company we have a "process" too (but it's very simple).. But when it gets in the way I break the rules and skip steps. No one will ever say anything to me as long as I don't break anything by taking shortcuts because I get results.

share|improve this answer
add comment


Literally, most engineering is putting together well established pieces in a set order in response to a particular problem. This is most obvious if you ask an ME what they do all day. You're confusing engineering with architecture or early-stage product development (in any field). I have two observations about your question.

  1. You seem to have been assigned to bug fixing work, not architecture or design work.
  2. Your coworkers seem to have come up with limited workarounds (combining bug fix VMs) to make them more efficient, you don't seem to be following their sensible example.

It's simply a fact that in any constructive endeavor that takes a large number of people, some people get to do the design, and a larger group 'get' to do the implementation. Sorry, but you're in that second group.

As other commentators have noted, CVS is not the best tool for the job for a highly-branched development model, but I also note that you aren't responsible for merging... so don't worry about it.

Most of your complaints essentially seem to be "I don't want to test, before, during or after development!" Let's break down your steps, point by point.

  • To fix a single bug, first I obtain a clean, new virtual machine. A test environment
  • Then I create a branch for that single bug fix, based on another branch described in the Bugzilla report. - You duplicate the environment the bug was found in... how is this unreasonable?
  • I install the branch on the machine, set that up. I fix the bug. I check it in - The basic dev process
  • ...leaving it and the machine for others to test with. - Your merge team probably wants this to be able to verify the fix if the merge goes south.
  • Then I have to go into the bug control software and explain what I did - If you're at a shop that doesn't do this... why are you tracking bugs at all?
  • and write a test case with all the steps. - I could be wrong, but that seems to be the direction all of the 'cool kids' are going anyway
  • Eventually someone else merges it with a release. - And several of the steps above are for making their jobs easier

Someone else in front of you obviously does bug triage to make sure that a bug that is found isn't re-fixed in another release, or broken in a later release (that's what the test-cases are for).

The only thing even remotely unusual or overzealous about this process is the VM thing. There's a fair chance that would be considered reasonable if we knew what domain you were working in.

share|improve this answer
add comment

Interesting. Do you have testers? They should be doing some of that. I am one, and where I work we have a decent solution.

For a functional defect, like you're explaining, our process goes like this:

  1. I test for the defect in a VM (either reported by a customer, or during my exploratory testing, or w/e)
  2. Bug is found/repro'd.
  3. I create the bug. Bugs include:
    • Full repro, from the point of installation to the point of seeing the bug.
    • A link to a snapshot of the VM (if I think the dev will need it... I snapshot it anyway, just in case they ask for it).
    • System info, any special settings that I needed to make.
  4. That bug gets assigned to the dev. While they're working on a fix I:
    • Create a test case
    • Write (or convert) the manual test into an automated test

Then I wait for a resolution, and help the dev in any way that they need. When it comes back as resolved, I:

  1. Run the automated test case and a manual version (sometimes) to confirm the fix.
  2. Close the bug.

TL;DR: If you don't have testers, then you need some. If you do have some, then they're not 'doing their part'.

share|improve this answer
add comment

Tom DeMarco:

...My early metrics book... most quoted line is its first sentence: "You can't control what you can't measure." This line contains a real truth, but I've become increasingly uncomfortable with my use of it. Implicit in the quote (and indeed in the book's title) is that control is an important aspect, maybe the most important, of any software project. But it isn't.

...the more you focus on control, the more likely you're working on a project that's striving to deliver something of relatively minor value.

To my mind, the question that's much more important than how to control a software project is, why on earth are we doing so many projects that deliver such marginal value?...

Software Engineering: An Idea Whose Time Has Come and Gone?

share|improve this answer
show 1 more comment

"Then I create a branch for that single bug fix, "

There is no need to make a branch for the single bug fix. First create the bug in bugzilla. Check out the release branch. Make the fix. Perform the commit with the commit message containing the bug number, which updates the bug and marks it "fixed, needs test" or "fixed, tested, needs merge" appropriately, depending on the text keywords written in the commit message. The bug database is the perfect tracking mechanism for all changes made and why they were made; reports can be run from the bug database to extract this info.

share|improve this answer
add comment

I think most of the process could be automated, so that the virtual machine and branch creation (including compiling code, setting up debuggers etc.) was all done for you before you started to work on the bug.

Describing what you done and how it should be tested is well worth it for all bug fixes. I have found just writing the text can catch issues, as it make me think about risk etc.

So I think the process may be a bit “over the top”, but that the real issue is the lack of custom automated tools to support the process.

share|improve this answer
add comment

Yo man, your thought process is right in that what you described is utterly crappy and a true demonstration of how wrong things can be in software. Here is the good news though, there are so many companies out there practicing Agile in true spirits. The company I work for is one such. There are many these days and thatz real good news.

The time you sense that things really aren't right at your workplace, therer are two things you can do - either you can influence positive change or leave that place and join a better one.

CVS or any configuration management system isn't bad. Unfortunately the way it is used, without really knowing its purpose, causes this kinda pain in the !@#$ for the entire organization.

To get a quick understanding of what Agile is really all about, peruse through the book, "Practices of an Agile developer" by Venkata Subramaniam. Its nicely written in easily understandable language.

Wish you good luck!

share|improve this answer
add comment

Your Answer


