Take the 2-minute tour ×

I've been reading a bit about Literate Programming recently, and it got me thinking... Well-written tests, especially BDD-style specs can do a better job at explaining what code does than prose does, and have the big advantage of verifying their own accuracy.

I've never seen tests written inline with the code that they test. Is this just because languages don't tend to make it simple to separate application and test code when written in the same source file (and nobody's made it easy), or is there a more principled reason that people separate test code from application code?

share|improve this question
Some programming languages like python with doctest allows you to do that. –  Simon Feb 25 '13 at 13:40
You may feel the BDD-style specs are better than prose at explaining the code, but that doesn't mean the combination of the two isn't better. –  JeffO Feb 25 '13 at 13:49
Half of the arguments here apply to inline documentation as well. –  CodesInChaos Feb 25 '13 at 14:23
@Simon doctests are too simplistic for serious testing, mostly because they aren't designed for it. They were intended for, and excel at, having code examples in the documentation that can be verified automatically. Now, some people use them for unit testing as well, but lately (as in, for the past years) this took a lot of flak, because it tends to end in fragile messes, overly verbose "documentation", and other messes. –  delnan Feb 25 '13 at 14:34
Design by Contract allows for inline specifications that make testing straightforward. –  Fuhrmanator Feb 25 '13 at 17:20

11 Answers 11

up vote 80 down vote accepted

The only advantage I can think of for inline tests would be reducing the number of files to be written. With modern IDEs this really isn't that big a deal.

There are, however, a number of obvious drawbacks to inline testing:

  • It violates separation of concerns. This may be debatable, but to me testing functionality is a different responsibility than implementing it.
  • You'd either have to introduce new language features to distinguish between tests/implementation, or you'd risk blurring the line between the two.
  • Larger source files are harder to work with: harder to read, harder to understand, you're more likely to have to deal with source control conflicts.
  • I think it would make it harder to put your "tester" hat on, so to speak. If you're looking at the implementation details, you'll be more tempted to skip implementing certain tests.
share|improve this answer
That's interesting. I guess the advantage I can see is that when you've got your "coder" hat on, you want to be thinking about tests, but it's a good point that the reverse is not true. –  Chris Devereux Feb 25 '13 at 14:24
Along these lines, it's possible (and perhaps desirable) to have one person creating the tests and a second actually implementing the code. Putting the tests inline makes this more difficult. –  Jim Nutt Feb 25 '13 at 16:07
would downvote if i could. How is this an answer in any way at all? Implementers not writing tests? People skipping tests if they look at implementation details? "Just too hard" Conflicts on big files?? ANd how in any way could a test be confused with an implementation detail??? –  bharal Feb 26 '13 at 11:15
@bharal Also, wrt to "Just too hard", masochism is a fool's virtue. I want everything to be easy except for the problem I'm actually trying to solve. –  deworde Feb 26 '13 at 11:33
Unit test can be considered documentation. That suggests unit tests should be included in the code for the same reason as comments - to improve readability. However, the problem with that is there tend to be a lot of unit tests, and a lot of test-implementation overhead that doesn't specify expected results. Even comments within code should be kept succinct, with larger explanations moved out of the way - to a comment block outside the function, to a separate file, or maybe to a design document. Unit tests are IMO rarely if ever short enough to keep in the tested code like comments. –  Steve314 May 5 '13 at 1:11

I can think of some:

  • Readability. Interspersing "real" code and tests will make it harder to read the real code.

  • Code bloat. Mixing "real" code and test code into the same files / classes / whatever is likely to result in larger compiled files, etc. This is particularly important for languages with late binding.

  • You may not want your customers / clients to see your test code. (I don't like this reason ... but if you are working on a closed source project, the test code is unlikely to help the customer anyway.)

Now there are possible workarounds for each of these issue. But IMO, it is simpler not to go there in the first place.

It is worth observing that in the early days, Java programmers used to do this kind of thing; e.g. including a main(...) method in a class to facilitate testing. This idea has almost completely disappeared. It is industry practice to implement tests separately using a test framework of some kind.

It is also worth observing that Literate Programming (as conceived by Knuth) has never caught on in the software engineering industry.

share|improve this answer
+1 Readability issues - test code might be proportionally greater to the implementation code, especially in OO designs. –  Fuhrmanator Feb 25 '13 at 17:22
+1 for pointing out using test frameworks. I can't imagine using a good test framework concurrently with production code. –  joshin4colours Feb 25 '13 at 17:23
RE: You may not want your customers / clients to see your test code. (I don't like this reason ... but if you are working on a closed source project, the test code is unlikely to help the customer anyway.) -- It may be desirable to run the tests on the clients machine. Running the tests may help to quickly identify what the issue is and to id differences in the clients env.. –  sixtyfootersdude Feb 25 '13 at 18:24
@sixtyfootersdude - that is a pretty unusual situation. And assuming that you were developing closed source, you wouldn't want to include your tests in your standard binary distro just in case. (You would create a separate bundle containing the tests that you want the customer to run.) –  Stephen C Feb 26 '13 at 0:02

For many of the same reasons that you try to avoid tight coupling between classes in your code, it's also a good idea to avoid unnecessary coupling between tests and code.

Creation: Tests and code may be written at different times, by different people.

Control: If tests are used to specify requirements, you'd certainly want them to be subject to different rules about who can change them and when than the actual code is.

Reusability: If you put the tests inline, you can't use them with another piece of code.

Imagine that you've got a chunk of code that does the job correctly, but leaves a lot to be desired in terms of performance, maintainability, whatever. You decide to replace that code with new and improved code. Using the same set of tests can help you verify that the new code produces the same results as the old code.

Selectability: Keeping the tests separate from the code makes it easier to choose which tests you want to run.

For example, you might have a small suite of tests that relate just to the code you're currently working on, and a larger suite that tests the entire project.

share|improve this answer
I'm puzzled about your reasons: TDD already says test creation happens before (or at the same time) as production code, and must be done by the same coder! They also hint that tests are pretty much like requirements. Of course, these objections don't apply if you don't subscribe to the TDD dogma (which would be acceptable, but you must make it clear!). Also, what exactly is a "reusable" test? Aren't tests, by definition, specific to the code they test? –  Andres F. May 5 '13 at 15:32
@AndresF. No, tests are not specific to the code they test; they're specific to the behavior that they test for. So, say you've got a Widget module complete with a set of tests that verify that Widget is behaving correctly. Your colleague comes up with BetterWidget, which purports to do the same thing as Widget but three times faster. If the tests for Widget are embedded in the Widget's source code in the same way that Literate Programming embeds documentation within source code, you can't very well apply those tests to BetterWidget to verify that it behaves the same was that Widget does. –  Caleb May 6 '13 at 3:54

Actually, you can think of Design By Contract as doing this. The problem is most programming languages don't let you write code like this :( It's very easy to test for preconditions by hand, but the post conditions are a real challenge without changing the way you write code (a huge negative IMO).

Michael Feathers has a presentation about this and this is one of the many ways he mentions you can improve code quality.

share|improve this answer

Here are some additional reasons I can think of:

  • having tests in a separate library makes it easier to link only that library against your testing framework, and not your production code (this could be avoided by some preprocessor, but why to build such a thing when the easier solution is to write the tests in a separate place)

  • tests of a function, a class, a library are typically written from a "users" point of view (a user of that function/class/library). Such "using code" is typically written in a separate file or library, and a test may be clearer or "more realistic" if it mimics that situation.

share|improve this answer

If tests were inline, it would be necessary to remove the code you need for testing when you ship the product to your customer. So an extra place where you store your tests simply separates between the code you need and the code your customer needs.

share|improve this answer
Not impossible. It would require an additional preprocessing phase, just as LP does. It could be done easily in C, or a compile-to-js language, for example. –  Chris Devereux Feb 25 '13 at 13:44
+1 for pointing that out to me. I've edited my answer to represent that. –  mri Feb 25 '13 at 15:23

This idea simply amounts to a "Self_Test" method within the context of an object-based or object-oriented design. If using a compiled object-based language like Ada, all the self-test code will be marked by the compiler as unused (never invoked) during the production compilation, and therefore it will all be optimized away - none of it will appear in the resulting executable.

Using a "Self_Test" method is an extremely good idea, and if programmers were really concerned with quality they would all be be doing it. One important issue, though, is that the "Self_Test" method needs to have intense discipline, in that it cannot access any of the implementation details and must instead rely only on all the other published methods within the object's specification. Obviously, if the self test fails, the implementation will need to change. The self test should be rigorously testing all the published properties of the object's methods, but never ever relying in any way upon any details of any specific implementation.

Object-based and object-oriented languages frequently do provide exactly that type of discipline with respect to methods external to the tested object (they enforce the object's specification, preventing any access to its implementation details and raising a compilation error if any such attempt is detected). But the object's own internal methods are all given complete access to every implementation detail. So the self test method is in a unique situation: it needs to be an internal method because of its nature (self test is obviously a method of the object being tested), yet it needs to receive all the compiler discipline of an external method (it has to be independent of the object's implementation details). Few if any programming languages provide the capability to discipline an object's internal method as if it were an external method. So this is an important programming language design issue.

In the absence of proper programming language support, the best way to do it is to create a companion object. In other words, for every object you code (let's call it "Big_Object"), you also create a second, companion object whose name consists of a standard suffix concatenated with the name of the "real" object (in this case, "Big_Object_Self_Test"), and whose specification consists of a single method ("Big_Object_Self_Test.Self_Test (This_Big_Object : Big_Object) return Boolean;"). The companion object will then depend upon the specification of the main object, and the compiler will fully enforce all of the discipline of that specification against the companion object's implementation.

share|improve this answer

We use inline tests with our Perl code. There's a module, Test::Inline, that generates test files from the inline code.

I'm not particularly good at organizing my tests, and have found them easier and more likely to be maintained when inlined.

Responding to a couple of the concerns raised:

  • The inlined tests are written in POD sections, so they're not part of the actual code. They are ignored by the interpreter, so there's no code bloat.
  • We use Vim folding to hide the test sections. The only thing you see is a single line above each method being tested like +-- 33 lines: #test----. When you want to work with the test, you just expand it.
  • The Test::Inline module "compiles" the tests to normal TAP-compatible files, so they can co-exist with traditional tests.

For reference:

share|improve this answer

Erlang 2 does actually support inline tests. Any boolean expression in the code which is not used (e.g. assigned to a variable or passed) is automatically treated as a test and evaluated by the compiler; if the expression is false, the code doesn't compile.

share|improve this answer

This is in response to a large number of comments suggesting that inline tests are not done because it's difficult to impossible to remove the test code from release builds. This is untrue. Almost all compilers and assemblers support this already, with compiled languages, such as C, C++, C#, this is done with what are called compiler directives.

In the case of c# ( I believe c++ as well, the syntax might've slightly different depending on what compiler you are using) this is how you can do it.

#define DEBUG //  = true if c++ code
#define TEST /* can also be defined in the make file for c++ or project file for c# and applies to all associated .cs/.cpp files */

//somewhere in your code
// debug only code
#elif TEST
// test only code

Because this uses compiler directives, the code will not exist in the executable files that are built if the flags are not set. This is also how you make "write once , compile twice" programs for multiple platforms/hardware.

share|improve this answer

Another reason for separating tests is that you often use additional or even different libraries for testing than for the actual implementation. If you mix tests and implementation, accidential usage of test libraries in the implementation cannot be caught by the compiler.

Also, tests tend to have way more code lines than the implementation parts they test, so you'll have trouble finding the implementation between all the tests. :-)

share|improve this answer

protected by World Engineer May 4 '13 at 23:45


Would you like to answer one of these unanswered questions instead?
