I've seen too many cases when a student has completed a programming exercise (without cheating) but can't say, for example, what type of values a variable contains during program runtime (in case of Python programs).

It looks like the ability to solve exercises should not be the only (or even main?) criteria for assessment.

Do you know any research papers (or books or posts) that tackle this issue and/or recommend other assessment techniques / exercise types?

  • $\begingroup$ Comments are not for extended discussion; this conversation has been moved to chat. $\endgroup$ – thesecretmaster Aug 26 '17 at 10:49
  • $\begingroup$ IMHO, if a student can prove that their program satisfies a specification, they understand the program. Proof of correctness requires knowledge of types of values, conditions on values, and much more. $\endgroup$ – beroal Sep 10 '17 at 11:12

If I understand your problem correctly, it's that students can create programs that behave correctly without understanding why they behave correctly. I assume that they do this by some combination of brute force trying things and SO search.

Using program correctness for grading has the desirable qualities of being objective and automation friendly.

So how could you keep these qualities while testing for actual understanding? While I agree that good code structure, variable naming, and possibly comments can offer proof of understanding, they are all highly subjective.

What I would do instead is construct exam exercises that are relatively small requirements changes to homework assignments that students have completed earlier, ideally at least 2 weeks earlier. Then give each student access to the code (s)he submitted for the homework as a starting point for completing the exam exercises. Tune the original homework size and the exam requirements changes so that a student who understands the code and has well-written code can complete the exam exercise quickly, but one that started from scratch would not likely be able to complete the work. This accomplishes 4 things:

  1. It solves your original problem of testing whether students understand the code they have written.
  2. It maintains the objectivity and automation friendliness of grading on program behavior.
  3. It rewards production of readable, maintainable code.
  4. It prepares students for the world of coding outside of a class environment, where the majority of time spent is not on isolated exercises, but extending an existing code base.

Disclaimer: I work in industry and have no research to support this.

  • 5
    $\begingroup$ I'm also an engineer and not an instructor, but I'd be concerned about using specific homework assignments as a basis for an exam. If you have a student who struggled with the homework assignment, then they're already at a large disadvantage for the exam. Exam questions should probably stand on their own and the students are either able to answer them or unable to answer them. Instead of using the student's homework, simply provide your own program code instead. $\endgroup$ – Ellesedil Aug 25 '17 at 21:29
  • $\begingroup$ @Ellesedil It's true that students could be punished twice for doing a bad job on homework. I would mitigate that by not making too much depend on any one piece of homework or test question. Also, you could communicate how you test ahead of time, and have office hours or class time set aside for improving homework even after it has been turned in, which would improve the code for the exam, but not for the original homework grade. Generally, I think the issue can be managed/mitigated. Using your own code would be ok sometimes, but doesn't incent students to create readable, maintainable code. $\endgroup$ – ShawnMartin Aug 28 '17 at 14:40

I have seen my share of this 'program gets output' but the programmer has no clue how she/he got there. It's funny how that happens so many times.

This is what I have done to at least handle the issue.

Before assessments, I break down the evaluation to include the following.

  • Simply getting the output gets them the bare passing grade.
  • Those who can use meaningful names for their variables, function calls get extra.
  • Those that can back up their loops, method calls and variable declarations with expansive ( 2 to 3 lines at least) comments get extra points.
  • In addition to that, those that can write psuedocode above the program code (perhaps in a big block of comments) get some more.

Obviously, this makes me their least favourite faculty in terms of evaluation and gets me a lot of complaints but I stick with it.

Update : Please note that, during evaluation, I don't expect all of them to be done. Say, a student is already using meaningful names everywhere, I would not mark her/him down for not writing comments.

  • $\begingroup$ Comments are for suggesting improvements; this conversation has been moved to chat. $\endgroup$ – thesecretmaster Aug 26 '17 at 10:46

I am wondering how much of this is because they can not express in natural language (don't know terminology). How much is because of just fiddle until it works programming.

Learning to express in natural language

This is important, to allow them to communicate with a larger team, to allow them to look stuff up on an internet search, and to answer some of the exam questions.

Programming by fiddling / tinkering

This is an important technique to use some of the time. However it should then be followed by evaluation.

For example: Trying to find the angle of a triangle, when drawing using turtle graphics.

  • Made a first guess (common first guesses: 30, 60, 45, 90, 100 )
    • Try it.
    • Decide higher of lower (this is often decided wrong, but that is ok)
    • Guess a new value.
  • repeat until you find the answer.
  • record answer with result for square.

Table of angles:

| number of sides | angle |
|        3        |       |
|        4        |  90   |
|        5        |       |
|        6        |       |
|        7        |       |

You can discuss what the best strategy is [binary search]. One you have a completed table (4 or 5 entries), you can start to look for a pattern. It there a general formula for a polygon. Also getting pupils to pretend to be the turtle can help, or teacher is turtle. In ether case another pupil is the instructor, that tells turtle to move and turn. While doing this they gain a deeper understanding of how the code works.


I'm afraid my answer here will suggest that you completely revamp how you teach.

The sort of problems that result in issues like this, seem to me to be problems that treat the computer as a fancy calculator. Problems given to students are of the "math-y" type. Some require tricky thinking, of course, but they are unlike the sorts of problems that people in the real world write programs to solve.

My suggestion, is to, instead, use simulation as the basis of your teaching. There are many ways to do this, but one of the best and easiest to introduce is the Greenfoot system. It provides both an IDE (for Java) and a simulation framework.

There is also an organization for users, Greenroom, who contribute simulation frameworks that you can start with and modify. There are hundreds of such simulations, some with teaching materials, even videos. Note that the Greenroom is a membership site. You will need to join.

Here are a few examples. I have used some of them, but not all.

Create Flappy Bird Game

Fuel Depot Question from APCS-A

Greeps - A Programming Competition One of the originals.

Karel J Robot meets Greenfoot A robot simulation - from the book.

2d Platformer Similar to Mario.

There are hundreds more, both at the main Greenfoot site and the Greenroom. Only the Greenroom requires (free) membership. With these sorts of frameworks and the programming that it involves, the problem you discuss simply won't arise.

It has been noted that I often promote Greenfoot. I'm not affiliated with the site or its developers, nor have I ever been. I know some of the developers and I have produced materials for use with Greenfoot. I also occasionally submit bug reports and feature requests to the site.

But Greenfoot aligns well with my general teaching philosophy and makes Object-Oriented Programming easy to teach to novices in an interesting and engaging way.


It's time for a little "code review." Have a student present his code in front of class and talk about how he made it work. Hey, this happens in the professional world. There is no time like the present to begin learning this vital skill.

  • 1
    $\begingroup$ @Davor I understood the answer to mean that every student would present the same amount of times, with a random or fair algorithm used to choose the presenter; not, "your code was especially bad this week, stand up there so we can make fun of you." In other words, part of the assignment is understanding your code well enough to explain it to others. $\endgroup$ – drewbenn Aug 25 '17 at 20:47
  • 2
    $\begingroup$ @drewbenn - even if that was the case, id adds no value to someone's education to be forced to present his work in front of audience. And no, despite many claims of the opposite, you don't really do that when working as a programmer. $\endgroup$ – Davor Aug 25 '17 at 20:54
  • 2
    $\begingroup$ @Davor I disagree with both your statements. But that's why we get to vote on answers here :) $\endgroup$ – drewbenn Aug 25 '17 at 21:02
  • 1
    $\begingroup$ @drewbenn - unfortunately cs academics drown out actual people from the industry here, so the answers you get are the same nonsense you get in the academic setting in the first place. "Democracy" has severe downsides. $\endgroup$ – Davor Aug 26 '17 at 20:54
  • $\begingroup$ @davor Learning to present your ideas is an important skill. This often brings interesting discussion of design for programs that is both productive and useful. NCSSM has a nice environment; kids do not have anything to fear going in front of their peers. They are used to this. $\endgroup$ – ncmathsadist Aug 27 '17 at 1:04

I also encounter this issue (just encountered it yesterday in a lab exam). This is how I differentiate between someone who has done his work and just not been able to explain it and someone who has crammed/cheated:

  1. Deleting some part of code and asking student to rewrite it
  2. Asking student to modify a snippet of code
  3. Explain this line

Usually Point 2 is enough to differentiate. Kindly ensure to let him/her sit calmly in order to not let nerves get hold of him/her - it works for me!

  • 1
    $\begingroup$ I'm using the second point as the base for grading first-year computer science students. In the setting I usually teach, students prepare their solutions as a homework, so it's all too easy for them to just find a solution on the web; so they don't get a grade for just providing a solution that works. Most of their grade comes from implementing small modifications to their code (e.g. "Please change the order from ascending to descending" for an assignment on sorting) and bonus points for code clarity (if they can explain why; typical question is: "why is this identifier named this way?"). $\endgroup$ – liori Aug 25 '17 at 14:13
  • 1
    $\begingroup$ I got a very clear example of #1 back when I was a lab assistant. This was in the era of floppy disks, on single drive machines backup meant a lot of swapping. The idiotic backup routine couldn't sense the wrong disk in the drive and someone not paying attention could easily copy the target directory over the source--deleting everything. I knew how to rebuild them and thus got a fair number of students crying for help. (Continued) $\endgroup$ – Loren Pechtel Aug 25 '17 at 21:06
  • 1
    $\begingroup$ Everyone would freak when I sat them down in front of a binary dump of their data and asked them to identify it but after a bit of paging through it they would start recognizing pieces and I would build up the map I needed. One gal absolutely couldn't get it. Somehow it wasn't a surprise when later I found out she had copied everything. $\endgroup$ – Loren Pechtel Aug 25 '17 at 21:12

Isn't this what lab reports are for. Computer Science is a science and should come with some basic rigors. Copying some code and hacking away might get an output, but forcing students to do analysis, flow charts, etc will prove how much they really understand the process. Even simple things like writing a paragraph on the theory and another on the basic algorithm to solve the problem would go a long way.


Minimal: 'Provide a code solution', (does not necessarily need to be working).

Slightly better: 'Show how your answer works, or does not work, with detailed explanation and/or comments.

Best: 'Show how you tested your solution, describe what problems you found, and how you tried to debug and fix them'.

Awarding a large marks boost for details of testing and debugging renders SO/web copypasta answers much less useful. Testing and debugging are absolutely essential skills when developing software, and the sooner that students practice those skills, the better for us all.

The number of SO posts with non-trivial code, but demonstrating no debug skills whatsoever, is embarrassing. Too many cannot, (or will not), summon up enough programming skill to print out variables that may help diagnose a problem, never mind acually use a debugger. Educators should be pressing home the message 'If you cannot debug, you cannot develop programs'.


protected by thesecretmaster Aug 27 '17 at 10:21


Would you like to answer one of these unanswered questions instead?
