CodeRunner Documentation (V3.3.0)

13 Supporting or implementing new languages

Most authors seem to assume that if they wish to write CodeRunner questions that use a language not directly supported by the Jobe sandbox, they must first modify Jobe to support the new language. That is not the case. A much easier, more convenient and more maintainable approach is to use a Python question type that compiles (if necessary) and runs the student code in a Python subprocess. Indeed, in recent years at the University of Canterbury nearly all new question types have been scripted in Python, even those using languages built into Jobe. For example, the question type we now use for all question in the C programming course is a Python3 question type parses the student's C program and performs lots of checks on things like function length and use of various C constructs before compiling and running the submitted code.

The template code below shows a simple example in which a Python question prototype is used to define a new question type c_via_python that mimics the built-in c_program question type but provides more flexibility. To create the new question type using this template:

  1. Create a new CodeRunner question.
  2. Choose the question type python3
  3. Click Customise
  4. Replace the contents of the Template text area with the template code below.
  5. Uncheck the Is combinator checkbox
  6. Enter DEMO_PROTOTYPE_C_using_python as the question name
  7. Enter whatever text you wish to use to describe the question type in the Question text area. This text will be displayed to any authors using this new question type if they open the Question type details section of the question authoring form.
  8. Open Advanced Customisation
  9. Set Is prototype? to Yes (user defined)
  10. Set Question type to c_via_python.
  11. Set Ace language to c, so that the students' code will be edited as C even though the prototype is in Python.
  12. Save the question.

You should now find the new question type c_via_python appearing in the Question type dropdown of the author edit form for a new CodeRunner question. This new question type should behave like the built-in c_program question type but is more flexible; for example, it can easily be extended to perform checks on the submitted C code prior to compilation.

The full question prototype for the c_via_python question type is included in the samples folder of the CodeRunner distribution.

""" The template for a question type that compiles and runs a student-submitted
    C program. 

import subprocess, sys

# Write the student code to a file prog.c
with open("prog.c", "w") as src:
    print(student_answer, file=src)

# Compile
{% if QUESTION.parameters.cflags is defined %}
cflags = """{{ QUESTION.parameters.cflags | e('py') }}"""
{% else %}
cflags = "-std=c99 -Wall -Werror"
{% endif %}
return_code ="gcc {0} -o prog prog.c".format(cflags).split())
if return_code != 0:
    print("** Compilation failed. Testing aborted **", file=sys.stderr)

# If compile succeeded, run the code. Since this is a per-test template,
# stdin is already set up for the stdin text specified in the test case,
# so we can run the compiled program directly.
if return_code == 0:
        output = subprocess.check_output(["./prog"], universal_newlines=True)
    except subprocess.CalledProcessError as e:
        if e.returncode > 0:
            # Ignore non-zero positive return codes
            if e.output:
            # But negative return codes are signals - abort
            if e.output:
                print(e.output, file=sys.stderr)
            if e.returncode < 0:
                print("Task failed with signal", -e.returncode, file=sys.stderr)
            print("** Further testing aborted **", file=sys.stderr)