<p>This is currently a very active field in programming languages research.</p>

<p>I know of these two different approaches that look at the problem:
 - <a href="http://arxiv.org/pdf/1409.0166.pdf" rel="nofollow">http://arxiv.org/pdf/1409.0166.pdf</a>
 - <a href="http://research.microsoft.com/en-us/um/people/sumitg/pubs/cacm14.pdf" rel="nofollow">http://research.microsoft.com/en-us/um/people/sumitg/pubs/cacm14.pdf</a> (this is actually only one of very many papers by Sumit and his group)</p>

<p>You may want to look at these things to find something that could help with your problem (and edit this answer to make it more useful).</p>
Python evaluate and grade code from students
<p>For a class, I would like to automatically evaluate (parts) of the coding assignments of students. The setup I had in mind is something like:</p>

<ol>
<li>Students get a class skeleton, which they have to fill in. </li>
<li>Students ``upload'' this class definition to a server (or via webinterface)</li>
<li>The server runs a script an test on specific functions, eg class.sigmoid(x), and checks if the output of the function is correct and might give suggestions.</li>
</ol>

<p>This setup brings a whole lot of problems, since you're evaluating untrusted code. However, it would be extremely useful, for many of my classes, so I'm willing to spend some time in thinking it trough. I remember Coursera had something similar for matlab/octace assignments, but I can't get the details of that.</p>

<p>I've looked at many online python interfaces (eg, codecademy.com, ideone.com, c9.io); while they seem perfect to learn and or share code, with online evaluation. I do miss the option, that the evaluation script is "hidden" from the students (ie the evaluation script should contain a correct reference implementation to compare output on random generated data). Moreover, the course I give requires some data (eg images) and packages (sklearn / numpy), which is not always available.</p>

<p>Specifically, my questions are</p>

<ol>
<li>Do I miss an online environment which actually offers such a functionality. (that would be easiest)</li>
<li>To set this up myself, I was thinking to host it at (eg) amazon cloud (so no problem with infrastructure at University), but are there any python practices you could recommend on sandboxing the evaluation? </li>
</ol>

<p>Thanks in advance for any suggestions!</p>

<hr>

<p>Pity to hear that the question is not suitable for StackOverflow. Thanks to the people (partially) answering the question.</p>

<p>After some more feedback via other channels, I think my approach will become as follows:</p>

<ol>
<li>Student gets skeleton and fills it in</li>
<li>Student also has the evaluation script.</li>
<li>In the script, some connections with a server are made to
<ul>
<li>login  </li>
<li>obtain some random data</li>
<li>check if the output of the students code is numerically identical to what the server expects.</li>
</ul></li>
</ol>

<p>In this way the students code is evaluated locally, but only output is send to the server. This limits the kind of evaluations possible, but still allows for kind of automatic evaluation of code.</p>