Testing tool for PATWIC
Grader is a framework for creating coding exercises in Python. A coding exercise results in a type that can be used to decorate a callable with extra functionality. The behaviour of the decorated callable can be automatically verified and tested against the exercise’s constraints and expectations.
Setting up a Development Environment
0. Ensure that virtualenv and pip are installed (already installed on the terminal servers)
1. Create a virtualenv (only needed if it already doesn’t exist)
virtualenv -p python3 ~/venv/grader
2. Activate the virtualenv
Activating the virtualenv will start a new shell process with updated PATH parameters pointing to the bin directory in the virtualenv. This bin directory contains a python binary wrapper that will only use the standard library modules and any modules installed in the virtualenv.
3. Clone the grader git (if not already done)
mkdir ~/grader-dev cd ~/grader-dev git clone firstname.lastname@example.org:patwic/grader.git
5. Check out the assets git and install it into the virtualenv(use life here as example)
cd ~/grader-dev git clone email@example.com:patwic/life-assets.git cd life-assets pip install -e .
6. Checkout the git for the exercises, which the students later will clone
cd ~/grader-dev git clone firstname.lastname@example.org:patwic/life.git cd life
At this point, a python interpreter will have access to the life code as it is in the current directory, but the grader and life-asset code will also be on the PYTHONPATH via the virtualenv.
Creating an Assignment
A grader assignment consists of a number of exercises, each of which has a number of tests that need to pass.
To create an assignment you follow these steps:
- Create a reference implementation of an exercise
- Create some test cases
- Create the exercise by supplying input data, tests and implementation.
- Create the assignment by supplying exercises.
Write a reference implementation
A reference implementation is any callable object. Let’s assume that the assignment includes creating a function that can add two integers. The reference implementation might look like below.
def ref_add(a, b): return a + b
Create test cases
A test case is created using an instance of the class TestBuilder, which has several methods for building a test piece by piece:
- The constructor - Takes two arguments, the latter of which is optional: a string that is a name or description of the test, and the expected result of the test.
- with_exp - Sets the expected result of the test, in case this wasn’t done in the constructor.
- with_args - Takes any number of positional arguments to be used as input to the callable being tested.
- with_kwargs - Takes any number of keyword arguments to be used as input to the callable being tested.
- with_hint_tests - Takes any number of HintTest objects (see below) to provide specialized hints for special cases.
- with_func - Takes a function to be used to determine the success of the test. This function should be callable with the syntax func(result, expected, *args, **kwargs) where args and kwargs are set by the next two methods below, and return a boolean showing if the test succeeded and a string to be displayed if it failed. If with_func is never called, a simple equality check is used as a default.
- with_func_args - Takes any number of positional arguments to be used with the test success check function set with with_func.
- with_func_kwargs - Takes any number of keyword arguments to be used with the test success check function set with with_func.
Once all the wanted attributes have been added, the create method returns the finished test.
Following the previous example we create some tests for our add exercise.
test1 = TestBuilder("add 1 and 2", 3).with_args(1, 2).create() test2 = TestBuilder("add 2 and 3 as kwargs").with_exp(5).with_kwargs(b = 3, a = 2).create() def close_enough(result, expected, max_diff): return (abs(result - expected) <= max_diff), "Difference too large" test3 = TestBuilder("4 + 2 is in the range 6 +/- 2", 6).with_args(4, 2).with_func(close_enough).with_func_args(2).create()
Create hint tests
A hint test is an object containing a function and a hint string, and it is used to provide specialized hints for special cases. It is created using the HintTest constructor, which takes four arguments (the last three of which is optional):
- A string containing the hint to be displayed if the hint test passes.
- A function determining whether or not to display the hint. It should be callable using the syntax func(result, *args, **kwargs) where args and kwargs are the next two arguments in the constructor. If not provided, it defaults to an equality check between result and a single parameter to be provided in args.
- A list of positional arguments to be used in the function.
- A dict of keyword arguments to be used in the function.
def less_than(a, b): return a < b hint1 = HintTest("1 + 1 != 1", args = ) hint2 = HintTest("1 + 1 > 0", less_than, ) test4 = TestBuilder("add 1 and 1", 2).with_args(1, 1).with_hint_tests(hint1, hint2).create()
Create an exercise
Similar to the test cases, an exercise is created using an instance of the class ExerciseBuilder, which has the following methods:
- The constructor - Takes two arguments: a string that serves as a description of the exercise, and the reference implementation object.
- with_tests - Adds any number of Test objects to the exercise. The tests are run in the same order as they are added.
- with_hint - Adds a string that is shown as a default hint if a test fails and none of its HintTest hints are shown.
Again, the create method is used to get the finished exercise.
exercise_1 = ExerciseBuilder("add two integers", ref_add).with_tests(test1, test2, test3, test4).with_hint("Use plus").create()
Create an assignment
First, create an ExerciseDict object and add all exercises to it. The add method takes two arguments: the exercise itself and an ID to be associated with it. Then, call the create method on the Assignment class object. It takes three arguments, the last of which is optional: a string to serve as the assignment name, a dict of id: Exercise pairs (which can be easily acquired by calling the get_dict method on the ExerciseDict object), and (optionally) a function to be executed if all tests in all exercises succeed.
exercises = ExerciseDict() exercises.add(exercise_1, 1) assignment = Assignment.create("example", exercises.get_dict())
Testing the exercise
The reference implementation is used to self-test the exercise when a solution is created. If the reference implementation does not satisfy the tests the creation will fail.
The Runtime object
During the execution of the tests, information will be stored in a Runtime object that can be accessed at any time with the function getRuntime (which needs to be imported from the grader module). The main use for this when creating an assignment is that it can be used to access the function that is currently being tested through the method get_func, which means that a function provided to a Test or HintTest can determine whether the test succeeds depending on the code of the function being tested (using e.g. the inspect module) rather than just its output.
Using an assignment (as a Student)
Assume this assignment (in a module called arithmex)
from grader import * def ref_add(a, b): return a + b test0 = TestBuilder("adding 1 and 2 should be 3", 3).with_args(1, 2).create() test1 = TestBuilder("adding 5 and -3 should be 2", 2).with_args(5, -3).create() exercises = ExerciseDict() ex = ExerciseBuilder("Add function", ref_add).with_tests(test0, test1).with_hint("some hint").create() exercises.add(ex, 1) arit = Assignment.create("ass1", exercises.get_dict())
The arit object is a decorator that takes an exercise id. Students use the decorator to mark their solution to a specific exercise:
Here is the code a student would write.
from arithmex import arit @arit(1) def my_add_solution(x, y): res = x + y return res
The arit object has a test method that can take an optional id. Calling this method will test all exercise solutions or the one who’s id was given:
The testing can also be done directly from the terminal. The below two commands are mostly equivalent to the function calls above, with the slight difference that the second command performs separate calls to test for each exercise.
grader filename.py arit.1 grader filename.py
An exercise’s description can be shown by calling the show_instr method. This method shows all exercises in a solution or if given an id as argument the instructions for that exercise.
Releasing Assets Modules
1. Create or update the setup.py file
To define a releasable packet, Python requires a setup.py file to be present in the same directory as the source code.
Example of a setup.py file is show here:
#!/usr/bin/env python __author__ = 'dod' from setuptools import setup setup(name='func_assets', version='1.1', description='Function exercises for PATWIC', py_modules=['numbers_tests', 'races_tests', 'binary_tests', "simple_tests"], )
At least the name, version, description and py_modules attributes must be defined.
If the file already exists, then update the version number to the new release version.
Add and commit the updated files.
2. Set an annotatated git tag
Mark the release in git with:
git tag -a <version>