Computing lab demo tutorial: how to read feedback messages¶

If you prefer a video over reading the text below, you can watch video 42.

Introduction¶

For every assignment submissions, there will be (automatic) feedback on the correctness of the submission.

Checks are done by machine and thus objective and fair (i.e. exactly the same testing strictness is employed for all students). They are also strict.

You can submit your assignments at any time of the day. You can submit as often as you like. Ideally, you try to improve your assignment after every submission.

The testing system will stop testing for each task once it has come across a single check that has failed. Once you have fixed that problem, you should submit your assignment again.

The auto testing is carried out by a machine and as such the feedback it provides is expressed in a particular way that we need to learn to interpret.

This page and the examples below provide training how to read the protocol from failed checks, and how to understand what the error is that the autotesting system complains about. (This will not explain what is wrong in your code, it will just say that the way the code reacts to particular input values appears to be wrong.)

We encourage students to ask demonstrators in the laboratory sessions to explain the error messages you get from the feedback system, in particular in the beginning of the course.

How to read the feedback (if your solution is not perfect)¶

The messages from the autotesting system can tell you exactly the symptom of a problem/bug/missing feature in your code, and you should understand and use that information to use your time effectively when trying to debug your code. In principle, you should try to repeat the test that the testing system carried out at your computer, and then debug your code until that test produces the right output. We show some examples below.
We have tried to help you by adding comments in the testing code that outline what is being tested. These may be helpful to read as well. You may need to scroll up a little in the reported error from the testing system to come to understand what test is being carried out. Comments are line that start with a hash (#)
While the actual testing is carried out by a machine, the tests have been written by a human. It is possible that one or more are incorrect.

If you do not understand why your program fails a particular test, then there is a chance that the test is wrong. Please get in touch with demonstrators/lecturer to address this, and we will investigate.

This page: walk through of `add` exercise from `demo` assignment¶

On this page, we look at a hypothetical task of the lab called demo where students are asked to implement a function add.

We show a correct solution (i.e. a function add) to the task, and a number of incorrect solutions (each of which contains a mistake).

For all of these, we show the feedback that the feedback system would provide. In particular for the first few examples, we explain in great detail how to read the feedback from the system.

The task¶

Imagine we are given the following task:

Write a function add(a, b) that takes two numbers a and b and returns their sum.

Correct solution¶

A possible solution is given here:

def add(a, b):
  """This is my add function. It returns the sum of a and b."""
  return a + b

There is nothing wrong with this solution. The testing system would provide a report (by email), that looks like this:

Testing of your submitted code has been completed:

Overview
========

add                  passed   -> 1

Success rate for this lab: 1/1 = 100.0%

This tells us that one exercise has been tested (the exercise function was called add), and this has been tested ok. This means there were no mistakes found in that function.

Most labs have more than one exercises per submission, so there will be multiple lines of pass/fail information: one for each exercise.

In the following, we describe a number of mistakes and the associated error messages.

Function returning wrong value (function carrying out wrong calculation)¶

Imagine you have misunderstood the question and written the following code which computes the average value of a and b (where the sum was requested):

def add(a, b):
    """This is my function. It returns the mean of a and b."""
    return (a + b) * 0.5

The test system will provide a report like this:

 Overview
 ========
3
 add                  failed   -> 0
5
 Success rate for this lab: 0/1 = 0.0%
7
8
 Detailed test report:
10
============================= test session starts ==============================
platform linux -- Python 3.12.6, pytest-8.3.3, pluggy-1.5.0 -- /.pixi/envs/default/bin/python3.12
rootdir: /io
plugins: reportlog-0.1.2, timeout-2.3.1
timeout: 10.0s
timeout method: signal
timeout func_only: False
collecting ... collected 1 item
19
test_add.py::test_add FAILED                                             [100%]
21
22
  def test_add():
      assert s.add(0, 0) == 0
>      assert s.add(1, 2) == 3
E      assert 1.5 == 3
E       +  where 1.5 = <function add at 0x7ff136713560>(1, 2)
E       +    where <function add at 0x7ff136713560> = s.add
29
30
test_add.py:14: AssertionError
-------------- generated report log file: pytest_reportlog.jsonl ---------------
=========================== short test summary info ============================
FAILED test_add.py::test_add - assert 1.5 == 3
+  where 1.5 = <function add at 0x7ff136713560>(1, 2)
+    where <function add at 0x7ff136713560> = s.add
============================== 1 failed in 0.05s ===============================

We have added line numbers to the output above to make the explanation easier:

lines 1 provides a summary of all questions ‘passed’ (i.e. no mistake) or ‘failed’ (i.e. some mistake).
line 5 shows a list of all exercises and for each of those whether they have passed or failed. In this case, there was only one exercise (to write the function add) and thus there is only line 5 reporting that this function has failed a test.
from line 9 down, a detailed report is provided for every question that failed a check. In this case, there is only the testing of the add function and the report for this starts in line 12.
Lines 11 to 18 include diagnostic information which you can ignore.
Line 20 reminds us that some error was reported when testing the add function.
the actual code that carries out the testing is written in Python and shown starting in line 31. Because it is TESTing the function ADD, the test function is called test_add.
the testing system will go through a number of assert statements: each of those will check a certain property or behaviour that the function add should fulfil (the condition given to the assert statement should evaluate to True for a check to pass.)
the first such assert statement is in line 24: assert s.add(0, 0) == 0
- the file you submitted is known in the test system as s – you can think of this as s for Submission (or s for student). The function s.add() is thus the function you have written and submitted.
- line 24 the assert statement assert s.add(0, 0) == 0 will call s.add with arguments (0, 0) and expects that the result of this is the same as 0 (== 0).
- Our implementation of add is thus called with a=0 and b=0 and will return 0 (because the average of 0 and 0 is zero). This 0 is compared with the other zero (on the right hand sign of the comparison operator ==) and the check is passed.
This first check done here (i.e. checking that 0 + 0 = 0) does not find the bug in the function (which computes (a + b) * 0.5 instead of the required sum a + b) and is passed.
the next check (line 25) will fail though:
- line 25: the assert statement assert s.add(1, 2) == 3 will call s.add with arguments (1, 2) and expects that the result of this is the same as 3 (== 3).
In more detail, the assert statement
- calls the function s.add(1, 2) and replaces s.add(1, 2) with the value that the function returns.
- the return value of s.add(1, 2) is 1.5 (as we can see line 27 on the left of the comparison operator ==)
- the assert statement will compare this value (i.e. 1.5) with 3 (line 26) and notice that they are not the same, thus raising an AssertionError (the AssertionError is mentioned in line 31).
the submission system reports that a test has failed. In particular, we need to look for the greater than sign (>) in the left most column which points to exactly the assert statement that has failed. In this case, this is line 25.

Lines 26, 27 and 28 provide more details on intermediate steps in evaluating the assert statement that has failed.
So what has happened? The code we submitted returns 1.5 when given a=1 and b=2 because it (wrongly) computes the mean of a and b, rather than the sum. The testing system knows that the sum of 1 and 2 should be 3, and this is exactly the test carried out in line 25.

A single failed assert statement will result in an overall fail for that exercise. Once a test has failed, no further tests will be carried out for that function.

In the case here, the question you should ask (to find out what is wrong) is “why does my code return 1.5 where as it should return 3 for input arguments a=1 and b=2”. You can actually copy the test to Python prompt and run:

>>> add(1, 2) == 3

For the code with the bug shown above, this should display:

>>> add(1, 2) == 3
False

where as once the bug is fixed, you should see:

>>> add(1, 2) == 3
True

Often, it may be more instructive to not carry out the comparison, but just display the return value, i.e.

>>> add(1, 2)

where the incorrect response would be:

>>> add(1, 2)
1.5

and the correct answer should read:

>>> add(1, 2)
3

The feedback from the testing system should help you debugging your code.

Function not defined (no attribute `add`)¶

Let’s consider another possible mistake. Imagine we have implemented the right code, but given the function a different name:

def add_those_numbers(a, b):
    """This is my add function. It returns the sum of a and b."""
    return a + b

The feedback for this submission reads:

___________________________________ test_add ___________________________________
2
   def test_add():
>       assert s.add(0, 0) == 0
E       AttributeError: module 's' has no attribute 'add'

Here, the very first assert statement (line 4) fails: it as trying to call the function s.add() with arguments (0, 0). However, that function object does not exist: the only function object that does exist in s (which is the submitted file) is called add_those_numbers.
The system thus reports (line 5) 'module' object has no attribute 'add'.

The ‘module’ refers to the module s (i.e. your code), and the remainder of the message simply says: there is no add in module s.

As always, the greater than sign (>) shows the line in which an assert statement has failed, and subsequent lines give more information about the type of failure. In this case, the test fails because the function that should be tested (i.e. add) appears to be missing.

This kind of error will always be raised if a particular question has not been attempted.

Function returning wrong data type (string)¶

The function add needs to return the value of a+b. The type of a and b has not been specified in the instructions. We would thus expect the function to work at least for Python data types int and long (both integers), floating point numbers float and complex numbers.

(Dynamic typing will ensure that the return value is of the same type as the input arguments a and b (if a and b are of the same type), and if the plus operator can deal with all the types.)

Let’s assume the following mistake: the function add that we submit returns the result of the calculation as a string (this may seem like a good idea but it is not: if the result of a calculation is required as a string, it can be converted into a string later):

def add(a, b):
    """Takes numbers a and b and returns a+b."""
    return str(a + b)

This code would produce the following error message when being auto-tested:

___________________________________ test_add ___________________________________
2
    def test_add():
>       assert s.add(0, 0) == 0
E       AssertionError: assert '0' == 0
E        +  where '0' = <function add at 0x7efbfe537560>(0, 0)
E        +    where <function add at 0x7efbfe537560> = s.add
8
9
test_add.py:13: AssertionError

The > tells us that the assert statement that fails is in line 4. It attempts to compare s.add(0, 0) with 0.
line 5 (6 and 7) tells us that the call of s.add(0,0) results in the value '0.0' which is the compared against the number 0:
```
assert '0.0' == 0
```
Note the subtlety that '0.0' (on the left hand side of the comparison operator ==) is a string but 0 (on the right hand side) is a number.

They are thus not the same, and the test fails.
We summarise that it is important to pay close attention to the details as to why a particular assert statement has failed (in this case the clue is in line 5).

Function returning wrong data type (list)¶

A minor variation on Function returning wrong data type (string) is that the function returns [a + b] (which is a list with one element, and this element will be the sum of a and b) instead of a + b:

def add(a, b):
    """Takes numbers a and b and returns a+b."""
    return [a + b]

The error message reads:

  ___________________________________ test_add ___________________________________
2
   def test_add():
>       assert s.add(0, 0) == 0
E       assert [0] == 0
E        +  where [0] = <function add at 0x7fd1bc397560>(0, 0)
E        +    where <function add at 0x7fd1bc397560> = s.add
8
test_add.py:13: AssertionError

The revealing line is number 5 which shows that a comparison of [0] and 0 is attempted: the list with one element ([0]) is what s.add(0, 0) returned.

Function not returning a value¶

Another common mistake is that a function prints a value to the screen rather than returning it to the calling level using the return command.

Suppose our function would print the value of a+b instead of returning it:

def add(a, b):
    """Takes numbers a and b and returns a+b."""
    print(a + b)

The error report would read:

test_add
--------
def test_add():
>      assert s.add(0, 0) == 0
E      assert None == 0
E        +  where None = <function add at 0x838009c>(0, 0)
E        +    where <function add at 0x838009c> = s.add
8
  test_demo.py:4: AssertionError
------------------------------- Captured stdout -------------------------------
0

the assert in line 4 fails (we know that because of the > on the left)
line 5 contains the clue: the call of s.add(0, 0) evaluates to None (the special Python object None).

The comparison of None with 0 fails (i.e. None == 0 evaluates to False).
The underlying problem is that the submitted definition of the add function does not use the return keyword to return the value of a + b (instead the value is printed). Any function not using the return keyword will return the special value None.
Remark: Note that in this case (where the function prints something) the new section Captured stdout appears in line 10: this shows the 0 that was printed when s.add(0, 0) was called.

Function not having a docstring¶

Suppose the function is implemented correctly, but no docstring has been provided:

def add(a, b):
    return a + b

The error message reads:

___________________________________ test_add ___________________________________
2
    def test_add():
        assert s.add(0, 0) == 0
        assert s.add(1, 2) == 3
        assert s.add(0, 1) == 1
        assert s.add(1, 1) == 2
        assert s.add(1.5, 1.0) == 2.5
9
       # Does the function have a documentation string?
>       assert s.add.__doc__ is not None
E       assert None is not None
E        +  where None = <function add at 0x7f28a5023560>.__doc__
E        +    where <function add at 0x7f28a5023560> = s.add

We observe that

all the numerical tests (lines 4 to 8) are passed.
the assert statement in line 11 fails
line 10 contains a comment we have put before the test to give a hint about the following test. It suggests that the test in the next line is to see whether the function has a documentation string.
line 11 reads:
```
assert s.add.__doc__ is not None
```
The object that is being tested here is s.add.__doc__, i.e. the __doc__ attribute of the function add in the submitted file s.

This is the documentation string: if it is defined, it is a string. If it is undefined, this attribute s.add.__doc__ is the special Python value None.

The documentation string is what is displayed when we pass the function object add to the help function, for example:
```
>>> help(add)
```
The comparison operator (!=) checks for inequality, i.e.
```
s.add.__doc__ is not None
```
will evaluate to be True if s.add.__doc__ is NOT None, i.e. if the s.add.__doc__ exists (so all is well, because then the documentation string is defined; and this is what the test is for).

If, however, s.add.__doc__ is None (because it was not defined as in our example here), then the whole expression s.add.__doc__ is not None evaluates to False and thus the assert statement fails.

Occasionally, assert statements are preceded by Python comments in the line above (as here where the comment is in line 10 and the assert statement in line 11). These comments have been placed there to help understanding what is being tested: they are worth reading – often there is no need to look into the detailed error message if that line provides enough context, or at least knowing about the context will make understanding the purpose of the assert statement easier.

When the source code file has syntax errors¶

The testing system will try to import the student file under the name s before running any tests. I.e. if the submitted file is demo.py, then the testing file will want to run a command like:

import demo as s

If there are syntax errors in the demo.py file, this will be impossible, and the testing cannot even be started. For example, here is a “Python” program with incorrect syntax:

define add(a, b):
  """This is my function. It returns the mean of a and b."""
  return a + b

The error is the use of define instead of the correct term def in line 1. (Many other syntax errors can be made, of course.)

For this example (and more generally if the import of the demo.py file fails), an email along these lines is returned:

Error importing files
=====================
3
File 'demo.py'
--------------
Traceback (most recent call last):
 File "<string>", line 1, in <module>
 File "/io/s.py", line 1
   define add(a, b):
         ^^^
SyntaxError: invalid syntax

The error message is trying to tell us that there is a SyntaxError (line 11) and the three hats pointing upwards (^^^) point to to line 9 to indicate the error has been noticed in that line. (Not always is the error at the point the interpreter points to, but often it is correct or at least close to the problem.)

The best way to avoid/fix this type of import error, is to make sure that the file demo.py is valid Python, i.e. executing it does not raise any error messages.

Writing code is easy…¶

… but debugging it is (a lot) harder.

Please seek help for interpreting error messages if needed: the error messages you receive in the feedback are standard Python error messages. Learning how to read those will benefit you for the practical exercises and also far beyond this module.