cs136l module 2.

Terminology.

• Testing: Checking over your code for errors and bugs.

• Debugging: Act of finding the cause of the bug and fixing it is called debugging.

They are NOT the same.

Black-Box Testing: Refers to tests where the tester has no/minimal knowledge of the implementation

• testing edge-cases ( corner of ranges )

White-Box Testing: Refers to tests where the tester HAS knowledge of the implementation

Unit Testing: Specific Components of a program are tested in isolation ( specific helper or one part of function is testing )

Regression Testing: Philsophy that after making changes to code, all tests should be rerun as the changes can introduce new defects

Complie Error: Not a bug. Your program does not complie since it doesn't follow the rules of language.

Runtime Error: A Bug. Occur during execution of a program where an illegal operation happens e.g. dividng by zero.

Logical Error: A Bug, occurs when your program produces unexpected behaviour.

• Can Run but It doesn't have right output.

Memory Error: Occurs when your program ends up using your memory in an unexpected/illegal way.

It is common for other developers besides yourself to test your code !

• Learn about Linkers.

1. Tips and Tricks to avoid bugs.

   1. Create/think list of test-cases and edge-cases BEFORE making the program.

Assert Function

• Stops the program and outputs something if some condition holds false.

• Tool for testing.

  • Confirm pre- and post- conditions for a function

  • Return a boolean expression assessing correctness

• Asserts are like check-expect in racket

// constants that describe the maximum window size
const int SCREEN_WIDTH = 300;
const int SCREEN_HEIGHT = 500;

// global variables for the current rectangle being drawn
int rect_width = 0;
int rect_height = 0;
int start_x_coordinate = 0;
int start_y_coordinate = 0;

// create_rectangle() obtains input from user and creates a new rectangle
// that will be completely visible in the available window
// effects: modifies rect_width, rect_height,
//          start_x_coordinate, start_y_coordinate
// While C has more elegant ways of doing this, we use global variables 
// because students have not yet learned other features
void create_rectangle(void){
  // .....
}

int main(void){
  create_rectangle();
  assert(start_x_coordinate >= 0);
  assert(start_y_coordinate >= 0);
  assert(start_x_coordinate + rect_width <= SCREEN_WIDTH);
  assert(start_y_coordinate + rect_height <= SCREEN_HEIGHT);

  // more code
}

//sample tests
assert(total(-1,0,0,0,0)==-1); // illegal value for hundred's
assert(total(0,0,-5,0,0) == -1); // illegal value for twenty's
assert(total(0,0,0,0,0) == 0); // corner case, all ranges at lowest value
assert(total(1,0,0,0,0) == 100); //corner case
assert(total(0,1,0,0,0) == 50); //corner case
assert(total(0,0,1,0,0) == 20); //corner case
assert(total(0,0,0,1,0) == 10); //corner case
assert(total(0,0,0,0,1) == 5); //corner case
assert(total(1,1,1,1,1) == 185); //lowest positive values  
assert(total(1,1,1,1,0) == 180); //edge case, five's are zero 
assert(total(1,1,1,0,1) == 175); //edge case, ten's are zero 
assert(total(1,1,0,1,1) == 165); //edge case, twenty's are zero 
assert(total(1,0,1,1,1) == 135); //edge case, fifty's are zero 
assert(total(0,1,1,1,1) == 165); //edge case, hundred's are zero 
assert(total(4,8,15,16,23) == 1375); //random values, or are they? hmmm 

• Note this is an example of black-box testing. Not directly testing every path based on the code, instead generic cases that are generally good for all types of codes ( i.e. edge cases etc... )

Keep testings convientely together for a quick way to turn them off in production ( or keep them up ).

const bool perform_tests = true;
// tester() is a helper function that has assertion-based tests 
// for the sqrt() function
void sqrt_tester(void){
  assert(sqrt(49) == 7);
  assert(sqrt(25) == 5));
  // etc.
}
int main(void){
  if (perform_tests){
    sqrt_tester();
  }
  //program code goes here
}

(assert [condition]), if condition is false, breaks!

I/O Testing

• Running the program to be tested with different test inputs.

• Test Output compared with Expected outputs.

Main focus:

• writing good tests ( what does this mean ? )

• automate I/O testing

Test Harness:

• Used for testing specific parts of the program, without having the other parts of the program being completed.

• Program used to test a specific functionality of a program or subset.

• We can explicitly choose which part of the code to test with these harnesses.

E.g.

#include "cs136.h"

// sqr(n) computes n^2
int sqr(int n) {
  return n * n;
}

// abs(n) computes |n|
int abs(int n) {
  if (n < 0) {
    return -n;
  } else {
    return n;
  }
}

int main(void){
  int n = read_int(); // read which operation to perform
  if (n == READ_INT_FAIL){
    return 0;
  } else if ( n == 1 ) { // testing square
    int val = read_int();
    printf("Squaring %d gets us %d\n",val,sqr(val));
  } else if ( n == 2) { // testing abs
    int val = read_int();
    printf("Absolute value for %d is %d\n",val,abs(val));
  } else {
      return 0;
  }
  return 0;
}

---

While using Test Harnesses, try not to test the harness instead of code

Basically don't get caught up making test-harness perfect, its merely there to test the actual useful code.

First step in automating I/O testing:

1. Create a new file for different tests

In CS136/136L, if an input is stored in a file to be used for a program, it typically ends with the .in extension. This is the convention. Expected output files end with .expect

e.g.

$>cat sqr_test1.in
1 5
  
$>cat sqr_test1.expect
Squaring 5 gets us 25

$>cat abs_test1.in
2 -42

$>cat abs_test1.expect
Absolute value for -42 is 42

Two ways to run these tests in ".in" and ".expect" files.

1. edX environment

WHEN "Run with Tests" is called, it scans for every unique pair of .in and .expect files.

And, for a hypothetical pair, test.in, and test.expect, edX runs the following command with the pair:

./myprogram < test.in > test.out

After,

diff test.out test.expect

The diff command checks the difference between the two files.

For a given pair of test files, say mytest.in and mytest.expect, "mytest" is the STEM.

1. Second way of automating testing, USING LINUX SHELL

• Easiest way is to just do what edX does.

• You have to manually do it tho.

• Later we will be learning SHELL Scripts, for automation of test-checking, just like the RUN TESTS btn of edX.

Improvements to the previous test-harness:

$>cat test1.in
sqr 5
sqr 10
abs -42
abs 0

$>cat test.expect
Squaring 5 gets us 25
Squaring 10 gets us 100
Absolute value for -42 is 42
Absolute value for 0 is 0

which focused more on readability for other people !

the issue?

• Making one test test many things, will mean a lack of clarity of finding out the root of the program later on.

Writing short tests that test for one specific functionality is prefered as the failure of such a test quickly tells us what is wrong.

Don't try to test unspecified/undefined behaviour— behaviour not made clear in problem-statement or inputs that are not addressed, such test would be meaningless.

Make sure tests are correct

asserts are typically safer and better tests, as one doesn't have to look where code was broken !

BUT asserts can only test through conditional statements, and cannot test SIDE-EFFECTS like output on the output stream.

Try to actually break your code. Maintain the mentality to find and learn from "GOTCHA" moments.

Since it is evidence of correctness that we are looking for, we can view the testing of software as a jury or judge views evidence: the better quality of evidence (not necessarily the greater quantity), the stronger the case that the program satisfies the requirements. As the programmer, we are the person on trial: innocent until proven guilty. As the tester, we are the accuser: we must provide substantial evidence that the program does not meet the specifications. If strong evidence cannot be presented to this end, the program is deemed to be "correct enough" and it is set free into the world. - Troy Vasiga

#include "cs136.h"

// total(hundreds, fifties, twenties, tens, fives) returns the total amount of money being 
//   deposited. The function returns -1 if any of the parameters has an illegal value
int total(int hundreds, int fifties, int twenties, int tens, int fives);

// read_and_execute_test(hundreds,fifties,twenties,tens,five) is a recursive function that 
//   at each recursive step will read a number. If the number is -1 the function calls the 
//   total function. Instead, if the number read is a valid denomination, the function reads 
//   another number which will represent the number of notes of that denomination. The function 
//   recurses until it encounters -1. For first integer inputs other than -1 and valid 
//   denominations, the function returns -1
int read_and_execute_test(int hundreds, int fifties, int twenties, int tens, int fives){
  int n = read_int(); // read the denomination or end of test
  if (n == READ_INT_FAIL){
    // read failed
    return -1;
  }
  if ( n == -1){
    // this indicates no more input
    // call total and return the answer
    return total(hundreds,fifties,twenties,tens,fives);
  }

  // since the number was not -1, we expect the number to be a 
  // denomination and expect to be able to read a second integer
  // which will represent number of notes of that denomination
  int val = read_int();
  if (val == READ_INT_FAIL){ //read failed
    return -1;
  }
  if ( n == 100){
    hundreds = val;
  } else if (n == 50){
    fifties = val;
  } else if (n == 20){
    twenties = val;
  } else if (n == 10){
    tens = val;
  } else if (n == 5){
    fives = val;
  } else {
    return -1; //invalid input
  } 
  // recurse
  return read_and_execute_test(hundreds,fifties,twenties,tens,fives);
}

int main(void){
  int answer = read_and_execute_test(0,0,0,0,0);
  printf("%d\n",answer);
  return 0;
}

2.4. Debugging

• Don't randomly fix the bug, understand it.

  • What was I expecting the program to do, and what did it do instead?

    • answer through Assert testing is obvious ( why? )

    • answer through I/O testing is obvious ( why? )

• Now we understood WHAT went wrong, now we think about why is this happening?

To Debug, and find the path taken by the program, the most rudimentary method is using print statements, useful locations:

• Beginning of Helper Functions

• First Statement in a loop's body

• after assignment of important variables

Think of the reasons why.

• By putting print statements and assertions along the intended sequence a program takes, they achieve a second goal:

  • Building a Memory Model of what the program's internal state is along the path that leads to a bug.

    • Allows us to accurately determine the first instance the Program's internal state differs from expected one.

Common Mistakes

• Using = instead of ==

• Using less than (<) where less than or equal (<=) or greater than (>) is needed, or vice versa

• Using greater than (>) where greater than or equal (>=) or less than (<) is needed, or vice versa

• Off by one errors; often due to the two mistakes highlighted above

• Using bitwise or (|) where they intended to use logical or (||)

• Using bitwise and (&) where they intended to use logical and (&&)

• Not checking return values e.g. read_int or scanf returns whether the read was successful

• Dereferencing a NULL pointer

• Dereferencing an incorrect or dangling pointer

(It is okay if you do not know about pointers yet. But once you do, watch out for NULL and incorrect/dangling pointers)

Normally, commenting out/remaking/deleting print statements is inefficient. Thus, C has Tracer Libraries.

i.e. in CS136 header:

#include <cs136.h>
//trace functions will tell you more than normal print statements.
// i.e.more details of where the print comes from, 
//i.e., the output will tell you the file name,
// function name, line number and the actual value you printed.

//the function trace_off turns off the tracings. 

trace functions print on the error stream.

Alternate approach to tracers:

• Writing Print Statements that are conditionally compiled using C preprocessor's #define and #ifdef preprocessor directives. We learn these concepts later.

Notes

• Output Stream is buffered (it flushes the produced output to the screen after a certain number of bytes have been requested to be sent to the screen)

  • Error Stream is NOT

  • resulting in async between output stream and error stream as outstream may be buffered at unknown times, leading to unlined up outputs/outputs in the wrong order.

• Solution:

  • trace_sync function

    • CS136 trace library provides a trace_sync function which flushes the standard output buffer so that outputs generated on either streams appear interwoven in exactly the order that the statements that generated the output are executed.