Script for bisecting conformance test failures

[Francois Gouget]

Would something like this be useful to include in Wine?
If that's deemed interesting I can clean it up and submit it. I'm 
posting it here early in case it's useful for WineConf.


It's a tool which you can use with git bisect to find the patch that 
causes a regression in a conformance test. It's used like this:

    git bisect start <bad> <good>
    git bisect run ./tools/testbisect dlls/gdi32/tests bitmap

Come back a bit later and you have the bad patch.

It can also be used with flaky tests by specifying the number of times 
to run a test. So if a test fails in 30% of the time you could only 
consider that it works if it succeeds 10 times in a row. You can even 
grep for a specific error if the test has other errors you don't care 
about.

Finally you can also use it without git just to figure out if a test 
is flaky or not. For instance:

   $ ./tools/testbisect dlls/random/tests flaky 100 100
   Failed 42 time(s) in 100 run(s)

 -> definitely flaky


-- 
Francois Gouget <fgouget at free.fr>              http://fgouget.free.fr/
    Indifference will certainly be the downfall of mankind, but who cares?
-------------- next part --------------
#!/bin/sh
# Copyright (C) 2011 Francois Gouget
name0=`basename "$0"`


error()
{
    echo "$name0:error:" "$@" >&2
}


#
# Command line parsing
#

opt_build="1"
opt_grep=""
opt_dir=""
opt_test=""
opt_runs=""
opt_threshold=""
usage=""
while [ $# -gt 0 ]
do
    arg="$1"
    shift
    case "$arg" in
    --no-build)
        opt_build=0
        ;;
    --grep)
        if [ -n "$opt_error" ]
        then
            error "--grep can only be specified once"
            usage=2 # but continue processing options
        fi
        if [ $# -eq 0 ]
        then
            error "missing value for --grep"
            usage=2
        else
            opt_grep="$1"
            shift
        fi
        ;;
    --help|-h|-\?)
        usage=0
        ;;
    -*)
        error "unknown option '$arg'"
	usage=2
        ;;
    *)
        if [ -z "$opt_dir" ]
        then
            opt_dir="$arg"
        elif [ -z "$opt_test" ]
        then
            opt_test="$arg"
        elif [ -z "$opt_runs" ]
        then
            opt_runs="$arg"
        elif [ -z "$opt_threshold" ]
        then
            opt_threshold="$arg"
        else
            error "unexpected parameter '$arg'"
        fi
        ;;
    esac
done

if [ -z "$usage" ]
then
    if [ -z "$opt_dir" ]
    then
        error "you must specify the test directory"
        usage=2
    elif [ ! -d "$opt_dir" ]
    then
        error "'$opt_dir' is not a directory"
        usage=2
    fi
    if [ -z "$opt_test" ]
    then
        error "you must specify the test to run"
        usage=2
    elif [ ! -f "$opt_dir/$opt_test.c" ]
    then
        error "'$opt_dir/$opt_test.c' does not exist"
        usage=2
    fi
    if [ -n "$opt_runs" ]
    then
        expr $opt_runs + 1 >/dev/null 2>&1
        if [ $? -ne 0 ]
        then
            error "the number of runs '$opt_runs' must be an integer"
            usage=2
        fi
    else
        opt_runs=1
    fi
    if [ -n "$opt_threshold" ]
    then
        expr $opt_threshold + 1 >/dev/null 2>&1
        if [ $? -ne 0 ]
        then
            error "the threshold '$opt_threshold' must be an integer"
            usage=2
        fi
    else
        opt_threshold=1
    fi
fi

if [ -n "$usage" ]
then
    if [ $usage -ne 0 ]
    then
        error "try '$name0 --help' for more information"
        exit $usage
    fi
    echo "Usage: $name0 [--stats] [--no-build] [--error ERR] DIR TEST [RUNS [THRES]]"
    echo
    echo "Rebuilds Wine and runs the specified test, checking for an error for use with git bisect."
    echo
    echo "Where:"
    echo "  DIR          "
    echo "  TEST         "
    echo "  RUNS         "
    echo "  THRES        "
    echo "  --grep RE"
    # FIXME
    exit 0
fi


#
# Rebuild
#

if [ "$opt_build" = "1" ]
then
    ./configure || exit 125
    if [ -z "$MAKE" ] && type cxmake >/dev/null 2>&1
    then
        cxmake || exit 125
    else
        make || exit 125
    fi
fi


#
# Run the test
#

cd "$opt_dir"
i=0
errors=0
while [ $i -lt $opt_runs ]
do
    i=`expr $i + 1`

    # Run the test
    failed="1"
    rm -f $opt_test.ok
    make $opt_test.ok >$opt_test.log 2>&1

    # Check if it failed
    if [ $? -eq 0 ]
    then
        failed=""
    elif [ -n "$opt_grep" ]
    then
        grep ': Test failed:' $opt_test.log | grep "$opt_grep"
        [ $? -eq 0 ] || failed=""
    fi

    # Collect statistics and check if we're done
    if [ -n "$failed" ]
    then
        errors=`expr $errors + 1`
        if [ $errors -ge $opt_threshold ]
        then
            echo "Failed $errors time(s) in $i run(s)"
            exit 1
        fi
    fi
done

echo "Failed $errors time(s) in $opt_runs run(s)"
exit 0