Skip to content

lindig/quest

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

28 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Quest - A tool for testing C compilers

This is Quest, a tool that generates C code for testing C compilers. A test is run by compiling it with the compiler under test and subsequently executing the binary that the compiler produced. A compiler bug manifests itself by an assertion failure during the execution of the binary.

The code generated tests one specific part of a C compiler: how it implements passing values to a function by function call and returning a value from it using the return statement. Stated simpler, the generated code by Quests checks that a value passed to a function is indeed received unaltered from the caller and likewise when a valuereturned to the caller. Conceptually, the code generated by Quest looks like this:

static int   i = 1;
static char  c = 'c'; 
static float r = 3.14; 

static float f(int ii, char cc)
{
    assert(ii == i);
    assert(cc == c);
    return r;
}

static void test(void)
{
    float rr;
    rr = f(i ,c);
    assert(rr == r); 
}

int main(int argc, char **argv)
{
    test();
}

By compiling and runnung the code with a C compiler we can test that values are passed and returned to and from a function as expected:

$ quest > testcase.c
$ gcc -o testcase testcase.c
$ ./testcase

If the compiler fails to generate correct code, executing the code will lead to an assertion failure that would highlight the problem. Quest generates tests in a random yet controlled way. Generating a test case, compiling and running it can be done in a loop until a bug is found.

The Issue

Why is compiling function calls even an issue? What looks simple on the surface of C source code is quite complicated inside a compiler. How values are supposed to be passed between functions is governed by a calling convention. Such a convention specifies how values are passed in registers and on the stack using a specific alignment. C calling conventions in particular are complicted when structured values are passed, like structures, or var args. As a consequence, even mature compilers exhibit bugs and Quest helps to find them. The beauty is that we don't have to know the details of a calling convention in order to test it: generating test cases and executing them is enough.

Directory doc/ contains a paper whith more details:

Christian Lindig. Random  Testing of C Calling Conventions. In
Jong Deok Choi and Raimondas Lencevicius, editors, Sixth
Internation Symposium on  Automated and Analysis-Driven
Debugging", ACM Press, Monterey, CA,  USA, September 2005. 

Directory bugs contains a few compiler bugs that were uncovered with Quest in the past. This provides also examples for code generated by Quest.

Installation from Opam

Quest is available from OCaml's package manager Opam:

$ opam install quest

This will install the quest binary.

Installation from GitHub

Quest is implemened in OCaml as a literate program. It should build on Unix systems without a problem and does not depend on libraries outside the OCaml standard libraries and its tools.

$ opam install lipsum
$ make PREFIX=$HOME     # see the Makefile

Quick Start

To be written. See doc/quest.pod for now.

Documentation

  • The manual page quest.pod in doc/ explains Quest in detail.

  • If you have a Quest binary you can view the manual page also with

    quest -man

    which emits it to stdout. A short summary of options is available with:

    quest -help

  • The Lua 2.5 reference manual in doc/ introduces the Lua scripting language. It is only required to define new generators beyond the existing ones.

Discussion

Recently testing calling concentions was discussed in a thread on Hacker News:

Quest Features

  • Quest pretty prints C code to keep it readable.

  • Quest can split test cases into two files such that they may be compiled by different compilers to check their interoperability.

  • Neither Quest itself, nor the generated code requires any special run-time libraries or configuration files.

  • Quest is developed on Mach OS X but contains no system-specific source code. However, the build process needs to be adapted for other systems.

  • Quest is controlled by an embedded Lua interpreter. This makes it easy to tailor the generation of test cases.

Author

Christian Lindig [email protected]. Please report bugs on GitHub https://github.com/lindig/quest

Opam

quest - generates C code for testing a C compiler's calling convention

Quest generates C code for testing C compilers. A test is run by compiling the generated code it with the compiler under test and subsequently executing the binary that the compiler produced. A compiler bug manifests itself by an assertion failure during the execution of the binary. This leads to automatic testing: generate and execute tests until an assertion failure is found.

The code generated by Quest tests one specific part of a C compiler: how it implements passing values to a function by function call and returning a value from it using the return statement. Stated simpler, the generated code by Quests checks that a value passed to a function is indeed received unaltered from the caller and likewise when a valuereturned to the caller.

About

A tool for testing C compilers automatically

Resources

License

Stars

Watchers

Forks

Packages

No packages published