testing-in-gem5.tex

\subsection[Testing in gem5]{Testing in gem5\footnote{by Bobby R. Bruce}}
\label{sec:testing}

In order to ensure the quality of gem5, we have continued to improve testing.
Testing ensures errors during development are caught and rectified early, prior
to release. A good testing infrastructure is essential for open-source projects
such as gem5, due frequent contributions, from many individuals, with varying
levels of expertise and familiarity with the project. Testing gives assurances
that these regular contributions are not breaking key pieces of functionality,
thus allowing for a relatively high throughput of changes.

Furthermore, due to gem5's age and size, tests give developers a degree of confidence when undertaking engineering on seldom touched component, thereby improving the productivity of all developers.
Good tests and regular testing are thereby critical in ensuring the smooth running of the project both presently and into the future.

\subsubsection{The TestLib Framework}

The majority of gem5 tests are run via \emph{TestLib}, our python-based
testing framework. TestLib runs compiled instances of gem5 on specific
computer architecture simulations, and verifies that they run as intended. It
has been designed with extendibility in-mind, allowing for tests to be easily
added as they are required. While powerful, it is important to note that
TestLib runs system-level tests, and is therefore not well-suited to testing
individual gem5 components.

\subsubsection{Unit Tests}

In order to test at a finer granularity, unit tests have been developed and
incorporated into our testing processes. Unlike our TestLib tests, these unit
tests ensure the functionality of gem5 classes and functions. Thus, if broken,
the source of the bugs can more quickly be determined. Unit Test coverage is
improving over time, and will play a greater role in improving stability in
future releases of gem5.

\subsubsection{Continuous Integration}

Via our Gerrit code-review system, we run tests for each
submitted patch prior to merging into the gem5 code base. These tests
compile gem5 against key ISA targets and run a suite of tests designed to
cover all major gem5 components (in practice, all unit tests and a subset of
the TestLib tests). This procedure supplements our code review system, thereby
reducing the possibility of new contributions introducing bugs.

In addition to the tests run prior to submission, a larger set of tests are
available for more ``in-depth'' testing. These can take several hours to complete
execution, and test functionality such as
full-system booting. We run these tests regularly to ensure gem5 meets our
standards. While these are currently triggered manually, we hope to
automatically run these tests nightly and automatically send error reports to
the developers.