Adding and running tests

A high-quality suite of tests is crucial in ensuring correctness and robustness of the codebase. Here, we provide instructions how to run unit tests, and also how to add a new one.

Contents

Adding a new unit test

Python package: pytest

Add your test under the directory tests/python/ or tests/python-gpu/ (if you are testing GPU code). Refer to the PyTest tutorial to learn how to write tests for Python code.

You may try running your test by following instructions in this section.

C++: Google Test

Add your test under the directory tests/cpp/. Refer to this excellent tutorial on using Google Test.

You may try running your test by following instructions in this section. Note. Google Test version 1.8.1 or later is required.

JVM packages: JUnit / scalatest

The JVM packages for XGBoost (XGBoost4J / XGBoost4J-Spark) use the Maven Standard Directory Layout. Specifically, the tests for the JVM packages are located in the following locations:

To write a test for Java code, see JUnit 5 tutorial. To write a test for Scala, see Scalatest tutorial.

You may try running your test by following instructions in this section.

R package: testthat

Add your test under the directory R-package/tests/testthat. Refer to this excellent tutorial on testthat.

You may try running your test by following instructions in this section.

Running Unit Tests Locally

R package

Run

make Rcheck

at the root of the project directory.

JVM packages

As part of the building process, tests are run:

mvn package

Python package: pytest

To run Python unit tests, first install pytest package:

pip3 install pytest

Then compile XGBoost according to instructions in Building the Shared Library. Finally, invoke pytest at the project root directory:

# Tell Python where to find XGBoost module
export PYTHONPATH=./python-package
pytest -v -s --fulltrace tests/python

In addition, to test CUDA code, run:

# Tell Python where to find XGBoost module
export PYTHONPATH=./python-package
pytest -v -s --fulltrace tests/python-gpu

(For this step, you should have compiled XGBoost with CUDA enabled.)

C++: Google Test

To build and run C++ unit tests enable tests while running CMake:

mkdir build
cd build
cmake -DGOOGLE_TEST=ON -DUSE_DMLC_GTEST=ON  ..
make
make test

To enable tests for CUDA code, add -DUSE_CUDA=ON and -DUSE_NCCL=ON (CUDA toolkit required):

mkdir build
cd build
cmake -DGOOGLE_TEST=ON -DUSE_DMLC_GTEST=ON -DUSE_CUDA=ON -DUSE_NCCL=ON ..
make
make test

One can also run all unit test using ctest tool which provides higher flexibility. For example:

ctest --verbose

Sanitizers: Detect memory errors and data races

By default, sanitizers are bundled in GCC and Clang/LLVM. One can enable sanitizers with GCC >= 4.8 or LLVM >= 3.1, But some distributions might package sanitizers separately. Here is a list of supported sanitizers with corresponding library names:

  • Address sanitizer: libasan

  • Undefined sanitizer: libubsan

  • Leak sanitizer: liblsan

  • Thread sanitizer: libtsan

Memory sanitizer is exclusive to LLVM, hence not supported in XGBoost. With latest compilers like gcc-9, when sanitizer flags are specified, the compiler driver should be able to link the runtime libraries automatically.

How to build XGBoost with sanitizers

One can build XGBoost with sanitizer support by specifying -DUSE_SANITIZER=ON. By default, address sanitizer and leak sanitizer are used when you turn the USE_SANITIZER flag on. You can always change the default by providing a semicolon separated list of sanitizers to ENABLED_SANITIZERS. Note that thread sanitizer is not compatible with the other two sanitizers.

cmake -DUSE_SANITIZER=ON -DENABLED_SANITIZERS="address;leak" /path/to/xgboost

By default, CMake will search regular system paths for sanitizers, you can also supply a specified SANITIZER_PATH.

cmake -DUSE_SANITIZER=ON -DENABLED_SANITIZERS="address;leak" \
-DSANITIZER_PATH=/path/to/sanitizers /path/to/xgboost

How to use sanitizers with CUDA support

Runing XGBoost on CUDA with address sanitizer (asan) will raise memory error. To use asan with CUDA correctly, you need to configure asan via ASAN_OPTIONS environment variable:

ASAN_OPTIONS=protect_shadow_gap=0 ${BUILD_DIR}/testxgboost

Other sanitizer runtime options

By default undefined sanitizer doesn’t print out the backtrace. You can enable it by exporting environment variable:

UBSAN_OPTIONS=print_stacktrace=1 ${BUILD_DIR}/testxgboost

For details, please consult official documentation for sanitizers.