XGBoost GPU Support

This page contains information about GPU algorithms supported in XGBoost. To install GPU support, checkout the build and installation instructions.

CUDA Accelerated Tree Construction Algorithms

This plugin adds GPU accelerated tree construction and prediction algorithms to XGBoost.

Usage

Specify the ‘tree_method’ parameter as one of the following algorithms.

Algorithms

tree_method Description
gpu_exact The standard XGBoost tree construction algorithm. Performs exact search for splits. Slower and uses considerably more memory than ‘gpu_hist’
gpu_hist Equivalent to the XGBoost fast histogram algorithm. Faster and uses considerably less memory. Splits may be less accurate.

Supported parameters

parameter gpu_exact gpu_hist
subsample
colsample_bytree
colsample_bylevel
max_bin
gpu_id
n_gpus
predictor

GPU accelerated prediction is enabled by default for the above mentioned ‘tree_method’ parameters but can be switched to CPU prediction by setting ‘predictor’:’cpu_predictor’. This could be useful if you want to conserve GPU memory. Likewise when using CPU algorithms, GPU accelerated prediction can be enabled by setting ‘predictor’:’gpu_predictor’.

The device ordinal can be selected using the ‘gpu_id’ parameter, which defaults to 0.

Multiple GPUs can be used with the grow_gpu_hist parameter using the n_gpus parameter. which defaults to 1. If this is set to -1 all available GPUs will be used. If gpu_id is specified as non-zero, the gpu device order is mod(gpu_id + i) % n_visible_devices for i=0 to n_gpus-1. As with GPU vs. CPU, multi-GPU will not always be faster than a single GPU due to PCI bus bandwidth that can limit performance. For example, when n_features * n_bins * 2^depth divided by time of each round/iteration becomes comparable to the real PCI 16x bus bandwidth of order 4GB/s to 10GB/s, then AllReduce will dominant code speed and multiple GPUs become ineffective at increasing performance. Also, CPU overhead between GPU calls can limit usefulness of multiple GPUs.

This plugin currently works with the CLI version and python version.

Python example:

param['gpu_id'] = 0
param['max_bin'] = 16
param['tree_method'] = 'gpu_hist'

Benchmarks

To run benchmarks on synthetic data for binary classification:

$ python tests/benchmark/benchmark.py

Training time time on 1,000,000 rows x 50 columns with 500 boosting iterations and 0.25/0.75 test/train split on i7-6700K CPU @ 4.00GHz and Pascal Titan X.

tree_method Time (s)
gpu_hist 13.87
hist 63.55
gpu_exact 161.08
exact 1082.20

See here for additional performance benchmarks of the ‘gpu_exact’ tree_method.

Author

Rory Mitchell Jonathan C. McKinney Shankara Rao Thejaswi Nanditale Vinay Deshpande ... and the rest of the H2O.ai and NVIDIA team.

Please report bugs to the xgboost/issues page.