DMatrix is the basic data storage for XGBoost used by all XGBoost algorithms including both training, prediction and explanation. There are a few variants of DMatrix including normal DMatrix, which is a CSR matrix, QuantileDMatrix, which is used by histogram-based tree methods for saving memory, and lastly the experimental external-memory-based DMatrix, which reads data in batches during training. For the last two variants, see the Streaming group. More...

Collaboration diagram for DMatrix:

Modules
	Streaming
	Quantile DMatrix and external memory DMatrix can be created from batches of data.

Functions
int	XGDMatrixCreateFromFile (const char fname, int silent, DMatrixHandle out)
	load a data matrix More...

int	XGDMatrixCreateFromURI (char const config, DMatrixHandle out)
	load a data matrix More...

int	XGDMatrixCreateFromCSREx (const size_t indptr, const unsigned indices, const float data, size_t nindptr, size_t nelem, size_t num_col, DMatrixHandle out)
	create a matrix content from CSR format More...

int	XGDMatrixCreateFromColumnar (char const data, char const config, DMatrixHandle *out)
	Create a DMatrix from columnar data. (table) More...

int	XGDMatrixCreateFromCSR (char const indptr, char const indices, char const data, bst_ulong ncol, char const config, DMatrixHandle *out)
	Create a DMatrix from CSR matrix. More...

int	XGDMatrixCreateFromDense (char const data, char const config, DMatrixHandle *out)
	Create a DMatrix from dense array. More...

int	XGDMatrixCreateFromCSC (char const indptr, char const indices, char const data, bst_ulong nrow, char const config, DMatrixHandle *out)
	Create a DMatrix from a CSC matrix. More...

int	XGDMatrixCreateFromCSCEx (const size_t col_ptr, const unsigned indices, const float data, size_t nindptr, size_t nelem, size_t num_row, DMatrixHandle out)
	create a matrix content from CSC format More...

int	XGDMatrixCreateFromMat (const float data, bst_ulong nrow, bst_ulong ncol, float missing, DMatrixHandle out)
	create matrix content from dense matrix More...

int	XGDMatrixCreateFromMat_omp (const float data, bst_ulong nrow, bst_ulong ncol, float missing, DMatrixHandle out, int nthread)
	create matrix content from dense matrix More...

int	XGDMatrixCreateFromCudaColumnar (char const data, char const config, DMatrixHandle *out)
	Create DMatrix from CUDA columnar format. (cuDF) More...

int	XGDMatrixCreateFromCudaArrayInterface (char const data, char const config, DMatrixHandle *out)
	Create DMatrix from CUDA array. More...

int	XGDMatrixSliceDMatrix (DMatrixHandle handle, const int idxset, bst_ulong len, DMatrixHandle out)
	create a new dmatrix from sliced content of existing matrix More...

int	XGDMatrixSliceDMatrixEx (DMatrixHandle handle, const int idxset, bst_ulong len, DMatrixHandle out, int allow_groups)
	create a new dmatrix from sliced content of existing matrix More...

int	XGDMatrixFree (DMatrixHandle handle)
	free space in data matrix More...

int	XGDMatrixSaveBinary (DMatrixHandle handle, const char *fname, int silent)
	Save the DMatrix object into a file. `QuantileDMatrix` and external memory DMatrix are not supported. More...

int	XGDMatrixSetInfoFromInterface (DMatrixHandle handle, char const field, char const data)
	Set content in array interface to a content in info. More...

int	XGDMatrixSetFloatInfo (DMatrixHandle handle, const char field, const float array, bst_ulong len)
	set float vector to a content in info More...

int	XGDMatrixSetUIntInfo (DMatrixHandle handle, const char field, const unsigned array, bst_ulong len)

int	XGDMatrixSetStrFeatureInfo (DMatrixHandle handle, const char field, const char *features, const bst_ulong size)
	Set string encoded information of all features. More...

int	XGDMatrixGetStrFeatureInfo (DMatrixHandle handle, const char field, bst_ulong size, const char ***out_features)
	Get string encoded information of all features. More...

int	XGDMatrixSetDenseInfo (DMatrixHandle handle, const char field, void const data, bst_ulong size, int type)

int	XGDMatrixGetFloatInfo (const DMatrixHandle handle, const char field, bst_ulong out_len, const float **out_dptr)
	get float info vector from matrix. More...

int	XGDMatrixGetUIntInfo (const DMatrixHandle handle, const char field, bst_ulong out_len, const unsigned **out_dptr)
	get uint32 info vector from matrix More...

int	XGDMatrixNumRow (DMatrixHandle handle, bst_ulong *out)
	get number of rows. More...

int	XGDMatrixNumCol (DMatrixHandle handle, bst_ulong *out)
	get number of columns More...

int	XGDMatrixNumNonMissing (DMatrixHandle handle, bst_ulong *out)
	Get number of valid values from DMatrix. More...

int	XGDMatrixDataSplitMode (DMatrixHandle handle, bst_ulong *out)
	Get the data split mode from DMatrix. More...

int	XGDMatrixGetDataAsCSR (DMatrixHandle const handle, char const config, bst_ulong out_indptr, unsigned out_indices, float out_data)
	Get the predictors from DMatrix as CSR matrix for testing. If this is a quantized DMatrix, quantized values are returned instead. More...

int	XGDMatrixGetQuantileCut (DMatrixHandle const handle, char const config, char const out_indptr, char const *out_data)
	Export the quantile cuts used for training histogram-based models like `hist` and `approx`. Useful for model compression. More...

Detailed Description

DMatrix is the basic data storage for XGBoost used by all XGBoost algorithms including both training, prediction and explanation. There are a few variants of DMatrix including normal DMatrix, which is a CSR matrix, QuantileDMatrix, which is used by histogram-based tree methods for saving memory, and lastly the experimental external-memory-based DMatrix, which reads data in batches during training. For the last two variants, see the Streaming group.

Function Documentation

◆ XGDMatrixCreateFromColumnar()

int XGDMatrixCreateFromColumnar	(	char const *	data,
		char const *	config,
		DMatrixHandle *	out
	)

Create a DMatrix from columnar data. (table)

A special type of input to the DMatrix is the columnar format, which refers to column-based dataframes. XGBoost can accept both numeric data types like integers and floats, along with the categorical type, called dictionary in arrow's term. The addition of categorical type is introduced in 3.1.0. The dataframe is represented by a list array interfaces with one object for each column.

A categorical type is represented by 3 buffers, the validity mask, the names of the categories (called index for most of the dataframe implementation), and the codes used to represent the categories in the rows. XGBoost consumes a categorical column by accepting two JSON-encoded arrow arrays in a list. The first item in the list is a JSON object with {"offsets": IntegerArray, "values": StringArray } representing the string names defined by the arrow columnar format. The second buffer is an masked integer array that stores the categorical codes along with the validity mask:

[
  // categorical column, represented as an array (list)
  [
    {
      'offsets':
      {
        'data': (129412626415808, True),
        'typestr': '<i4', 'version': 3, 'strides': None, 'shape': (3,), 'mask': None
      },
      'values':
      {
        'data': (129412626416000, True),
        'typestr': '<i1', 'version': 3, 'strides': None, 'shape': (7,), 'mask': None
      }
    },
    {
      'data': (106200854378448, True),
      'typestr': '<i1', 'version': 3, 'strides': None, 'shape': (2,), 'mask': None
    }
  ],
  // numeric column, represented as an object, same number of rows as the previous column (2)
  {
    'data': (106200854378448, True),
    'typestr': '<f4', 'version': 3, 'strides': None, 'shape': (2,), 'mask': None
  }
]

As for numeric inputs, it's the same as dense array.

Parameters

data	A list of JSON-encoded array interfaces.
config	See XGDMatrixCreateFromDense for details.
out	The created DMatrix.

Returns: 0 when success, -1 when failure happens

◆ XGDMatrixCreateFromCSC()

int XGDMatrixCreateFromCSC	(	char const *	indptr,
		char const *	indices,
		char const *	data,
		bst_ulong	nrow,
		char const *	config,
		DMatrixHandle *	out
	)

Create a DMatrix from a CSC matrix.

Parameters

indptr	JSON encoded array_interface to column pointers in CSC.
indices	JSON encoded array_interface to row indices in CSC.
data	JSON encoded array_interface to values in CSC.
nrow	The number of rows in the matrix.
config	See XGDMatrixCreateFromDense for details.
out	The created dmatrix.

Returns: 0 when success, -1 when failure happens

Examples: c-api-demo.c.

◆ XGDMatrixCreateFromCSCEx()

int XGDMatrixCreateFromCSCEx	(	const size_t *	col_ptr,
		const unsigned *	indices,
		const float *	data,
		size_t	nindptr,
		size_t	nelem,
		size_t	num_row,
		DMatrixHandle *	out
	)

create a matrix content from CSC format

Deprecated:: since 2.0.0

See also: XGDMatrixCreateFromCSC()

◆ XGDMatrixCreateFromCSR()

int XGDMatrixCreateFromCSR	(	char const *	indptr,
		char const *	indices,
		char const *	data,
		bst_ulong	ncol,
		char const *	config,
		DMatrixHandle *	out
	)

Create a DMatrix from CSR matrix.

Parameters

indptr	JSON encoded array_interface to row pointers in CSR.
indices	JSON encoded array_interface to column indices in CSR.
data	JSON encoded array_interface to values in CSR.
ncol	The number of columns.
config	See XGDMatrixCreateFromDense for details.
out	The created dmatrix

Returns: 0 when success, -1 when failure happens

Examples: c-api-demo.c.

◆ XGDMatrixCreateFromCSREx()

int XGDMatrixCreateFromCSREx	(	const size_t *	indptr,
		const unsigned *	indices,
		const float *	data,
		size_t	nindptr,
		size_t	nelem,
		size_t	num_col,
		DMatrixHandle *	out
	)

create a matrix content from CSR format

Deprecated:: since 2.0.0

See also: XGDMatrixCreateFromCSR()

◆ XGDMatrixCreateFromCudaArrayInterface()

int XGDMatrixCreateFromCudaArrayInterface	(	char const *	data,
		char const *	config,
		DMatrixHandle *	out
	)

Create DMatrix from CUDA array.

Parameters

data	JSON encoded cuda_array_interface for array data.
config	JSON encoded configuration. Required values are: missing: Which value to represent missing value. nthread (optional): Number of threads used for initializing DMatrix. data_split_mode (optional): Whether the data was split by row or column beforehand. Default to row.
out	created dmatrix

Returns: 0 when success, -1 when failure happens

◆ XGDMatrixCreateFromCudaColumnar()

int XGDMatrixCreateFromCudaColumnar	(	char const *	data,
		char const *	config,
		DMatrixHandle *	out
	)

Create DMatrix from CUDA columnar format. (cuDF)

See XGDMatrixCreateFromColumnar for a brief description of the columnar format.

Parameters

data	A list of JSON-encoded array interfaces.
config	See XGDMatrixCreateFromDense for details.
out	Created dmatrix

Returns: 0 when success, -1 when failure happens

◆ XGDMatrixCreateFromDense()

int XGDMatrixCreateFromDense	(	char const *	data,
		char const *	config,
		DMatrixHandle *	out
	)

Create a DMatrix from dense array.

The array interface is defined in https://numpy.org/doc/2.1/reference/arrays.interface.html We encode the interface as a JSON object.

Parameters

data	JSON encoded array_interface to array values.
config	JSON encoded configuration. Required values are: missing: Which value to represent missing value. nthread (optional): Number of threads used for initializing DMatrix. data_split_mode (optional): Whether the data was split by row or column beforehand. Default to row.
out	The created DMatrix

Returns: 0 when success, -1 when failure happens

Examples: inference.c.

◆ XGDMatrixCreateFromFile()

int XGDMatrixCreateFromFile	(	const char *	fname,
		int	silent,
		DMatrixHandle *	out
	)

load a data matrix

Deprecated:: since 2.0.0

See also: XGDMatrixCreateFromURI()

Parameters

fname	the name of the file
silent	whether print messages during loading
out	a loaded data matrix

Returns: 0 when success, -1 when failure happens

Examples: c-api-demo.c.

◆ XGDMatrixCreateFromMat()

int XGDMatrixCreateFromMat	(	const float *	data,
		bst_ulong	nrow,
		bst_ulong	ncol,
		float	missing,
		DMatrixHandle *	out
	)

create matrix content from dense matrix

Parameters

data	pointer to the data space
nrow	number of rows
ncol	number columns
missing	which value to represent missing value
out	created dmatrix

Returns: 0 when success, -1 when failure happens

Examples: c-api-demo.c.

◆ XGDMatrixCreateFromMat_omp()

int XGDMatrixCreateFromMat_omp	(	const float *	data,
		bst_ulong	nrow,
		bst_ulong	ncol,
		float	missing,
		DMatrixHandle *	out,
		int	nthread
	)

create matrix content from dense matrix

Parameters

data	pointer to the data space
nrow	number of rows
ncol	number columns
missing	which value to represent missing value
out	created dmatrix
nthread	number of threads (up to maximum cores available, if <=0 use all cores)

Returns: 0 when success, -1 when failure happens

◆ XGDMatrixCreateFromURI()

int XGDMatrixCreateFromURI	(	char const *	config,
		DMatrixHandle *	out
	)

load a data matrix

Parameters

config

JSON encoded parameters for DMatrix construction. Accepted fields are:

uri: The URI of the input file. The URI parameter format is required when loading text data.

embed:rst:leading-asterisk
*            See :doc:`/tutorials/input_format` for more info.
*

silent (optional): Whether to print message during loading. Default to true.
data_split_mode (optional): Whether the file was split by row or column beforehand for distributed computing. Default to row.

out a loaded data matrix

Returns: 0 when success, -1 when failure happens

◆ XGDMatrixDataSplitMode()

int XGDMatrixDataSplitMode	(	DMatrixHandle	handle,
		bst_ulong *	out
	)

Get the data split mode from DMatrix.

Parameters

handle	the handle to the DMatrix
out	The output of the data split mode

Returns: 0 when success, -1 when failure happens

◆ XGDMatrixFree()

int XGDMatrixFree ( DMatrixHandle handle )

free space in data matrix

Returns: 0 when success, -1 when failure happens

Examples: c-api-demo.c, external_memory.c, and inference.c.

◆ XGDMatrixGetDataAsCSR()

int XGDMatrixGetDataAsCSR	(	DMatrixHandle const	handle,
		char const *	config,
		bst_ulong *	out_indptr,
		unsigned *	out_indices,
		float *	out_data
	)

Get the predictors from DMatrix as CSR matrix for testing. If this is a quantized DMatrix, quantized values are returned instead.

Unlike most of XGBoost C functions, caller of XGDMatrixGetDataAsCSR is required to allocate the memory for return buffer instead of using thread local memory from XGBoost. This is to avoid allocating a huge memory buffer that can not be freed until exiting the thread.

Since: 1.7.0

Parameters

handle	the handle to the DMatrix
config	JSON configuration string. At the moment it should be an empty document, preserved for future use.
out_indptr	indptr of output CSR matrix.
out_indices	Column index of output CSR matrix.
out_data	Data value of CSR matrix.

Returns: 0 when success, -1 when failure happens

◆ XGDMatrixGetFloatInfo()

int XGDMatrixGetFloatInfo	(	const DMatrixHandle	handle,
		const char *	field,
		bst_ulong *	out_len,
		const float **	out_dptr
	)

get float info vector from matrix.

Parameters

handle	a instance of data matrix
field	field name
out_len	used to set result length
out_dptr	pointer to the result

Returns: 0 when success, -1 when failure happens

Examples: c-api-demo.c.

◆ XGDMatrixGetQuantileCut()

int XGDMatrixGetQuantileCut	(	DMatrixHandle const	handle,
		char const *	config,
		char const **	out_indptr,
		char const **	out_data
	)

Export the quantile cuts used for training histogram-based models like hist and approx. Useful for model compression.

Since: 2.0.0

Parameters

handle	the handle to the DMatrix
config	JSON configuration string. At the moment it should be an empty document, preserved for future use.
out_indptr	indptr of output CSC matrix represented by a JSON encoded __(cuda_)array_interface__.
out_data	Data value of CSC matrix represented by a JSON encoded __(cuda_)array_interface__.

◆ XGDMatrixGetStrFeatureInfo()

int XGDMatrixGetStrFeatureInfo	(	DMatrixHandle	handle,
		const char *	field,
		bst_ulong *	size,
		const char ***	out_features
	)

Get string encoded information of all features.

Accepted fields are:

feature_name
feature_type

Caller is responsible for copying out the data, before next call to any API function of XGBoost.

Parameters

handle	An instance of data matrix
field	Field name
size	Size of output pointer `features` (number of strings returned).
out_features	Address of a pointer to array of strings. Result is stored in thread local memory.

Returns: 0 when success, -1 when failure happens

char const **c_out_features = NULL;
bst_ulong out_size = 0;
 
// Asumming the feature names are already set by `XGDMatrixSetStrFeatureInfo`.
XGDMatrixGetStrFeatureInfo(handle, "feature_name", &out_size,
                           &c_out_features)
 
for (bst_ulong i = 0; i < out_size; ++i) {
  // Here we are simply printing the string.  Copy it out if the feature name is
  // useful after printing.
  printf("feature %lu: %s\n", i, c_out_features[i]);
}

◆ XGDMatrixGetUIntInfo()

int XGDMatrixGetUIntInfo	(	const DMatrixHandle	handle,
		const char *	field,
		bst_ulong *	out_len,
		const unsigned **	out_dptr
	)

get uint32 info vector from matrix

Parameters

handle	a instance of data matrix
field	field name
out_len	The length of the field.
out_dptr	pointer to the result

Returns: 0 when success, -1 when failure happens

◆ XGDMatrixNumCol()

int XGDMatrixNumCol	(	DMatrixHandle	handle,
		bst_ulong *	out
	)

get number of columns

Parameters

handle	the handle to the DMatrix
out	The output of number of columns

Returns: 0 when success, -1 when failure happens

◆ XGDMatrixNumNonMissing()

int XGDMatrixNumNonMissing	(	DMatrixHandle	handle,
		bst_ulong *	out
	)

Get number of valid values from DMatrix.

Parameters

handle	the handle to the DMatrix
out	The output of number of non-missing values

Returns: 0 when success, -1 when failure happens

◆ XGDMatrixNumRow()

int XGDMatrixNumRow	(	DMatrixHandle	handle,
		bst_ulong *	out
	)

get number of rows.

Parameters

handle	the handle to the DMatrix
out	The address to hold number of rows.

Returns: 0 when success, -1 when failure happens

◆ XGDMatrixSaveBinary()

int XGDMatrixSaveBinary	(	DMatrixHandle	handle,
		const char *	fname,
		int	silent
	)

Save the DMatrix object into a file. QuantileDMatrix and external memory DMatrix are not supported.

Parameters

handle	a instance of data matrix
fname	file name
silent	print statistics when saving

Returns: 0 when success, -1 when failure happens

◆ XGDMatrixSetDenseInfo()

int XGDMatrixSetDenseInfo	(	DMatrixHandle	handle,
		const char *	field,
		void const *	data,
		bst_ulong	size,
		int	type
	)

Deprecated:: since 2.1.0

Use XGDMatrixSetInfoFromInterface instead.

Examples: external_memory.c, and inference.c.

◆ XGDMatrixSetFloatInfo()

int XGDMatrixSetFloatInfo	(	DMatrixHandle	handle,
		const char *	field,
		const float *	array,
		bst_ulong	len
	)

set float vector to a content in info

Parameters

handle	a instance of data matrix
field	field name, can be label, weight
array	pointer to float vector
len	length of array

Returns: 0 when success, -1 when failure happens

◆ XGDMatrixSetInfoFromInterface()

int XGDMatrixSetInfoFromInterface	(	DMatrixHandle	handle,
		char const *	field,
		char const *	data
	)

Set content in array interface to a content in info.

Parameters

handle	An instance of data matrix
field	Field name.
data	JSON encoded array_interface to values in the dense matrix/vector.

Returns: 0 when success, -1 when failure happens

◆ XGDMatrixSetStrFeatureInfo()

int XGDMatrixSetStrFeatureInfo	(	DMatrixHandle	handle,
		const char *	field,
		const char **	features,
		const bst_ulong	size
	)

Set string encoded information of all features.

Accepted fields are:

feature_name
feature_type

Parameters

handle	An instance of data matrix
field	Field name
features	Pointer to array of strings.
size	Size of `features` pointer (number of strings passed in).

Returns: 0 when success, -1 when failure happens

char const* feat_names [] {"feat_0", "feat_1"};
XGDMatrixSetStrFeatureInfo(handle, "feature_name", feat_names, 2);
 
// i for integer, q for quantitive, c for categorical.  Similarly "int" and "float"
// are also recognized.
char const* feat_types [] {"i", "q"};
XGDMatrixSetStrFeatureInfo(handle, "feature_type", feat_types, 2);

◆ XGDMatrixSetUIntInfo()

int XGDMatrixSetUIntInfo	(	DMatrixHandle	handle,
		const char *	field,
		const unsigned *	array,
		bst_ulong	len
	)

Deprecated:: since 2.1.0

Use XGDMatrixSetInfoFromInterface instead.

◆ XGDMatrixSliceDMatrix()

int XGDMatrixSliceDMatrix	(	DMatrixHandle	handle,
		const int *	idxset,
		bst_ulong	len,
		DMatrixHandle *	out
	)

create a new dmatrix from sliced content of existing matrix

Parameters

handle	instance of data matrix to be sliced
idxset	index set
len	length of index set
out	a sliced new matrix

Returns: 0 when success, -1 when failure happens

◆ XGDMatrixSliceDMatrixEx()

int XGDMatrixSliceDMatrixEx	(	DMatrixHandle	handle,
		const int *	idxset,
		bst_ulong	len,
		DMatrixHandle *	out,
		int	allow_groups
	)

create a new dmatrix from sliced content of existing matrix

Parameters

handle	instance of data matrix to be sliced
idxset	index set
len	length of index set
out	a sliced new matrix
allow_groups	allow slicing of an array with groups

Returns: 0 when success, -1 when failure happens

Modules

Functions

Detailed Description

Function Documentation

◆ XGDMatrixCreateFromColumnar()

◆ XGDMatrixCreateFromCSC()

◆ XGDMatrixCreateFromCSCEx()

◆ XGDMatrixCreateFromCSR()

◆ XGDMatrixCreateFromCSREx()

◆ XGDMatrixCreateFromCudaArrayInterface()

◆ XGDMatrixCreateFromCudaColumnar()

◆ XGDMatrixCreateFromDense()

◆ XGDMatrixCreateFromFile()

◆ XGDMatrixCreateFromMat()

◆ XGDMatrixCreateFromMat_omp()

◆ XGDMatrixCreateFromURI()

◆ XGDMatrixDataSplitMode()

◆ XGDMatrixFree()

◆ XGDMatrixGetDataAsCSR()

◆ XGDMatrixGetFloatInfo()

◆ XGDMatrixGetQuantileCut()

◆ XGDMatrixGetStrFeatureInfo()

◆ XGDMatrixGetUIntInfo()

◆ XGDMatrixNumCol()

◆ XGDMatrixNumNonMissing()

◆ XGDMatrixNumRow()

◆ XGDMatrixSaveBinary()

◆ XGDMatrixSetDenseInfo()

◆ XGDMatrixSetFloatInfo()

◆ XGDMatrixSetInfoFromInterface()

◆ XGDMatrixSetStrFeatureInfo()

◆ XGDMatrixSetUIntInfo()

◆ XGDMatrixSliceDMatrix()

◆ XGDMatrixSliceDMatrixEx()