-
-
Notifications
You must be signed in to change notification settings - Fork 8.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
terminate called without an active exception #3500
Comments
Do you have a script for reproduction? |
The usage is basic:
Unfortunately, I am not able to share the data that encounter the problem as they are proprietary. But I can mention a few characteristics of them:
The crash behavior is deterministic, meaning that the same dataset and hyperparameters can generate the same crash at the same training step (always at step 0). |
Thanks. Are you using the version from pip or building from source? There were a few bug fixes recently that would be only in the source version. Would it be possible to take a minimal sample of data that reproduces the issue and then anonymise it? You could remove column labels and add noise to features. I can't see how to resolve it without having a reproducible example. |
That makes sense. Let me look into what will be the viable path to share some fragment of data. The version is built from source just two weeks ago with the flag NCCL. If you know the exact dates of those fixes applied, I can check whether my version includes them or not. Thank you for looking into the issue! I will try to get back to you shortly. |
I made a notable fix in #3472 |
Hi @RAMitchell , Report back here: the bug fix #3472 does not help resolve this issue. I figure out a sample snippet that can reproduce the crash import os
import numpy as np
import scipy as sp
import xgboost
N = 10000
d = 50
thres = 0.5
density = 0.10
features = sp.sparse.random(N, d, density=density)
labels = np.random.rand(N)
labels[labels>thres] = 1
labels[labels<=thres] = 0
dtrain = xgboost.DMatrix(data=features, label=labels)
xgb_params = {
'tree_method': 'gpu_hist',
# 'n_gpus': 2,
'n_estimators': 700,
'learning_rate': 0.035,
'max_depth': 16,
'min_split_loss': 10,
'min_child_weight': 100,
'colsample_bytree': 0.9,
'reg_lambda':5
}
bst = xgboost.train(xgb_params, dtrain) The overall messages are as follows
Note that this happens only when the sparse matrix density is high enough, say, as in the above snippet, 0.1. When it is as low as 0.01, the code runs fine. It behaves the same when using multiple GPUs. Please let me know if this is enough for investigation. Thanks a lot again for looking into the problem! |
version:
xgboost:0.72
compiled with NCCL.Saw this error message 'deterministically' with some hyperparameters for "gpu_hist". But the same set of hyperparameters works for "hist" (cpu).
The training does not start even the first iteration. No other error messages printed either. The compete messages are just as above.
It seems related to the gpu-hist implementation. This was observed for single gpu training.
Any idea?
The text was updated successfully, but these errors were encountered: