1. 概述
在竞赛题中，我们知道XGBoost算法非常热门，是很多的比赛的大杀器，但是在使用过程中，其训练耗时很长，内存占用比较大。在2017年年1月微软在GitHub的上开源了LightGBM。该算法在不降低准确率的前提下，速度提升了10倍左右，占用内存下降了3倍左右。LightGBM是个快速的，分布式的，高性能的基于决策树算法的梯度提升算法。可用于排序，分类，回归以及很多其他的机器学习任务中。其详细的原理及操作内容详见：LightGBM 中文文档。
本文主要讲解LightGBM的两种调参方法。
下面几张表为重要参数的含义和如何应用
| 学习控制参数 | 含义 | 用法 |
|---|---|---|
max_depth |
树的最大深度 | 当模型过拟合时,可以考虑首先降低 max_depth
|
min_data_in_leaf |
叶子可能具有的最小记录数 | 默认20，过拟合时用 |
feature_fraction |
例如 为0.8时，意味着在每次迭代中随机选择80％的参数来建树 | boosting 为 random forest 时用 |
bagging_fraction |
每次迭代时用的数据比例 | 用于加快训练速度和减小过拟合 |
early_stopping_round |
如果一次验证数据的一个度量在最近的early_stopping_round 回合中没有提高，模型将停止训练 |
加速分析，减少过多迭代 |
| lambda | 指定正则化 | 0～1 |
min_gain_to_split |
描述分裂的最小 gain | 控制树的有用的分裂 |
max_cat_group |
在 group 边界上找到分割点 | 当类别数量很多时，找分割点很容易过拟合时 |
| 核心参数 | 含义 | 用法 |
|---|---|---|
| Task | 数据的用途 | 选择 train 或者 predict |
| application | 模型的用途 | 选择 regression: 回归时，binary: 二分类时，multiclass: 多分类时 |
| boosting | 要用的算法 | gbdt， rf: random forest， dart: Dropouts meet Multiple Additive Regression Trees， goss: Gradient-based One-Side Sampling
|
num_boost_round |
迭代次数 | 通常 100+ |
learning_rate |
如果一次验证数据的一个度量在最近的 early_stopping_round 回合中没有提高，模型将停止训练 |
常用 0.1, 0.001, 0.003… |
num_leaves |
默认 31 | |
| device | cpu 或者 gpu | |
| metric | mae: mean absolute error ， mse: mean squared error ， binary_logloss: loss for binary classification ， multi_logloss: loss for multi classification |
| IO 参数 | 含义 |
|---|---|
max_bin |
表示 feature 将存入的 bin 的最大数量 |
categorical_feature |
如果 categorical_features = 0,1,2， 则列 0，1，2是 categorical 变量 |
ignore_column |
与 categorical_features 类似，只不过不是将特定的列视为categorical，而是完全忽略 |
save_binary |
这个参数为 true 时，则数据集被保存为二进制文件，下次读数据时速度会变快 |
调参
| IO parameter | 含义 |
|---|---|
num_leaves |
取值应 <= 2 ^（max_depth）， 超过此值会导致过拟合 |
min_data_in_leaf |
将它设置为较大的值可以避免生长太深的树，但可能会导致 underfitting，在大型数据集时就设置为数百或数千 |
max_depth |
这个也是可以限制树的深度 |
下表对应了 Faster Speed ，better accuracy ，over-fitting 三种目的时，可以调的参数
| Faster Speed | better accuracy | over-fitting |
|---|---|---|
将 max_bin 设置小一些 |
用较大的 max_bin
|
max_bin 小一些 |
num_leaves 大一些 |
num_leaves 小一些 |
|
用 feature_fraction 来做 sub-sampling
|
用 feature_fraction
|
|
用 bagging_fraction 和 bagging_freq
|
设定 bagging_fraction 和 bagging_freq
|
|
| training data 多一些 | training data 多一些 | |
用 save_binary 来加速数据加载 |
直接用 categorical feature | 用 gmin_data_in_leaf 和 min_sum_hessian_in_leaf
|
| 用 parallel learning | 用 dart | 用 lambda_l1, lambda_l2 ，min_gain_to_split 做正则化 |
num_iterations 大一些，learning_rate 小一些 |
用 max_depth 控制树的深度 |
调参
LightGBM的调参过程和RF、GBDT等类似，其基本流程如下：
-
首先选择较高的学习率，大概0.1附近，这样是为了加快收敛的速度。这对于调参是很有必要的。
-
对决策树基本参数调参
-
正则化参数调参
-
最后降低学习率，这里是为了最后提高准确率
第一步：学习率和迭代次数
我们先把学习率先定一个较高的值，这里取 learning_rate = 0.1，其次确定估计器boosting/boost/boosting_type的类型，不过默认都会选gbdt。
迭代的次数，也可以说是残差树的数目，参数名为n_estimators/num_iterations/num_round/num_boost_round。我们可以先将该参数设成一个较大的数，然后在cv结果中查看最优的迭代次数，具体如代码。
在这之前，我们必须给其他重要的参数一个初始值。初始值的意义不大，只是为了方便确定其他参数。下面先给定一下初始值：
以下参数根据具体项目要求定：
'boosting_type'/'boosting': 'gbdt'
'objective': 'binary'
'metric': 'auc'
以下是我选择的初始值：
'max_depth': 5 # 由于数据集不是很大，所以选择了一个适中的值，其实4-10都无所谓。
'num_leaves': 30 # 由于lightGBM是leaves_wise生长，官方说法是要小于2^max_depth
'subsample'/'bagging_fraction':0.8 # 数据采样
'colsample_bytree'/'feature_fraction': 0.8 # 特征采样
下面用LightGBM的cv函数进行确定：
import pandas as pd
import lightgbm as lgb
from import load_breast_cancer
from sklearn.cross_validation import train_test_split
canceData=load_breast_cancer()
X=
y=
X_train,X_test,y_train,y_test=train_test_split(X,y,random_state=0,test_size=0.2)
params = {
'boosting_type': 'gbdt',
'objective': 'binary',
'metric': 'auc',
'nthread':4,
'learning_rate':0.1,
'num_leaves':30,
'max_depth': 5,
'subsample': 0.8,
'colsample_bytree': 0.8,
}
data_train = (X_train, y_train)
cv_results = (params, data_train, num_boost_round=1000, nfold=5, stratified=False, shuffle=True, metrics='auc',early_stopping_rounds=50,seed=0)
print('best n_estimators:', len(cv_results['auc-mean']))
print('best cv score:', (cv_results['auc-mean']).max())输出结果如下：
('best n_estimators:', 188)
('best cv score:', 0.99134716298085424)
我们根据以上结果，取n_estimators=188。
第二步：确定max_depth和num_leaves
这是提高精确度的最重要的参数。这里我们引入sklearn里的GridSearchCV()函数进行搜索。
from sklearn.grid_search import GridSearchCVparams_test1={'max_depth': range(3,8,1), 'num_leaves':range(5, 100, 5)}
gsearch1 = GridSearchCV(estimator = (boosting_type='gbdt',objective='binary',metrics='auc',learning_rate=0.1, n_estimators=188, max_depth=6, bagging_fraction = 0.8,feature_fraction = 0.8),
param_grid = params_test1, scoring='roc_auc',cv=5,n_jobs=-1)
(X_train,y_train)
gsearch1.grid_scores_, gsearch1.best_params_, gsearch1.best_score_结果如下：（结果较长，只显示部分内容）
([mean: 0.99248, std: 0.01033, params: {'num_leaves': 5, 'max_depth': 3},
mean: 0.99227, std: 0.01013, params: {'num_leaves': 10, 'max_depth': 3},
mean: 0.99227, std: 0.01013, params: {'num_leaves': 15, 'max_depth': 3},
······
mean: 0.99331, std: 0.00775, params: {'num_leaves': 85, 'max_depth': 7},
mean: 0.99331, std: 0.00775, params: {'num_leaves': 90, 'max_depth': 7},
mean: 0.99331, std: 0.00775, params: {'num_leaves': 95, 'max_depth': 7}],
{'max_depth': 4, 'num_leaves': 10},
0.9943573667711598)根据结果，我们取max_depth=4，num_leaves=10。
第三步：确定min_data_in_leaf和max_bin in
params_test2={'max_bin': range(5,256,10), 'min_data_in_leaf':range(1,102,10)}
gsearch2 = GridSearchCV(estimator = (boosting_type='gbdt',objective='binary',metrics='auc',learning_rate=0.1, n_estimators=188, max_depth=4, num_leaves=10,bagging_fraction = 0.8,feature_fraction = 0.8),
param_grid = params_test2, scoring='roc_auc',cv=5,n_jobs=-1)
(X_train,y_train)
gsearch2.grid_scores_, gsearch2.best_params_, gsearch2.best_score_结果如下：（结果较长，只显示部分内容）
([mean: 0.98715, std: 0.01044, params: {'min_data_in_leaf': 1, 'max_bin': 5},
mean: 0.98809, std: 0.01095, params: {'min_data_in_leaf': 11, 'max_bin': 5},
mean: 0.98809, std: 0.00952, params: {'min_data_in_leaf': 21, 'max_bin': 5},
······
mean: 0.99363, std: 0.00812, params: {'min_data_in_leaf': 81, 'max_bin': 255},
mean: 0.99133, std: 0.00788, params: {'min_data_in_leaf': 91, 'max_bin': 255},
mean: 0.98882, std: 0.01223, params: {'min_data_in_leaf': 101, 'max_bin': 255}],
{'max_bin': 15, 'min_data_in_leaf': 51},
0.9952978056426331)根据结果，我们取min_data_in_leaf=51，max_bin in=15。
第四步：确定feature_fraction、bagging_fraction、bagging_freq
params_test3={'feature_fraction': [0.6,0.7,0.8,0.9,1.0],
'bagging_fraction': [0.6,0.7,0.8,0.9,1.0],
'bagging_freq': range(0,81,10)
}
gsearch3 = GridSearchCV(estimator = (boosting_type='gbdt',objective='binary',metrics='auc',learning_rate=0.1, n_estimators=188, max_depth=4, num_leaves=10,max_bin=15,min_data_in_leaf=51),
param_grid = params_test3, scoring='roc_auc',cv=5,n_jobs=-1)
(X_train,y_train)
gsearch3.grid_scores_, gsearch3.best_params_, gsearch3.best_score_结果如下：（结果较长，只显示部分内容）
([mean: 0.99467, std: 0.00710, params: {'bagging_freq': 0, 'bagging_fraction': 0.6, 'feature_fraction': 0.6},
mean: 0.99415, std: 0.00804, params: {'bagging_freq': 0, 'bagging_fraction': 0.6, 'feature_fraction': 0.7},
mean: 0.99530, std: 0.00722, params: {'bagging_freq': 0, 'bagging_fraction': 0.6, 'feature_fraction': 0.8},
······
mean: 0.99530, std: 0.00722, params: {'bagging_freq': 80, 'bagging_fraction': 1.0, 'feature_fraction': 0.8},
mean: 0.99383, std: 0.00731, params: {'bagging_freq': 80, 'bagging_fraction': 1.0, 'feature_fraction': 0.9},
mean: 0.99383, std: 0.00766, params: {'bagging_freq': 80, 'bagging_fraction': 1.0, 'feature_fraction': 1.0}],
{'bagging_fraction': 0.6, 'bagging_freq': 0, 'feature_fraction': 0.8},
0.9952978056426331)第五步：确定lambda_l1和lambda_l2
params_test4={'lambda_l1': [1e-5,1e-3,1e-1,0.0,0.1,0.3,0.5,0.7,0.9,1.0],
'lambda_l2': [1e-5,1e-3,1e-1,0.0,0.1,0.3,0.5,0.7,0.9,1.0]
}
gsearch4 = GridSearchCV(estimator = (boosting_type='gbdt',objective='binary',metrics='auc',learning_rate=0.1, n_estimators=188, max_depth=4, num_leaves=10,max_bin=15,min_data_in_leaf=51,bagging_fraction=0.6,bagging_freq= 0, feature_fraction= 0.8),
param_grid = params_test4, scoring='roc_auc',cv=5,n_jobs=-1)
(X_train,y_train)
gsearch4.grid_scores_, gsearch4.best_params_, gsearch4.best_score_解果如下：（结果较长，只显示部分内容）
([mean: 0.99530, std: 0.00722, params: {'lambda_l1': 1e-05, 'lambda_l2': 1e-05},
mean: 0.99415, std: 0.00804, params: {'lambda_l1': 1e-05, 'lambda_l2': 0.001},
mean: 0.99331, std: 0.00826, params: {'lambda_l1': 1e-05, 'lambda_l2': 0.1},
·····
mean: 0.99049, std: 0.01047, params: {'lambda_l1': 1.0, 'lambda_l2': 0.7},
mean: 0.99049, std: 0.01013, params: {'lambda_l1': 1.0, 'lambda_l2': 0.9},
mean: 0.99070, std: 0.01071, params: {'lambda_l1': 1.0, 'lambda_l2': 1.0}],
{'lambda_l1': 1e-05, 'lambda_l2': 1e-05},
0.9952978056426331)第六步：确定 min_split_gain
params_test5={'min_split_gain':[0.0,0.1,0.2,0.3,0.4,0.5,0.6,0.7,0.8,0.9,1.0]}
gsearch5 = GridSearchCV(estimator = (boosting_type='gbdt',objective='binary',metrics='auc',learning_rate=0.1, n_estimators=188, max_depth=4, num_leaves=10,max_bin=15,min_data_in_leaf=51,bagging_fraction=0.6,bagging_freq= 0, feature_fraction= 0.8,
lambda_l1=1e-05,lambda_l2=1e-05),
param_grid = params_test5, scoring='roc_auc',cv=5,n_jobs=-1)
(X_train,y_train)
gsearch5.grid_scores_, gsearch5.best_params_, gsearch5.best_score_结果如下：
([mean: 0.99530, std: 0.00722, params: {'min_split_gain': 0.0},
mean: 0.99415, std: 0.00810, params: {'min_split_gain': 0.1},
mean: 0.99394, std: 0.00898, params: {'min_split_gain': 0.2},
mean: 0.99373, std: 0.00918, params: {'min_split_gain': 0.3},
mean: 0.99404, std: 0.00845, params: {'min_split_gain': 0.4},
mean: 0.99300, std: 0.00958, params: {'min_split_gain': 0.5},
mean: 0.99258, std: 0.00960, params: {'min_split_gain': 0.6},
mean: 0.99227, std: 0.01071, params: {'min_split_gain': 0.7},
mean: 0.99342, std: 0.00872, params: {'min_split_gain': 0.8},
mean: 0.99206, std: 0.01062, params: {'min_split_gain': 0.9},
mean: 0.99206, std: 0.01064, params: {'min_split_gain': 1.0}],
{'min_split_gain': 0.0},
0.9952978056426331)第七步：降低学习率，增加迭代次数，验证模型
model=(boosting_type='gbdt',objective='binary',metrics='auc',learning_rate=0.01, n_estimators=1000, max_depth=4, num_leaves=10,max_bin=15,min_data_in_leaf=51,bagging_fraction=0.6,bagging_freq= 0, feature_fraction= 0.8,
lambda_l1=1e-05,lambda_l2=1e-05,min_split_gain=0)
(X_train,y_train)
y_pre=(X_test)
print("acc:",metrics.accuracy_score(y_test,y_pre))
print("auc:",metrics.roc_auc_score(y_test,y_pre))结果如下：
('acc:', 0.97368421052631582)
('auc:', 0.9744363289933311)
而使用默认参数时，模型表现如下：
model=()
(X_train,y_train)
y_pre=(X_test)
print("acc:",metrics.accuracy_score(y_test,y_pre))
print("auc:",metrics.roc_auc_score(y_test,y_pre))
('acc:', 0.96491228070175439)
('auc:', 0.96379803112099083)我们可以看出在准确率和AUC得分都有所提高。
的cv函数调参
这种方式比较省事儿，写好代码自动寻优，但需要有调参经验，如何设置较好的参数范围有一定的技术含量，这里直接给出代码。
import pandas as pd
import lightgbm as lgb
from sklearn import metrics
from import load_breast_cancer
from sklearn.cross_validation import train_test_split
canceData=load_breast_cancer()
X=
y=
X_train,X_test,y_train,y_test=train_test_split(X,y,random_state=0,test_size=0.2)
### 数据转换
print('数据转换')
lgb_train = (X_train, y_train, free_raw_data=False)
lgb_eval = (X_test, y_test, reference=lgb_train,free_raw_data=False)
### 设置初始参数--不含交叉验证参数
print('设置参数')
params = {
'boosting_type': 'gbdt',
'objective': 'binary',
'metric': 'auc',
'nthread':4,
'learning_rate':0.1
}
### 交叉验证(调参)
print('交叉验证')
max_auc = float('0')
best_params = {}
# 准确率
print("调参1：提高准确率")
for num_leaves in range(5,100,5):
for max_depth in range(3,8,1):
params['num_leaves'] = num_leaves
params['max_depth'] = max_depth
cv_results = (
params,
lgb_train,
seed=1,
nfold=5,
metrics=['auc'],
early_stopping_rounds=10,
verbose_eval=True
)
mean_auc = (cv_results['auc-mean']).max()
boost_rounds = (cv_results['auc-mean']).idxmax()
if mean_auc >= max_auc:
max_auc = mean_auc
best_params['num_leaves'] = num_leaves
best_params['max_depth'] = max_depth
if 'num_leaves' and 'max_depth' in best_params.keys():
params['num_leaves'] = best_params['num_leaves']
params['max_depth'] = best_params['max_depth']
# 过拟合
print("调参2：降低过拟合")
for max_bin in range(5,256,10):
for min_data_in_leaf in range(1,102,10):
params['max_bin'] = max_bin
params['min_data_in_leaf'] = min_data_in_leaf
cv_results = (
params,
lgb_train,
seed=1,
nfold=5,
metrics=['auc'],
early_stopping_rounds=10,
verbose_eval=True
)
mean_auc = (cv_results['auc-mean']).max()
boost_rounds = (cv_results['auc-mean']).idxmax()
if mean_auc >= max_auc:
max_auc = mean_auc
best_params['max_bin']= max_bin
best_params['min_data_in_leaf'] = min_data_in_leaf
if 'max_bin' and 'min_data_in_leaf' in best_params.keys():
params['min_data_in_leaf'] = best_params['min_data_in_leaf']
params['max_bin'] = best_params['max_bin']
print("调参3：降低过拟合")
for feature_fraction in [0.6,0.7,0.8,0.9,1.0]:
for bagging_fraction in [0.6,0.7,0.8,0.9,1.0]:
for bagging_freq in range(0,50,5):
params['feature_fraction'] = feature_fraction
params['bagging_fraction'] = bagging_fraction
params['bagging_freq'] = bagging_freq
cv_results = (
params,
lgb_train,
seed=1,
nfold=5,
metrics=['auc'],
early_stopping_rounds=10,
verbose_eval=True
)
mean_auc = (cv_results['auc-mean']).max()
boost_rounds = (cv_results['auc-mean']).idxmax()
if mean_auc >= max_auc:
max_auc=mean_auc
best_params['feature_fraction'] = feature_fraction
best_params['bagging_fraction'] = bagging_fraction
best_params['bagging_freq'] = bagging_freq
if 'feature_fraction' and 'bagging_fraction' and 'bagging_freq' in best_params.keys():
params['feature_fraction'] = best_params['feature_fraction']
params['bagging_fraction'] = best_params['bagging_fraction']
params['bagging_freq'] = best_params['bagging_freq']
print("调参4：降低过拟合")
for lambda_l1 in [1e-5,1e-3,1e-1,0.0,0.1,0.3,0.5,0.7,0.9,1.0]:
for lambda_l2 in [1e-5,1e-3,1e-1,0.0,0.1,0.4,0.6,0.7,0.9,1.0]:
params['lambda_l1'] = lambda_l1
params['lambda_l2'] = lambda_l2
cv_results = (
params,
lgb_train,
seed=1,
nfold=5,
metrics=['auc'],
early_stopping_rounds=10,
verbose_eval=True
)
mean_auc = (cv_results['auc-mean']).max()
boost_rounds = (cv_results['auc-mean']).idxmax()
if mean_auc >= max_auc:
max_auc=mean_auc
best_params['lambda_l1'] = lambda_l1
best_params['lambda_l2'] = lambda_l2
if 'lambda_l1' and 'lambda_l2' in best_params.keys():
params['lambda_l1'] = best_params['lambda_l1']
params['lambda_l2'] = best_params['lambda_l2']
print("调参5：降低过拟合2")
for min_split_gain in [0.0,0.1,0.2,0.3,0.4,0.5,0.6,0.7,0.8,0.9,1.0]:
params['min_split_gain'] = min_split_gain
cv_results = (
params,
lgb_train,
seed=1,
nfold=5,
metrics=['auc'],
early_stopping_rounds=10,
verbose_eval=True
)
mean_auc = (cv_results['auc-mean']).max()
boost_rounds = (cv_results['auc-mean']).idxmax()
if mean_auc >= max_auc:
max_auc=mean_auc
best_params['min_split_gain'] = min_split_gain
if 'min_split_gain' in best_params.keys():
params['min_split_gain'] = best_params['min_split_gain']
print(best_params)
结果如下：
{'bagging_fraction': 0.7,
'bagging_freq': 30,
'feature_fraction': 0.8,
'lambda_l1': 0.1,
'lambda_l2': 0.0,
'max_bin': 255,
'max_depth': 4,
'min_data_in_leaf': 81,
'min_split_gain': 0.1,
'num_leaves': 10}我们将训练得到的参数代入模型
model=(boosting_type='gbdt',objective='binary',metrics='auc',learning_rate=0.01, n_estimators=1000, max_depth=4, num_leaves=10,max_bin=255,min_data_in_leaf=81,bagging_fraction=0.7,bagging_freq= 30, feature_fraction= 0.8,
lambda_l1=0.1,lambda_l2=0,min_split_gain=0.1)
(X_train,y_train)
y_pre=(X_test)
print("acc:",metrics.accuracy_score(y_test,y_pre))
print("auc:",metrics.roc_auc_score(y_test,y_pre))结果如下：
('acc:', 0.98245614035087714)
('auc:', 0.98189901556049541)