“This is the 20th day of my participation in the Gwen Challenge in November. Check out the details: The Last Gwen Challenge in 2021.”

Optuna is an automatic hyperparametric optimization framework specifically designed for machine learning, deep learning, and user API with scripting language features. As a result, OpTUNA’s code is highly modular, and users can dynamically construct hyperparametric search Spaces as they wish. Optuna was also a popular tuner in the Kaggle competition, appearing frequently in excellent solutions. Say goodbye to violent GridSearch callbacks.

Project address: github.com/optuna/optu…

Characteristics of framework

  • Small, lightweight, versatile and platform-independent
  • Hyperparameter space search in Python form
  • Efficient optimization algorithm
  • It’s easy to write in parallel
  • Quick visualization

The installation

Optuna supports Python 3.6 and above

PIP install Optuna

pip install optuna
Copy the code

Conda installation Optuna

conda install -c conda-forge optuna
Copy the code

The sample

Generally, hyperparameters of tree-based models can be adjusted in three steps:

1. Adjust the tree structure

Max_depth: tree depth. If you do not control the tree depth, it is easy to overfit. Generally, set it to 3-8.

Num_leaves: the maximum num_leaves should not exceed 2^(maxdepth) because the tree model is binary.

Min_data_in_leaf: the minimum number of leaf nodes. If set to 50, then the tree will stop growing when the number reaches 50. Therefore, the size of this value is related to over-fitting, and its size is also related to num_leaves.

2. Accuracy adjustment

Learning_rate: indicates the learning rate

N_estimators: The number of trees

These two parameters are frequently adjusted, and you generally need to find the best combination of N_ESTIMators and Learning_rate.

Max_bin: This parameter can also be used to improve accuracy, but there is a risk of overfitting.

3. Over-fitting adjustment

Lambda_l1: indicates the L1 regular

Lambda_l2: indicates that L2 is regular

The regularization has an influence on over-fitting and is difficult to adjust. The general search range is set at (0,100).

Min_gain_to_split: minimum gain for splitting. If the calculated gain is not high, splitting will not continue.

Bagging_fraction: Train the percentage of training samples for each tree to make each tree good and different.

Feature_fraction: The percentage of features to be sampled for each tree, so that the features of each sub-tree are differentiated and resistant to over-fitting.

Optuna will simplify the optimizer into three simple steps: Objective, trial, and study.

The objective function

import numpy as np
import pandas as pd
from sklearn.metrics import accuracy_score,f1_score
from sklearn.model_selection import  StratifiedKFold
import lightgbm as lgbm

def objective(trial, data, target) :
    X_train, X_test, y_train, y_test=train_test_split(data, target, train_size=0.3)# Data set partitioning
    # Parameter grid
    param_grid = {
        "n_estimators": trial.suggest_categorical("n_estimators"[10000]),
        "learning_rate": trial.suggest_float("learning_rate".0.01.0.3),
        "num_leaves": trial.suggest_int("num_leaves".20.3000, step=20),
        "max_depth": trial.suggest_int("max_depth".3.12),
        "min_data_in_leaf": trial.suggest_int("min_data_in_leaf".200.10000, step=100),
        "lambda_l1": trial.suggest_int("lambda_l1".0.100, step=5),
        "lambda_l2": trial.suggest_int("lambda_l2".0.100, step=5),
        "min_gain_to_split": trial.suggest_float("min_gain_to_split".0.15),
        "bagging_fraction": trial.suggest_float("bagging_fraction".0.2.0.95, step=0.1),
        "bagging_freq": trial.suggest_categorical("bagging_freq"[1]),
        "feature_fraction": trial.suggest_float("feature_fraction".0.2.0.95, step=0.1),
        "random_state": 2021,
    }
    model = lgbm.LGBMClassifier(objective="gbdt", **param_grid)
    model.fit(
        X_train,
        y_train,
        eval_set=[(X_test, y_test)],
        eval_metric="multi_logloss",
        early_stopping_rounds=100,
        verbose=False
    )
    pred_lgb = model.predict(X_test)
    scores = f1_score(y_true=y_test, y_pred=pred_lgb,average='macro')
    return scores
Copy the code

Word experiment and research:

n_trials=50
train = pd.read_csv("train.csv")
X,y = train.drop(["id"."label"],axis=1),train["label"]
study = optuna.create_study(direction="maximize", study_name="LGBM Classifier") # minimize
study.optimize(func, n_trials=n_trials)
Copy the code

Print the best parameters:

print(f"\tBest value (f1): {study.best_value:. 5f}")
print(f"\tBest params:")

for key, value in study.best_params.items():
    print(f"\t\t{key}: {value}")
Copy the code

visualization

optuna.visualization.plot_optimization_history(study)
Copy the code

optuna.visualization.plot_parallel_coordinate(study)
Copy the code

optuna.visualization.plot_param_importances(study)
Copy the code