These scoring metrics are then computed for each model using Monte Carlo cross validation, with three training/testing splits of 70%/30%. Depending on the training mode, the best model is determined as follows:
Note:
For the mathematical expressions listed below, model_score represents the scoring metric from step 2, while model_time is the time spent training the model.
Training Mode |
Expression for Model Comparison |
TIME |
(model_score)/(model_time^1.2) |
BALANCE |
(model_score)/(model_time) |
SCORE |
model_score |
For example, if the following three models were being compared:
Model |
model_score |
model_time |
Model A |
0.7 |
500 |
Model B |
0.85 |
600 |
Model C |
0˙.87 |
800 |
In the TIME training mode, Model A would be selected.
In the BALANCE training mode, Model B would be selected.
In the SCORE training mode, Model C would be selected.