About AutoML

Copy link to this sectionNatural Language Processing

AutoML leverages natural language processing (NLP) to turn your text features into numeric features for your predictive models. AutoML uses Term frequency-inverse document frequency (TFIDF) to evaluate key words in text and list columns.

Copy link to this sectionMulti-Hot Encoding

While most of our data is sparse, machine learning algorithms can only understand dense data. In most data modeling workflows, data scientists are burdened with performing difficult, and cumbersome, manual transformations to convert their sparse data into dense data.

Unlike many workflows that require this manual step, AutoML performs this conversion seamlessly. Lists and one-to-many relationships are smartly “multi-hot encoded” to account for columns that are representing more than a single value.

For instance, assume a table that contains a list of medical conditions for each person:

Many machine learning functions treat these lists as separate entities, with one-hot encoding resulting in the following conversion:

Instead, AutoML uses bag-of-words to create a separate column for each value in each list:

While other functions would have treated each person as having a separate list of medical conditions, AutoML’s method allows a model to properly find patterns between each of these persons’ set of medical conditions.

AutoML assumes that order does not matter. Person C and Person D share the same set of medical conditions, but just ordered differently. While other functions treat those two lists differently, AutoML identifies that they are the same.

Copy link to this sectionColumn Type Classification

AutoML first examines the columns in the dataset and classifies them as a particular Python data type. For information about the conversion from DDL to Python data types, see DDL to Python Type Conversion.

The column types, along with how their classifications are made, are listed below:

Copy link to this sectionData Transformation

The Transform Function transforms the entire dataset into the form to be used by the machine learning models. It is applied on the training set before training, and on any future datasets before predictions are made.

Additional Date/Time columns are created. For every datetime column, the following separate columns are added whenever applicable:

• Hour of day.

• Day of week.

• Month of year.

AutoML also creates duration columns. Each column added represents one of the original date/time columns, and each value in this column is the duration between the dates of that particular date/time column and all other date/time columns. For example, consider patient data that has three date/time columns:

• Date of birth.

• Time of admission.

• Time of exit.

AutoML creates two useful duration columns from these columns: age (duration between date of birth and time of admission) and length of stay (duration between time of admission and exit).

Finally, for each list column present, another column is added simply with the size of the lists. That is, each value in the new column is the length of the corresponding list in the old column.

Datasets can often be incomplete, with missing values in some of their columns. To help compensate for this and improve performance, AutoML fills in missing/NULL values:

• For categorical and date columns, AutoML replaces missing values with the mode (most popular value) of the column.

• For numeric and duration columns, AutoML replaces missing values with the mean (average) of the column.

• For list and text columns, AutoML replaces missing values with an empty string.

For each numeric column, a standard scalar is fit. These include the original numeric columns, along with the duration and list size columns as well.

Numerical column values are also binned and then used as categorical values. These new categorical bin columns are added on separately in addition to the already present numerical columns. Each numerical column is separated into four bins, each representing a quartile of the values in that column. The new binned columns are treated as categorical columns.

For each text and list column, a vectorizer is fit to transform the data to the appropriate form needed for training. This is done with SciKit Learn’s TFIDF Vectorizer. Please see their documentation.

The following parameters are used:

Binary columns are simply transformed to be composed of 1’s and 0’s, with true values mapping to 1’s.

Categorical columns are one-hot encoded before being used for training.

As the last step before training, feature elimination is performed to remove redundancy, improve training speed, and improve the accuracy of models. This is done using Scikit Learn’s SelectFPR function.

The following parameters are used:

Copy link to this sectionXGBRegressor

For regression models, AutoML uses XGBoost’s XGBRegressor class.

The model hyperparameters are detailed below:

Copy link to this sectionNeural Network

For the Neural Network model, AutoML uses TensorFlow with Keras as a wrapper.

The input layer has its size based on the number of features. This layer is then densely connected to a single hidden layer composed of 100 neurons, which implement the ReLU Activation Function. This hidden layer is densely connected to the final output layer, which implements the Softmax Activation Function. The number of neurons in the output layer is equivalent to the number of classes present for classification.

The model hyperparameters are detailed below:

Copy link to this sectionLogistic Regression

For the Logistic Regression Model, AutoML uses SciKit Learn’s Logistic Regression class.

The model hyperparameters are detailed below:

Copy link to this sectionRandom Forest Classifier

For the Random Forest Classifier model, AutoML uses SciKit Learn’s Random Forest Classifier class.

The model hyperparameters are detailed below: