Feature Engineering

For many organizations, data is an overwhelming wave of information. It’s a chaotic mess that’s impossible to realize
any benefit from. With feature engineering, organizations can make sense of their data and turn it into something
beneficial.

The term feature engineering refers to the process of applying domain knowledge to data by generating features that
transform the data to make it easier to understand and interpret. It usually occurs after the data gathering and
cleaning process and before training machine learning models.

Feature engineering is often part of the ML problem solving workflow:

1. Gather data
2. Clean it
3. Perform feature engineering
4. Define the model
5. Train the model
6. Run tests
7. Predict the output

Most of the information used by Artificial Intelligence (AI) is contained in tables. Each row is an observation, and
a column is a feature. Unfortunately, the data is often complicated, irrelevant, missing, or duplicated.

Feature engineering provides a process for transforming data into a format that better represents the underlying
problem. To do this, it makes the data more digestible, putting data into categories to better reflect a finite set of
outcomes or systematically replacing missing values with appropriate estimations.

This process of transforming data with feature engineering is often as much of an art as a science. For example, a
business may want to predict instances of fraud. Raw timestamped transactions could be entered into AI software, but
the output may not be meaningful or actionable. However, a bit of domain expertise helps the data scientist. The
scientist, using their knowledge of retail, creates a new feature that differentiates between the work week and
weekends, as there are always spikes in retail activity during the weekend. Once that context is manually established,
models are better able to spot anomalies, with fewer false positives. That’s the ‘art’ of feature engineering.

Done correctly, feature engineering amplifies the predictive power of Machine Learning (ML) algorithms. It achieves
this by fashioning features out of raw data that feeds and facilitates the ML process. It can be the differentiator
between a good data model and a bad one.

Breaking it down further, the feature engineering part comprises the following steps:

• Brainstorm new, possible features for the model
• Create the features
• Test how efficiently these features work with the model
• Tweak the features, repeat, or go back to the drawing board as needed
• Get the features to work seamlessly with the model

Feature engineering should not be considered a one-time step. It can be used throughout the data science process to
either clean data or enhance existing results. Feature engineering is an iterative process that is interwoven between
data selection, model evaluation, and re-evaluation. The process continues until the data is in a format that is
ingestible by ML models and enables those models to output actionable results.

Examples Of Feature Engineering For Machine Learning

ML algorithms learn solutions to specific problems using the sample data they are presented with. Feature engineering
helps an organization arrange the best representation of their sample data to give the model a chance to learn the
solution to any specific problem.

In feature engineering, representation and relationships matter, and there are four common engineering strategies:

– Resampling imbalanced data
– Creating new features
– Managing missing values
– Detecting outliers

Resampling Imbalanced Data

In its raw form, data is usually imbalanced. Most of the time this can easily be resolved with validation techniques.
But sometimes the imbalance can be large, affecting the outputs. Feature engineering can resolve this by artificially
generating samples in the minority groups. These samples can be used to help address variability or uncertainty in the
data.

Creation of New Features

Creating new features can just be restating data in a different format to match the context of the question. For
example, a company may have the departure and arrival times for trains and turn them into total travel time. Combining
the timestamps into one new feature enables the algorithm to fit the business need and produce more actionable results

Users can also combine two moderately useful features or two features that by themselves are not useful on their own
to create one feature that helps the machine learn better. An example of this is in healthcare where a variety of risk
factors are present but, on their own, don’t indicate a likelihood of a medical event. For example, age, hypertension,
and being a smoker individually don’t predict having a stroke, but the three factors together do.

Feature selection is simply about picking the right independent features that correlate the most with the dependent
feature. All these things combine to make the best possible predictive model. Heatmaps, univariate selection, and the
ExtraTreesClassifier method are all tried-and-tested methods for identifying the features that are related
appropriately.

Feature engineering also helps pick which buckets to create so that the machine can accurately map relevant data to
the right bucket. This includes removing and weeding out unwanted features and noise that helps the model to function
more smoothly.

Managing Missing Values

Missing values are a frequent problem in data, but there are many methods to adequately resolving them during the
data cleansing process.

There are also several advanced engineering techniques that can use existing data to accurately recreate missing
values and complete the dataset, ensuring the data is in a form that models can better utilize.

One method is data deletion. With this method, Feature engineers can remove samples that have missing values. This
works best when only a few samples are incomplete. The more missing values a dataset contains, the more problematic
this method becomes.

Another technique involves replacing missing data with a variable of the mean or median. While this approach resolves
missing data, it can skew the results. If data has a gaussian distribution, then the missing results could be imputed
(a model within a model) so that they match the normal distribution.

These are the two main methods. While there are other methods that can be used to manage missing values, the general
approach is to remove data or input estimated values.

Outlier Detection

Outlier detection is another process that crosses the cleansing/engineering barrier. In the data cleansing step, AI
may simply remove the outliers, suggesting they are errors, or a sample that’s not relevant to the data. However,
that’s a blunt tool and could miss essential information.

In data science, key factors that influence a model’s performance are data handling and data processing. A model
without proper data handling results in an accuracy of about 70%. When feature engineering is applied to the same
model, the performance can greatly improve.

But a good understanding of the data is still needed for feature engineering as it allows a data scientist to specify
thresholds where the data is still logical. For example, a business may have a customer who is 100 years old but
definitely not 1,000 years old. A machine may disregard both data points while a data scientist knows the extra zero
is likely an input error.

This part of the feature engineering process can be long, frustrating, and rely on the skill and domain knowledge of
a data scientist. This is why some view feature engineering in ML as nothing less than an art form.

Advantages Of Feature Engineering

As the adage goes, AI and ML models are only as good as the data they receive. Including feature engineering in the
modeling process can ensure the quality and relevance models receive help them solve real-world problems. But there
are two important things to keep in mind as you proceed:

• Framing The Problem Correctly: Using the right objective measures to estimate the accuracy of the output
• Inter-Dependencies Within The Model: The inherent, underlying structures in the organization’s data. Good structure always provides far better results.

Once these things are considered when selecting or designing features, the advantages of feature engineering include:

• More flexibility and less complexity in models
• Faster processing
• Clear, easy-to-understand models
• Simpler models that are easier to maintain
• A better understanding of the underlying problem
• Better representation of all the available data that is helpful in characterizing the underlying problem

Challenges of Feature Engineering

Data is often unstructured and messy, containing outliers, redundancy, and missing values. Because data comes from
multiple sources, making redundancy and duplicate data are a given. Since data is the starting point for ML, this
results in the following challenges for feature engineering:

• Enormous amounts of data from multiple sources that must be cleansed, aggregated, and analyzed
• Data must be organized into a recognizable structure that models and tools can work with
• Business context and processes must be understood to discern patterns and facilitate analysis
• Insights given must be relevant and actionable for the organization
• Data should be presented in a way that’s easy for people to understand, such as dashboards or graphs
• Timeliness can be a problem, with results taking so long that the results are no longer applicable
• Processes are labor intensive and often must be completed by a data scientist

The Future Of Feature Engineering

Modern technologies are improving the performance of feature engineering. Deep learning as a subset of ML is starting
to reshape the process. Autoencoders and restricted Boltzmann machines are showing promise, automatically learning
abstract feature representation.

The more that computers ‘think’ like humans, the more helpful their feature engineering becomes. Taking heavily
manual tasks from data scientists and allocating them to machines removes cost and time constraints. This means that
data forms such as images, videos, objects, and speech, which are not easily understood by traditional AI that relies
on tables, may be accurately interpreted by machines soon.

New ML models are increasingly offering human-like thought processes, better feature analysis, and higher model
accuracy.

But for now, the field is still reliant on data scientists. The best interpretations of data not only require
knowledge of data science, but also industry or domain knowledge, making this subset of AI a specialized field. Data
interpretation is vital to organizations wanting accurate predictions, and this is the best way to get valid results.

Does Your Organization Need More Accurate Predictions?

Alteryx’s machine learning package offers
Deep Feature Synthesis. This helps to create more accurate models by understanding relationships within your data and
detecting high quality features.

These algorithms give a step up for organizations needing accurate models and predictions, allowing for better
explanations, decision making, and future plans.