This article is excerpted from the course “Fundamental Machine Learning,” part of the Machine Learning Specialist certification program from Arcitura Education. It is the eleventh part of the 13-part series, “Using machine learning algorithms, practices and patterns.”
Model optimization techniques are used to improve the performance of machine learning models. This and the next, final article in this series cover a set of optimization techniques that are normally applied toward the end of a machine learning problem-solving task, after a given model has been trained but when there exist opportunities to make it more effective.
This article describes the first two of four optimization practices: the ensemble learning and the frequent model retraining techniques. As explained in Part 4, these techniques are documented in a standard pattern profile format.
In the context of a classification or regression task, different models carry different strengths depending on whether they are trained using the same or different algorithms, and capture different aspects and relationships hidden in a data set. Even if the model with the highest accuracy is chosen, there is no guarantee that it will generalize well when exposed to unseen data in the production environment. Consequently, relying on a single model generally results in fluctuating accuracy based on whether the model sees the same type of unseen data that it is good at predicting. (See Figure 1.)
A single meta-model is generated that builds upon the strengths of each of its constituent models. The constituent models can be generated either by using the same type of algorithm for all models (each model differs in terms of its parameters) or different algorithms (applicable to the same type of machine learning problem, such as regression or classification) for each model. The results of all the constituent models are combined together using strategies such as voting or averaging. The meta-model works best when the constituent models carry a high accuracy but disagree among themselves as voting or averaging the same set of results does not bring any added value when compared with the results obtained from a single model.
This pattern can be applied via one of the following strategies for creating a meta-model:
The ensemble learning pattern can further benefit from the application of the baseline modeling pattern. With an established baseline, it is easier to select the right combination of constituent models. (See Figure 2.)
Once a model has been developed and tested to provide valuable insights, it is deployed in the production environment. However, over time, the model does not remain as accurate as it was when it was first deployed and provides less accurate results. With a classification model, the false positive or false negative rate may increase. With a regression model, the mean squared error (MSE) may increase. [Editor’s note: MSE is a measure that determines how close the line of best fit is to the actual values of the response variable. See Part 9 of this series for a further explanation.] The increase of MSE suggests the model is losing its efficacy, which can result in lost business opportunities or the inability to address a threat in a timely manner. (See Figure 3.)
The decrease in the efficacy of a model after its initial deployment can be attributed to the changing nature of the input data, which itself can be traced back to a change in the data-generating process. To keep up with these changes, the model is periodically refreshed. This involves preparing a training data set that is comprised of both the current and historical data, then retraining the model. Periodic retraining of the model ensures that the model keeps capturing the true concept the data encompasses and thus remains accurate.
The application of this pattern is dependent upon the existence of a feedback loop that informs on the effectiveness of the model. This requires setting up a process that records the predicted and actual values, keeping in mind that the actual values will only be available after the fact. For model retraining, a representative data set needs to be gathered to include a blend of both newly observed data and historic data. Based on the nature of the business problem at hand, either the same algorithm can be reused for the next iteration or a different algorithm can be used. The latter generally happens if the current algorithm is unable to fully capture the underlying concept and the error rate keeps increasing when retrained with new data. If the same algorithm suffices, the entire model retraining process can be automated. However, in both the cases, it is vital that the performance of the model is tracked on a continuous basis.
As retraining a model from scratch can consume valuable time and effort, the frequent model retraining pattern can further benefit from the application of the incremental model learning pattern (covered in the next article), provided that fit-for-purpose algorithmic implementations are available for the task at hand (Figure 4).
The next article covers the second pair of model optimization techniques: the lightweight model implementation and the incremental model learning patterns.
This lesson is one in a 13-part series on using machine learning algorithms, practices and patterns. Click the titles below to read the other available lessons.
Lesson 1: Introduction to using machine learning
Lesson 2: The “supervised” approach to machine learning
Lesson 3: Unsupervised machine learning: Dealing with unknown data
Lesson 4: Common ML patterns: Central tendency and variability
Lesson 5: Associativity, graphical summary computations aid ML insights
Lesson 6: How feature selection, extraction improve ML predictions
Lesson 7: 2 data-wrangling techniques for better machine learning
Lesson 8: Wrangling data with feature discretization, standardization
Lesson 9: 2 supervised learning techniques that aid value predictions
Lesson 10: Discover 2 unsupervised techniques that help categorize data
Lesson 11: 3 ways to evaluate and improve machine learning models
Data science, machine learning and AI are central to analytics and other enterprise uses. Here’s what each involves and how …
Data science teams typically include various analytics and data professionals and can be set up in different ways, as explained …
Data scientists engage in various applications to analyze data and create technologies. These are eight common ones, with …
In this week’s antitrust roundup, South Korean officials focus in on Apple and Google app store practices, while U.S. officials …
With its high speeds, low latency and other advantages, 5G is delivering value in the enterprise. Now, business leaders view the …
Hybrid work is here to stay. To keep employees happy and productive, CIOs should partner with stakeholders companywide to design …
Rockset previews new features in its cloud database platform to help organizations more easily use real-time data to support data…
Cribl looks to grow its LogStream platform capabilities with new funding that will help the vendor grow operations and advance …
Data warehouse BI benefits include data storage, summarization and transformation and can be unlocked with these six strategies …
In this Q&A, Pritam Dutta of ArcLight Consulting explains why some ERP cloud migrations run into trouble, and how integrating HCM…
As with any job, there are various skills supply chain professionals need to succeed in their chosen field. Here are the five …
‘Supply Chain Management Best Practices’ author David Blanchard discusses pandemic lessons, how the SCOR model can help and why …
All Rights Reserved, Copyright 2018 – 2021, TechTarget