{"id":173426,"date":"2023-11-13T13:57:12","date_gmt":"2023-11-13T12:57:12","guid":{"rendered":"https:\/\/datascientest.com\/en\/?p=173426"},"modified":"2023-11-13T13:57:19","modified_gmt":"2023-11-13T12:57:19","slug":"navigating-data-drift-strategies-to-tackle-shifting-data-landscapes","status":"publish","type":"post","link":"https:\/\/datascientest.com\/en\/navigating-data-drift-strategies-to-tackle-shifting-data-landscapes","title":{"rendered":"Navigating Data Drift: Strategies to Tackle Shifting Data Landscapes"},"content":{"rendered":"\t\t<div data-elementor-type=\"wp-post\" data-elementor-id=\"173426\" class=\"elementor elementor-173426\" data-elementor-post-type=\"post\">\n\t\t\t\t\t\t\t<div class=\"elementor-element elementor-element-3807b5be e-flex e-con-boxed e-con\" data-id=\"3807b5be\" data-element_type=\"container\" data-settings=\"{&quot;content_width&quot;:&quot;boxed&quot;}\">\n\t\t\t\t\t<div class=\"e-con-inner\">\n\t\t\t\t<div class=\"elementor-element elementor-element-1d787302 elementor-widget elementor-widget-heading\" data-id=\"1d787302\" data-element_type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t<style>\/*! elementor - v3.15.0 - 02-08-2023 *\/\n.elementor-heading-title{padding:0;margin:0;line-height:1}.elementor-widget-heading .elementor-heading-title[class*=elementor-size-]>a{color:inherit;font-size:inherit;line-height:inherit}.elementor-widget-heading .elementor-heading-title.elementor-size-small{font-size:15px}.elementor-widget-heading .elementor-heading-title.elementor-size-medium{font-size:19px}.elementor-widget-heading .elementor-heading-title.elementor-size-large{font-size:29px}.elementor-widget-heading .elementor-heading-title.elementor-size-xl{font-size:39px}.elementor-widget-heading .elementor-heading-title.elementor-size-xxl{font-size:59px}<\/style><h2 class=\"elementor-heading-title elementor-size-default\">Data drift occurs when the data on which the model is run differs too greatly from the training data. This problem must be detected and anticipated, as it degrades prediction performance over time. Find out all you need to know about data drift: definition, dangers, solutions...<\/h2>\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-79156bf7 elementor-widget elementor-widget-text-editor\" data-id=\"79156bf7\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t<style>\/*! elementor - v3.15.0 - 02-08-2023 *\/\n.elementor-widget-text-editor.elementor-drop-cap-view-stacked .elementor-drop-cap{background-color:#69727d;color:#fff}.elementor-widget-text-editor.elementor-drop-cap-view-framed .elementor-drop-cap{color:#69727d;border:3px solid;background-color:transparent}.elementor-widget-text-editor:not(.elementor-drop-cap-view-default) .elementor-drop-cap{margin-top:8px}.elementor-widget-text-editor:not(.elementor-drop-cap-view-default) .elementor-drop-cap-letter{width:1em;height:1em}.elementor-widget-text-editor .elementor-drop-cap{float:left;text-align:center;line-height:1;font-size:50px}.elementor-widget-text-editor .elementor-drop-cap-letter{display:inline-block}<\/style>\t\t\t\t<p><a href=\"https:\/\/datascientest.com\/en\/cracking-the-code-on-underfitting-in-machine-learning\">Machine Learning<\/a> involves using available data to train a model to recognise, predict or reproduce that data. However, things can get complicated when the model is deployed during the <strong>production process.<\/strong><\/p><p>Let&#8217;s take the most c<strong>ommon case of supervised learning:<\/strong> the training data is labelled and this is what helps our model quantify its errors when training on past data&#8230; However, once the model is deployed, the data to be predicted is current data and therefore has no label.<\/p><p>Regardless of the<strong> model&#8217;s level of accuracy,<\/strong> predictions can only be relevant if the data submitted to the model in production is similar or statistically equivalent to the data used for training. If this is not the case, the phenomenon of &#8220;data drift&#8221; occurs.<\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t<div class=\"elementor-element elementor-element-2988bd74 e-flex e-con-boxed e-con\" data-id=\"2988bd74\" data-element_type=\"container\" data-settings=\"{&quot;content_width&quot;:&quot;boxed&quot;}\">\n\t\t\t\t\t<div class=\"e-con-inner\">\n\t\t\t\t<div class=\"elementor-element elementor-element-58a137b6 elementor-widget elementor-widget-heading\" data-id=\"58a137b6\" data-element_type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t<h3 class=\"elementor-heading-title elementor-size-default\">What causes data drift?<\/h3>\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-70a83046 elementor-widget elementor-widget-text-editor\" data-id=\"70a83046\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p><strong>Data Drift<\/strong> is a variation of real-world data from the data used to test and validate the model before deploying it in production.<\/p><p>Many factors can cause data drift. One of the main causes is the time dimension, as a long period can elapse between the time when the <a href=\"https:\/\/datascientest.com\/en\/adversarial-training-what-you-didnt-know-yet\">training data<\/a> is collected and the time when the model trained on this data is used for prediction from the real data.<\/p><p>Another common cause of <strong>data drift<\/strong> is seasonality. Some of the data collected during the summer can be very different during the winter, if the temperature or the length of the day has an impact on it, for example.<\/p><p>Similarly, data collected before an event will not necessarily be valid afterwards. For example, much of the data collected before the emergence of Covid is now obsolete. If a <a href=\"https:\/\/datascientest.com\/en\/data-poisoning-a-threat-to-machine-learning-models\">Machine Learning model<\/a> is trained on this data and tested on current data, a data drift problem will most likely arise once it has been deployed.<\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-2cf53968 elementor-widget elementor-widget-image\" data-id=\"2cf53968\" data-element_type=\"widget\" data-widget_type=\"image.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t<style>\/*! elementor - v3.15.0 - 02-08-2023 *\/\n.elementor-widget-image{text-align:center}.elementor-widget-image a{display:inline-block}.elementor-widget-image a img[src$=\".svg\"]{width:48px}.elementor-widget-image img{vertical-align:middle;display:inline-block}<\/style>\t\t\t\t\t\t\t\t\t<figure class=\"wp-caption\">\n\t\t\t\t\t\t\t\t\t\t<img decoding=\"async\" width=\"641\" height=\"509\" src=\"https:\/\/datascientest.com\/en\/files\/2023\/11\/image3.png\" class=\"attachment-full size-full wp-image-173432\" alt=\"\" srcset=\"https:\/\/datascientest.com\/en\/files\/2023\/11\/image3.png 641w, https:\/\/datascientest.com\/en\/files\/2023\/11\/image3-300x238.png 300w\" sizes=\"(max-width: 641px) 100vw, 641px\" \/>\t\t\t\t\t\t\t\t\t\t\t<figcaption class=\"widget-image-caption wp-caption-text\">Example: A possible cause of data drift: seasonality<\/figcaption>\n\t\t\t\t\t\t\t\t\t\t<\/figure>\n\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-594ae908 elementor-widget elementor-widget-text-editor\" data-id=\"594ae908\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p>Another example: A <a href=\"https:\/\/datascientest.com\/en\/mushroom-recognition\">Computer Vision<\/a> model trained on a dataset linked to European roads will not be suitable for roads in the United States. The choice of training data is therefore extremely important.<\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t<div class=\"elementor-element elementor-element-7a7f19c e-flex e-con-boxed e-con\" data-id=\"7a7f19c\" data-element_type=\"container\" data-settings=\"{&quot;content_width&quot;:&quot;boxed&quot;}\">\n\t\t\t\t\t<div class=\"e-con-inner\">\n\t\t\t\t<div class=\"elementor-element elementor-element-2939b0ea elementor-widget elementor-widget-heading\" data-id=\"2939b0ea\" data-element_type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t<h3 class=\"elementor-heading-title elementor-size-default\">What are the risks of Data Drift?<\/h3>\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-865ba93 elementor-widget elementor-widget-text-editor\" data-id=\"865ba93\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p>If a <strong>data drift<\/strong> is not identified in time, the model&#8217;s predictions will be wrong. The decisions taken on the basis of these predictions will therefore have a negative impact.<\/p><p>For example, if your tastes have changed and<strong> Netflix<\/strong> doesn&#8217;t have access to your most recent data, its model is likely to suggest films that you won&#8217;t like, or at least won&#8217;t like any more.<\/p><p>This example is not very worrying, but the consequences can be much more serious. For example, a trader misdirected by a <strong>data drift<\/strong> can invest colossal sums in a share with no current value.<\/p><p>Depending on the nature, extent and type of data drift, the effort required varies. Sometimes it is possible to solve the problem by re-training the model on new data. However, it may also be necessary to start from scratch.<\/p><p>In addition to the data, the model itself can also drift. This is the case, for example, of a model used before Covid to predict which students will opt for online courses. If used during the Covid crisis, this model would be completely useless.<\/p><p>This is known as &#8220;Concept Drift&#8221;. This problem can be solved through e-learning, by re-training the model on each observation.<\/p><p>It is important to build a reusable<strong> process for identifying data drift.<\/strong> You can also define thresholds for the percentage of drift not to be exceeded, or configure an alert system to take the necessary action before it&#8217;s too late.<\/p><p>Data drift can be detected from the evolution of the data itself, or when the model&#8217;s predictions are incorrect.<\/p><p>However, incorrect predictions can only be identified if a manual method of finding the correct label is available.<\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t<div class=\"elementor-element elementor-element-44531b0f e-flex e-con-boxed e-con\" data-id=\"44531b0f\" data-element_type=\"container\" data-settings=\"{&quot;content_width&quot;:&quot;boxed&quot;}\">\n\t\t\t\t\t<div class=\"e-con-inner\">\n\t\t\t\t<div class=\"elementor-element elementor-element-482f27fb elementor-widget elementor-widget-heading\" data-id=\"482f27fb\" data-element_type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t<h3 class=\"elementor-heading-title elementor-size-default\">How do you detect Data Drifts?<\/h3>\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-18fc12eb elementor-widget elementor-widget-text-editor\" data-id=\"18fc12eb\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p>When <a href=\"https:\/\/datascientest.com\/en\/feature-engineering-importance-for-machine-learning\">deploying a Machine Learning model,<\/a> it is important to regularly monitor its performance. Let&#8217;s take the example of a <a href=\"https:\/\/datascientest.com\/en\/u-net-computer-visions-neural-network\">Computer Vision model<\/a> built from a dataset of 100 different dog breeds.<\/p><p>To check whether its performance is decreasing due to data drift, it is necessary to review all the <strong>new images and predictions<\/strong> ingested by the feedback loop system. The predictions must then be examined, edited or validated.<\/p><p>By comparing the initial <strong>predictions with the edited predictions<\/strong>, it is possible to check whether or not performance is better than average. However, in order to validate all the predictions in real time, it is necessary to entrust access to the data to a large number of people. A large workforce is therefore required for this operation.<\/p><p><strong>Data Drifts<\/strong> can be identified using sequential analysis methods, model-based methods, or methods based on temporal distribution.<\/p><p>Sequential analysis methods such as <strong>DDM (Drift Detection Method) or EDDM (Early DDM)<\/strong> detect drift based on a fixed error rate.<\/p><p>A model-based method, on the other hand, uses a customised <a href=\"https:\/\/datascientest.com\/en\/unraveling-machine-learning-vs-deep-learning-key-differences-explained\">Machine Learning model<\/a> to identify drift, by determining the similarity between a point or group of points and a reference baseline.<\/p><p>Data that has been used to build the current model in production needs to be labelled as &#8216;0&#8217;, and real-time data needs to be labelled as &#8216;1&#8217;.<\/p><p>A model is then built to evaluate the results. If the model is highly accurate, this means that it can easily discriminate <a href=\"https:\/\/datascientest.com\/en\/datasets-top-5-places-to-find-quality-datasets\">between the two sets of data.<\/a><\/p><p>We can conclude that a drift has occurred, and that the model needs to be recalibrated.<\/p><p>On the other hand, if the accuracy of the model is around 0.5, this indicates that there has been no significant drift between the two datasets. The model can therefore continue to be used.<\/p><p>This method is effective, but requires the model to be trained and tested each time <strong>new data becomes available,<\/strong> which can be very costly.<\/p><p>Finally, a method based on time window distribution takes into account the timestamp and occurrence of events. The ADWIN (Adaptive Windowing) technique, for example, consists of starting to dynamically increase the W window as long as there is no apparent change in the context, and reducing it when a change is detected.<\/p><p>The <a href=\"https:\/\/datascientest.com\/en\/boosting-business-with-3-essential-machine-learning-algorithms\">ADWIN algorithm<\/a> attempts to find two sub-windows with distinct averages, and abandons the oldest sub-window.<\/p><p>The main methods for calculating the difference between two populations are the Population Stability Index, the <strong>Kullback-Leibler Divergence, the Jensen-Shannon Divergence<\/strong>, the Kolmogorov-Smirnov Test and the Wasserstein Distance.<\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t<div class=\"elementor-element elementor-element-634b3300 e-flex e-con-boxed e-con\" data-id=\"634b3300\" data-element_type=\"container\" data-settings=\"{&quot;content_width&quot;:&quot;boxed&quot;}\">\n\t\t\t\t\t<div class=\"e-con-inner\">\n\t\t\t\t<div class=\"elementor-element elementor-element-38b23c8b elementor-widget elementor-widget-heading\" data-id=\"38b23c8b\" data-element_type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t<h3 class=\"elementor-heading-title elementor-size-default\">What is the Kolmogorov-Smirnov Test?<\/h3>\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-2b19c484 elementor-widget elementor-widget-text-editor\" data-id=\"2b19c484\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p>The <strong>Kolmogorov-Smirnov Test (KS Test)<\/strong> is a statistical test that requires no user input. It is used to compare continuous or discontinuous probability distributions.<\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-38230e4f elementor-widget elementor-widget-image\" data-id=\"38230e4f\" data-element_type=\"widget\" data-widget_type=\"image.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t\t\t\t<figure class=\"wp-caption\">\n\t\t\t\t\t\t\t\t\t\t<img decoding=\"async\" width=\"543\" height=\"616\" src=\"https:\/\/datascientest.com\/en\/files\/2023\/11\/image4.png\" class=\"attachment-full size-full wp-image-173433\" alt=\"\" srcset=\"https:\/\/datascientest.com\/en\/files\/2023\/11\/image4.png 543w, https:\/\/datascientest.com\/en\/files\/2023\/11\/image4-264x300.png 264w\" sizes=\"(max-width: 543px) 100vw, 543px\" \/>\t\t\t\t\t\t\t\t\t\t\t<figcaption class=\"widget-image-caption wp-caption-text\">Illustration of the Kolmogorov-Smirnov test<\/figcaption>\n\t\t\t\t\t\t\t\t\t\t<\/figure>\n\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-5d57c900 elementor-widget elementor-widget-text-editor\" data-id=\"5d57c900\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p>It can be <strong>used to compare the distribution<\/strong> of a sample with the reference probability distribution, or two samples of population distributions with equal variability, to check whether they are drawn from different population distributions of unknown parameters.<\/p><p>This method takes its name from Andrey Kolmogorov and <strong>Nikolai Smirnov<\/strong>, who were the first to propose its use for very large arrays of random numbers generated by humans.<\/p><p>The Kolmogorov-Smirnov statistic is used to quantify the distance between the distribution functions of real-world sample data and the data distributions of the training set.<\/p><p>It yields a list of &#8216;p&#8217; values for the training data and the real-world data, and the distance between the two curves reveals the data drift.<\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t<div class=\"elementor-element elementor-element-60fc6d62 e-flex e-con-boxed e-con\" data-id=\"60fc6d62\" data-element_type=\"container\" data-settings=\"{&quot;content_width&quot;:&quot;boxed&quot;}\">\n\t\t\t\t\t<div class=\"e-con-inner\">\n\t\t\t\t<div class=\"elementor-element elementor-element-5f17d117 elementor-widget elementor-widget-heading\" data-id=\"5f17d117\" data-element_type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t<h3 class=\"elementor-heading-title elementor-size-default\">How do I solve a data drift problem?<\/h3>\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-50450d44 elementor-widget elementor-widget-text-editor\" data-id=\"50450d44\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p><strong>Data drift<\/strong> can be detected quickly using the various methods described above. It is then important to take steps to resolve the problem.<\/p><p>However, there are several things to consider before taking action. First of all, check the data in the feedback loop.<\/p><p>It is essential not to inject deteriorated data into the pipeline.<\/p><p>If the <strong>quality of the data<\/strong> is correct, check the model&#8217;s performance for a specific business use case. If the model is still capable of generating reliable predictions, you might consider adding a percentage of the new data ingested via the feedback loop to the dataset and retraining the model.<\/p><p>This gives a better representation of reality, while maintaining the same degree of accuracy.<\/p><p>If the model no longer performs as expected, this means that the<a href=\"https:\/\/datascientest.com\/en\/the-importance-of-cross-validation\"> <strong>data used for training<\/strong> and validation<\/a> were not representative of the reality of your use case. It is therefore necessary to rebuild a complete dataset with the new data ingested, or to wait until you have enough data to train the model again. <\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t<div class=\"elementor-element elementor-element-1e87ef73 e-flex e-con-boxed e-con\" data-id=\"1e87ef73\" data-element_type=\"container\" data-settings=\"{&quot;content_width&quot;:&quot;boxed&quot;}\">\n\t\t\t\t\t<div class=\"e-con-inner\">\n\t\t\t\t<div class=\"elementor-element elementor-element-18ca16f8 elementor-widget elementor-widget-heading\" data-id=\"18ca16f8\" data-element_type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t<h3 class=\"elementor-heading-title elementor-size-default\">Data Drift vs Concept Drift: what's the difference?<\/h3>\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-6d21db80 elementor-widget elementor-widget-text-editor\" data-id=\"6d21db80\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p><strong>Data drift<\/strong> occurs when there is a mismatch between a model&#8217;s training data and real-world data. This change in the data degrades the model&#8217;s performance.<\/p><p><strong>Concept drift<\/strong> occurs when the target of the model&#8217;s predictions or its statistical properties change over time. The model has learned a function mapping the target variable during training, but has forgotten them over time or is unable to use the patterns in a new environment.<\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-3d0296cb elementor-widget elementor-widget-image\" data-id=\"3d0296cb\" data-element_type=\"widget\" data-widget_type=\"image.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t\t\t\t<figure class=\"wp-caption\">\n\t\t\t\t\t\t\t\t\t\t<img decoding=\"async\" width=\"921\" height=\"401\" src=\"https:\/\/datascientest.com\/en\/files\/2023\/11\/image1-1-1.png\" class=\"attachment-full size-full wp-image-173434\" alt=\"\" srcset=\"https:\/\/datascientest.com\/en\/files\/2023\/11\/image1-1-1.png 921w, https:\/\/datascientest.com\/en\/files\/2023\/11\/image1-1-1-300x131.png 300w, https:\/\/datascientest.com\/en\/files\/2023\/11\/image1-1-1-768x334.png 768w\" sizes=\"(max-width: 921px) 100vw, 921px\" \/>\t\t\t\t\t\t\t\t\t\t\t<figcaption class=\"widget-image-caption wp-caption-text\">Concept drift v\/s Data Drift<\/figcaption>\n\t\t\t\t\t\t\t\t\t\t<\/figure>\n\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-55884810 elementor-widget elementor-widget-text-editor\" data-id=\"55884810\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p>For example, spam detection models may need to be adjusted if the definition of spam evolves. Concept Drift can occur seasonally (such as clothing fashions), suddenly (such as consumer behaviour after the <strong>Covid pandemic) or gradually.<\/strong><\/p><p><strong>Concept Drift<\/strong> can be measured by continuously monitoring training data and identifying changes within dataset relationships. Popular algorithms for detecting concept drift include <strong>ADWIN (ADaptive WINdowing)<\/strong> for streaming data, and the KS or chi-square test for batch data.<\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t<div class=\"elementor-element elementor-element-7fc0f0ba e-flex e-con-boxed e-con\" data-id=\"7fc0f0ba\" data-element_type=\"container\" data-settings=\"{&quot;content_width&quot;:&quot;boxed&quot;}\">\n\t\t\t\t\t<div class=\"e-con-inner\">\n\t\t\t\t<div class=\"elementor-element elementor-element-44d9477d elementor-widget elementor-widget-heading\" data-id=\"44d9477d\" data-element_type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t<h3 class=\"elementor-heading-title elementor-size-default\">Conclusion<\/h3>\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-22ff5d94 elementor-widget elementor-widget-text-editor\" data-id=\"22ff5d94\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p>And there you have it, you now know where <strong>data drift<\/strong> comes from, its causes, its characteristics and the possible solutions to be tested to resolve it. <strong>Data drift<\/strong> and concept drift are among the most common problems encountered in training <a href=\"https:\/\/datascientest.com\/en\/boosting-business-with-3-essential-machine-learning-algorithms\">Machine Learning models.<\/a><\/p><p>To become an expert in this field, you can choose DataScientest.<\/p><p>Our various training courses enable you to master Data Science and Machine Learning. By choosing one of our programmes, you can acquire the skills required to work as a Data Analyst, Data Scientist, <strong>Data Engineer or Machine Learning Engineer.<\/strong><\/p><p>At the end of the course, you&#8217;ll know everything there is to know about <a href=\"https:\/\/datascientest.com\/en\/what-is-a-dataset-how-do-i-work-with-it\">managing datasets<\/a> and training Machine Learning models. You&#8217;ll be ready to apply your knowledge in the workplace, just like the 80% of our alumni who found a job immediately.<\/p><p>Our innovative Blended Learning approach combines online learning on a coached platform and group Masterclasses.<\/p><p>What&#8217;s more, you can choose between an intensive BootCamp or Continuing Education. All our courses are entirely distance learning.<\/p><p>Our state-recognised organisation is <strong>eligible for state funding.<\/strong> Don&#8217;t wait any longer, and discover DataScientest!<\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-71fb6864 elementor-align-center button-principale elementor-widget elementor-widget-button\" data-id=\"71fb6864\" data-element_type=\"widget\" data-widget_type=\"button.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<div class=\"elementor-button-wrapper\">\n\t\t\t<a class=\"elementor-button elementor-button-link elementor-size-sm elementor-animation-grow\" href=\"https:\/\/datascientest.com\/en\/data-courses\">\n\t\t\t\t\t\t<span class=\"elementor-button-content-wrapper\">\n\t\t\t\t\t\t<span class=\"elementor-button-text\">Discover our training courses<\/span>\n\t\t<\/span>\n\t\t\t\t\t<\/a>\n\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t\t\t<\/div>\n\t\t","protected":false},"excerpt":{"rendered":"<p>Data drift occurs when the data on which the model is run differs too greatly from the training data. This problem must be detected and anticipated, as it degrades prediction performance over time. Find out all you need to know about data drift: definition, dangers, solutions&#8230; Machine Learning involves using available data to train a [&hellip;]<\/p>\n","protected":false},"author":67,"featured_media":173428,"comment_status":"open","ping_status":"open","sticky":false,"template":"elementor_theme","format":"standard","meta":{"content-type":"","inline_featured_image":false},"categories":[2425],"tags":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO Premium plugin v21.7 (Yoast SEO v21.7) - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>Navigating Data Drift: Strategies to Tackle Shifting Data Landscapes<\/title>\n<meta name=\"description\" content=\"Uncover the mysteries of data drift and learn effective solutions to keep your analytics on course. Explore what data drift is, understand ..\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/datascientest.com\/en\/navigating-data-drift-strategies-to-tackle-shifting-data-landscapes\" \/>\n<meta property=\"og:locale\" content=\"en_GB\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Navigating Data Drift: Strategies to Tackle Shifting Data Landscapes\" \/>\n<meta property=\"og:description\" content=\"Data drift occurs when the data on which the model is run differs too greatly from the training data. This problem must be detected and anticipated, as it\" \/>\n<meta property=\"og:url\" content=\"https:\/\/datascientest.com\/en\/navigating-data-drift-strategies-to-tackle-shifting-data-landscapes\" \/>\n<meta property=\"og:site_name\" content=\"Data Science Courses | DataScientest\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/DataScientestEN\" \/>\n<meta property=\"article:published_time\" content=\"2023-11-13T12:57:12+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2023-11-13T12:57:19+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/datascientest.com\/en\/files\/2023\/11\/data-drift.webp\" \/>\n\t<meta property=\"og:image:width\" content=\"1024\" \/>\n\t<meta property=\"og:image:height\" content=\"563\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/webp\" \/>\n<meta name=\"author\" content=\"Melanie\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@DataScientest__\" \/>\n<meta name=\"twitter:site\" content=\"@DataScientest__\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Melanie\" \/>\n\t<meta name=\"twitter:label2\" content=\"Estimated reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"9 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/datascientest.com\/en\/navigating-data-drift-strategies-to-tackle-shifting-data-landscapes#article\",\"isPartOf\":{\"@id\":\"https:\/\/datascientest.com\/en\/navigating-data-drift-strategies-to-tackle-shifting-data-landscapes\"},\"author\":{\"name\":\"Melanie\",\"@id\":\"https:\/\/datascientest.com\/en\/#\/schema\/person\/19d5f7ade9407eb69af3bf6d1ae1731d\"},\"headline\":\"Navigating Data Drift: Strategies to Tackle Shifting Data Landscapes\",\"datePublished\":\"2023-11-13T12:57:12+00:00\",\"dateModified\":\"2023-11-13T12:57:19+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/datascientest.com\/en\/navigating-data-drift-strategies-to-tackle-shifting-data-landscapes\"},\"wordCount\":1782,\"commentCount\":0,\"publisher\":{\"@id\":\"https:\/\/datascientest.com\/en\/#organization\"},\"articleSection\":[\"Data Science\"],\"inLanguage\":\"en-GB\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/datascientest.com\/en\/navigating-data-drift-strategies-to-tackle-shifting-data-landscapes#respond\"]}],\"copyrightYear\":\"2023\",\"copyrightHolder\":{\"@id\":\"https:\/\/datascientest.com\/#organization\"}},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/datascientest.com\/en\/navigating-data-drift-strategies-to-tackle-shifting-data-landscapes\",\"url\":\"https:\/\/datascientest.com\/en\/navigating-data-drift-strategies-to-tackle-shifting-data-landscapes\",\"name\":\"Navigating Data Drift: Strategies to Tackle Shifting Data Landscapes\",\"isPartOf\":{\"@id\":\"https:\/\/datascientest.com\/en\/#website\"},\"datePublished\":\"2023-11-13T12:57:12+00:00\",\"dateModified\":\"2023-11-13T12:57:19+00:00\",\"description\":\"Uncover the mysteries of data drift and learn effective solutions to keep your analytics on course. Explore what data drift is, understand ..\",\"inLanguage\":\"en-GB\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/datascientest.com\/en\/navigating-data-drift-strategies-to-tackle-shifting-data-landscapes\"]}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/datascientest.com\/en\/#website\",\"url\":\"https:\/\/datascientest.com\/en\/\",\"name\":\"Data Science Courses | DataScientest\",\"description\":\"Leader in Data Science Courses\",\"publisher\":{\"@id\":\"https:\/\/datascientest.com\/en\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/datascientest.com\/en\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-GB\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/datascientest.com\/en\/#organization\",\"name\":\"DataScientest\",\"url\":\"https:\/\/datascientest.com\/en\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-GB\",\"@id\":\"https:\/\/datascientest.com\/en\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/datascientest.com\/en\/wp-content\/blogs.dir\/9\/files\/2024\/02\/DataScientest_icon_blue-2.png\",\"contentUrl\":\"https:\/\/datascientest.com\/en\/wp-content\/blogs.dir\/9\/files\/2024\/02\/DataScientest_icon_blue-2.png\",\"width\":301,\"height\":301,\"caption\":\"DataScientest\"},\"image\":{\"@id\":\"https:\/\/datascientest.com\/en\/#\/schema\/logo\/image\/\"},\"sameAs\":[\"https:\/\/www.facebook.com\/DataScientestEN\",\"https:\/\/twitter.com\/DataScientest__\",\"https:\/\/www.linkedin.com\/school\/datascientest\/mycompany\/\",\"https:\/\/www.tiktok.com\/@datascientest\"]},{\"@type\":\"Person\",\"@id\":\"https:\/\/datascientest.com\/en\/#\/schema\/person\/19d5f7ade9407eb69af3bf6d1ae1731d\",\"name\":\"Melanie\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-GB\",\"@id\":\"https:\/\/datascientest.com\/en\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/8d3d660a0698de06f13ecf4823519da1?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/8d3d660a0698de06f13ecf4823519da1?s=96&d=mm&r=g\",\"caption\":\"Melanie\"},\"url\":\"https:\/\/datascientest.com\/en\/author\/melaniekrause\"}]}<\/script>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"Navigating Data Drift: Strategies to Tackle Shifting Data Landscapes","description":"Uncover the mysteries of data drift and learn effective solutions to keep your analytics on course. Explore what data drift is, understand ..","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/datascientest.com\/en\/navigating-data-drift-strategies-to-tackle-shifting-data-landscapes","og_locale":"en_GB","og_type":"article","og_title":"Navigating Data Drift: Strategies to Tackle Shifting Data Landscapes","og_description":"Data drift occurs when the data on which the model is run differs too greatly from the training data. This problem must be detected and anticipated, as it","og_url":"https:\/\/datascientest.com\/en\/navigating-data-drift-strategies-to-tackle-shifting-data-landscapes","og_site_name":"Data Science Courses | DataScientest","article_publisher":"https:\/\/www.facebook.com\/DataScientestEN","article_published_time":"2023-11-13T12:57:12+00:00","article_modified_time":"2023-11-13T12:57:19+00:00","og_image":[{"width":1024,"height":563,"url":"https:\/\/datascientest.com\/en\/files\/2023\/11\/data-drift.webp","type":"image\/webp"}],"author":"Melanie","twitter_card":"summary_large_image","twitter_creator":"@DataScientest__","twitter_site":"@DataScientest__","twitter_misc":{"Written by":"Melanie","Estimated reading time":"9 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/datascientest.com\/en\/navigating-data-drift-strategies-to-tackle-shifting-data-landscapes#article","isPartOf":{"@id":"https:\/\/datascientest.com\/en\/navigating-data-drift-strategies-to-tackle-shifting-data-landscapes"},"author":{"name":"Melanie","@id":"https:\/\/datascientest.com\/en\/#\/schema\/person\/19d5f7ade9407eb69af3bf6d1ae1731d"},"headline":"Navigating Data Drift: Strategies to Tackle Shifting Data Landscapes","datePublished":"2023-11-13T12:57:12+00:00","dateModified":"2023-11-13T12:57:19+00:00","mainEntityOfPage":{"@id":"https:\/\/datascientest.com\/en\/navigating-data-drift-strategies-to-tackle-shifting-data-landscapes"},"wordCount":1782,"commentCount":0,"publisher":{"@id":"https:\/\/datascientest.com\/en\/#organization"},"articleSection":["Data Science"],"inLanguage":"en-GB","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/datascientest.com\/en\/navigating-data-drift-strategies-to-tackle-shifting-data-landscapes#respond"]}],"copyrightYear":"2023","copyrightHolder":{"@id":"https:\/\/datascientest.com\/#organization"}},{"@type":"WebPage","@id":"https:\/\/datascientest.com\/en\/navigating-data-drift-strategies-to-tackle-shifting-data-landscapes","url":"https:\/\/datascientest.com\/en\/navigating-data-drift-strategies-to-tackle-shifting-data-landscapes","name":"Navigating Data Drift: Strategies to Tackle Shifting Data Landscapes","isPartOf":{"@id":"https:\/\/datascientest.com\/en\/#website"},"datePublished":"2023-11-13T12:57:12+00:00","dateModified":"2023-11-13T12:57:19+00:00","description":"Uncover the mysteries of data drift and learn effective solutions to keep your analytics on course. Explore what data drift is, understand ..","inLanguage":"en-GB","potentialAction":[{"@type":"ReadAction","target":["https:\/\/datascientest.com\/en\/navigating-data-drift-strategies-to-tackle-shifting-data-landscapes"]}]},{"@type":"WebSite","@id":"https:\/\/datascientest.com\/en\/#website","url":"https:\/\/datascientest.com\/en\/","name":"Data Science Courses | DataScientest","description":"Leader in Data Science Courses","publisher":{"@id":"https:\/\/datascientest.com\/en\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/datascientest.com\/en\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-GB"},{"@type":"Organization","@id":"https:\/\/datascientest.com\/en\/#organization","name":"DataScientest","url":"https:\/\/datascientest.com\/en\/","logo":{"@type":"ImageObject","inLanguage":"en-GB","@id":"https:\/\/datascientest.com\/en\/#\/schema\/logo\/image\/","url":"https:\/\/datascientest.com\/en\/wp-content\/blogs.dir\/9\/files\/2024\/02\/DataScientest_icon_blue-2.png","contentUrl":"https:\/\/datascientest.com\/en\/wp-content\/blogs.dir\/9\/files\/2024\/02\/DataScientest_icon_blue-2.png","width":301,"height":301,"caption":"DataScientest"},"image":{"@id":"https:\/\/datascientest.com\/en\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/DataScientestEN","https:\/\/twitter.com\/DataScientest__","https:\/\/www.linkedin.com\/school\/datascientest\/mycompany\/","https:\/\/www.tiktok.com\/@datascientest"]},{"@type":"Person","@id":"https:\/\/datascientest.com\/en\/#\/schema\/person\/19d5f7ade9407eb69af3bf6d1ae1731d","name":"Melanie","image":{"@type":"ImageObject","inLanguage":"en-GB","@id":"https:\/\/datascientest.com\/en\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/8d3d660a0698de06f13ecf4823519da1?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/8d3d660a0698de06f13ecf4823519da1?s=96&d=mm&r=g","caption":"Melanie"},"url":"https:\/\/datascientest.com\/en\/author\/melaniekrause"}]}},"_links":{"self":[{"href":"https:\/\/datascientest.com\/en\/wp-json\/wp\/v2\/posts\/173426"}],"collection":[{"href":"https:\/\/datascientest.com\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/datascientest.com\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/datascientest.com\/en\/wp-json\/wp\/v2\/users\/67"}],"replies":[{"embeddable":true,"href":"https:\/\/datascientest.com\/en\/wp-json\/wp\/v2\/comments?post=173426"}],"version-history":[{"count":0,"href":"https:\/\/datascientest.com\/en\/wp-json\/wp\/v2\/posts\/173426\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/datascientest.com\/en\/wp-json\/wp\/v2\/media\/173428"}],"wp:attachment":[{"href":"https:\/\/datascientest.com\/en\/wp-json\/wp\/v2\/media?parent=173426"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/datascientest.com\/en\/wp-json\/wp\/v2\/categories?post=173426"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/datascientest.com\/en\/wp-json\/wp\/v2\/tags?post=173426"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}