{"id":184236,"date":"2024-04-10T06:30:00","date_gmt":"2024-04-10T05:30:00","guid":{"rendered":"https:\/\/datascientest.com\/en\/?p=184236"},"modified":"2024-06-11T11:00:53","modified_gmt":"2024-06-11T10:00:53","slug":"introduction-to-kubeflow-for-mlops","status":"publish","type":"post","link":"https:\/\/datascientest.com\/en\/introduction-to-kubeflow-for-mlops","title":{"rendered":"Introduction to Kubeflow for MLOps<br><i><small>by Tony Ruiz, GCP Customer Engineer at Google<\/small><\/i>"},"content":{"rendered":"\t\t<div data-elementor-type=\"wp-post\" data-elementor-id=\"184236\" class=\"elementor elementor-184236\" data-elementor-post-type=\"post\">\n\t\t\t\t\t\t\t<div class=\"elementor-element elementor-element-c39f041 e-flex e-con-boxed e-con\" data-id=\"c39f041\" data-element_type=\"container\" data-settings=\"{&quot;content_width&quot;:&quot;boxed&quot;}\">\n\t\t\t\t\t<div class=\"e-con-inner\">\n\t\t\t\t<div class=\"elementor-element elementor-element-df8b61d elementor-widget elementor-widget-heading\" data-id=\"df8b61d\" data-element_type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t<style>\/*! elementor - v3.15.0 - 02-08-2023 *\/\n.elementor-heading-title{padding:0;margin:0;line-height:1}.elementor-widget-heading .elementor-heading-title[class*=elementor-size-]>a{color:inherit;font-size:inherit;line-height:inherit}.elementor-widget-heading .elementor-heading-title.elementor-size-small{font-size:15px}.elementor-widget-heading .elementor-heading-title.elementor-size-medium{font-size:19px}.elementor-widget-heading .elementor-heading-title.elementor-size-large{font-size:29px}.elementor-widget-heading .elementor-heading-title.elementor-size-xl{font-size:39px}.elementor-widget-heading .elementor-heading-title.elementor-size-xxl{font-size:59px}<\/style><h3 class=\"elementor-heading-title elementor-size-default\">Motivation<\/h3>\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-3c8e032 elementor-widget elementor-widget-text-editor\" data-id=\"3c8e032\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t<style>\/*! elementor - v3.15.0 - 02-08-2023 *\/\n.elementor-widget-text-editor.elementor-drop-cap-view-stacked .elementor-drop-cap{background-color:#69727d;color:#fff}.elementor-widget-text-editor.elementor-drop-cap-view-framed .elementor-drop-cap{color:#69727d;border:3px solid;background-color:transparent}.elementor-widget-text-editor:not(.elementor-drop-cap-view-default) .elementor-drop-cap{margin-top:8px}.elementor-widget-text-editor:not(.elementor-drop-cap-view-default) .elementor-drop-cap-letter{width:1em;height:1em}.elementor-widget-text-editor .elementor-drop-cap{float:left;text-align:center;line-height:1;font-size:50px}.elementor-widget-text-editor .elementor-drop-cap-letter{display:inline-block}<\/style>\t\t\t\t<p><span style=\"font-weight: 400;\"><a href=\"https:\/\/datascientest.com\/en\/manifold-learning-what-is-this-machine-learning-method\">Machine Learning Development<\/a> goes beyond developing a model. Productionizing a model quickly becomes a multi-faceted, indeed even a multi-disciplinary process that involves several key stakeholders across both the business and IT. As depicted in the image below, the code to actually develop a model is a small component compared to other complexities such as configuring serving infrastructure, data verification, and monitoring. <\/span><\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-f7cf0cf elementor-widget elementor-widget-image\" data-id=\"f7cf0cf\" data-element_type=\"widget\" data-widget_type=\"image.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t<style>\/*! elementor - v3.15.0 - 02-08-2023 *\/\n.elementor-widget-image{text-align:center}.elementor-widget-image a{display:inline-block}.elementor-widget-image a img[src$=\".svg\"]{width:48px}.elementor-widget-image img{vertical-align:middle;display:inline-block}<\/style>\t\t\t\t\t\t\t\t\t\t\t\t<img decoding=\"async\" width=\"1074\" height=\"452\" src=\"https:\/\/datascientest.com\/en\/files\/2024\/04\/image1.png\" class=\"attachment-full size-full wp-image-184239\" alt=\"\" srcset=\"https:\/\/datascientest.com\/en\/files\/2024\/04\/image1.png 1074w, https:\/\/datascientest.com\/en\/files\/2024\/04\/image1-300x126.png 300w, https:\/\/datascientest.com\/en\/files\/2024\/04\/image1-1024x431.png 1024w, https:\/\/datascientest.com\/en\/files\/2024\/04\/image1-768x323.png 768w\" sizes=\"(max-width: 1074px) 100vw, 1074px\" \/>\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t<div class=\"elementor-element elementor-element-750864c e-flex e-con-boxed e-con\" data-id=\"750864c\" data-element_type=\"container\" data-settings=\"{&quot;content_width&quot;:&quot;boxed&quot;}\">\n\t\t\t\t\t<div class=\"e-con-inner\">\n\t\t\t\t<div class=\"elementor-element elementor-element-ca7fff4 elementor-widget elementor-widget-heading\" data-id=\"ca7fff4\" data-element_type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t<h3 class=\"elementor-heading-title elementor-size-default\">Purpose<\/h3>\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-656e11c elementor-widget elementor-widget-text-editor\" data-id=\"656e11c\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p><span style=\"font-weight: 400;\">The goal of this article is to serve as a gentle introduction to <strong>Kubeflow pipelines<\/strong> and how it can address the challenges in <strong>ML Learning Lifecycle<\/strong> development. To address all the components in the image above would go beyond the scope of a short blog. For the sake of simplicity, we are going to reference Vertex Pipelines on Google Cloud for our example. Note that <a href=\"https:\/\/www.kubeflow.org\/docs\/started\/installing-kubeflow\/\">Kubeflow<\/a> is an open source project and can be deployed in several different environments.<\/span><\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t<div class=\"elementor-element elementor-element-aa9bc61 e-flex e-con-boxed e-con\" data-id=\"aa9bc61\" data-element_type=\"container\" data-settings=\"{&quot;content_width&quot;:&quot;boxed&quot;}\">\n\t\t\t\t\t<div class=\"e-con-inner\">\n\t\t\t\t<div class=\"elementor-element elementor-element-4be6862 elementor-widget elementor-widget-heading\" data-id=\"4be6862\" data-element_type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t<h3 class=\"elementor-heading-title elementor-size-default\">What is Kubeflow ?<\/h3>\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-5f5a31d elementor-widget elementor-widget-text-editor\" data-id=\"5f5a31d\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p><span style=\"font-weight: 400;\">Kubeflow is an open-source <a href=\"https:\/\/datascientest.com\/en\/mlops-devops-applied-to-machine-learning-projects\">MLOps<\/a> (<strong>Machine Learning Operations<\/strong>) platform built on top of <a href=\"https:\/\/datascientest.com\/en\/why-kubernetes-has-become-an-indispensable-tool-in-data-science\">Kubernetes<\/a>. It simplifies and automates various stages of the machine learning lifecycle, <strong>from data preparation and model training to deployment and monitoring<\/strong>. Think of it as a toolkit that helps you build and manage your ML workflows efficiently and portably across different platforms.<\/span><\/p><p><span style=\"font-weight: 400;\">Here are some key aspects of Kubeflow:<\/span><\/p><ul><li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\"><strong>Components<\/strong>: It provides a collection of components for different stages of the ML lifecycle, such as pipelines for <a href=\"https:\/\/datascientest.com\/en\/modernising-data-processing-with-new-tools\">data processing<\/a>, notebooks for experimentation, training jobs for model training, and <strong>KFServing<\/strong> for model deployment.<\/span><\/li><li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\"><strong>Kubernetes integration<\/strong>: It leverages Kubernetes for resource <strong>management<\/strong>, <strong>scheduling<\/strong>, and <strong>scaling<\/strong>. This makes Kubeflow workflows portable and scalable across different environments.<\/span><\/li><li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\"><strong>Flexibility and customization<\/strong>: You can tailor your <strong>ML workflows<\/strong> by choosing and configuring the specific components you need.<\/span><\/li><li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\"><strong>Community-driven<\/strong>: Kubeflow is an active open-source project with a <strong>vibrant community<\/strong> that contributes to its development and supports users.<\/span><\/li><\/ul>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-0886de3 elementor-widget elementor-widget-text-editor\" data-id=\"0886de3\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p><span style=\"font-weight: 400;\">Several different <a href=\"https:\/\/datascientest.com\/en\/choosing-the-right-cloud-provider-aws-vs-azure-vs-gcp-unveiled\">cloud providers<\/a> (<a href=\"https:\/\/datascientest.com\/en\/amazon-web-services-aws-unveiling-the-power-of-the-amazon-cloud\">AWS<\/a>, <a href=\"https:\/\/datascientest.com\/en\/google-cloud-platform-monitoring-features-and-benefits\">Google Cloud<\/a>, <a href=\"https:\/\/datascientest.com\/en\/microsoft-azure-empower-yourself-with-knowledge\">Microsoft Azure<\/a> ) offer pre-configured Kubeflow distributions that can be installed on a Kubernetes cluster. For this article, <a href=\"https:\/\/cloud.google.com\/vertex-ai\/docs\/pipelines\/introduction\">Vertex Pipelines<\/a> will be used since it is a<\/span><b> fully-managed<\/b><span style=\"font-weight: 400;\"> Kubeflow service. Fully managed cloud services are cloud services that are handled by a cloud service provider using automations, typically meaning that a developer doesn&#8217;t have to set up and manage machines, patching, or backing up clusters.\u00a0<\/span><\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t<div class=\"elementor-element elementor-element-ebbc05a e-flex e-con-boxed e-con\" data-id=\"ebbc05a\" data-element_type=\"container\" data-settings=\"{&quot;content_width&quot;:&quot;boxed&quot;}\">\n\t\t\t\t\t<div class=\"e-con-inner\">\n\t\t\t\t<div class=\"elementor-element elementor-element-5f09bc3 elementor-widget elementor-widget-heading\" data-id=\"5f09bc3\" data-element_type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t<h3 class=\"elementor-heading-title elementor-size-default\">Pipelines and components<\/h3>\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-596be48 elementor-widget elementor-widget-text-editor\" data-id=\"596be48\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p><span style=\"font-weight: 400;\">A Kubeflow Pipeline is a platform-agnostic way to define, orchestrate, and manage repeatable, end-to-end machine learning (ML) workflows based on containers.<\/span><\/p><p><span style=\"font-weight: 400;\">Think of a pipeline as the workflow for your machine learning job. A <\/span><b>pipeline<\/b><span style=\"font-weight: 400;\"> is built up from <\/span><b>components<\/b><span style=\"font-weight: 400;\">. Components are light weight abstractions of a task in a pipeline. These tasks can be isolated containers or implementations of a function.\u00a0 This is useful because now we can reap several benefits in our machine learning workflows<\/span><\/p><ul><li><b>Modularity<\/b><span style=\"font-weight: 400;\">: Components break down complex machine learning workflows into manageable, reusable steps. This improves organization and scalability.<\/span><\/li><li><b>Reusability<\/b><span style=\"font-weight: 400;\">: Well-defined components can be used across various pipelines, reducing code duplication and promoting efficiency.<\/span><\/li><li><b>Flexibility<\/b><span style=\"font-weight: 400;\">: You can develop Kubeflow components in various ways, giving you flexibility in your technology choices.<\/span><\/li><\/ul>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-5940939 elementor-align-center button-principal elementor-widget elementor-widget-button\" data-id=\"5940939\" data-element_type=\"widget\" data-widget_type=\"button.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<div class=\"elementor-button-wrapper\">\n\t\t\t<a class=\"elementor-button elementor-button-link elementor-size-md\" href=\"https:\/\/datascientest.com\/en\/data-courses\">\n\t\t\t\t\t\t<span class=\"elementor-button-content-wrapper\">\n\t\t\t\t\t\t<span class=\"elementor-button-text\">Browse our training courses<\/span>\n\t\t<\/span>\n\t\t\t\t\t<\/a>\n\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t<div class=\"elementor-element elementor-element-960d8e6 e-flex e-con-boxed e-con\" data-id=\"960d8e6\" data-element_type=\"container\" data-settings=\"{&quot;content_width&quot;:&quot;boxed&quot;}\">\n\t\t\t\t\t<div class=\"e-con-inner\">\n\t\t\t\t<div class=\"elementor-element elementor-element-48130e1 elementor-widget elementor-widget-heading\" data-id=\"48130e1\" data-element_type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t<h3 class=\"elementor-heading-title elementor-size-default\">Hello Kubeflow - A Rudimentary Example<\/h3>\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-8f6186c elementor-widget elementor-widget-text-editor\" data-id=\"8f6186c\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p><span style=\"font-weight: 400;\">In this example we are going to create 3 functions\u00a0<\/span><\/p><ul><li><span style=\"font-weight: 400;\">Example_string will take a string and append some text to the string<\/span><\/li><li><span style=\"font-weight: 400;\">Example_number will take a number and add 10\u00a0<\/span><\/li><li><span style=\"font-weight: 400;\">Example_combo will take the output of both of those function return a string of the out<\/span><\/li><\/ul>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-ac9f0bc elementor-widget elementor-widget-image\" data-id=\"ac9f0bc\" data-element_type=\"widget\" data-widget_type=\"image.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<img decoding=\"async\" width=\"621\" height=\"458\" src=\"https:\/\/datascientest.com\/en\/files\/2024\/04\/image3.png\" class=\"attachment-full size-full wp-image-184246\" alt=\"\" srcset=\"https:\/\/datascientest.com\/en\/files\/2024\/04\/image3.png 621w, https:\/\/datascientest.com\/en\/files\/2024\/04\/image3-300x221.png 300w\" sizes=\"(max-width: 621px) 100vw, 621px\" \/>\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-97aa332 elementor-widget elementor-widget-text-editor\" data-id=\"97aa332\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<span style=\"font-weight: 400;\">A couple of points here\u00a0\u00a0<\/span>\n<ul>\n \t<li><span style=\"font-weight: 400;\">We are using the <a href=\"https:\/\/www.kubeflow.org\/docs\/components\/pipelines\/v2\/components\/lightweight-python-components\/\">kfp.dsl.component decorator<\/a> to transform our python functions into components<\/span><\/li>\n \t<li><span style=\"font-weight: 400;\">Each of these components will be executed in their own container, note that we are defining our base image as well as our packages to install.<\/span><\/li>\n<\/ul>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-a85166b elementor-widget elementor-widget-text-editor\" data-id=\"a85166b\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p><span style=\"font-weight: 400;\">Next we are going to define our pipeline, which will in turn define how the components are going to interact with each other. In this case, we are going to create a <a href=\"https:\/\/en.wikipedia.org\/wiki\/Directed_acyclic_graph\">DAG<\/a> pipeline which will take the output of the <i>example_string()<\/i> and <i>example_number()<\/i> functions and feed them into a the <i>example_combo()<\/i> function. Note that the pipeline creation is happening inside the <i>example_pipeline()<\/i> function and we are using the <a href=\"https:\/\/www.kubeflow.org\/docs\/components\/pipelines\/v2\/pipelines\/pipeline-basics\/\">kfp.dsl.pipeline<\/a> decorator.<\/span><\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-bb607b4 elementor-widget elementor-widget-image\" data-id=\"bb607b4\" data-element_type=\"widget\" data-widget_type=\"image.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<img decoding=\"async\" width=\"953\" height=\"612\" src=\"https:\/\/datascientest.com\/en\/files\/2024\/04\/image2.png\" class=\"attachment-full size-full wp-image-184247\" alt=\"\" srcset=\"https:\/\/datascientest.com\/en\/files\/2024\/04\/image2.png 953w, https:\/\/datascientest.com\/en\/files\/2024\/04\/image2-300x193.png 300w, https:\/\/datascientest.com\/en\/files\/2024\/04\/image2-768x493.png 768w\" sizes=\"(max-width: 953px) 100vw, 953px\" \/>\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-a096ae6 elementor-widget elementor-widget-text-editor\" data-id=\"a096ae6\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p><span style=\"font-weight: 400;\">After defining the pipeline function, the next step is to <\/span><b>compile<\/b><span style=\"font-weight: 400;\"> the pipeline. The output of this compilation is a YAML template (can also be JSON) that will give the instructions to the Kubeflow service on how to create the pipeline and its components. In this example, we are submitting the template to <strong>Google Cloud\u2019s Vertex Pipelines<\/strong> for execution. <\/span><\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-8be4b1c elementor-widget elementor-widget-image\" data-id=\"8be4b1c\" data-element_type=\"widget\" data-widget_type=\"image.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<img decoding=\"async\" width=\"529\" height=\"228\" src=\"https:\/\/datascientest.com\/en\/files\/2024\/04\/image5.png\" class=\"attachment-full size-full wp-image-184251\" alt=\"\" srcset=\"https:\/\/datascientest.com\/en\/files\/2024\/04\/image5.png 529w, https:\/\/datascientest.com\/en\/files\/2024\/04\/image5-300x129.png 300w\" sizes=\"(max-width: 529px) 100vw, 529px\" \/>\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t<div class=\"elementor-element elementor-element-af4c811 e-flex e-con-boxed e-con\" data-id=\"af4c811\" data-element_type=\"container\" data-settings=\"{&quot;content_width&quot;:&quot;boxed&quot;}\">\n\t\t\t\t\t<div class=\"e-con-inner\">\n\t\t\t\t<div class=\"elementor-element elementor-element-a1a793f elementor-widget elementor-widget-heading\" data-id=\"a1a793f\" data-element_type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t<h3 class=\"elementor-heading-title elementor-size-default\">A Minimalistic End-to-End Pipeline<\/h3>\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-4b92741 elementor-widget elementor-widget-text-editor\" data-id=\"4b92741\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<span style=\"font-weight: 400;\">If you are new to Google Cloud, refer to <a href=\"https:\/\/cloud.google.com\/docs\/get-started\">this page<\/a> to get started. In this example, we are going to be developing our pipeline on a <a href=\"https:\/\/cloud.google.com\/vertex-ai-notebooks?hl=en\">Vertex AI Notebook<\/a>. <\/span>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-61b1d7a elementor-widget elementor-widget-heading\" data-id=\"61b1d7a\" data-element_type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t<h4 class=\"elementor-heading-title elementor-size-medium\">Installation<\/h4>\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-13697d7 elementor-widget elementor-widget-code-highlight\" data-id=\"13697d7\" data-element_type=\"widget\" data-widget_type=\"code-highlight.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<div class=\"prismjs-default copy-to-clipboard \">\n\t\t\t<pre data-line=\"\" class=\"highlight-height language-python \">\n\t\t\t\t<code readonly=\"true\" class=\"language-python\">\n\t\t\t\t\t<xmp>! pip3 install --no-cache-dir --upgrade \"kfp>2\" \\\r\n                                        google-cloud-aiplatform\r\n\r\n\r\n\r\nimport google.cloud.aiplatform as aiplatform\r\nimport kfp\r\nfrom kfp import compiler, dsl\r\nfrom kfp.dsl import Artifact, Dataset, Input, Metrics, Model, Output, component\r\n\r\nfrom kfp.dsl import ClassificationMetrics\r\nfrom typing import NamedTuple\r\nimport os\r\n\r\nPROJECT_ID = '<YOUR-PROJECT-ID>' # replace with project ID\r\nREGION = '<YOUR-REGION>'\r\nEXPERIMENT = 'vertex-pipelines'\r\nSERIES = 'dev'\r\n\r\n# gcs bucket\r\nGCS_BUCKET = PROJECT_ID\r\nBUCKET_URI = f\"gs:\/\/{PROJECT_ID}-bucket\"  # @param {type:\"string\"}\r\n\r\naiplatform.init(project=PROJECT_ID, staging_bucket=BUCKET_URI)<\/xmp>\n\t\t\t\t<\/code>\n\t\t\t<\/pre>\n\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-18bb5c9 elementor-widget elementor-widget-text-editor\" data-id=\"18bb5c9\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p><span style=\"font-weight: 400;\">Note that Vertex Pipelines require a Google Cloud Storage bucket to store pipeline metadata and artifacts. If you don\u2019t have one created yet, you can create a bucket manually through the console or use this gsutil command line :<\/span><\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-12f50b5 elementor-widget elementor-widget-code-highlight\" data-id=\"12f50b5\" data-element_type=\"widget\" data-widget_type=\"code-highlight.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<div class=\"prismjs-default copy-to-clipboard \">\n\t\t\t<pre data-line=\"\" class=\"highlight-height language-python \">\n\t\t\t\t<code readonly=\"true\" class=\"language-python\">\n\t\t\t\t\t<xmp>! gsutil mb -l $REGION -p $PROJECT_ID $BUCKET_URI<\/xmp>\n\t\t\t\t<\/code>\n\t\t\t<\/pre>\n\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-a61b73a elementor-align-center button-principal elementor-widget elementor-widget-button\" data-id=\"a61b73a\" data-element_type=\"widget\" data-widget_type=\"button.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<div class=\"elementor-button-wrapper\">\n\t\t\t<a class=\"elementor-button elementor-button-link elementor-size-md\" href=\"https:\/\/datascientest.com\/en\/data-courses\">\n\t\t\t\t\t\t<span class=\"elementor-button-content-wrapper\">\n\t\t\t\t\t\t<span class=\"elementor-button-text\">Discover our training courses<\/span>\n\t\t<\/span>\n\t\t\t\t\t<\/a>\n\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-b8a658c elementor-widget elementor-widget-heading\" data-id=\"b8a658c\" data-element_type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t<h4 class=\"elementor-heading-title elementor-size-medium\">Defining the components<\/h4>\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-a949342 elementor-widget elementor-widget-text-editor\" data-id=\"a949342\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p><span style=\"font-weight: 400;\">We are going to create 4 different components\u00a0<\/span><\/p><ul><li><span style=\"font-weight: 400;\"><strong>get_data()<\/strong> &#8211; this will retrieve a curated breast cancer dataset from kaggle and perform a train\/test split. Note that this example is using the <a href=\"https:\/\/www.kubeflow.org\/docs\/components\/pipelines\/v2\/data-types\/artifacts\/\">Output[Dataset] artifact<\/a> to store the metadata (such as the path of where the dataset is stored).\u00a0<\/span><\/li><li><span style=\"font-weight: 400;\"><strong>train_model()<\/strong> &#8211; this will take the dataset artifact created in <\/span><i><span style=\"font-weight: 400;\">get_data(), <\/span><\/i><span style=\"font-weight: 400;\">train an XGBoost classification model, and finally use the Model Artifact to store the model and it\u2019s metadata in a google cloud storage bucket. Note that custom metadata, such as the training score and framework, is being defined inside of the model artifact.<\/span><\/li><li><span style=\"font-weight: 400;\"><strong>eval_model()<\/strong> &#8211; this function will take the test dataset created from the get_data() and\u00a0 the model artifact created in the train_mode() as inputs and create an evaluation step. In an MLOps process, we may want to ensure that the machine learning model is meeting a certain performance threshold before deploying the model to an\u00a0 environment\u00a0<\/span><\/li><li><span style=\"font-weight: 400;\"><strong>deploy_xgboost_model()<\/strong> &#8211; finally, if our model meets our conditions established in the eval_model(), the model will then be deployed to an Vertex AI Endpoint where it will service requests.<\/span><\/li><\/ul>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-45a7ec9 elementor-widget elementor-widget-text-editor\" data-id=\"45a7ec9\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p><span style=\"font-weight: 400;\">Note that this is a simplified and minimalistic example. In practice, instead of deploying the model directly in the same environment, there may be using several different environments (<strong>Dev UAT<\/strong>) that the model artifact has to traverse before being deployed to a production environment.<\/span><\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-ab0ad20 elementor-widget elementor-widget-code-highlight\" data-id=\"ab0ad20\" data-element_type=\"widget\" data-widget_type=\"code-highlight.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<div class=\"prismjs-default copy-to-clipboard \">\n\t\t\t<pre data-line=\"\" class=\"highlight-height language-python \">\n\t\t\t\t<code readonly=\"true\" class=\"language-python\">\n\t\t\t\t\t<xmp>@dsl.component(base_image='python:3.8',\r\npackages_to_install=[\r\n    \"pandas==1.3.4\",\r\n    \"scikit-learn==1.0.1\",\r\n    \"google-cloud-bigquery==3.13.0\",\r\n    \"db-dtypes==1.1.1\"\r\n  ],\r\n)\r\ndef get_data(\r\n    project_id: str,\r\n    dataset_train: Output[Dataset],\r\n    dataset_test: Output[Dataset]\r\n) -> None:\r\n\r\n    \"\"\" Loads data from BigQuery, splits it into training and test sets,\r\n    and saves them as CSV files.\r\n    \r\n    Args:\r\n      project_id: str\r\n      dataset_train: Output[Dataset] for the training set.\r\n      dataset_test: Output[Dataset] for the test set.\r\n    \"\"\"\r\n    \r\n     from sklearn import datasets\r\n     from sklearn.model_selection import train_test_split\r\n     import pandas as pd\r\n    \r\n    \r\n     from google.cloud import bigquery\r\n\r\n     # Construct a BigQuery client object.\r\n     client = bigquery.Client(project=project_id)\r\n     job_config = bigquery.QueryJobConfig()\r\n     query = \"\"\"\r\n     \r\n        SELECT\r\n      * EXCEPT(fullVisitorId)\r\n     FROM\r\n     \r\n      # features\r\n      (SELECT\r\n       fullVisitorId,\r\n       IFNULL(totals.bounces, 0) AS bounces,\r\n       IFNULL(totals.timeOnSite, 0) AS time_on_site\r\n      FROM\r\n       `data-to-insights.ecommerce.web_analytics`\r\n\r\n      WHERE\r\n       totals.newVisits = 1\r\n       AND date BETWEEN '20160801' AND '20170430') # train on first 9 months\r\n      JOIN\r\n      (SELECT\r\n       fullvisitorid,\r\n       IF(COUNTIF(totals.transactions > 0 AND totals.newVisits IS NULL) > 0, 1, 0) AS will_buy_on_return_visit\r\n      FROM\r\n        `data-to-insights.ecommerce.web_analytics`\r\n      GROUP BY fullvisitorid)\r\n      USING (fullVisitorId)\r\n      LIMIT 10000\r\n     ;\r\n     \"\"\"\r\n     \r\n     query_job = client.query(query, job_config=job_config)\r\n     df = query_job.to_dataframe()\r\n    \r\n     # Split Data\r\n     train, test = train_test_split(df, test_size=0.3, random_state=42)\r\n    \r\n     # Save to Outputs\r\n     train.to_csv(dataset_train.path, index=False)\r\n     test.to_csv(dataset_test.path, index=False)\r\n\r\n\r\n\r\n@dsl.component(base_image='python:3.8',\r\n  packages_to_install=[\r\n    \"xgboost==1.6.2\",\r\n    \"pandas==1.3.5\",\r\n    \"joblib==1.1.0\",\r\n    \"scikit-learn==1.0.2\",\r\n  ],\r\n)\r\ndef train_model(\r\n  dataset: Input[Dataset],\r\n  model_artifact: Output[Model]\r\n) -> None:\r\n\r\n    \"\"\"Trains an XGBoost classifier on a given dataset and saves the model artifact.\r\n    \r\n    Args:\r\n      dataset: Input[Dataset]\r\n        The training dataset as a Kubeflow component input.\r\n      model_artifact: Output[Model]\r\n        A Kubeflow component output for saving the trained model.\r\n    \r\n    Returns:\r\n      None\r\n        This function doesn't have a return value; its primary purpose is to produce a model artifact.\r\n    \"\"\"\r\n    import os\r\n    import joblib\r\n    import pandas as pd\r\n    from xgboost import XGBClassifier\r\n\r\n    # Load Training Data\r\n    data = pd.read_csv(dataset.path)\r\n\r\n    # Train XGBoost Model\r\n    model = XGBClassifier(objective=\"binary:logistic\")\r\n    model.fit(data.drop(columns=[\"will_buy_on_return_visit\"]), data.will_buy_on_return_visit)\r\n    \r\n    # Evaluate and Log Metrics\r\n    score = model.score(data.drop(columns=[\"will_buy_on_return_visit\"]), data.will_buy_on_return_visit)\r\n    \r\n    # Save the Model Artifact\r\n    os.makedirs(model_artifact.path, exist_ok=True)\r\n    joblib.dump(model, os.path.join(model_artifact.path, \"model.joblib\"))\r\n    \r\n    # Metadata for the Artifact\r\n    model_artifact.metadata[\"train_score\"] = float(score)\r\n    model_artifact.metadata[\"framework\"] = \"XGBoost\"\r\n\r\n    \r\n@dsl.component(base_image='python:3.8',\r\n packages_to_install=[\r\n   \"xgboost==1.6.2\",\r\n   \"pandas==1.3.5\",\r\n   \"joblib==1.1.0\",\r\n   \"scikit-learn==1.0.2\",\r\n   \"google-cloud-storage==2.13.0\",\r\n ],\r\n)\r\ndef eval_model(\r\n  test_set: Input[Dataset],\r\n  xgb_model: Input[Model],\r\n  metrics: Output[ClassificationMetrics],\r\n  smetrics: Output[Metrics],\r\n  bucket_name: str,\r\n  score_threshold: float = 0.8\r\n) -> NamedTuple(\"Outputs\", [(\"deploy\", str)]):\r\n\r\n\r\n   \"\"\"Evaluates an XGBoost model on a test dataset, logs metrics, and decides whether to deploy.\r\n   \r\n   Args:\r\n     test_set: Input[Dataset]\r\n        The test dataset as a Kubeflow component input.\r\n     xgb_model: Input[Model]\r\n        The trained XGBoost model as a Kubeflow component input.\r\n     metrics: Output[ClassificationMetrics]\r\n        A Kubeflow component output for logging classification metrics.\r\n     smetrics: Output[Metrics]\r\n        A Kubeflow component output for logging scalar metrics.\r\n     bucket_name: str\r\n        The name of the Google Cloud Storage bucket containing the model.\r\n     score_threshold: float, default=0.8\r\n        The minimum score required for deployment.\r\n    \r\n    Returns:\r\n      NamedTuple(\"Outputs\", [(\"deploy\", str)])\r\n        A named tuple with a single field:\r\n        * deploy: str\r\n           A string indicating whether to deploy the model (\"true\" or \"false\").\r\n    \"\"\"\r\n    \r\n    from google.cloud import storage\r\n    import joblib\r\n    import pandas as pd\r\n    from sklearn.metrics import roc_curve, confusion_matrix\r\n    from collections import namedtuple\r\n\r\n\r\n    # Load Test Data and Model\r\n    data = pd.read_csv(test_set.path)\r\n    \r\n    client = storage.Client()\r\n    bucket = client.get_bucket(bucket_name)\r\n    blob_path = xgb_model.uri.replace(f\"gs:\/\/{bucket_name}\/\", \"\")\r\n    smetrics.log_metric(\"blob_path\", str(blob_path))\r\n    \r\n    blob = bucket.blob(f\"{blob_path}\/model.joblib\")\r\n    with blob.open(mode=\"rb\") as file:\r\n      model = joblib.load(file)\r\n\r\n    # Evaluation and Metrics\r\n    y_scores = model.predict_proba(data.drop(columns=[\"will_buy_on_return_visit\"]))[:, 1]\r\n    y_pred = model.predict(data.drop(columns=[\"will_buy_on_return_visit\"]))\r\n    score = model.score(data.drop(columns=[\"will_buy_on_return_visit\"]), data.will_buy_on_return_visit)\r\n    \r\n    fpr, tpr, thresholds = roc_curve(data.will_buy_on_return_visit.to_numpy(), y_scores, pos_label=True)\r\n    metrics.log_roc_curve(fpr.tolist(), tpr.tolist(), thresholds.tolist())\r\n    \r\n    cm = confusion_matrix(data.will_buy_on_return_visit, y_pred)\r\n    metrics.log_confusion_matrix([\"False\", \"True\"], cm.tolist())\r\n    smetrics.log_metric(\"score\", float(score))\r\n\r\n    # ----- 3. Deployment Decision Logic -----\r\n    deploy = \"true\" if score >= score_threshold else \"false\"\r\n\r\n    # ----- 4. Metadata Update -----\r\n    xgb_model.metadata[\"test_score\"] = float(score)\r\n    \r\n    Outputs = namedtuple(\"Outputs\", [\"deploy\"])\r\n    return Outputs(deploy)\r\n\r\n\r\n@dsl.component(base_image='python:3.8',\r\n packages_to_install=[\"google-cloud-aiplatform==1.25.0\"],\r\n)\r\ndef deploy_xgboost_model(\r\n   model: Input[Model],\r\n   project_id: str,\r\n   vertex_endpoint: Output[Artifact],\r\n   vertex_model: Output[Model]\r\n) -> None:\r\n   \"\"\"Deploys an XGBoost model to Vertex AI Endpoint.\r\n\r\n   Args:\r\n     model: The model to deploy.\r\n     project_id: The Google Cloud project ID.\r\n     vertex_endpoint: Output[Artifact] representing the deployed Vertex AI Endpoint.\r\n     vertex_model: Output[Model] representing the deployed Vertex AI Model.\r\n  \"\"\"\r\n  \r\n  from google.cloud import aiplatform\r\n\r\n  # Initialize AI Platform with project\r\n  aiplatform.init(project=project_id)\r\n\r\n  # Upload the Model\r\n  deployed_model = aiplatform.Model.upload(\r\n    display_name=\"xgb-classification\",\r\n    artifact_uri=model.uri,\r\n    serving_container_image_uri=\"us-docker.pkg.dev\/vertex-ai\/prediction\/xgboost-cpu.1-6:latest\",\r\n  )\r\n\r\n   # Deploy the Model to an Endpoint\r\n   endpoint = deployed_model.deploy(machine_type=\"n1-standard-4\")\r\n\r\n   # Save Outputs\r\n   vertex_endpoint.uri = endpoint.resource_name\r\n   vertex_model.uri = deployed_model.resource_name<\/xmp>\n\t\t\t\t<\/code>\n\t\t\t<\/pre>\n\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-91754bb elementor-widget elementor-widget-heading\" data-id=\"91754bb\" data-element_type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t<h4 class=\"elementor-heading-title elementor-size-medium\">Defining the pipeline<\/h4>\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-21e9044 elementor-widget elementor-widget-text-editor\" data-id=\"21e9044\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p><span style=\"font-weight: 400;\">Next step is to define the pipeline function. Similar to the initial example, the pipeline components are stitched together to create a Directed Acyclic graph of steps. The <a href=\"https:\/\/www.kubeflow.org\/docs\/components\/pipelines\/v2\/pipelines\/control-flow\/\">dsl.Condition<\/a> is introduced as a control flow step for the evaluation function. In this case, if the evaluation criteria is met (returns true), the pipeline will proceed with the deployment.<\/span><\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-4a7467d elementor-widget elementor-widget-code-highlight\" data-id=\"4a7467d\" data-element_type=\"widget\" data-widget_type=\"code-highlight.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<div class=\"prismjs-default copy-to-clipboard \">\n\t\t\t<pre data-line=\"\" class=\"highlight-height language-python \">\n\t\t\t\t<code readonly=\"true\" class=\"language-python\">\n\t\t\t\t\t<xmp>PIPELINE_ROOT = BUCKET_NAME + \"\/pipeline_root\/\" # path to pipeline metadata @dsl.pipeline(\r\n    # Default pipeline root. You can override it when submitting the pipeline.\r\n    pipeline_root=PIPELINE_ROOT + \"xgboost-pipeline-v2\",\r\n    # A name for the pipeline. Use to determine the pipeline Context.\r\n    name=\"xgboost-pipeline-with-deployment-v2\",\r\n)\r\ndef pipeline():\r\n    dataset_op = get_data()\r\n    training_op = train_model(dataset = dataset_op.outputs[\"dataset_train\"])\r\n    eval_op = eval_model(\r\n        test_set=dataset_op.outputs[\"dataset_test\"],\r\n        xgb_model=training_op.outputs[\"model_artifact\"],\r\n        bucket_name = \"kubeflow-mlops-410520-bucket\"\r\n    )\r\n\r\n    with dsl.Condition(\r\n        eval_op.outputs[\"deploy\"] == \"true\",\r\n        name=\"deploy\",\r\n    ):\r\n\r\n      deploy_op = deploy_xgboost_model(model = training_op.outputs[\"model_artifact\"],\r\n                         project_id = PROJECT_ID,\r\n                        )<\/xmp>\n\t\t\t\t<\/code>\n\t\t\t<\/pre>\n\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-83bdd9c elementor-widget elementor-widget-heading\" data-id=\"83bdd9c\" data-element_type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t<h4 class=\"elementor-heading-title elementor-size-medium\">Compiling and executing the pipeline<\/h4>\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-c3bc0e7 elementor-widget elementor-widget-text-editor\" data-id=\"c3bc0e7\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p><span style=\"font-weight: 400;\">Finally, the pipeline is compiled to create the YAML template. That template is submitted to the Vertex AI platform service. The model is then trained and deployed to the endpoint :<\/span><\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-6a0b5df elementor-widget elementor-widget-code-highlight\" data-id=\"6a0b5df\" data-element_type=\"widget\" data-widget_type=\"code-highlight.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<div class=\"prismjs-default copy-to-clipboard \">\n\t\t\t<pre data-line=\"\" class=\"highlight-height language-python \">\n\t\t\t\t<code readonly=\"true\" class=\"language-python\">\n\t\t\t\t\t<xmp>compiler.Compiler().compile(pipeline_func=pipeline, package_path=\"pipeline-breast-cancer.yaml\")\r\n\r\nfrom google.cloud.aiplatform import pipeline_jobs\r\n\r\njob = aiplatform.PipelineJob(\r\n    display_name=\"breast-cancer-demo-pipeline\",\r\n    template_path=\"pipeline-breast-cancer.yaml\",\r\n    pipeline_root=PIPELINE_ROOT,\r\n)\r\n\r\njob.run()<\/xmp>\n\t\t\t\t<\/code>\n\t\t\t<\/pre>\n\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-1dc2a97 elementor-widget elementor-widget-image\" data-id=\"1dc2a97\" data-element_type=\"widget\" data-widget_type=\"image.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<img decoding=\"async\" width=\"889\" height=\"978\" src=\"https:\/\/datascientest.com\/en\/files\/2024\/04\/image4.png\" class=\"attachment-full size-full wp-image-184255\" alt=\"\" srcset=\"https:\/\/datascientest.com\/en\/files\/2024\/04\/image4.png 889w, https:\/\/datascientest.com\/en\/files\/2024\/04\/image4-273x300.png 273w, https:\/\/datascientest.com\/en\/files\/2024\/04\/image4-768x845.png 768w\" sizes=\"(max-width: 889px) 100vw, 889px\" \/>\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t<div class=\"elementor-element elementor-element-8f69859 e-flex e-con-boxed e-con\" data-id=\"8f69859\" data-element_type=\"container\" data-settings=\"{&quot;content_width&quot;:&quot;boxed&quot;}\">\n\t\t\t\t\t<div class=\"e-con-inner\">\n\t\t\t\t<div class=\"elementor-element elementor-element-4117992 elementor-widget elementor-widget-heading\" data-id=\"4117992\" data-element_type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t<h3 class=\"elementor-heading-title elementor-size-default\">Conclusion<\/h3>\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-8a568a7 elementor-widget elementor-widget-text-editor\" data-id=\"8a568a7\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p><span style=\"font-weight: 400;\">This article covers a lot of ground. To summarize we covered some basic components Kubeflow pipelines and their part to play in creating a modularize, reusable, and flexible machine learning workflows. We also created a minimalistic end-to-end pipeline on Googe Cloud. Congratulations !<\/span><\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-a25b734 elementor-align-center button-principal elementor-widget elementor-widget-button\" data-id=\"a25b734\" data-element_type=\"widget\" data-widget_type=\"button.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<div class=\"elementor-button-wrapper\">\n\t\t\t<a class=\"elementor-button elementor-button-link elementor-size-md\" href=\"https:\/\/datascientest.com\/en\/data-courses\">\n\t\t\t\t\t\t<span class=\"elementor-button-content-wrapper\">\n\t\t\t\t\t\t<span class=\"elementor-button-text\">Start a training course<\/span>\n\t\t<\/span>\n\t\t\t\t\t<\/a>\n\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t<div class=\"elementor-element elementor-element-ae98e43 e-flex e-con-boxed e-con\" data-id=\"ae98e43\" data-element_type=\"container\" data-settings=\"{&quot;content_width&quot;:&quot;boxed&quot;}\">\n\t\t\t\t\t<div class=\"e-con-inner\">\n\t\t\t\t<div class=\"elementor-element elementor-element-14ed86c elementor-widget elementor-widget-heading\" data-id=\"14ed86c\" data-element_type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t<h3 class=\"elementor-heading-title elementor-size-default\">References :<\/h3>\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-d0d63d4 elementor-widget elementor-widget-text-editor\" data-id=\"d0d63d4\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<ul><li><a class=\"c-link\" href=\"https:\/\/cloud.google.com\/vertex-ai\/docs\/pipelines\/introduction\" target=\"_blank\" rel=\"noopener noreferrer\" data-stringify-link=\"https:\/\/cloud.google.com\/vertex-ai\/docs\/pipelines\/introduction\" data-sk=\"tooltip_parent\">https:\/\/cloud.google.com\/vertex-ai\/docs\/pipelines\/introduction<\/a><\/li><\/ul><ul class=\"p-rich_text_list p-rich_text_list__bullet\" data-stringify-type=\"unordered-list\" data-indent=\"0\" data-border=\"0\"><li data-stringify-indent=\"0\" data-stringify-border=\"0\"><a href=\"https:\/\/github.com\/GoogleCloudPlatform\/vertex-ai-samples\/blob\/main\/notebooks\/official\/pipelines\/kfp2_pipeline.ipynb\">https:\/\/github.com\/GoogleCloudPlatform\/vertex-ai-samples\/blob\/main\/notebooks\/official\/pipelines\/kfp2_pipeline.ipynb<\/a><\/li><li data-stringify-indent=\"0\" data-stringify-border=\"0\">Code was referenced Sascha Heyer from\u00a0<a class=\"c-link\" href=\"https:\/\/medium.com\/google-cloud\/google-vertex-ai-the-easiest-way-to-run-ml-pipelines-3a41c5ed153\" target=\"_blank\" rel=\"noopener noreferrer\" data-stringify-link=\"https:\/\/medium.com\/google-cloud\/google-vertex-ai-the-easiest-way-to-run-ml-pipelines-3a41c5ed153\" data-sk=\"tooltip_parent\">https:\/\/medium.com\/google-cloud\/google-vertex-ai-the-easiest-way-to-run-ml-pipelines-3a41c5ed153<\/a><\/li><\/ul>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t<div class=\"elementor-element elementor-element-7d8270a e-flex e-con-boxed e-con\" data-id=\"7d8270a\" data-element_type=\"container\" data-settings=\"{&quot;content_width&quot;:&quot;boxed&quot;}\">\n\t\t\t\t\t<div class=\"e-con-inner\">\n\t\t\t\t<div class=\"elementor-element elementor-element-e0120c1 elementor-widget elementor-widget-html\" data-id=\"e0120c1\" data-element_type=\"widget\" data-widget_type=\"html.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t<script type=\"application\/ld+json\">\r\n{\r\n  \"@context\": \"https:\/\/schema.org\",\r\n  \"@type\": \"FAQPage\",\r\n  \"mainEntity\": [{\r\n    \"@type\": \"Question\",\r\n    \"name\": \"Motivation\",\r\n    \"acceptedAnswer\": {\r\n      \"@type\": \"Answer\",\r\n      \"text\": \"Machine Learning Development goes beyond developing a model. Productionizing a model quickly becomes a multi-faceted, indeed even a multi-disciplinary process that involves several key stakeholders across both the business and IT. As depicted in the image below, the code to actually develop a model is a small component compared to other complexities such as  configuring serving infrastructure, data verification, and monitoring.\"\r\n    }\r\n  },{\r\n    \"@type\": \"Question\",\r\n    \"name\": \"Purpose\",\r\n    \"acceptedAnswer\": {\r\n      \"@type\": \"Answer\",\r\n      \"text\": \"The goal of this article is to serve as a gentle introduction to Kubeflow pipelines and how it can address the challenges in ML Learning Lifecycle development. To address all the components in the image above would go beyond the scope of a short blog. For the sake of simplicity, we are going to reference Vertex Pipelines on Google Cloud for our example. Note that Kubeflow is an open source project and can be deployed in several different environments.\"\r\n    }\r\n  },{\r\n    \"@type\": \"Question\",\r\n    \"name\": \"What is Kubeflow ?\",\r\n    \"acceptedAnswer\": {\r\n      \"@type\": \"Answer\",\r\n      \"text\": \"Kubeflow is an open-source MLOps (Machine Learning Operations) platform built on top of Kubernetes. It simplifies and automates various stages of the machine learning lifecycle, from data preparation and model training to deployment and monitoring. Think of it as a toolkit that helps you build and manage your ML workflows efficiently and portably across different platforms.\"\r\n    }\r\n  }]\r\n}\r\n<\/script>\r\n\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t\t\t<\/div>\n\t\t","protected":false},"excerpt":{"rendered":"<p>Motivation Machine Learning Development goes beyond developing a model. Productionizing a model quickly becomes a multi-faceted, indeed even a multi-disciplinary process that involves several key stakeholders across both the business and IT. As depicted in the image below, the code to actually develop a model is a small component compared to other complexities such as [&hellip;]<\/p>\n","protected":false},"author":47,"featured_media":184238,"comment_status":"open","ping_status":"open","sticky":false,"template":"elementor_theme","format":"standard","meta":{"content-type":"","inline_featured_image":false},"categories":[2425],"tags":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO Premium plugin v21.7 (Yoast SEO v21.7) - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>Introduction to Kubeflow for MLOpsby Tony Ruiz, GCP Customer Engineer at Google<\/title>\n<meta name=\"description\" content=\"Motivation Machine Learning Development goes beyond developing a model. Productionizing a model quickly becomes a multi-faceted, indeed even a\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/datascientest.com\/en\/introduction-to-kubeflow-for-mlops\" \/>\n<meta property=\"og:locale\" content=\"en_GB\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Introduction to Kubeflow for MLOpsby Tony Ruiz, GCP Customer Engineer at Google\" \/>\n<meta property=\"og:description\" content=\"Motivation Machine Learning Development goes beyond developing a model. Productionizing a model quickly becomes a multi-faceted, indeed even a\" \/>\n<meta property=\"og:url\" content=\"https:\/\/datascientest.com\/en\/introduction-to-kubeflow-for-mlops\" \/>\n<meta property=\"og:site_name\" content=\"Data Science Courses | DataScientest\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/DataScientestEN\" \/>\n<meta property=\"article:published_time\" content=\"2024-04-10T05:30:00+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2024-06-11T10:00:53+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/datascientest.com\/en\/files\/2024\/04\/kubeflow_for_mlops.png\" \/>\n\t<meta property=\"og:image:width\" content=\"1200\" \/>\n\t<meta property=\"og:image:height\" content=\"600\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"author\" content=\"Adriano R\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@DataScientest__\" \/>\n<meta name=\"twitter:site\" content=\"@DataScientest__\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Adriano R\" \/>\n\t<meta name=\"twitter:label2\" content=\"Estimated reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"7 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/datascientest.com\/en\/introduction-to-kubeflow-for-mlops#article\",\"isPartOf\":{\"@id\":\"https:\/\/datascientest.com\/en\/introduction-to-kubeflow-for-mlops\"},\"author\":{\"name\":\"Adriano R\",\"@id\":\"https:\/\/datascientest.com\/en\/#\/schema\/person\/3d2560836c47019eb12b7b77f3a15378\"},\"headline\":\"Introduction to Kubeflow for MLOps by Tony Ruiz, GCP Customer Engineer at Google\",\"datePublished\":\"2024-04-10T05:30:00+00:00\",\"dateModified\":\"2024-06-11T10:00:53+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/datascientest.com\/en\/introduction-to-kubeflow-for-mlops\"},\"wordCount\":1253,\"commentCount\":0,\"publisher\":{\"@id\":\"https:\/\/datascientest.com\/en\/#organization\"},\"articleSection\":[\"Data Science\"],\"inLanguage\":\"en-GB\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/datascientest.com\/en\/introduction-to-kubeflow-for-mlops#respond\"]}],\"copyrightYear\":\"2024\",\"copyrightHolder\":{\"@id\":\"https:\/\/datascientest.com\/#organization\"}},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/datascientest.com\/en\/introduction-to-kubeflow-for-mlops\",\"url\":\"https:\/\/datascientest.com\/en\/introduction-to-kubeflow-for-mlops\",\"name\":\"Introduction to Kubeflow for MLOps by Tony Ruiz, GCP Customer Engineer at Google\",\"isPartOf\":{\"@id\":\"https:\/\/datascientest.com\/en\/#website\"},\"datePublished\":\"2024-04-10T05:30:00+00:00\",\"dateModified\":\"2024-06-11T10:00:53+00:00\",\"description\":\"Motivation Machine Learning Development goes beyond developing a model. Productionizing a model quickly becomes a multi-faceted, indeed even a\",\"inLanguage\":\"en-GB\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/datascientest.com\/en\/introduction-to-kubeflow-for-mlops\"]}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/datascientest.com\/en\/#website\",\"url\":\"https:\/\/datascientest.com\/en\/\",\"name\":\"Data Science Courses | DataScientest\",\"description\":\"Leader in Data Science Courses\",\"publisher\":{\"@id\":\"https:\/\/datascientest.com\/en\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/datascientest.com\/en\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-GB\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/datascientest.com\/en\/#organization\",\"name\":\"DataScientest\",\"url\":\"https:\/\/datascientest.com\/en\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-GB\",\"@id\":\"https:\/\/datascientest.com\/en\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/datascientest.com\/en\/wp-content\/blogs.dir\/9\/files\/2024\/02\/DataScientest_icon_blue-2.png\",\"contentUrl\":\"https:\/\/datascientest.com\/en\/wp-content\/blogs.dir\/9\/files\/2024\/02\/DataScientest_icon_blue-2.png\",\"width\":301,\"height\":301,\"caption\":\"DataScientest\"},\"image\":{\"@id\":\"https:\/\/datascientest.com\/en\/#\/schema\/logo\/image\/\"},\"sameAs\":[\"https:\/\/www.facebook.com\/DataScientestEN\",\"https:\/\/twitter.com\/DataScientest__\",\"https:\/\/www.linkedin.com\/school\/datascientest\/mycompany\/\",\"https:\/\/www.tiktok.com\/@datascientest\"]},{\"@type\":\"Person\",\"@id\":\"https:\/\/datascientest.com\/en\/#\/schema\/person\/3d2560836c47019eb12b7b77f3a15378\",\"name\":\"Adriano R\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-GB\",\"@id\":\"https:\/\/datascientest.com\/en\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/fbfbc3a7811c63caf3333396a123058f?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/fbfbc3a7811c63caf3333396a123058f?s=96&d=mm&r=g\",\"caption\":\"Adriano R\"},\"sameAs\":[\"user\"],\"url\":\"https:\/\/datascientest.com\/en\/author\/adriano-r\"}]}<\/script>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"Introduction to Kubeflow for MLOpsby Tony Ruiz, GCP Customer Engineer at Google","description":"Motivation Machine Learning Development goes beyond developing a model. Productionizing a model quickly becomes a multi-faceted, indeed even a","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/datascientest.com\/en\/introduction-to-kubeflow-for-mlops","og_locale":"en_GB","og_type":"article","og_title":"Introduction to Kubeflow for MLOpsby Tony Ruiz, GCP Customer Engineer at Google","og_description":"Motivation Machine Learning Development goes beyond developing a model. Productionizing a model quickly becomes a multi-faceted, indeed even a","og_url":"https:\/\/datascientest.com\/en\/introduction-to-kubeflow-for-mlops","og_site_name":"Data Science Courses | DataScientest","article_publisher":"https:\/\/www.facebook.com\/DataScientestEN","article_published_time":"2024-04-10T05:30:00+00:00","article_modified_time":"2024-06-11T10:00:53+00:00","og_image":[{"width":1200,"height":600,"url":"https:\/\/datascientest.com\/en\/files\/2024\/04\/kubeflow_for_mlops.png","type":"image\/png"}],"author":"Adriano R","twitter_card":"summary_large_image","twitter_creator":"@DataScientest__","twitter_site":"@DataScientest__","twitter_misc":{"Written by":"Adriano R","Estimated reading time":"7 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/datascientest.com\/en\/introduction-to-kubeflow-for-mlops#article","isPartOf":{"@id":"https:\/\/datascientest.com\/en\/introduction-to-kubeflow-for-mlops"},"author":{"name":"Adriano R","@id":"https:\/\/datascientest.com\/en\/#\/schema\/person\/3d2560836c47019eb12b7b77f3a15378"},"headline":"Introduction to Kubeflow for MLOps by Tony Ruiz, GCP Customer Engineer at Google","datePublished":"2024-04-10T05:30:00+00:00","dateModified":"2024-06-11T10:00:53+00:00","mainEntityOfPage":{"@id":"https:\/\/datascientest.com\/en\/introduction-to-kubeflow-for-mlops"},"wordCount":1253,"commentCount":0,"publisher":{"@id":"https:\/\/datascientest.com\/en\/#organization"},"articleSection":["Data Science"],"inLanguage":"en-GB","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/datascientest.com\/en\/introduction-to-kubeflow-for-mlops#respond"]}],"copyrightYear":"2024","copyrightHolder":{"@id":"https:\/\/datascientest.com\/#organization"}},{"@type":"WebPage","@id":"https:\/\/datascientest.com\/en\/introduction-to-kubeflow-for-mlops","url":"https:\/\/datascientest.com\/en\/introduction-to-kubeflow-for-mlops","name":"Introduction to Kubeflow for MLOps by Tony Ruiz, GCP Customer Engineer at Google","isPartOf":{"@id":"https:\/\/datascientest.com\/en\/#website"},"datePublished":"2024-04-10T05:30:00+00:00","dateModified":"2024-06-11T10:00:53+00:00","description":"Motivation Machine Learning Development goes beyond developing a model. Productionizing a model quickly becomes a multi-faceted, indeed even a","inLanguage":"en-GB","potentialAction":[{"@type":"ReadAction","target":["https:\/\/datascientest.com\/en\/introduction-to-kubeflow-for-mlops"]}]},{"@type":"WebSite","@id":"https:\/\/datascientest.com\/en\/#website","url":"https:\/\/datascientest.com\/en\/","name":"Data Science Courses | DataScientest","description":"Leader in Data Science Courses","publisher":{"@id":"https:\/\/datascientest.com\/en\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/datascientest.com\/en\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-GB"},{"@type":"Organization","@id":"https:\/\/datascientest.com\/en\/#organization","name":"DataScientest","url":"https:\/\/datascientest.com\/en\/","logo":{"@type":"ImageObject","inLanguage":"en-GB","@id":"https:\/\/datascientest.com\/en\/#\/schema\/logo\/image\/","url":"https:\/\/datascientest.com\/en\/wp-content\/blogs.dir\/9\/files\/2024\/02\/DataScientest_icon_blue-2.png","contentUrl":"https:\/\/datascientest.com\/en\/wp-content\/blogs.dir\/9\/files\/2024\/02\/DataScientest_icon_blue-2.png","width":301,"height":301,"caption":"DataScientest"},"image":{"@id":"https:\/\/datascientest.com\/en\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/DataScientestEN","https:\/\/twitter.com\/DataScientest__","https:\/\/www.linkedin.com\/school\/datascientest\/mycompany\/","https:\/\/www.tiktok.com\/@datascientest"]},{"@type":"Person","@id":"https:\/\/datascientest.com\/en\/#\/schema\/person\/3d2560836c47019eb12b7b77f3a15378","name":"Adriano R","image":{"@type":"ImageObject","inLanguage":"en-GB","@id":"https:\/\/datascientest.com\/en\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/fbfbc3a7811c63caf3333396a123058f?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/fbfbc3a7811c63caf3333396a123058f?s=96&d=mm&r=g","caption":"Adriano R"},"sameAs":["user"],"url":"https:\/\/datascientest.com\/en\/author\/adriano-r"}]}},"_links":{"self":[{"href":"https:\/\/datascientest.com\/en\/wp-json\/wp\/v2\/posts\/184236"}],"collection":[{"href":"https:\/\/datascientest.com\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/datascientest.com\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/datascientest.com\/en\/wp-json\/wp\/v2\/users\/47"}],"replies":[{"embeddable":true,"href":"https:\/\/datascientest.com\/en\/wp-json\/wp\/v2\/comments?post=184236"}],"version-history":[{"count":0,"href":"https:\/\/datascientest.com\/en\/wp-json\/wp\/v2\/posts\/184236\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/datascientest.com\/en\/wp-json\/wp\/v2\/media\/184238"}],"wp:attachment":[{"href":"https:\/\/datascientest.com\/en\/wp-json\/wp\/v2\/media?parent=184236"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/datascientest.com\/en\/wp-json\/wp\/v2\/categories?post=184236"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/datascientest.com\/en\/wp-json\/wp\/v2\/tags?post=184236"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}