{"id":185048,"date":"2024-06-11T14:20:59","date_gmt":"2024-06-11T13:20:59","guid":{"rendered":"https:\/\/datascientest.com\/en\/?p=185048"},"modified":"2024-06-11T14:21:01","modified_gmt":"2024-06-11T13:21:01","slug":"ci-cd-for-kubeflow","status":"publish","type":"post","link":"https:\/\/datascientest.com\/en\/ci-cd-for-kubeflow","title":{"rendered":"CI\/CD for Kubeflow (part 2) <\/br><i><small>by Tony Ruiz, GCP Customer Engineer at Google<\/small><\/i>"},"content":{"rendered":"\t\t<div data-elementor-type=\"wp-post\" data-elementor-id=\"185048\" class=\"elementor elementor-185048\" data-elementor-post-type=\"post\">\n\t\t\t\t\t\t\t<div class=\"elementor-element elementor-element-b872fc9 e-flex e-con-boxed e-con\" data-id=\"b872fc9\" data-element_type=\"container\" data-settings=\"{&quot;content_width&quot;:&quot;boxed&quot;}\">\n\t\t\t\t\t<div class=\"e-con-inner\">\n\t\t\t\t<div class=\"elementor-element elementor-element-0c36a96 elementor-widget elementor-widget-text-editor\" data-id=\"0c36a96\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t<style>\/*! elementor - v3.15.0 - 02-08-2023 *\/\n.elementor-widget-text-editor.elementor-drop-cap-view-stacked .elementor-drop-cap{background-color:#69727d;color:#fff}.elementor-widget-text-editor.elementor-drop-cap-view-framed .elementor-drop-cap{color:#69727d;border:3px solid;background-color:transparent}.elementor-widget-text-editor:not(.elementor-drop-cap-view-default) .elementor-drop-cap{margin-top:8px}.elementor-widget-text-editor:not(.elementor-drop-cap-view-default) .elementor-drop-cap-letter{width:1em;height:1em}.elementor-widget-text-editor .elementor-drop-cap{float:left;text-align:center;line-height:1;font-size:50px}.elementor-widget-text-editor .elementor-drop-cap-letter{display:inline-block}<\/style>\t\t\t\t<p><span style=\"font-weight: 400;\">This is the second part of a series\u00a0<\/span><\/p><ul><li style=\"font-weight: 400;\" aria-level=\"1\"><a href=\"https:\/\/datascientest.com\/en\/introduction-to-kubeflow-for-mlops\">Part 1 here<\/a><\/li><\/ul>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t<div class=\"elementor-element elementor-element-ad25ad5 e-flex e-con-boxed e-con\" data-id=\"ad25ad5\" data-element_type=\"container\" data-settings=\"{&quot;content_width&quot;:&quot;boxed&quot;}\">\n\t\t\t\t\t<div class=\"e-con-inner\">\n\t\t\t\t<div class=\"elementor-element elementor-element-4b78b6d elementor-widget elementor-widget-heading\" data-id=\"4b78b6d\" data-element_type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t<style>\/*! elementor - v3.15.0 - 02-08-2023 *\/\n.elementor-heading-title{padding:0;margin:0;line-height:1}.elementor-widget-heading .elementor-heading-title[class*=elementor-size-]>a{color:inherit;font-size:inherit;line-height:inherit}.elementor-widget-heading .elementor-heading-title.elementor-size-small{font-size:15px}.elementor-widget-heading .elementor-heading-title.elementor-size-medium{font-size:19px}.elementor-widget-heading .elementor-heading-title.elementor-size-large{font-size:29px}.elementor-widget-heading .elementor-heading-title.elementor-size-xl{font-size:39px}.elementor-widget-heading .elementor-heading-title.elementor-size-xxl{font-size:59px}<\/style><h3 class=\"elementor-heading-title elementor-size-default\">Motivation<\/h3>\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-31f1dd5 elementor-widget elementor-widget-text-editor\" data-id=\"31f1dd5\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p><span style=\"font-weight: 400;\">The previous article discusses the complexities of the <a href=\"https:\/\/datascientest.com\/en\/bagging-machine-learning-what-is-it-about\">machine learning life cycle<\/a> and how <strong>Vertex Pipelines<\/strong>, a managed version of Kubeflow on Google Cloud, can address operationalizing a machine learning model. The next step in the journey is to automate the training and deployment of the system in a manner that facilitates and enables reproducibility and time to market. In traditional software development, this capabilities would be enabled by a system that allows for\u00a0<\/span><\/p><ul><li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\"><strong>Continuous Integration (CI)<\/strong> &#8211; the ability to test and validate checked in code that needs to be merged into a central repository<\/span><\/li><li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\"><strong>Continuous Deployment (CD)<\/strong> &#8211; Creating and deploying the system to an appropriate environment <\/span><\/li><\/ul>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-6ac7011 elementor-widget elementor-widget-text-editor\" data-id=\"6ac7011\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p><span style=\"font-weight: 400;\">Since machine learning models are dependent on the data, it must be accounted for in the <a href=\"https:\/\/datascientest.com\/en\/all-about-ci-cd\">CI\/CD process<\/a>. Particularly,<\/span><\/p><ul><li><span style=\"font-weight: 400;\">Data must be tested and validated for quality and <a href=\"https:\/\/developers.google.com\/machine-learning\/guides\/rules-of-ml#rule_37_measure_trainingserving_skew\">training\/serving skew<\/a><\/span><\/li><li><span style=\"font-weight: 400;\">A model must be continuously trained on new data to account for changes in the world<\/span><\/li><li><span style=\"font-weight: 400;\">The model must be validated, tested and deployed to a prediction service<\/span><\/li><\/ul>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-40123ac elementor-widget elementor-widget-image\" data-id=\"40123ac\" data-element_type=\"widget\" data-widget_type=\"image.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t<style>\/*! elementor - v3.15.0 - 02-08-2023 *\/\n.elementor-widget-image{text-align:center}.elementor-widget-image a{display:inline-block}.elementor-widget-image a img[src$=\".svg\"]{width:48px}.elementor-widget-image img{vertical-align:middle;display:inline-block}<\/style>\t\t\t\t\t\t\t\t\t<figure class=\"wp-caption\">\n\t\t\t\t\t\t\t\t\t\t<img decoding=\"async\" width=\"800\" height=\"240\" src=\"https:\/\/datascientest.com\/en\/files\/2024\/05\/image1-1024x307.png\" class=\"attachment-large size-large wp-image-185061\" alt=\"\" srcset=\"https:\/\/datascientest.com\/en\/files\/2024\/05\/image1-1024x307.png 1024w, https:\/\/datascientest.com\/en\/files\/2024\/05\/image1-300x90.png 300w, https:\/\/datascientest.com\/en\/files\/2024\/05\/image1-768x230.png 768w, https:\/\/datascientest.com\/en\/files\/2024\/05\/image1.png 1313w\" sizes=\"(max-width: 800px) 100vw, 800px\" \/>\t\t\t\t\t\t\t\t\t\t\t<figcaption class=\"widget-image-caption wp-caption-text\">Image from <a href=\"https:\/\/cloud.google.com\/architecture\/architecture-for-mlops-using-tfx-kubeflow-pipelines-and-cloud-build\">Architecture for MLOps using TensorFlow Extended, Vertex AI Pipelines, and Cloud Build<\/a><\/figcaption>\n\t\t\t\t\t\t\t\t\t\t<\/figure>\n\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-506bad3 elementor-widget elementor-widget-text-editor\" data-id=\"506bad3\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p><span style=\"font-weight: 400;\">This also introduces the notion of continuous training. Depending on the use case, we need our model to be trained and updated on a frequency. <\/span><\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t<div class=\"elementor-element elementor-element-a1c82a1 e-flex e-con-boxed e-con\" data-id=\"a1c82a1\" data-element_type=\"container\" data-settings=\"{&quot;content_width&quot;:&quot;boxed&quot;}\">\n\t\t\t\t\t<div class=\"e-con-inner\">\n\t\t\t\t<div class=\"elementor-element elementor-element-6bd329e elementor-widget elementor-widget-heading\" data-id=\"6bd329e\" data-element_type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t<h3 class=\"elementor-heading-title elementor-size-default\">Purpose<\/h3>\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-b94bb23 elementor-widget elementor-widget-text-editor\" data-id=\"b94bb23\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p><span style=\"font-weight: 400;\">The goal of this article is to serve as a gentle introduction to <strong>CI\/CD for the Vertex Pipelines<\/strong> created in the previous article. To address all the components in a comprehensive CI\/CD workflow in the image above would go beyond the scope of a short blog. This article assumes the reader has foundational knowledge on concepts surrounding git, docker, and cloud technologies. <\/span><\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t<div class=\"elementor-element elementor-element-39a6637 e-flex e-con-boxed e-con\" data-id=\"39a6637\" data-element_type=\"container\" data-settings=\"{&quot;content_width&quot;:&quot;boxed&quot;}\">\n\t\t\t\t\t<div class=\"e-con-inner\">\n\t\t\t\t<div class=\"elementor-element elementor-element-6e6f684 elementor-widget elementor-widget-heading\" data-id=\"6e6f684\" data-element_type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t<h3 class=\"elementor-heading-title elementor-size-default\">What is Cloud Build ?<\/h3>\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-061cdcc elementor-widget elementor-widget-text-editor\" data-id=\"061cdcc\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p><span style=\"font-weight: 400;\">Cloud build is a <strong>fully managed, serverless CI\/CD platform<\/strong>. Other potential include Jenkins, <a href=\"https:\/\/datascientest.com\/en\/azure-devops-definitions-devops-methods\">Azure DevOps<\/a> and Github Actions. A CI\/CD platform allows for teams to effectively and iteratively develop and ship software to different environments. This tutorial will use Google Cloud Build alongside a github repository to build and run containers using the code that was pushed to a main branch.<\/span><\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t<div class=\"elementor-element elementor-element-86d92f0 e-flex e-con-boxed e-con\" data-id=\"86d92f0\" data-element_type=\"container\" data-settings=\"{&quot;content_width&quot;:&quot;boxed&quot;}\">\n\t\t\t\t\t<div class=\"e-con-inner\">\n\t\t\t\t<div class=\"elementor-element elementor-element-f3bb749 elementor-widget elementor-widget-heading\" data-id=\"f3bb749\" data-element_type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t<h3 class=\"elementor-heading-title elementor-size-default\">A Minimalistic CI\/CD\/CT Workflow<\/h3>\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-e9812d0 elementor-widget elementor-widget-heading\" data-id=\"e9812d0\" data-element_type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t<h4 class=\"elementor-heading-title elementor-size-medium\">Prerequisites<\/h4>\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-fb24ea6 elementor-widget elementor-widget-text-editor\" data-id=\"fb24ea6\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<ul><li><span style=\"font-weight: 400;\">A github account and repository\u00a0<\/span><ul><li style=\"font-weight: 400;\" aria-level=\"2\"><span style=\"font-weight: 400;\">This article assumes the reader has some foundational knowledge in git. <a href=\"https:\/\/www.atlassian.com\/git\/tutorials\/setting-up-a-repository\">https:\/\/www.atlassian.com\/git\/tutorials\/setting-up-a-repository<\/a><\/span><\/li><\/ul><\/li><li><span style=\"font-weight: 400;\">Google cloud environment<\/span><\/li><li style=\"list-style-type: none;\"><ul><li><span style=\"font-weight: 400;\">If you are new to Google Cloud, refer to <a href=\"https:\/\/cloud.google.com\/docs\/get-started\">this page<\/a> to get started. In this example, we are going to be developing our pipeline on a <a href=\"https:\/\/cloud.google.com\/vertex-ai-notebooks?hl=en\">Vertex AI Notebook<\/a>.<\/span><\/li><\/ul><\/li><\/ul>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-8cb92b9 elementor-widget elementor-widget-text-editor\" data-id=\"8cb92b9\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p><span style=\"font-weight: 400;\">Recall from the previous post, that the goal was to build an end-to-end ML workflow where\u00a0<\/span><\/p><ul><li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">A model is trained\u00a0<\/span><\/li><li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Evaluated against a metric\u00a0<\/span><\/li><li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Deployed to a computer endpoint where predictions can be made<\/span><\/li><\/ul>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-a23a5aa elementor-widget elementor-widget-text-editor\" data-id=\"a23a5aa\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p><span style=\"font-weight: 400;\">Note that in some instances, the deployment workflow may be separate from the training workflow. For example, you may want your machine learning model to be trained on a weekly basis but go through an extensive quality assurance process before deploying. <\/span><\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-50f9817 elementor-widget elementor-widget-heading\" data-id=\"50f9817\" data-element_type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t<h4 class=\"elementor-heading-title elementor-size-medium\">Rewrite our pipeline workflow<\/h4>\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-40ce3de elementor-widget elementor-widget-text-editor\" data-id=\"40ce3de\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p><span style=\"font-weight: 400;\">The first thing we will need to do is <strong>transform the<\/strong> <strong>python notebook file (.ipynb)<\/strong> <strong>into a normal python file (.py)<\/strong>.\u00a0 This python file contains all the executable code to recreate a compiled pipeline file that will actually be used to execute our machine learning workflow. This is because python files work much better with text files than with mixtures of text and binaries, which is what notebook files are. Refer to the github repository here for the full code.<\/span><\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-b9f26cc elementor-align-center button-principal elementor-widget elementor-widget-button\" data-id=\"b9f26cc\" data-element_type=\"widget\" data-widget_type=\"button.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<div class=\"elementor-button-wrapper\">\n\t\t\t<a class=\"elementor-button elementor-button-link elementor-size-md\" href=\"https:\/\/datascientest.com\/en\/data-courses\">\n\t\t\t\t\t\t<span class=\"elementor-button-content-wrapper\">\n\t\t\t\t\t\t<span class=\"elementor-button-text\">Browse our training courses<\/span>\n\t\t<\/span>\n\t\t\t\t\t<\/a>\n\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-31be371 elementor-widget elementor-widget-heading\" data-id=\"31be371\" data-element_type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t<h4 class=\"elementor-heading-title elementor-size-medium\">Create a Dockerfile<\/h4>\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-c50e83a elementor-widget elementor-widget-text-editor\" data-id=\"c50e83a\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p><span style=\"font-weight: 400;\">The next step is to create a Dockerfile that contains our python file <a href=\"http:\/\/pipeline.py\">pipeline.py<\/a> and our requirements.txt. Note that we are creating a separate folder (.\/kpi-cli) within our workspace to house the files for our docker image. <\/span><\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-9d5423d elementor-widget elementor-widget-code-highlight\" data-id=\"9d5423d\" data-element_type=\"widget\" data-widget_type=\"code-highlight.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<div class=\"prismjs-default copy-to-clipboard word-wrap\">\n\t\t\t<pre data-line=\"\" class=\"highlight-height language-python \">\n\t\t\t\t<code readonly=\"true\" class=\"language-python\">\n\t\t\t\t\t<xmp>%%writefile kfp-cli\/Dockerfile\r\n\r\n\r\nFROM gcr.io\/deeplearning-platform-release\/base-cpu\r\nWORKDIR \/kfp-cli\r\nADD pipeline.py .\/pipeline.py\r\nADD requirements.txt .\/requirements.txt\r\nRUN pip install -r requirements.txt\r\nCMD python .\/pipeline.py<\/xmp>\n\t\t\t\t<\/code>\n\t\t\t<\/pre>\n\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-304f22a elementor-widget elementor-widget-heading\" data-id=\"304f22a\" data-element_type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t<h4 class=\"elementor-heading-title elementor-size-medium\">Build a local image and push it to Container Registry<\/h4>\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-6e6cfc2 elementor-widget elementor-widget-text-editor\" data-id=\"6e6cfc2\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p><span style=\"font-weight: 400;\">Now that a Dockerfile is defined and the dependencies exist within a directory, create a local image and push it to <a href=\"https:\/\/cloud.google.com\/artifact-registry\">Artifact Registry<\/a>.<\/span><\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-618ae0d elementor-widget elementor-widget-code-highlight\" data-id=\"618ae0d\" data-element_type=\"widget\" data-widget_type=\"code-highlight.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<div class=\"prismjs-default copy-to-clipboard \">\n\t\t\t<pre data-line=\"\" class=\"highlight-height language-python \">\n\t\t\t\t<code readonly=\"true\" class=\"language-python\">\n\t\t\t\t\t<xmp>IMAGE_NAME='kfp-truiz-mlops-v2'\r\nTAG='latest'\r\nIMAGE_URI='gcr.io\/{}\/{}:{}'.format(PROJECT_ID, IMAGE_NAME, TAG)\r\n\r\n\r\n\r\n\r\n!gcloud builds submit --timeout 15m --tag {IMAGE_URI} .\/kfp-cli<\/xmp>\n\t\t\t\t<\/code>\n\t\t\t<\/pre>\n\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-e63df07 elementor-widget elementor-widget-text-editor\" data-id=\"e63df07\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p><span style=\"font-weight: 400;\">The cloudbuild product page within the console will contain the information on the build.<\/span><\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-863d70c elementor-widget elementor-widget-heading\" data-id=\"863d70c\" data-element_type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t<h4 class=\"elementor-heading-title elementor-size-medium\">Build a cloudbuild.yaml file<\/h4>\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-1b30b42 elementor-widget elementor-widget-text-editor\" data-id=\"1b30b42\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p><span style=\"font-weight: 400;\">The next step is to create the <strong>cloudbuild.yaml file<\/strong>. This yaml file will contain the steps to execute the build when code gets pushed to the remote repository. For more information on creating the cloudbuild, refer to the <a href=\"https:\/\/cloud.google.com\/build\/docs\/configuring-builds\/create-basic-configuration\">documentation<\/a>.\u00a0<\/span><\/p><p><span style=\"font-weight: 400;\">In short, this will do a few things<\/span><\/p><ul><li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Build the docker image using the code in the repository. If there are errors in code, this build step will fail<\/span><\/li><li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Push the most recent version of the docker image back to Artifact Registry<\/span><\/li><li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Use the image that was created to run the python script in the repository. The output of this execution will be the pipeline yaml file\u00a0<\/span><\/li><li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">The next step is to write the pipeline yaml file to a google cloud storage bucket. This yaml file can then be picked up by a scheduler to trigger the pipeline.<\/span><\/li><\/ul>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-cc72683 elementor-widget elementor-widget-code-highlight\" data-id=\"cc72683\" data-element_type=\"widget\" data-widget_type=\"code-highlight.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<div class=\"prismjs-default copy-to-clipboard word-wrap\">\n\t\t\t<pre data-line=\"\" class=\"highlight-height language-python \">\n\t\t\t\t<code readonly=\"true\" class=\"language-python\">\n\t\t\t\t\t<xmp>steps:\r\n  # Step 1: Build the Docker image\r\n  - name: 'gcr.io\/cloud-builders\/docker'\r\n    args: ['build', '-t', 'gcr.io\/$_PROJECT_ID\/kfp-truiz-mlops-v2:latest', '.\/kfp-cli']\r\n  \r\n  # Step 2: Push the image back to container registry \r\n  - name: 'gcr.io\/cloud-builders\/docker'\r\n    args: ['push', 'gcr.io\/$_PROJECT_ID\/kfp-truiz-mlops-v2:latest']\r\n\r\n  # Step 3: Run the Python file within the container\r\n  - name: 'gcr.io\/$_PROJECT_ID\/kfp-truiz-mlops-v2:latest'\r\n    args: ['python', '.\/kfp-cli\/pipeline.py']  \r\n    env:\r\n    - 'PROJECT_ID=$_PROJECT_ID'\r\n    - 'REGION=$_REGION'\r\n    \r\n  # Step 4: Write results to Google Cloud Storage\r\n  - name: 'gcr.io\/cloud-builders\/gsutil'\r\n    args: ['cp', '.\/xgb-pipeline.yaml', 'gs:\/\/$_PROJECT_ID-cloudbuild-pipelines'] \r\n\r\noptions:\r\n    logging: CLOUD_LOGGING_ONLY\r\n    \r\nsubstitutions:\r\n   _PROJECT_ID: '<PROJECT_ID>'\r\n   _REGION: 'us-central1'<\/xmp>\n\t\t\t\t<\/code>\n\t\t\t<\/pre>\n\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-869f24c elementor-widget elementor-widget-heading\" data-id=\"869f24c\" data-element_type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t<h4 class=\"elementor-heading-title elementor-size-medium\">Submit a local job to the Cloud Build<\/h4>\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-aea280e elementor-widget elementor-widget-text-editor\" data-id=\"aea280e\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p><span style=\"font-weight: 400;\">Once the docker file is defined and the cloudbuild yaml created, a local build can be submitted using gcloud command line interface. This:<\/span><\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-f0b49c1 elementor-widget elementor-widget-code-highlight\" data-id=\"f0b49c1\" data-element_type=\"widget\" data-widget_type=\"code-highlight.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<div class=\"prismjs-default copy-to-clipboard \">\n\t\t\t<pre data-line=\"\" class=\"highlight-height language-python \">\n\t\t\t\t<code readonly=\"true\" class=\"language-python\">\n\t\t\t\t\t<xmp>!gcloud builds submit .\/kfp-cli --config cloudbuild.yaml<\/xmp>\n\t\t\t\t<\/code>\n\t\t\t<\/pre>\n\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-49e6c42 elementor-align-center button-principal elementor-widget elementor-widget-button\" data-id=\"49e6c42\" data-element_type=\"widget\" data-widget_type=\"button.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<div class=\"elementor-button-wrapper\">\n\t\t\t<a class=\"elementor-button elementor-button-link elementor-size-md\" href=\"https:\/\/datascientest.com\/en\/data-courses\">\n\t\t\t\t\t\t<span class=\"elementor-button-content-wrapper\">\n\t\t\t\t\t\t<span class=\"elementor-button-text\">Discover our training courses<\/span>\n\t\t<\/span>\n\t\t\t\t\t<\/a>\n\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-189664a elementor-widget elementor-widget-heading\" data-id=\"189664a\" data-element_type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t<h4 class=\"elementor-heading-title elementor-size-medium\">Connect Google Cloud to our Github Repository<\/h4>\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-3a12538 elementor-widget elementor-widget-text-editor\" data-id=\"3a12538\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p><span style=\"font-weight: 400;\">Now that the cloudbuild file is defined, there needs to be an integration between the cloudbuild environment and the git repository. Fortunately this is pretty simple and straightforward to set up <a href=\"https:\/\/cloud.google.com\/build\/docs\/automating-builds\/github\/connect-repo-github#connecting_a_github_host\">via the console<\/a>.\u00a0<\/span><\/p><p style=\"padding-left: 40px;\"><span style=\"font-weight: 400;\">The first step is <strong>to create a host connection to the git environment<\/strong>. This will in turn handle the authentication mechanism to the repository. You will be redirected to your github repository to allow the connection. An authentication token from GitHub will be created and stored in this project as a Secret Manager secret. You may revoke access to Cloud Build through GitHub at any time.\u00a0<\/span><\/p><p style=\"padding-left: 40px;\"><span style=\"font-weight: 400;\">The second step is <strong>to link the repository<\/strong>. You will need to <a href=\"https:\/\/github.com\/marketplace\/google-cloud-build\">install the Google Cloud build<\/a> application within Github to allow the integration to the selected repository.<\/span><\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-6eaa4fb elementor-widget elementor-widget-image\" data-id=\"6eaa4fb\" data-element_type=\"widget\" data-widget_type=\"image.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<img decoding=\"async\" width=\"1342\" height=\"1313\" src=\"https:\/\/datascientest.com\/en\/files\/2024\/05\/image3.png\" class=\"attachment-full size-full wp-image-185089\" alt=\"\" \/>\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-48cf2f6 elementor-widget elementor-widget-text-editor\" data-id=\"48cf2f6\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p><span style=\"font-weight: 400;\">After that step is complete, you will be able to link the repository to your Google Cloud Build host connection.<\/span><\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-cb47564 elementor-widget elementor-widget-image\" data-id=\"cb47564\" data-element_type=\"widget\" data-widget_type=\"image.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<img decoding=\"async\" width=\"988\" height=\"356\" src=\"https:\/\/datascientest.com\/en\/files\/2024\/05\/image2.png\" class=\"attachment-full size-full wp-image-185093\" alt=\"\" srcset=\"https:\/\/datascientest.com\/en\/files\/2024\/05\/image2.png 988w, https:\/\/datascientest.com\/en\/files\/2024\/05\/image2-300x108.png 300w, https:\/\/datascientest.com\/en\/files\/2024\/05\/image2-768x277.png 768w\" sizes=\"(max-width: 988px) 100vw, 988px\" \/>\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-4bf9121 elementor-widget elementor-widget-heading\" data-id=\"4bf9121\" data-element_type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t<h4 class=\"elementor-heading-title elementor-size-medium\">Create a Cloud Build Trigger<\/h4>\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-a896e99 elementor-widget elementor-widget-text-editor\" data-id=\"a896e99\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p><a href=\"https:\/\/cloud.google.com\/build\/docs\/automating-builds\/create-manage-triggers\">https:\/\/cloud.google.com\/build\/docs\/automating-builds\/create-manage-triggers<\/a><\/p><p><span style=\"font-weight: 400;\">Now every time a push is made to the main branch, a build process will kick off. This is done through <a href=\"https:\/\/docs.google.com\/document\/u\/0\/d\/1gAWZIUw8TYzoC1QZX4QVo4ziTQ0uPZ2mFEs2GxrzenM\/edit\">Cloud build triggers<\/a>, which allow the user to trigger specific actions on connected repositories. In the image below, a trigger is created to execute a build every time there\u2019s a push to a new branch. Note that specific branches can be defined using regex. The cloudbuild.yaml file is also specified as the configuration for the build.<\/span><\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-6c05bf1 elementor-widget elementor-widget-image\" data-id=\"6c05bf1\" data-element_type=\"widget\" data-widget_type=\"image.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<img decoding=\"async\" width=\"1341\" height=\"1312\" src=\"https:\/\/datascientest.com\/en\/files\/2024\/05\/image5.png\" class=\"attachment-full size-full wp-image-185097\" alt=\"\" srcset=\"https:\/\/datascientest.com\/en\/files\/2024\/05\/image5.png 1341w, https:\/\/datascientest.com\/en\/files\/2024\/05\/image5-300x294.png 300w, https:\/\/datascientest.com\/en\/files\/2024\/05\/image5-1024x1002.png 1024w, https:\/\/datascientest.com\/en\/files\/2024\/05\/image5-768x751.png 768w\" sizes=\"(max-width: 1341px) 100vw, 1341px\" \/>\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-693fcc2 elementor-widget elementor-widget-text-editor\" data-id=\"693fcc2\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p><span style=\"font-weight: 400;\">Once the connection to the remote repository is established and a trigger is created, the next step is to push a commit to the remote repository. This will execute a build on the environment within the remote git repository. The changes that were pushed to the folder will be used to create a new image, push it to the container registry, run the pipeline file, and move the compiled pipeline specification to storage. All that is needed now is to execute the pipeline using a scheduler or an event.<\/span><\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-4db8d67 elementor-widget elementor-widget-image\" data-id=\"4db8d67\" data-element_type=\"widget\" data-widget_type=\"image.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<img decoding=\"async\" width=\"1822\" height=\"1313\" src=\"https:\/\/datascientest.com\/en\/files\/2024\/05\/image4.png\" class=\"attachment-full size-full wp-image-185101\" alt=\"\" srcset=\"https:\/\/datascientest.com\/en\/files\/2024\/05\/image4.png 1822w, https:\/\/datascientest.com\/en\/files\/2024\/05\/image4-300x216.png 300w, https:\/\/datascientest.com\/en\/files\/2024\/05\/image4-1024x738.png 1024w, https:\/\/datascientest.com\/en\/files\/2024\/05\/image4-768x553.png 768w, https:\/\/datascientest.com\/en\/files\/2024\/05\/image4-1536x1107.png 1536w\" sizes=\"(max-width: 1822px) 100vw, 1822px\" \/>\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-0a0ae6b elementor-widget elementor-widget-heading\" data-id=\"0a0ae6b\" data-element_type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t<h4 class=\"elementor-heading-title elementor-size-medium\">Configure a schedule to execute Pipeline<\/h4>\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-2839db1 elementor-widget elementor-widget-text-editor\" data-id=\"2839db1\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p><span style=\"font-weight: 400;\">The next step is to actually execute the pipeline file. How this is done can vary depending on the use case.\u00a0<\/span><\/p><ul><li style=\"font-weight: 400;\" aria-level=\"1\">We may want to execute a scheduled training pipeline weekly, and deploy our model only when a model artifact has made it up the proper approval channels.<\/li><li style=\"font-weight: 400;\" aria-level=\"1\">We may want to retrain a model after a batch process is complete. For example, when data in our data warehouse gets updated, a trigger would occur that trains the model.We can schedule a recurring schedule with the Scheduler API or we can create an event driven workflow using a messaging\/queue service like Pub\/Sub. For our use case, a weekly batch process will suffice.<\/li><\/ul>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-da727b5 elementor-widget elementor-widget-code-highlight\" data-id=\"da727b5\" data-element_type=\"widget\" data-widget_type=\"code-highlight.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<div class=\"prismjs-default copy-to-clipboard \">\n\t\t\t<pre data-line=\"\" class=\"highlight-height language-python \">\n\t\t\t\t<code readonly=\"true\" class=\"language-python\">\n\t\t\t\t\t<xmp>pipeline_job = aiplatform.PipelineJob(\r\n    template_path=COMPILED_PIPELINE_PATH,\r\n    pipeline_root=PIPELINE_ROOT_PATH,\r\n    display_name=\"xgb-pipeline\",\r\n)\r\n\r\npipeline_job_schedule = pipeline_job.create_schedule(\r\n    display_name=\"weekly_training_and_deployment\",\r\n    cron=\"30 18 * * 6\",  # weekly 6pm UTC time on Saturday\r\n    max_concurrent_run_count=1,\r\n    max_run_count=1,\r\n)\r\n<\/xmp>\n\t\t\t\t<\/code>\n\t\t\t<\/pre>\n\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t<div class=\"elementor-element elementor-element-697d12c e-flex e-con-boxed e-con\" data-id=\"697d12c\" data-element_type=\"container\" data-settings=\"{&quot;content_width&quot;:&quot;boxed&quot;}\">\n\t\t\t\t\t<div class=\"e-con-inner\">\n\t\t\t\t<div class=\"elementor-element elementor-element-84f5eb9 elementor-widget elementor-widget-heading\" data-id=\"84f5eb9\" data-element_type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t<h3 class=\"elementor-heading-title elementor-size-default\">Conclusion<\/h3>\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-537ff70 elementor-widget elementor-widget-text-editor\" data-id=\"537ff70\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p>Congratulations, this article covered a lot of ground. We went from training a machine learning model through a manual process to automating the training, deployment and integrating our workflow with a code repository. This article did not cover an extensive list of all that can be done in a <strong>CICD workflow<\/strong>. For example, a more mature pipeline would handle things like unit tests, deployment to different environments, testing the data for training\/serving skew, and so forth, however, it did demonstrate a starting point to work towards those capabilities.<\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-7a2acbc elementor-align-center button-principal elementor-widget elementor-widget-button\" data-id=\"7a2acbc\" data-element_type=\"widget\" data-widget_type=\"button.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<div class=\"elementor-button-wrapper\">\n\t\t\t<a class=\"elementor-button elementor-button-link elementor-size-md\" href=\"https:\/\/datascientest.com\/en\/data-courses\">\n\t\t\t\t\t\t<span class=\"elementor-button-content-wrapper\">\n\t\t\t\t\t\t<span class=\"elementor-button-text\">Start a training course<\/span>\n\t\t<\/span>\n\t\t\t\t\t<\/a>\n\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t<div class=\"elementor-element elementor-element-b4b76bc e-flex e-con-boxed e-con\" data-id=\"b4b76bc\" data-element_type=\"container\" data-settings=\"{&quot;content_width&quot;:&quot;boxed&quot;}\">\n\t\t\t\t\t<div class=\"e-con-inner\">\n\t\t<div class=\"elementor-element elementor-element-d1cc582 e-flex e-con-boxed e-con\" data-id=\"d1cc582\" data-element_type=\"container\" data-settings=\"{&quot;content_width&quot;:&quot;boxed&quot;}\">\n\t\t\t\t\t<div class=\"e-con-inner\">\n\t\t\t\t<div class=\"elementor-element elementor-element-06f8cf4 elementor-widget elementor-widget-heading\" data-id=\"06f8cf4\" data-element_type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t<h3 class=\"elementor-heading-title elementor-size-default\">References :<\/h3>\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-ba138cd elementor-widget elementor-widget-text-editor\" data-id=\"ba138cd\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<ul class=\"p-rich_text_list p-rich_text_list__bullet\" data-stringify-type=\"unordered-list\" data-indent=\"0\" data-border=\"0\"><li><a href=\"https:\/\/medium.com\/google-cloud\/enterprise-mlops-with-google-cloud-vertex-ai-part-3-ci-cd-33d5e6e774a7\"><span style=\"font-weight: 400;\">https:\/\/medium.com\/google-cloud\/enterprise-mlops-with-google-cloud-vertex-ai-part-3-ci-cd-33d5e6e774a7<\/span><\/a><\/li><li style=\"font-weight: 400;\" aria-level=\"1\"><a href=\"https:\/\/cloud.google.com\/architecture\/architecture-for-mlops-using-tfx-kubeflow-pipelines-and-cloud-build\"><span style=\"font-weight: 400;\">Architecture for MLOps using TensorFlow Extended, Vertex AI Pipelines, and Cloud Build<\/span><\/a><\/li><li style=\"font-weight: 400;\" aria-level=\"1\"><a href=\"https:\/\/medium.com\/google-cloud\/how-to-implement-ci-cd-for-your-vertex-ai-pipeline-27963bead8bd\"><span style=\"font-weight: 400;\">https:\/\/medium.com\/google-cloud\/how-to-implement-ci-cd-for-your-vertex-ai-pipeline-27963bead8bd<\/span><\/a><\/li><li style=\"font-weight: 400;\" aria-level=\"1\"><a href=\"https:\/\/medium.com\/google-cloud\/how-to-implement-ci-cd-for-your-vertex-ai-pipeline-27963bead8bd\"><span style=\"font-weight: 400;\">https:\/\/medium.com\/google-cloud\/how-to-implement-ci-cd-for-your-vertex-ai-pipeline-27963bead8bd<\/span><\/a><\/li><\/ul>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-d8bbfdf elementor-widget elementor-widget-html\" data-id=\"d8bbfdf\" data-element_type=\"widget\" data-widget_type=\"html.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t<script type=\"application\/ld+json\">\r\n{\r\n  \"@context\": \"https:\/\/schema.org\",\r\n  \"@type\": \"FAQPage\",\r\n  \"mainEntity\": [{\r\n    \"@type\": \"Question\",\r\n    \"name\": \"Motivation\",\r\n    \"acceptedAnswer\": {\r\n      \"@type\": \"Answer\",\r\n      \"text\": \"The previous article discusses the complexities of the machine learning life cycle and how Vertex Pipelines, a managed version of Kubeflow on Google Cloud, can address operationalizing a machine learning model. The next step in the journey is to automate the training and deployment of the system in a manner that facilitates and enables reproducibility and time to market.\"\r\n    }\r\n  },{\r\n    \"@type\": \"Question\",\r\n    \"name\": \"Purpose\",\r\n    \"acceptedAnswer\": {\r\n      \"@type\": \"Answer\",\r\n      \"text\": \"The goal of this article is to serve as a gentle introduction to CICD for the Vertex Pipelines created in the previous article. To address all the components in a comprehensive CICD workflow in the image above would go beyond the scope of a short blog. This article assumes the reader has foundational knowledge on concepts surrounding git, docker, and cloud technologies.\"\r\n    }\r\n  },{\r\n    \"@type\": \"Question\",\r\n    \"name\": \"What is Cloud Build ?\",\r\n    \"acceptedAnswer\": {\r\n      \"@type\": \"Answer\",\r\n      \"text\": \"Cloud build is a fully managed, serverless CICD platform. Other potential include Jenkins, Azure DevOps and Github Actions. A CICD platform allows for teams to effectively and iteratively develop and ship software to different environments. This tutorial will use Google Cloud Build alongside a github repository to build and run containers using the code that was pushed to a main branch.\"\r\n    }\r\n  }]\r\n}\r\n<\/script>\r\n\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t\t\t<\/div>\n\t\t","protected":false},"excerpt":{"rendered":"<p>This is the second part of a series&nbsp; Part 1 here Motivation The previous article discusses the complexities of the machine learning life cycle and how Vertex Pipelines, a managed version of Kubeflow on Google Cloud, can address operationalizing a machine learning model. The next step in the journey is to automate the training and [&hellip;]<\/p>\n","protected":false},"author":45,"featured_media":185054,"comment_status":"open","ping_status":"open","sticky":false,"template":"elementor_theme","format":"standard","meta":{"content-type":"","inline_featured_image":false},"categories":[2425],"tags":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO Premium plugin v21.7 (Yoast SEO v21.7) - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>CI\/CD for Kubeflow (part 2) by Tony Ruiz, GCP Customer Engineer at Google<\/title>\n<meta name=\"description\" content=\"This is the second part of a series&nbsp; Part 1 here Motivation The previous article discusses the complexities of the machine learning life cycle and\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/datascientest.com\/en\/ci-cd-for-kubeflow\" \/>\n<meta property=\"og:locale\" content=\"en_GB\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"CI\/CD for Kubeflow (part 2) by Tony Ruiz, GCP Customer Engineer at Google\" \/>\n<meta property=\"og:description\" content=\"This is the second part of a series&nbsp; Part 1 here Motivation The previous article discusses the complexities of the machine learning life cycle and\" \/>\n<meta property=\"og:url\" content=\"https:\/\/datascientest.com\/en\/ci-cd-for-kubeflow\" \/>\n<meta property=\"og:site_name\" content=\"DataScientest\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/DataScientestEN\" \/>\n<meta property=\"article:published_time\" content=\"2024-06-11T13:20:59+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2024-06-11T13:21:01+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/datascientest.com\/en\/files\/2024\/05\/CI_CD_for_kubeflow.png\" \/>\n\t<meta property=\"og:image:width\" content=\"1200\" \/>\n\t<meta property=\"og:image:height\" content=\"600\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"author\" content=\"Daniel\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@DataScientest__\" \/>\n<meta name=\"twitter:site\" content=\"@DataScientest__\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Daniel\" \/>\n\t<meta name=\"twitter:label2\" content=\"Estimated reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"8 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/datascientest.com\/en\/ci-cd-for-kubeflow#article\",\"isPartOf\":{\"@id\":\"https:\/\/datascientest.com\/en\/ci-cd-for-kubeflow\"},\"author\":{\"name\":\"Daniel\",\"@id\":\"https:\/\/datascientest.com\/en\/#\/schema\/person\/5a86e30c5090234fb43dd8baa83c44b0\"},\"headline\":\"CI\/CD for Kubeflow (part 2) by Tony Ruiz, GCP Customer Engineer at Google\",\"datePublished\":\"2024-06-11T13:20:59+00:00\",\"dateModified\":\"2024-06-11T13:21:01+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/datascientest.com\/en\/ci-cd-for-kubeflow\"},\"wordCount\":1456,\"commentCount\":0,\"publisher\":{\"@id\":\"https:\/\/datascientest.com\/en\/#organization\"},\"articleSection\":[\"Data Science\"],\"inLanguage\":\"en-GB\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/datascientest.com\/en\/ci-cd-for-kubeflow#respond\"]}],\"copyrightYear\":\"2024\",\"copyrightHolder\":{\"@id\":\"https:\/\/datascientest.com\/#organization\"}},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/datascientest.com\/en\/ci-cd-for-kubeflow\",\"url\":\"https:\/\/datascientest.com\/en\/ci-cd-for-kubeflow\",\"name\":\"CI\/CD for Kubeflow (part 2) by Tony Ruiz, GCP Customer Engineer at Google\",\"isPartOf\":{\"@id\":\"https:\/\/datascientest.com\/en\/#website\"},\"datePublished\":\"2024-06-11T13:20:59+00:00\",\"dateModified\":\"2024-06-11T13:21:01+00:00\",\"description\":\"This is the second part of a series&nbsp; Part 1 here Motivation The previous article discusses the complexities of the machine learning life cycle and\",\"inLanguage\":\"en-GB\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/datascientest.com\/en\/ci-cd-for-kubeflow\"]}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/datascientest.com\/en\/#website\",\"url\":\"https:\/\/datascientest.com\/en\/\",\"name\":\"DataScientest\",\"description\":\"Leader in Data Science Courses\",\"publisher\":{\"@id\":\"https:\/\/datascientest.com\/en\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/datascientest.com\/en\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-GB\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/datascientest.com\/en\/#organization\",\"name\":\"DataScientest\",\"url\":\"https:\/\/datascientest.com\/en\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-GB\",\"@id\":\"https:\/\/datascientest.com\/en\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/datascientest.com\/en\/wp-content\/blogs.dir\/9\/files\/2024\/02\/DataScientest_icon_blue-2.png\",\"contentUrl\":\"https:\/\/datascientest.com\/en\/wp-content\/blogs.dir\/9\/files\/2024\/02\/DataScientest_icon_blue-2.png\",\"width\":301,\"height\":301,\"caption\":\"DataScientest\"},\"image\":{\"@id\":\"https:\/\/datascientest.com\/en\/#\/schema\/logo\/image\/\"},\"sameAs\":[\"https:\/\/www.facebook.com\/DataScientestEN\",\"https:\/\/twitter.com\/DataScientest__\",\"https:\/\/www.linkedin.com\/school\/datascientest\/mycompany\/\",\"https:\/\/www.tiktok.com\/@datascientest\"]},{\"@type\":\"Person\",\"@id\":\"https:\/\/datascientest.com\/en\/#\/schema\/person\/5a86e30c5090234fb43dd8baa83c44b0\",\"name\":\"Daniel\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-GB\",\"@id\":\"https:\/\/datascientest.com\/en\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/837acad4f2dc47e85d81cce837261670?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/837acad4f2dc47e85d81cce837261670?s=96&d=mm&r=g\",\"caption\":\"Daniel\"},\"sameAs\":[\"https:\/\/www.linkedin.com\/school\/datascientest\/\",\"user\"],\"jobTitle\":\"ing\u00e9nieur logiciel\",\"worksFor\":\"DataScientest\",\"url\":\"https:\/\/datascientest.com\/en\/author\/team-redac\"}]}<\/script>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"CI\/CD for Kubeflow (part 2) by Tony Ruiz, GCP Customer Engineer at Google","description":"This is the second part of a series&nbsp; Part 1 here Motivation The previous article discusses the complexities of the machine learning life cycle and","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/datascientest.com\/en\/ci-cd-for-kubeflow","og_locale":"en_GB","og_type":"article","og_title":"CI\/CD for Kubeflow (part 2) by Tony Ruiz, GCP Customer Engineer at Google","og_description":"This is the second part of a series&nbsp; Part 1 here Motivation The previous article discusses the complexities of the machine learning life cycle and","og_url":"https:\/\/datascientest.com\/en\/ci-cd-for-kubeflow","og_site_name":"DataScientest","article_publisher":"https:\/\/www.facebook.com\/DataScientestEN","article_published_time":"2024-06-11T13:20:59+00:00","article_modified_time":"2024-06-11T13:21:01+00:00","og_image":[{"width":1200,"height":600,"url":"https:\/\/datascientest.com\/en\/files\/2024\/05\/CI_CD_for_kubeflow.png","type":"image\/png"}],"author":"Daniel","twitter_card":"summary_large_image","twitter_creator":"@DataScientest__","twitter_site":"@DataScientest__","twitter_misc":{"Written by":"Daniel","Estimated reading time":"8 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/datascientest.com\/en\/ci-cd-for-kubeflow#article","isPartOf":{"@id":"https:\/\/datascientest.com\/en\/ci-cd-for-kubeflow"},"author":{"name":"Daniel","@id":"https:\/\/datascientest.com\/en\/#\/schema\/person\/5a86e30c5090234fb43dd8baa83c44b0"},"headline":"CI\/CD for Kubeflow (part 2) by Tony Ruiz, GCP Customer Engineer at Google","datePublished":"2024-06-11T13:20:59+00:00","dateModified":"2024-06-11T13:21:01+00:00","mainEntityOfPage":{"@id":"https:\/\/datascientest.com\/en\/ci-cd-for-kubeflow"},"wordCount":1456,"commentCount":0,"publisher":{"@id":"https:\/\/datascientest.com\/en\/#organization"},"articleSection":["Data Science"],"inLanguage":"en-GB","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/datascientest.com\/en\/ci-cd-for-kubeflow#respond"]}],"copyrightYear":"2024","copyrightHolder":{"@id":"https:\/\/datascientest.com\/#organization"}},{"@type":"WebPage","@id":"https:\/\/datascientest.com\/en\/ci-cd-for-kubeflow","url":"https:\/\/datascientest.com\/en\/ci-cd-for-kubeflow","name":"CI\/CD for Kubeflow (part 2) by Tony Ruiz, GCP Customer Engineer at Google","isPartOf":{"@id":"https:\/\/datascientest.com\/en\/#website"},"datePublished":"2024-06-11T13:20:59+00:00","dateModified":"2024-06-11T13:21:01+00:00","description":"This is the second part of a series&nbsp; Part 1 here Motivation The previous article discusses the complexities of the machine learning life cycle and","inLanguage":"en-GB","potentialAction":[{"@type":"ReadAction","target":["https:\/\/datascientest.com\/en\/ci-cd-for-kubeflow"]}]},{"@type":"WebSite","@id":"https:\/\/datascientest.com\/en\/#website","url":"https:\/\/datascientest.com\/en\/","name":"DataScientest","description":"Leader in Data Science Courses","publisher":{"@id":"https:\/\/datascientest.com\/en\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/datascientest.com\/en\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-GB"},{"@type":"Organization","@id":"https:\/\/datascientest.com\/en\/#organization","name":"DataScientest","url":"https:\/\/datascientest.com\/en\/","logo":{"@type":"ImageObject","inLanguage":"en-GB","@id":"https:\/\/datascientest.com\/en\/#\/schema\/logo\/image\/","url":"https:\/\/datascientest.com\/en\/wp-content\/blogs.dir\/9\/files\/2024\/02\/DataScientest_icon_blue-2.png","contentUrl":"https:\/\/datascientest.com\/en\/wp-content\/blogs.dir\/9\/files\/2024\/02\/DataScientest_icon_blue-2.png","width":301,"height":301,"caption":"DataScientest"},"image":{"@id":"https:\/\/datascientest.com\/en\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/DataScientestEN","https:\/\/twitter.com\/DataScientest__","https:\/\/www.linkedin.com\/school\/datascientest\/mycompany\/","https:\/\/www.tiktok.com\/@datascientest"]},{"@type":"Person","@id":"https:\/\/datascientest.com\/en\/#\/schema\/person\/5a86e30c5090234fb43dd8baa83c44b0","name":"Daniel","image":{"@type":"ImageObject","inLanguage":"en-GB","@id":"https:\/\/datascientest.com\/en\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/837acad4f2dc47e85d81cce837261670?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/837acad4f2dc47e85d81cce837261670?s=96&d=mm&r=g","caption":"Daniel"},"sameAs":["https:\/\/www.linkedin.com\/school\/datascientest\/","user"],"jobTitle":"ing\u00e9nieur logiciel","worksFor":"DataScientest","url":"https:\/\/datascientest.com\/en\/author\/team-redac"}]}},"_links":{"self":[{"href":"https:\/\/datascientest.com\/en\/wp-json\/wp\/v2\/posts\/185048"}],"collection":[{"href":"https:\/\/datascientest.com\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/datascientest.com\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/datascientest.com\/en\/wp-json\/wp\/v2\/users\/45"}],"replies":[{"embeddable":true,"href":"https:\/\/datascientest.com\/en\/wp-json\/wp\/v2\/comments?post=185048"}],"version-history":[{"count":0,"href":"https:\/\/datascientest.com\/en\/wp-json\/wp\/v2\/posts\/185048\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/datascientest.com\/en\/wp-json\/wp\/v2\/media\/185054"}],"wp:attachment":[{"href":"https:\/\/datascientest.com\/en\/wp-json\/wp\/v2\/media?parent=185048"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/datascientest.com\/en\/wp-json\/wp\/v2\/categories?post=185048"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/datascientest.com\/en\/wp-json\/wp\/v2\/tags?post=185048"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}