Open Source Summit Europe 2022

In Kubeflow Pipelines v1, the pipeline spec describing ML flow is platform-dependent, which makes no chance to deploy your ML pipelines on other pipeline frameworks. To lower the barrier, Intermediate Representation(IR) is available in kfp v2. It's a generic component/step-oriented specification that fits container orchestration frameworks. More importantly, a new pipeline orchestration engine is created on the backend to support automatic lineage tracking and components that consume or produce metadata. In light of these new features, it gets a foot into ML pipeline portability. You can port the necessary components to run your pipelines on the desired platforms. Currently, IR is in alpha version and used by Kubeflow Pipelines, and Google Vertex AI. Kubeflow Pipelines with Tekton will join the list soon. In this talk, we walk through the new IR spec, a list of components that constitute the new pipeline orchestration engine, and how we adapt the new features in another pipeline framework.