Polyaxon: An open source Deep Learning / Machine Learning stack on Kubernetes

Feb. 18, 2018, 4:09 a.m. By: Kirti Bakshi


Kubernetes is a service that runs on cloud infrastructures. It provides a single point of contact with the machinery of your cluster deployment and allows a user to specify the computational requirements that they need and then, it further handles the resources on the cluster and ensures that these resources are available always. If in case something does go down, Kubernetes will then try to automatically bring it back up.

Now, moving onto what Polyaxon is:

Polyaxon is basically a platform that is made use for building, training, as well as monitoring deep learning applications that are at a larger scale.

This platform easily deploys into any cloud provider, data center, or can be hosted and managed by Polyaxon itself as well, and to add to it, even more, it also supports all the major deep learning frameworks that include Tensorflow, MXNet, Caffe, Torch, etc.

Polyaxon, therefore, makes the development of deep learning applications faster, easier, and more efficient by managing workloads with the help of smart container and node management. And for your team or organization, it also turns GPU servers into shared, self-service resources.

How is it used with Kubernetes?

Polyaxon is installed into Kubernetes making the use of Helm that is the official package manager for Kubernetes, and a way of specifying Kubernetes objects with a standard template.

The advantages it comes with include the following:

  • Gain more productivity and ship faster: Polyaxon is a platform-as-a-Service for building deep learning models of any scale, it speeds up your data scientists job by eliminating infrastructure work.

  • Resources management that is Simplified: Polyaxon by running distributed models and concurrent experiments provide its users with a simple, transparent, and powerful way to maximize the cluster resources. The platform takes care of managing, scheduling as well as tracking all your jobs. So fewer worries!

  • The Benefit of Data privacy and elasticity: This platform gives you full control over your data as it runs everywhere, in the cloud and on-premise. The user can scale up and down the infrastructure as and when needed.

  • Supports Major Libraries: The platform can be run with all popular deep learning frameworks, that enables the user to push ideas to production very quickly.

Current Status of the Project:

Polyaxon is currently in its "beta" state. All interfaces, programming interfaces, and data structures may be changed. The best will be done to communicate potentially disruptive changes.

Now, what is actually required?

1. Requirements:

Just so you can run Polyaxon on the top of Kubernetes, the requirements are:

  • Unless you have all the images already, you need access to the internet to pull out images, that are required for running your experiments accessible to Kubernetes through a locally deployed registry.

  • Kubernetes with a version >= 1.8.0.

  • Helm with a version ≥ 2.5.

  • Persistence for data, outputs, logs, and code.

2. Computational Resources:

In order to run Polyaxon, you need some computational resources,

  • Computing

  • Disk space

  • GPU

  • Networking (both internal and external)

  • Creating, resizing, and deleting clusters

These resources could be your personal computer, university server, or some other organization that hosts computational resources that can be accessed remotely.

Polyaxon, when worked with custom cluster deployments or a cloud provider will give good result as well. So, any cluster with Kubernetes installed will work with Polyaxon for all these materials.

For More Information: GitHub

To Learn More About Polyaxon: Documentation.

Official Link: Polyaxon