{"action":"created","issue":{"url":"https://api.github.com/repos/gaia-pipeline/gaia/issues/107","repository_url":"https://api.github.com/repos/gaia-pipeline/gaia","labels_url":"https://api.github.com/repos/gaia-pipeline/gaia/issues/107/labels{/name}","comments_url":"https://api.github.com/repos/gaia-pipeline/gaia/issues/107/comments","events_url":"https://api.github.com/repos/gaia-pipeline/gaia/issues/107/events","html_url":"https://github.com/gaia-pipeline/gaia/issues/107","id":357382633,"node_id":"MDU6SXNzdWUzNTczODI2MzM=","number":107,"title":"RFC: Distributer Pipeline execution via Workers","user":{"login":"Skarlso","id":182850,"node_id":"MDQ6VXNlcjE4Mjg1MA==","avatar_url":"https://avatars0.githubusercontent.com/u/182850?v=4","gravatar_id":"","url":"https://api.github.com/users/Skarlso","html_url":"https://github.com/Skarlso","followers_url":"https://api.github.com/users/Skarlso/followers","following_url":"https://api.github.com/users/Skarlso/following{/other_user}","gists_url":"https://api.github.com/users/Skarlso/gists{/gist_id}","starred_url":"https://api.github.com/users/Skarlso/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Skarlso/subscriptions","organizations_url":"https://api.github.com/users/Skarlso/orgs","repos_url":"https://api.github.com/users/Skarlso/repos","events_url":"https://api.github.com/users/Skarlso/events{/privacy}","received_events_url":"https://api.github.com/users/Skarlso/received_events","type":"User","site_admin":false},"labels":[{"id":1023353376,"node_id":"MDU6TGFiZWwxMDIzMzUzMzc2","url":"https://api.github.com/repos/gaia-pipeline/gaia/labels/needs%20review","name":"needs review","color":"c2e0c6","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2018-09-05T20:05:35Z","updated_at":"2018-09-06T08:08:48Z","closed_at":null,"author_association":"MEMBER","body":"# Abstract\r\n\r\nThis document discusses the problem of executing pipelines in a distributed\r\nmanner.\r\n\r\n# Table of Contents\r\n\r\n1. [Introduction](#introduction)\r\n2. [Problem Statement](#problem-statement)\r\n3. [Terminology](#terminology)\r\n4. [Architecture Diagram](#architercutre-diagram)\r\n5. [Proposed Worker Distribution Model](#proposed-worker-distribution-model)\r\n6. [Managing Workers](#managing-workers)\r\n7. [The Worker RPC API](#the-worker-rpc-api)\r\n8. [Gaia Master - Agent](#gaia-master-agent)\r\n9. [Scheduling Jobs](#scheduling-jobs)\r\n9. [Implementation Approach](#implementation-approach)\r\n\r\n\r\n# Introduction\r\n\r\n# Problem Statement\r\n\r\nThe problem poses the following set of challenges for Gaia:\r\n\r\n1. Manage workers\r\n    - See what pipeline is running on which worker at any given point in time\r\n    - Add / Delete / Suspend workers\r\n    - Add specific environment variables to the worker\r\n2. Either automatically, or manually choose which pipeline should run on which\r\nworker.\r\n3. Label the workers so the user knows it's a windows machine or a linux machine\r\nor Go, Python, Java SDK is available on it... etc.\r\n\r\n# Terminology\r\n\r\n**Gaia Master**: The Gaia Master is a running instance of gaia launched via make or the\r\nreleased Gaia binary.\r\n**Worker**: A worker is a server which is connected to the Gaia Master and has\r\ncertain capabilities like, what kind of SDK it supports or what operating system\r\nis installed on it.\r\n**Pipeline**: A pipeline is a configured entity with a set of Jobs.\r\n**Job**: A job is a single running task like, create a user. A pipeline can have multiple jobs.\r\n**RPC**: Remote Procedure Call\r\n\r\n# Architecture Diagram\r\n\r\n![distributed workers](https://user-images.githubusercontent.com/182850/45118121-c890bd00-b157-11e8-9eec-04ed9e8e4a05.png)\r\n\r\n# Proposed Worker Distribution Model\r\n\r\nThe proposed model which aims to solve this problem is laid out as follows. \r\n\r\n# Managing Workers\r\n\r\nThe managing of the workers will happen through a set of API endpoints.\r\nAll workers are stored in the database with a designated set of labels\r\nassigned name and IP address.\r\n\r\nThese endpoints will be Delete / Suspend. Since adding will be taken care of\r\nby the Gaia agent, we don't support that operation here specifically.\r\n\r\n**Delete**: Delete will simply remove the server from the rotation. It won't restart\r\nthe server, or shut it down, it will just simply delete it from the database which\r\nholds the worker instances.\r\n**Suspend**: Suspending a worker will take it out of rotation but will not delete it.\r\nSuspended this worker will not be able to run any pipelines. This is a good option\r\nif some kind of maintenance needs to be performed on the machine.\r\n\r\n## The Worker RPC API\r\n\r\nThe Workers will talk to the Gaia Master via a set of defined RPC interfaces.\r\nThese are as follows:\r\n\r\n```go\r\n// RegisterWorker will take a worker struct which contains the following information:\r\n// IP: The address of the worker\r\n// Name: The name of the worker which typically can be `hostname`.\r\n// Operating System: The OS of the worker to save as a label.\r\n// SDK: The SDK the worker has.\r\nrpc RegisterWorker(Worker) {}\r\n\r\n// RunPipeline will take a pipeline, and execute it. This ia bi-directional endpoint.\r\n// Pipeline struct:\r\n// ID: Id of the pipeline\r\n// Repo: The git repository for the pipeline. This is needed because the worker needs\r\n// to build the pipeline.\r\nrpc RunPipeline(Pipeline) returns (Success) {}\r\n\r\nrpc GetAllPipelines(Worker) returns (Pipelines) {}\r\n```\r\n\r\n# Gaia Master - Agent\r\n\r\nThe current Gaia implementation will still hold and will be designated as Gaia Master.\r\nThe master will be a hub for the worker to connect to, get pipelines from, and report\r\nback on the current state of the pipelines they are running.\r\n\r\nAs such, Gaia Master will no longer be solely responsible to build and distribute\r\nbinaries. Since the operation system of the worker decides in what format the binaries\r\nwill be in, the workers will build their own binaries.\r\n\r\nWhich means a worker will get a repository to pull code from and do the whole thing\r\nthat Gaia does currently. This will not involve duplicating code however, since the\r\nwhole thing will be in the `worker` package. Gaia Master will use this package by\r\nsetting worker to `localhost`.\r\n\r\nThe Workers will need to have the go-plugin extracted because HashiCorp's plugin\r\nsystem does not support RPC calls over the network. Just strictly localhost communication\r\nis allowed. Pipeline execution and communication between jobs' running and state\r\nchanges are all through RPC.\r\n\r\n# Scheduling Jobs\r\n\r\nScheduling jobs will also have to be included into the workers. Workers will schedule\r\ntheir own parallel jobs execution model and Gaia Master will have to schedule and manage\r\nwhich worker to distribute pipelines to. This means that the workers will need an indicator\r\nto define when they are too busy to accept more pipelines.\r\n\r\n# Implementation Approach\r\n\r\n1. Extract all functionality regarding running and building pipelines including\r\nthe SDK and the go-plugin facility into a worker package. This should not change\r\nthe current behavior of Gaia. All tests should still pass. Including the WebHook\r\ncapability which should be able to still just call build. The worker package should\r\ntake care of building and distributing the binary.\r\n\r\n2. Create the API which handles most of the things worker related. But still don't\r\nbother extracting it.\r\n\r\n3. Create an Agent binary which calls back to master's RPC API and registers a\r\nserver as a worker.\r\n\r\n4. Implement the managing of the servers below settings on the left of the admin\r\nscreen."},"comment":{"url":"https://api.github.com/repos/gaia-pipeline/gaia/issues/comments/419004477","html_url":"https://github.com/gaia-pipeline/gaia/issues/107#issuecomment-419004477","issue_url":"https://api.github.com/repos/gaia-pipeline/gaia/issues/107","id":419004477,"node_id":"MDEyOklzc3VlQ29tbWVudDQxOTAwNDQ3Nw==","user":{"login":"michelvocks","id":2463119,"node_id":"MDQ6VXNlcjI0NjMxMTk=","avatar_url":"https://avatars1.githubusercontent.com/u/2463119?v=4","gravatar_id":"","url":"https://api.github.com/users/michelvocks","html_url":"https://github.com/michelvocks","followers_url":"https://api.github.com/users/michelvocks/followers","following_url":"https://api.github.com/users/michelvocks/following{/other_user}","gists_url":"https://api.github.com/users/michelvocks/gists{/gist_id}","starred_url":"https://api.github.com/users/michelvocks/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/michelvocks/subscriptions","organizations_url":"https://api.github.com/users/michelvocks/orgs","repos_url":"https://api.github.com/users/michelvocks/repos","events_url":"https://api.github.com/users/michelvocks/events{/privacy}","received_events_url":"https://api.github.com/users/michelvocks/received_events","type":"User","site_admin":false},"created_at":"2018-09-06T08:08:48Z","updated_at":"2018-09-06T08:08:48Z","author_association":"MEMBER","body":"Awesome work, @Skarlso . 🤗 \r\n\r\nA few hints from my side:\r\n\r\n1. `RegisterWorker` should pass something like a secret. Otherwise attackers could easily attach random workers to Gaia and use that to their advantage.\r\n2. Currently it's not really clear when pipelines are build on the worker. When a user opens the detailed view of a pipeline, he should be able to see all jobs from this pipeline. That means the pipeline needs to be started after creation and `GetJobs` must be executed to get this information. \r\n3. We might need an interface for `DeregisterWorker`. If you dynamically spawn new workers \"on-demand\", they need to be removed somehow. And a graceful removal can be done by the worker itself.\r\n4. Really love the Suspend functionality! 🤗 "}}
{"actor":{"display_login":"michelvocks"}}