{"action":"created","comment":{"url":"https://api.github.com/repos/awslabs/mxnet-model-server/pulls/comments/178605336","pull_request_review_id":108679305,"id":178605336,"diff_hunk":"@@ -0,0 +1,256 @@\n+# Serverless Inference with MMS on FARGATE\n+\n+This is self-contained step by step guide that shows how to create ECS with Fargate in order to do a serverless inference with MMS. Even though it is fully self contained we do expect reader to have some knowledge about the following topics:\n+\n+* [MMS](https://github.com/awslabs/mxnet-model-server)\n+* [ECS](https://aws.amazon.com/ecs)\n+* [Fargate](https://aws.amazon.com/fargate)\n+* [MXNet](https://aws.amazon.com/mxnet/)\n+* [Docker](https://www.docker.com/)\n+\n+Since we are doing inference, we need to have a pre-trained model that we can use to run inference. For the sake of this article we will be using [SqueezeNet model](https://github.com/awslabs/mxnet-model-server/blob/master/docs/model_zoo.md#squeezenet_v1.1). In short SqueezeNet is the model that allows you to recognize objects on the picture. \n+\n+Now, when we have the model chosen let's discuss on the high level how our end solution will looks like:\n+\n+![architecture](https://s3.amazonaws.com/mms-github-assets/MMS+with+Fargate+Article/architecture_2.png)\n+\n+Do not be worried if something is not clear from the picture. We are going to walk you step by step. And these steps are:\n+\n+1. Familiarize yourself with our containers\n+2. Create a SqueezeNet task definition (with the docker container of MMS) \n+3. Create Fargate ECS cluster\n+4. Create Application Load Balancer\n+5. Create Squeezenet Fargate service on the cluster\n+6. Profit!\n+\n+Let the show begin...\n+\n+## Familiarize Yourself With Our Containers \n+\n+With our current release of [MMS, 0.3](https://github.com/awslabs/mxnet-model-server/releases/tag/v0.3.0), we now providing official containers with the MMS preinstalled already. There are 2 containers:\n+\n+* [awsdeeplearningteam/mms_cpu](https://hub.docker.com/r/awsdeeplearningteam/mms_cpu/)\n+* [awsdeeplearningteam/mms_gpu](https://hub.docker.com/r/awsdeeplearningteam/mms_gpu/)\n+\n+Since we are doing inference we are going to use cpu container(mms_cpu). \n+\n+There are several constrains that one should consider when using Fargate:\n+\n+1. There is no GPU support at the moment\n+2. mms_cpu container is optimized for the Skylake Intel processors (that we have on our [C5 EC2 instances](https://aws.amazon.com/ec2/instance-types/c5/)). However since we are using Fargate, unfortunately there is no guarantee that the actual hardware will be Skylake  \n+\n+Our containers comes with pre-installed config of the SqueezeNet model (such a nice coincidence that we have decided to run inference for exactly this model :) ). Even though the config is pre-baked in the container we would highly recommend you to have a quick look on it, just to familiarize yourself, since for your own model most likely you will have to update it, plus it very-very simple, [check it yourself](https://github.com/awslabs/mxnet-model-server/blob/master/docker/mms_app_cpu.conf). To be more precise, here is the [line in the config](https://github.com/awslabs/mxnet-model-server/blob/master/docker/mms_app_cpu.conf#L3) that is pointing to the actual binary of the model. By a close look one can see that it is just a  public HTTPS link to the binary:\n+\n+```\n+https://s3.amazonaws.com/model-server/models/squeezenet_v1.1/squeezenet_v1.1.model\n+```\n+\n+So there is no need to pre-bake actual binary of the model to the container, you can just specify the HTTPS link to the binary.\n+\n+The last question that we need to address: how we should be starting our MMS within our container. And the answer is very simple, you just need to set the following ENTRYPOINT ([what is ENTRYPOINT?](https://docs.docker.com/engine/reference/builder/#entrypoint): \n+\n+```bash\n+mxnet-model-server start --mms-config /mxnet-model-server/mms_app_cpu.conf\n+```\n+\n+And this it, nothing else.\n+\n+At this point I think we are ready to start creating actual Task definition.\n+\n+## Create a SqueezeNet Task Definition\n+\n+This is the first task where we finally ready to start doing something:\n+\n+1. Log-in to the AWS console and go to the Elastic Cloud Service / Task Definitions and press “Create new Task Definition”:\n+\n+![task def](https://s3.amazonaws.com/mms-github-assets/MMS+with+Fargate+Article/1_Create_task_definition.png)\n+\n+2. Now you need to specify type of the task, surprise-surprise, we will be using the Fargate task:\n+\n+![](https://s3.amazonaws.com/mms-github-assets/MMS+with+Fargate+Article/2_Select_Fargate.png)\n+\n+3. Task requires some configuration, let's look on it step by step, first set the name:\n+\n+![](https://s3.amazonaws.com/mms-github-assets/MMS+with+Fargate+Article/3_Config_1.png)\n+\n+Now is important part, one need to create [IAM role](https://aws.amazon.com/iam) that will be used to publish metrics to CloudWatch:\n+\n+![](https://s3.amazonaws.com/mms-github-assets/MMS+with+Fargate+Article/Task+Execution+IAM+Role+.png)\n+\n+Our containers are optimized for 8 vCPUs, however in our example we going to use slightly smaller task with 4 vCPUs and 8 GB of RAM:\n+\n+![](https://s3.amazonaws.com/mms-github-assets/MMS+with+Fargate+Article/cpu+and+ram.png)\n+\n+2. Now it is time to configure the actual container that the task should be executing\n+\n+![](https://s3.amazonaws.com/mms-github-assets/MMS+with+Fargate+Article/container+step+1.png)\n+\n+3. Next one need to specify port mapping, we need to expose port 8080. This is the port that our container is listening to. If needed it can be configured via the config [here](https://github.com/awslabs/mxnet-model-server/blob/master/docker/mms_app_cpu.conf#L40).\n+\n+![](https://s3.amazonaws.com/mms-github-assets/MMS+with+Fargate+Article/port+8080.png)\n+\n+Now the health checks. MMS has pre-configured endpoint /ping that can be used for health checks.","path":"docs/mms_on_fargate.md","position":92,"original_position":92,"commit_id":"97863ae9612186b6e309844f1b39449f8b2c553b","original_commit_id":"97863ae9612186b6e309844f1b39449f8b2c553b","user":{"login":"aaronmarkham","id":5974205,"avatar_url":"https://avatars1.githubusercontent.com/u/5974205?v=4","gravatar_id":"","url":"https://api.github.com/users/aaronmarkham","html_url":"https://github.com/aaronmarkham","followers_url":"https://api.github.com/users/aaronmarkham/followers","following_url":"https://api.github.com/users/aaronmarkham/following{/other_user}","gists_url":"https://api.github.com/users/aaronmarkham/gists{/gist_id}","starred_url":"https://api.github.com/users/aaronmarkham/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/aaronmarkham/subscriptions","organizations_url":"https://api.github.com/users/aaronmarkham/orgs","repos_url":"https://api.github.com/users/aaronmarkham/repos","events_url":"https://api.github.com/users/aaronmarkham/events{/privacy}","received_events_url":"https://api.github.com/users/aaronmarkham/received_events","type":"User","site_admin":false},"body":"The next task if for the health checks. MMS has a\r\n\r\n`/ping`\r\n","created_at":"2018-04-02T18:00:18Z","updated_at":"2018-04-02T18:26:32Z","html_url":"https://github.com/awslabs/mxnet-model-server/pull/353#discussion_r178605336","pull_request_url":"https://api.github.com/repos/awslabs/mxnet-model-server/pulls/353","author_association":"CONTRIBUTOR","_links":{"self":{"href":"https://api.github.com/repos/awslabs/mxnet-model-server/pulls/comments/178605336"},"html":{"href":"https://github.com/awslabs/mxnet-model-server/pull/353#discussion_r178605336"},"pull_request":{"href":"https://api.github.com/repos/awslabs/mxnet-model-server/pulls/353"}}},"pull_request":{"url":"https://api.github.com/repos/awslabs/mxnet-model-server/pulls/353","id":178772033,"html_url":"https://github.com/awslabs/mxnet-model-server/pull/353","diff_url":"https://github.com/awslabs/mxnet-model-server/pull/353.diff","patch_url":"https://github.com/awslabs/mxnet-model-server/pull/353.patch","issue_url":"https://api.github.com/repos/awslabs/mxnet-model-server/issues/353","number":353,"state":"open","locked":false,"title":"MMS and Fargate article added","user":{"login":"b0noI","id":554101,"avatar_url":"https://avatars2.githubusercontent.com/u/554101?v=4","gravatar_id":"","url":"https://api.github.com/users/b0noI","html_url":"https://github.com/b0noI","followers_url":"https://api.github.com/users/b0noI/followers","following_url":"https://api.github.com/users/b0noI/following{/other_user}","gists_url":"https://api.github.com/users/b0noI/gists{/gist_id}","starred_url":"https://api.github.com/users/b0noI/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/b0noI/subscriptions","organizations_url":"https://api.github.com/users/b0noI/orgs","repos_url":"https://api.github.com/users/b0noI/repos","events_url":"https://api.github.com/users/b0noI/events{/privacy}","received_events_url":"https://api.github.com/users/b0noI/received_events","type":"User","site_admin":false},"body":"","created_at":"2018-04-02T04:41:08Z","updated_at":"2018-04-02T18:26:33Z","closed_at":null,"merged_at":null,"merge_commit_sha":"8f3eca9cecdb71c294f266962d607db80875913d","assignee":null,"assignees":[],"requested_reviewers":[],"requested_teams":[],"labels":[],"milestone":null,"commits_url":"https://api.github.com/repos/awslabs/mxnet-model-server/pulls/353/commits","review_comments_url":"https://api.github.com/repos/awslabs/mxnet-model-server/pulls/353/comments","review_comment_url":"https://api.github.com/repos/awslabs/mxnet-model-server/pulls/comments{/number}","comments_url":"https://api.github.com/repos/awslabs/mxnet-model-server/issues/353/comments","statuses_url":"https://api.github.com/repos/awslabs/mxnet-model-server/statuses/97863ae9612186b6e309844f1b39449f8b2c553b","head":{"label":"b0noI:master","ref":"master","sha":"97863ae9612186b6e309844f1b39449f8b2c553b","user":{"login":"b0noI","id":554101,"avatar_url":"https://avatars2.githubusercontent.com/u/554101?v=4","gravatar_id":"","url":"https://api.github.com/users/b0noI","html_url":"https://github.com/b0noI","followers_url":"https://api.github.com/users/b0noI/followers","following_url":"https://api.github.com/users/b0noI/following{/other_user}","gists_url":"https://api.github.com/users/b0noI/gists{/gist_id}","starred_url":"https://api.github.com/users/b0noI/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/b0noI/subscriptions","organizations_url":"https://api.github.com/users/b0noI/orgs","repos_url":"https://api.github.com/users/b0noI/repos","events_url":"https://api.github.com/users/b0noI/events{/privacy}","received_events_url":"https://api.github.com/users/b0noI/received_events","type":"User","site_admin":false},"repo":{"id":127702992,"name":"mxnet-model-server","full_name":"b0noI/mxnet-model-server","owner":{"login":"b0noI","id":554101,"avatar_url":"https://avatars2.githubusercontent.com/u/554101?v=4","gravatar_id":"","url":"https://api.github.com/users/b0noI","html_url":"https://github.com/b0noI","followers_url":"https://api.github.com/users/b0noI/followers","following_url":"https://api.github.com/users/b0noI/following{/other_user}","gists_url":"https://api.github.com/users/b0noI/gists{/gist_id}","starred_url":"https://api.github.com/users/b0noI/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/b0noI/subscriptions","organizations_url":"https://api.github.com/users/b0noI/orgs","repos_url":"https://api.github.com/users/b0noI/repos","events_url":"https://api.github.com/users/b0noI/events{/privacy}","received_events_url":"https://api.github.com/users/b0noI/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/b0noI/mxnet-model-server","description":"Model Server for Apache MXNet is a tool for serving neural net models for inference","fork":true,"url":"https://api.github.com/repos/b0noI/mxnet-model-server","forks_url":"https://api.github.com/repos/b0noI/mxnet-model-server/forks","keys_url":"https://api.github.com/repos/b0noI/mxnet-model-server/keys{/key_id}","collaborators_url":"https://api.github.com/repos/b0noI/mxnet-model-server/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/b0noI/mxnet-model-server/teams","hooks_url":"https://api.github.com/repos/b0noI/mxnet-model-server/hooks","issue_events_url":"https://api.github.com/repos/b0noI/mxnet-model-server/issues/events{/number}","events_url":"https://api.github.com/repos/b0noI/mxnet-model-server/events","assignees_url":"https://api.github.com/repos/b0noI/mxnet-model-server/assignees{/user}","branches_url":"https://api.github.com/repos/b0noI/mxnet-model-server/branches{/branch}","tags_url":"https://api.github.com/repos/b0noI/mxnet-model-server/tags","blobs_url":"https://api.github.com/repos/b0noI/mxnet-model-server/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/b0noI/mxnet-model-server/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/b0noI/mxnet-model-server/git/refs{/sha}","trees_url":"https://api.github.com/repos/b0noI/mxnet-model-server/git/trees{/sha}","statuses_url":"https://api.github.com/repos/b0noI/mxnet-model-server/statuses/{sha}","languages_url":"https://api.github.com/repos/b0noI/mxnet-model-server/languages","stargazers_url":"https://api.github.com/repos/b0noI/mxnet-model-server/stargazers","contributors_url":"https://api.github.com/repos/b0noI/mxnet-model-server/contributors","subscribers_url":"https://api.github.com/repos/b0noI/mxnet-model-server/subscribers","subscription_url":"https://api.github.com/repos/b0noI/mxnet-model-server/subscription","commits_url":"https://api.github.com/repos/b0noI/mxnet-model-server/commits{/sha}","git_commits_url":"https://api.github.com/repos/b0noI/mxnet-model-server/git/commits{/sha}","comments_url":"https://api.github.com/repos/b0noI/mxnet-model-server/comments{/number}","issue_comment_url":"https://api.github.com/repos/b0noI/mxnet-model-server/issues/comments{/number}","contents_url":"https://api.github.com/repos/b0noI/mxnet-model-server/contents/{+path}","compare_url":"https://api.github.com/repos/b0noI/mxnet-model-server/compare/{base}...{head}","merges_url":"https://api.github.com/repos/b0noI/mxnet-model-server/merges","archive_url":"https://api.github.com/repos/b0noI/mxnet-model-server/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/b0noI/mxnet-model-server/downloads","issues_url":"https://api.github.com/repos/b0noI/mxnet-model-server/issues{/number}","pulls_url":"https://api.github.com/repos/b0noI/mxnet-model-server/pulls{/number}","milestones_url":"https://api.github.com/repos/b0noI/mxnet-model-server/milestones{/number}","notifications_url":"https://api.github.com/repos/b0noI/mxnet-model-server/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/b0noI/mxnet-model-server/labels{/name}","releases_url":"https://api.github.com/repos/b0noI/mxnet-model-server/releases{/id}","deployments_url":"https://api.github.com/repos/b0noI/mxnet-model-server/deployments","created_at":"2018-04-02T04:35:50Z","updated_at":"2018-04-02T17:30:00Z","pushed_at":"2018-04-02T17:29:58Z","git_url":"git://github.com/b0noI/mxnet-model-server.git","ssh_url":"git@github.com:b0noI/mxnet-model-server.git","clone_url":"https://github.com/b0noI/mxnet-model-server.git","svn_url":"https://github.com/b0noI/mxnet-model-server","homepage":"","size":31175,"stargazers_count":0,"watchers_count":0,"language":"Python","has_issues":false,"has_projects":true,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":0,"mirror_url":null,"archived":false,"open_issues_count":0,"license":null,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"awslabs:master","ref":"master","sha":"ba5df68da09c91ba81e782cdff1c72c50e50cf34","user":{"login":"awslabs","id":3299148,"avatar_url":"https://avatars0.githubusercontent.com/u/3299148?v=4","gravatar_id":"","url":"https://api.github.com/users/awslabs","html_url":"https://github.com/awslabs","followers_url":"https://api.github.com/users/awslabs/followers","following_url":"https://api.github.com/users/awslabs/following{/other_user}","gists_url":"https://api.github.com/users/awslabs/gists{/gist_id}","starred_url":"https://api.github.com/users/awslabs/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/awslabs/subscriptions","organizations_url":"https://api.github.com/users/awslabs/orgs","repos_url":"https://api.github.com/users/awslabs/repos","events_url":"https://api.github.com/users/awslabs/events{/privacy}","received_events_url":"https://api.github.com/users/awslabs/received_events","type":"Organization","site_admin":false},"repo":{"id":105830539,"name":"mxnet-model-server","full_name":"awslabs/mxnet-model-server","owner":{"login":"awslabs","id":3299148,"avatar_url":"https://avatars0.githubusercontent.com/u/3299148?v=4","gravatar_id":"","url":"https://api.github.com/users/awslabs","html_url":"https://github.com/awslabs","followers_url":"https://api.github.com/users/awslabs/followers","following_url":"https://api.github.com/users/awslabs/following{/other_user}","gists_url":"https://api.github.com/users/awslabs/gists{/gist_id}","starred_url":"https://api.github.com/users/awslabs/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/awslabs/subscriptions","organizations_url":"https://api.github.com/users/awslabs/orgs","repos_url":"https://api.github.com/users/awslabs/repos","events_url":"https://api.github.com/users/awslabs/events{/privacy}","received_events_url":"https://api.github.com/users/awslabs/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/awslabs/mxnet-model-server","description":"Model Server for Apache MXNet is a tool for serving neural net models for inference","fork":false,"url":"https://api.github.com/repos/awslabs/mxnet-model-server","forks_url":"https://api.github.com/repos/awslabs/mxnet-model-server/forks","keys_url":"https://api.github.com/repos/awslabs/mxnet-model-server/keys{/key_id}","collaborators_url":"https://api.github.com/repos/awslabs/mxnet-model-server/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/awslabs/mxnet-model-server/teams","hooks_url":"https://api.github.com/repos/awslabs/mxnet-model-server/hooks","issue_events_url":"https://api.github.com/repos/awslabs/mxnet-model-server/issues/events{/number}","events_url":"https://api.github.com/repos/awslabs/mxnet-model-server/events","assignees_url":"https://api.github.com/repos/awslabs/mxnet-model-server/assignees{/user}","branches_url":"https://api.github.com/repos/awslabs/mxnet-model-server/branches{/branch}","tags_url":"https://api.github.com/repos/awslabs/mxnet-model-server/tags","blobs_url":"https://api.github.com/repos/awslabs/mxnet-model-server/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/awslabs/mxnet-model-server/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/awslabs/mxnet-model-server/git/refs{/sha}","trees_url":"https://api.github.com/repos/awslabs/mxnet-model-server/git/trees{/sha}","statuses_url":"https://api.github.com/repos/awslabs/mxnet-model-server/statuses/{sha}","languages_url":"https://api.github.com/repos/awslabs/mxnet-model-server/languages","stargazers_url":"https://api.github.com/repos/awslabs/mxnet-model-server/stargazers","contributors_url":"https://api.github.com/repos/awslabs/mxnet-model-server/contributors","subscribers_url":"https://api.github.com/repos/awslabs/mxnet-model-server/subscribers","subscription_url":"https://api.github.com/repos/awslabs/mxnet-model-server/subscription","commits_url":"https://api.github.com/repos/awslabs/mxnet-model-server/commits{/sha}","git_commits_url":"https://api.github.com/repos/awslabs/mxnet-model-server/git/commits{/sha}","comments_url":"https://api.github.com/repos/awslabs/mxnet-model-server/comments{/number}","issue_comment_url":"https://api.github.com/repos/awslabs/mxnet-model-server/issues/comments{/number}","contents_url":"https://api.github.com/repos/awslabs/mxnet-model-server/contents/{+path}","compare_url":"https://api.github.com/repos/awslabs/mxnet-model-server/compare/{base}...{head}","merges_url":"https://api.github.com/repos/awslabs/mxnet-model-server/merges","archive_url":"https://api.github.com/repos/awslabs/mxnet-model-server/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/awslabs/mxnet-model-server/downloads","issues_url":"https://api.github.com/repos/awslabs/mxnet-model-server/issues{/number}","pulls_url":"https://api.github.com/repos/awslabs/mxnet-model-server/pulls{/number}","milestones_url":"https://api.github.com/repos/awslabs/mxnet-model-server/milestones{/number}","notifications_url":"https://api.github.com/repos/awslabs/mxnet-model-server/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/awslabs/mxnet-model-server/labels{/name}","releases_url":"https://api.github.com/repos/awslabs/mxnet-model-server/releases{/id}","deployments_url":"https://api.github.com/repos/awslabs/mxnet-model-server/deployments","created_at":"2017-10-04T23:41:07Z","updated_at":"2018-04-01T03:34:59Z","pushed_at":"2018-04-02T17:29:59Z","git_url":"git://github.com/awslabs/mxnet-model-server.git","ssh_url":"git@github.com:awslabs/mxnet-model-server.git","clone_url":"https://github.com/awslabs/mxnet-model-server.git","svn_url":"https://github.com/awslabs/mxnet-model-server","homepage":"","size":31163,"stargazers_count":167,"watchers_count":167,"language":"Python","has_issues":true,"has_projects":false,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":33,"mirror_url":null,"archived":false,"open_issues_count":30,"license":null,"forks":33,"open_issues":30,"watchers":167,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/awslabs/mxnet-model-server/pulls/353"},"html":{"href":"https://github.com/awslabs/mxnet-model-server/pull/353"},"issue":{"href":"https://api.github.com/repos/awslabs/mxnet-model-server/issues/353"},"comments":{"href":"https://api.github.com/repos/awslabs/mxnet-model-server/issues/353/comments"},"review_comments":{"href":"https://api.github.com/repos/awslabs/mxnet-model-server/pulls/353/comments"},"review_comment":{"href":"https://api.github.com/repos/awslabs/mxnet-model-server/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/awslabs/mxnet-model-server/pulls/353/commits"},"statuses":{"href":"https://api.github.com/repos/awslabs/mxnet-model-server/statuses/97863ae9612186b6e309844f1b39449f8b2c553b"}},"author_association":"NONE"}}
{"actor":{"display_login":"aaronmarkham"}}