{"action":"created","comment":{"url":"https://api.github.com/repos/open-guides/og-aws/pulls/comments/82226375","pull_request_review_id":3148573,"id":82226375,"diff_hunk":"@@ -1256,6 +1256,35 @@ Billing and Cost Management\n -\t**Spot fleet:**\n \t-\tYou can realize even bigger cost reductions at the same time as improvements to fleet stability relative to regular Spot usage by using [Spot fleet](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/spot-fleet.html) to bid on instances across instance types, availability zones, and (through multiple Spot Fleet Requests) regions.\n \t-\tSpot fleet targets maintaining a specified (and weighted-by-instance-type) total capacity across a cluster of servers. If the Spot price of one instance type and availability zone combination rises above the weighted bid, it will rotate running instances out and bring up new ones of another type and location up in order to maintain the target capacity without going over target cluster cost.\n+\n+        -  Profile your application to figure out it's runtime characteristics. That would help give an understanding of the minimum cpu, memory, disk required. Having this information is critical before you try to optimize spot costs.\n+        - Once you know the minimum application requirements, instead of resorting to fixed instance types (r3.xlarge) you could bid across a variety of instance types (that gives you higher chances of getting a spot instance to run your application)\n+-  **Spot Usage Best Practices:**\n+\n+    - **Spot Price Monitoring and Intelligence:**\n+        - Spot Instance prices fluctuate depending on instance types, time of day, region and availability zone.  aws cli tools  and  api that allow you to describe spot price metadata given <Time, InstanceType,Region,AZ>.  \n+        -  Based on history of spot instance prices, you could potentially build a myriad of algorithms that would help you to pick an instance type that either\n+            - optimizes cost \n+            - maximizes availability\n+            - offers predictable performance\n+        - You could also track the number of times an instance of certain type got taken away (out bid) and plot that in graphite to improve your algorithm based on time of day.\n+\n+    - **Spot Machine Resource Utilization:**\n+        - For running spiky workloads (spark, map reduce jobs) that are schedule based and where failure is non critical, spot instances become the perfect candidates. \n+        - The time it takes to satisfy a spot instance could vary between 2-10 mins depending on the type of instance. \n+        - If you are running an infrastructure with 100s of  jobs of spiky nature, it is advisable to start pooling instances to optimize for cost, performance and most importantly time to acquire an instance. \n+        - Pooling implies creating and maintaining spot instances so that they do not get terminated after use. This promotes re-use of spot instances across jobs. This of course comes with the overhead of  lifecycle management.\n+        - Pooling has its own set of metrics that can be tracked to optimize resource utilization, efficiency and cost.\n+        - Typical pooling implementations give anywhere between 45-60% cost optimizations & 40% reduction in spot instance creationg time.\n+        - An excellent example of Pooling implementation is described here [credits to Netflix].\n+            *  http://techblog.netflix.com/2015/09/creating-your-own-ec2-spot-market.html\n+            * http://techblog.netflix.com/2015/11/creating-your-own-ec2-spot-market-part-2.html\n+\n+- **Spot Management Gotchas**\n+    - 🔸 **Lifetime** - There is no guarantee for the lifetime of a spot instance. It is purely based on bidding. If anyone outbids your price, the instance is taken away. Spot is not suitable for time sensitive jobs that have strong SLA. Instances will fail based on demand for spot at that time. ","path":"README.md","position":29,"original_position":29,"commit_id":"260dbc23ce14085dc8bb00dc20e5d560039cc7b0","original_commit_id":"260dbc23ce14085dc8bb00dc20e5d560039cc7b0","user":{"login":"nitingithub","id":3709637,"avatar_url":"https://avatars.githubusercontent.com/u/3709637?v=3","gravatar_id":"","url":"https://api.github.com/users/nitingithub","html_url":"https://github.com/nitingithub","followers_url":"https://api.github.com/users/nitingithub/followers","following_url":"https://api.github.com/users/nitingithub/following{/other_user}","gists_url":"https://api.github.com/users/nitingithub/gists{/gist_id}","starred_url":"https://api.github.com/users/nitingithub/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nitingithub/subscriptions","organizations_url":"https://api.github.com/users/nitingithub/orgs","repos_url":"https://api.github.com/users/nitingithub/repos","events_url":"https://api.github.com/users/nitingithub/events{/privacy}","received_events_url":"https://api.github.com/users/nitingithub/received_events","type":"User","site_admin":false},"body":"Done","created_at":"2016-10-06T16:16:24Z","updated_at":"2016-10-06T16:16:24Z","html_url":"https://github.com/open-guides/og-aws/pull/78#discussion_r82226375","pull_request_url":"https://api.github.com/repos/open-guides/og-aws/pulls/78","_links":{"self":{"href":"https://api.github.com/repos/open-guides/og-aws/pulls/comments/82226375"},"html":{"href":"https://github.com/open-guides/og-aws/pull/78#discussion_r82226375"},"pull_request":{"href":"https://api.github.com/repos/open-guides/og-aws/pulls/78"}}},"pull_request":{"url":"https://api.github.com/repos/open-guides/og-aws/pulls/78","id":88198438,"html_url":"https://github.com/open-guides/og-aws/pull/78","diff_url":"https://github.com/open-guides/og-aws/pull/78.diff","patch_url":"https://github.com/open-guides/og-aws/pull/78.patch","issue_url":"https://api.github.com/repos/open-guides/og-aws/issues/78","number":78,"state":"open","locked":false,"title":"Spot Instance Management, Monitoring and Gotchas","user":{"login":"nitingithub","id":3709637,"avatar_url":"https://avatars.githubusercontent.com/u/3709637?v=3","gravatar_id":"","url":"https://api.github.com/users/nitingithub","html_url":"https://github.com/nitingithub","followers_url":"https://api.github.com/users/nitingithub/followers","following_url":"https://api.github.com/users/nitingithub/following{/other_user}","gists_url":"https://api.github.com/users/nitingithub/gists{/gist_id}","starred_url":"https://api.github.com/users/nitingithub/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nitingithub/subscriptions","organizations_url":"https://api.github.com/users/nitingithub/orgs","repos_url":"https://api.github.com/users/nitingithub/repos","events_url":"https://api.github.com/users/nitingithub/events{/privacy}","received_events_url":"https://api.github.com/users/nitingithub/received_events","type":"User","site_admin":false},"body":"Pass 1 of Spot Instance Management, Monitoring and Gotchas. \r\nPass 2 will include a potential link to an external blog for spot utilization best practices and architecting using spot for high availability. (planning to write one on medium). ","created_at":"2016-10-06T04:49:07Z","updated_at":"2016-10-06T16:16:24Z","closed_at":null,"merged_at":null,"merge_commit_sha":"521f091b54d987dcd90cdbbb5e2d8a34971e766a","assignee":null,"assignees":[],"milestone":null,"commits_url":"https://api.github.com/repos/open-guides/og-aws/pulls/78/commits","review_comments_url":"https://api.github.com/repos/open-guides/og-aws/pulls/78/comments","review_comment_url":"https://api.github.com/repos/open-guides/og-aws/pulls/comments{/number}","comments_url":"https://api.github.com/repos/open-guides/og-aws/issues/78/comments","statuses_url":"https://api.github.com/repos/open-guides/og-aws/statuses/260dbc23ce14085dc8bb00dc20e5d560039cc7b0","head":{"label":"nitingithub:master","ref":"master","sha":"260dbc23ce14085dc8bb00dc20e5d560039cc7b0","user":{"login":"nitingithub","id":3709637,"avatar_url":"https://avatars.githubusercontent.com/u/3709637?v=3","gravatar_id":"","url":"https://api.github.com/users/nitingithub","html_url":"https://github.com/nitingithub","followers_url":"https://api.github.com/users/nitingithub/followers","following_url":"https://api.github.com/users/nitingithub/following{/other_user}","gists_url":"https://api.github.com/users/nitingithub/gists{/gist_id}","starred_url":"https://api.github.com/users/nitingithub/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nitingithub/subscriptions","organizations_url":"https://api.github.com/users/nitingithub/orgs","repos_url":"https://api.github.com/users/nitingithub/repos","events_url":"https://api.github.com/users/nitingithub/events{/privacy}","received_events_url":"https://api.github.com/users/nitingithub/received_events","type":"User","site_admin":false},"repo":{"id":70113407,"name":"og-aws","full_name":"nitingithub/og-aws","owner":{"login":"nitingithub","id":3709637,"avatar_url":"https://avatars.githubusercontent.com/u/3709637?v=3","gravatar_id":"","url":"https://api.github.com/users/nitingithub","html_url":"https://github.com/nitingithub","followers_url":"https://api.github.com/users/nitingithub/followers","following_url":"https://api.github.com/users/nitingithub/following{/other_user}","gists_url":"https://api.github.com/users/nitingithub/gists{/gist_id}","starred_url":"https://api.github.com/users/nitingithub/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nitingithub/subscriptions","organizations_url":"https://api.github.com/users/nitingithub/orgs","repos_url":"https://api.github.com/users/nitingithub/repos","events_url":"https://api.github.com/users/nitingithub/events{/privacy}","received_events_url":"https://api.github.com/users/nitingithub/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/nitingithub/og-aws","description":"📙 Amazon Web Services — a practical guide","fork":true,"url":"https://api.github.com/repos/nitingithub/og-aws","forks_url":"https://api.github.com/repos/nitingithub/og-aws/forks","keys_url":"https://api.github.com/repos/nitingithub/og-aws/keys{/key_id}","collaborators_url":"https://api.github.com/repos/nitingithub/og-aws/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/nitingithub/og-aws/teams","hooks_url":"https://api.github.com/repos/nitingithub/og-aws/hooks","issue_events_url":"https://api.github.com/repos/nitingithub/og-aws/issues/events{/number}","events_url":"https://api.github.com/repos/nitingithub/og-aws/events","assignees_url":"https://api.github.com/repos/nitingithub/og-aws/assignees{/user}","branches_url":"https://api.github.com/repos/nitingithub/og-aws/branches{/branch}","tags_url":"https://api.github.com/repos/nitingithub/og-aws/tags","blobs_url":"https://api.github.com/repos/nitingithub/og-aws/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/nitingithub/og-aws/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/nitingithub/og-aws/git/refs{/sha}","trees_url":"https://api.github.com/repos/nitingithub/og-aws/git/trees{/sha}","statuses_url":"https://api.github.com/repos/nitingithub/og-aws/statuses/{sha}","languages_url":"https://api.github.com/repos/nitingithub/og-aws/languages","stargazers_url":"https://api.github.com/repos/nitingithub/og-aws/stargazers","contributors_url":"https://api.github.com/repos/nitingithub/og-aws/contributors","subscribers_url":"https://api.github.com/repos/nitingithub/og-aws/subscribers","subscription_url":"https://api.github.com/repos/nitingithub/og-aws/subscription","commits_url":"https://api.github.com/repos/nitingithub/og-aws/commits{/sha}","git_commits_url":"https://api.github.com/repos/nitingithub/og-aws/git/commits{/sha}","comments_url":"https://api.github.com/repos/nitingithub/og-aws/comments{/number}","issue_comment_url":"https://api.github.com/repos/nitingithub/og-aws/issues/comments{/number}","contents_url":"https://api.github.com/repos/nitingithub/og-aws/contents/{+path}","compare_url":"https://api.github.com/repos/nitingithub/og-aws/compare/{base}...{head}","merges_url":"https://api.github.com/repos/nitingithub/og-aws/merges","archive_url":"https://api.github.com/repos/nitingithub/og-aws/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/nitingithub/og-aws/downloads","issues_url":"https://api.github.com/repos/nitingithub/og-aws/issues{/number}","pulls_url":"https://api.github.com/repos/nitingithub/og-aws/pulls{/number}","milestones_url":"https://api.github.com/repos/nitingithub/og-aws/milestones{/number}","notifications_url":"https://api.github.com/repos/nitingithub/og-aws/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/nitingithub/og-aws/labels{/name}","releases_url":"https://api.github.com/repos/nitingithub/og-aws/releases{/id}","deployments_url":"https://api.github.com/repos/nitingithub/og-aws/deployments","created_at":"2016-10-06T01:29:18Z","updated_at":"2016-10-06T01:29:19Z","pushed_at":"2016-10-06T04:11:19Z","git_url":"git://github.com/nitingithub/og-aws.git","ssh_url":"git@github.com:nitingithub/og-aws.git","clone_url":"https://github.com/nitingithub/og-aws.git","svn_url":"https://github.com/nitingithub/og-aws","homepage":"","size":2095,"stargazers_count":0,"watchers_count":0,"language":"Shell","has_issues":false,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"open-guides:master","ref":"master","sha":"7deaee51d10517ec59de2538d3daa4f17665d723","user":{"login":"open-guides","id":17152571,"avatar_url":"https://avatars.githubusercontent.com/u/17152571?v=3","gravatar_id":"","url":"https://api.github.com/users/open-guides","html_url":"https://github.com/open-guides","followers_url":"https://api.github.com/users/open-guides/followers","following_url":"https://api.github.com/users/open-guides/following{/other_user}","gists_url":"https://api.github.com/users/open-guides/gists{/gist_id}","starred_url":"https://api.github.com/users/open-guides/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/open-guides/subscriptions","organizations_url":"https://api.github.com/users/open-guides/orgs","repos_url":"https://api.github.com/users/open-guides/repos","events_url":"https://api.github.com/users/open-guides/events{/privacy}","received_events_url":"https://api.github.com/users/open-guides/received_events","type":"Organization","site_admin":false},"repo":{"id":63266213,"name":"og-aws","full_name":"open-guides/og-aws","owner":{"login":"open-guides","id":17152571,"avatar_url":"https://avatars.githubusercontent.com/u/17152571?v=3","gravatar_id":"","url":"https://api.github.com/users/open-guides","html_url":"https://github.com/open-guides","followers_url":"https://api.github.com/users/open-guides/followers","following_url":"https://api.github.com/users/open-guides/following{/other_user}","gists_url":"https://api.github.com/users/open-guides/gists{/gist_id}","starred_url":"https://api.github.com/users/open-guides/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/open-guides/subscriptions","organizations_url":"https://api.github.com/users/open-guides/orgs","repos_url":"https://api.github.com/users/open-guides/repos","events_url":"https://api.github.com/users/open-guides/events{/privacy}","received_events_url":"https://api.github.com/users/open-guides/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/open-guides/og-aws","description":"📙 Amazon Web Services — a practical guide","fork":false,"url":"https://api.github.com/repos/open-guides/og-aws","forks_url":"https://api.github.com/repos/open-guides/og-aws/forks","keys_url":"https://api.github.com/repos/open-guides/og-aws/keys{/key_id}","collaborators_url":"https://api.github.com/repos/open-guides/og-aws/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/open-guides/og-aws/teams","hooks_url":"https://api.github.com/repos/open-guides/og-aws/hooks","issue_events_url":"https://api.github.com/repos/open-guides/og-aws/issues/events{/number}","events_url":"https://api.github.com/repos/open-guides/og-aws/events","assignees_url":"https://api.github.com/repos/open-guides/og-aws/assignees{/user}","branches_url":"https://api.github.com/repos/open-guides/og-aws/branches{/branch}","tags_url":"https://api.github.com/repos/open-guides/og-aws/tags","blobs_url":"https://api.github.com/repos/open-guides/og-aws/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/open-guides/og-aws/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/open-guides/og-aws/git/refs{/sha}","trees_url":"https://api.github.com/repos/open-guides/og-aws/git/trees{/sha}","statuses_url":"https://api.github.com/repos/open-guides/og-aws/statuses/{sha}","languages_url":"https://api.github.com/repos/open-guides/og-aws/languages","stargazers_url":"https://api.github.com/repos/open-guides/og-aws/stargazers","contributors_url":"https://api.github.com/repos/open-guides/og-aws/contributors","subscribers_url":"https://api.github.com/repos/open-guides/og-aws/subscribers","subscription_url":"https://api.github.com/repos/open-guides/og-aws/subscription","commits_url":"https://api.github.com/repos/open-guides/og-aws/commits{/sha}","git_commits_url":"https://api.github.com/repos/open-guides/og-aws/git/commits{/sha}","comments_url":"https://api.github.com/repos/open-guides/og-aws/comments{/number}","issue_comment_url":"https://api.github.com/repos/open-guides/og-aws/issues/comments{/number}","contents_url":"https://api.github.com/repos/open-guides/og-aws/contents/{+path}","compare_url":"https://api.github.com/repos/open-guides/og-aws/compare/{base}...{head}","merges_url":"https://api.github.com/repos/open-guides/og-aws/merges","archive_url":"https://api.github.com/repos/open-guides/og-aws/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/open-guides/og-aws/downloads","issues_url":"https://api.github.com/repos/open-guides/og-aws/issues{/number}","pulls_url":"https://api.github.com/repos/open-guides/og-aws/pulls{/number}","milestones_url":"https://api.github.com/repos/open-guides/og-aws/milestones{/number}","notifications_url":"https://api.github.com/repos/open-guides/og-aws/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/open-guides/og-aws/labels{/name}","releases_url":"https://api.github.com/repos/open-guides/og-aws/releases{/id}","deployments_url":"https://api.github.com/repos/open-guides/og-aws/deployments","created_at":"2016-07-13T17:30:16Z","updated_at":"2016-10-06T16:08:00Z","pushed_at":"2016-10-06T04:49:08Z","git_url":"git://github.com/open-guides/og-aws.git","ssh_url":"git@github.com:open-guides/og-aws.git","clone_url":"https://github.com/open-guides/og-aws.git","svn_url":"https://github.com/open-guides/og-aws","homepage":"","size":2026,"stargazers_count":45,"watchers_count":45,"language":"Shell","has_issues":true,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":5,"mirror_url":null,"open_issues_count":45,"forks":5,"open_issues":45,"watchers":45,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/open-guides/og-aws/pulls/78"},"html":{"href":"https://github.com/open-guides/og-aws/pull/78"},"issue":{"href":"https://api.github.com/repos/open-guides/og-aws/issues/78"},"comments":{"href":"https://api.github.com/repos/open-guides/og-aws/issues/78/comments"},"review_comments":{"href":"https://api.github.com/repos/open-guides/og-aws/pulls/78/comments"},"review_comment":{"href":"https://api.github.com/repos/open-guides/og-aws/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/open-guides/og-aws/pulls/78/commits"},"statuses":{"href":"https://api.github.com/repos/open-guides/og-aws/statuses/260dbc23ce14085dc8bb00dc20e5d560039cc7b0"}}}}
{"actor":{"display_login":"nitingithub"}}