{"action":"closed","number":22009,"pull_request":{"url":"https://api.github.com/repos/ceph/ceph/pulls/22009","id":188143511,"node_id":"MDExOlB1bGxSZXF1ZXN0MTg4MTQzNTEx","html_url":"https://github.com/ceph/ceph/pull/22009","diff_url":"https://github.com/ceph/ceph/pull/22009.diff","patch_url":"https://github.com/ceph/ceph/pull/22009.patch","issue_url":"https://api.github.com/repos/ceph/ceph/issues/22009","number":22009,"state":"closed","locked":false,"title":"common/PriorityCache: First Step toward priority based caching","user":{"login":"markhpc","id":1286295,"node_id":"MDQ6VXNlcjEyODYyOTU=","avatar_url":"https://avatars0.githubusercontent.com/u/1286295?v=4","gravatar_id":"","url":"https://api.github.com/users/markhpc","html_url":"https://github.com/markhpc","followers_url":"https://api.github.com/users/markhpc/followers","following_url":"https://api.github.com/users/markhpc/following{/other_user}","gists_url":"https://api.github.com/users/markhpc/gists{/gist_id}","starred_url":"https://api.github.com/users/markhpc/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/markhpc/subscriptions","organizations_url":"https://api.github.com/users/markhpc/orgs","repos_url":"https://api.github.com/users/markhpc/repos","events_url":"https://api.github.com/users/markhpc/events{/privacy}","received_events_url":"https://api.github.com/users/markhpc/received_events","type":"User","site_admin":false},"body":"This PR depends on either the upstream rocksdb PR here:\r\n\r\nhttps://github.com/facebook/rocksdb/pull/3732\r\n\r\nor a PR against our fork here:\r\n\r\nhttps://github.com/ceph/rocksdb/pull/39\r\n\r\n# Problem:\r\n\r\nThe way that we currently cache data in bluestore is suboptimal.  Bluestore relies heavily on 3 different kinds of cache:\r\n\r\n1) metadata cache\r\n2) data cache\r\n3) rocksdb block cache\r\n\r\nCurrently, we implement a scheme where the user specifies a total amount of cache in bytes that bluestore should allocate and specifies ratios that should be allocated to each cache.  Uunfortunately there are corner cases where this scheme doesn't work well.  One example is during heavy RGW workloads where the total DB size becomes very large, and it's important to give rocksdb enough memory to cache all of the indexes and bloom filters.  Users are required to change the ratios assigned to each cache as the number of objects in RGW grows.  To help alleviate this, we implemented a \"minimum\" KV cache setting so that in most cases the users could just leave the default ratios while still making sure a minimum amount of memory was assigned to the KV store, however this meant that users were confused when different ratios were set than they specified.\r\n\r\nAnother issue is that during RBD workloads, you may need more or less memory than is assigned to the metadata cache to fully cache all bluestore onode metadata, and the amount of cache needed changes based on how much RBD data is present.  In master, bluestore can change the amount of memory allocated to data and metadata depending on how much is used.  This helps, but does not necessarily avoid the scenario where there is not enough cache to avoid caching all onodes and onodes end up double cached in rocksdb's cache and bluestore's cache.  It also further confuses the users, especially since even under memory pressure bluestore's caches tend to over-allocate memory to the metadata cache beyond what the user specified.\r\n\r\n![master_behavior](https://user-images.githubusercontent.com/1286295/40058491-e37ef3f6-5816-11e8-8e24-813edfe97915.png)\r\n\r\nUltimately, for optimal behavior, the ratios of caches must change to match the current workload, and so it's impossible for both us as developers and for users to specify static ratios that work well under all scenarios.  The result of all of this is that users have a hard time understanding how these different options and behaviors interact, how their settings ultimately affect the amount of memory assigned to different caches, and how they should configure these settings on their cluster.\r\n\r\n# What this PR Does\r\n\r\nThis PR introduces the concept of cache autotuning in Ceph, along with laying the foundation for a concept called priority based caching.  \r\n\r\n# Autotuning Disabled\r\n\r\nWhen autotuning is disabled, we should make bluestore's behavior as transparent to the user as possible.  IE given a certain cache size and certain ratios, set the cache sizes accordingly and don't mess with them beyond that.  This may result in less (or more) optimal behavior than current master, but does exactly what the user requests (and leaves it up to them to find the right ratios).\r\n\r\n# Autotuning Enabled\r\n\r\nWhen autotuning is enabled, bluestore will periodically query each cache for how much memory it would like for a given priority (typically how much memory is currently being used plus some extra space for expansion).  It will either allocate this amount or the user-defined fair ratio of remaining memory depending on which is larger.  If any caches don't get as much memory for this priority as they want and there is still memory available, that cache may get additional memory assigned to it to fulfill it's current priority needs before moving on to lower priorities.  Currently the only high priority items are rocksdb's index and filter blocks in rocksdb's high priority cache pool.  This PR lays the ground work for supporting an arbitrary number of generic priorities however.  A future PR for example may implement the concept of binned LRUs where each cache segments their LRU into different priority bins based on age.  If for example, the KV cache generally has older items than bluestore's onode cache, it would get a lower proportion of the available memory.\r\n\r\n# RBD Behavior - Autotuning Disabled\r\n\r\n![wip-bs-cache-autotune-40-40-20-false](https://user-images.githubusercontent.com/1286295/40060902-42ab50b2-581d-11e8-9981-d296e82317f5.png)\r\n\r\nThe behavior in the example above is similar to that of master, except that there are no longer any minimums for the user to set and the ratios assigned to each cache exactly match what the user specified.  In this example, 40% of the memory is devoted to bluestore metadata, 40% to RocksDB's block cache, and 20% to bluestore data.  There is not enough memory devoted to bluestore's metadata cache to cache all onode related metadata.  Once random writes start, rocksdb's block cache begins to fill with onode metadata due to cache misses at both the bluestore meta cache and rocksdb block cache levels.  In fact rocksdb is caching much of the same onode data that bluestore is caching.\r\n\r\n![wip-bs-cache-autotune-10-85-5-false](https://user-images.githubusercontent.com/1286295/40061409-7b49cf92-581e-11e8-90c2-ddf38fd1782f.png)\r\n\r\nSetting a larger onode ratio (85%) that is able to cache all onodes results in no double caching in rocksdb, but now we've assigned significantly more memory to bluestore's onode cache than is necessary to cache all onodes.  A significant portion of the available memory is underutilized when it could have been assigned to data cache.\r\n\r\n# RBD Behavior -Autotuning Enabled\r\n\r\n![wip-bs-cache-autotune-40-40-20-true](https://user-images.githubusercontent.com/1286295/40062311-8d03af30-5820-11e8-85f4-4620907965a9.png)\r\n![wip-bs-cache-autotune-10-85-5-true](https://user-images.githubusercontent.com/1286295/40062508-ff64390a-5820-11e8-9ba9-4f0448335f26.png)\r\n\r\nWhen autotuning is enabled, no matter what default ratios are specified, the autotuner converges on the same ratios.  The bluestore metadata cache is given roughly 51% of the available cache.  Because all onodes are cached in bluestore's onode cache, the KV cache is seldom utilized and remains small (~4%).  Any remaining memory is assigned to bluestore's data cache which receives a significantly larger chunk of the memory than was specified.\r\n\r\n# RBD Performance\r\n\r\n![wip-bs-cache-autotune-performance](https://user-images.githubusercontent.com/1286295/40069518-377b893c-5831-11e8-96e7-a68230c74aef.png)\r\n![wip-bs-cache-autotune-99th_latency](https://user-images.githubusercontent.com/1286295/40070757-96cafb2c-5834-11e8-9bfd-6848a9eea1de.png)\r\n\r\nThe most important performance consideration for the RBD workload is avoiding bluestore meta cache misses.  The autotuner does this implicitly though it can also be accomplished by setting a high enough meta cache ratio to ensure all onodes are cached (but this is difficult for users to do!).  the Autotuner is also able to do this while setting aside a large proportion of the memory for data cache.  While this doesn't appear to dramatically help the random read results here, it may have a more  profound affect in benchmarks that stress hotspot distributions like zipf.  One notable point to make is how much keeping onodes cached improves the 99th percentile latency for reads.  Presumably as latency in the read and write path is reduced via other methods, the importance of avoiding onode cache misses is going to become more pronounced.\r\n\r\n# RGW Behavior - Autotuning Disabled\r\n\r\n![wip-bs-cache-autotune-40-40-20-false-rgw](https://user-images.githubusercontent.com/1286295/40067374-a42d6632-582b-11e8-9369-5726f67a0029.png)\r\n![wip-bs-cache-autotune-10-85-5-false-rgw](https://user-images.githubusercontent.com/1286295/40067383-a888195c-582b-11e8-8419-46cc3e1a986d.png)\r\n\r\nDue to the large amount of onode and omap data during small object RGW workloads, all caches end up filling relatively quickly.  There is still likely a large amount of onode double caching going on, competing with OMAP data in rocksdb's block cache.\r\n\r\n# RGW Behavior - Autotuning Enabled\r\n\r\n![wip-bs-cache-autotune-40-40-20-true-rgw](https://user-images.githubusercontent.com/1286295/40067404-afa26fe4-582b-11e8-9922-8495f3dca6fa.png)\r\n![wip-bs-cache-autotune-10-85-5-true-rgw](https://user-images.githubusercontent.com/1286295/40067416-b32de800-582b-11e8-9edb-4b832799c739.png)\r\n\r\nThe autotuned results look much closer to the non-autotuned results for this RGW workload vs the previous RBD workload.  Because all caches end up competing for resources, the autotuner tends to converge toward the default specified ratios with one exception.  Because the autotuner prioritizes rocksdb index and filter blocks at a higher priority than typical cache items, the amount of memory devoted to rocksdb's block cache tends to increase over time.\r\n\r\n# RGW Performance\r\n\r\n![wip-bs-cache-autotune-performance-rgw](https://user-images.githubusercontent.com/1286295/40072207-a669025a-5838-11e8-9749-3377610f8459.png)\r\n![wip-bs-cache-autotune-avg_latency-rgw](https://user-images.githubusercontent.com/1286295/40072212-a9dc0a04-5838-11e8-8208-a045d4ec0e0f.png)\r\n\r\nNeither the cache ratios nor the autotuner appeared to have significant affect on the performance or latency of the RGW put workload.  A small effect may be present when memory is significantly skewed toward onode cache (to the point where indexes/filters are not cached), though it's also possible that there are other bottlenecks in the RGW path resulting in caching having minimal affect one way or the other.\r\n\r\n\r\n# Caveats\r\n\r\nThe idea behind the cache autotuner is that it adjusts the caches relatively slowly.  By default it waits 5 seconds between rebalances and asks for memory requests in 32MB chunks (with an extra chunk of headroom if a cache is at or above capacity before trim).  This plays relatively well with bluestore's caches but rocksdb's block cache behavior is less straightforward.  Compaction may drive rocksdb to exceed it's cache ratio for a short period of time, and it's unclear how the cache balancer should handle this.  More importantly, when rocksdb exceeds the block cache size (due to compaction), all indexes and filters in the high priority pool appear to be evicted and then rapidly re-added.  It's not yet clear why this only appears to happen only when the block_cache size is exceeded and not in other scenarios.  Both of these behaviors potentially can make the autotuner thrash in undesirable ways if encountered.  To help mitigate this, a high watermark is set and memory is requested as normal at the elevated level.  If the data in the pool (high priority pool or otherwise) rapidly shrinks, the memory request size degrades slowly over time in case the cache is quickly repopulated faster than the autotuner can adapt.\r\n\r\n# Future Work\r\n\r\nImagine a scenario in which a cluster receives heavy RGW writes to a large data set, then a series of small random RBD writes/reads.  Currently the autotuning mostly gives bluestore's metadata cache, data cache, and RocksDB's block cache equal priority at a single level (with the exception of RocksDB's indexes and filters).  After heavy RGW writes, each cache will likely request more memory than is available.  The autotuner will ultimately revert to the default ratios specified by the user as it has no knowledge of the relative importance of data in bluestore's onode cache and rocksdb's block cache.  A future PR may introduce the concept of binned LRU caches.  The idea being that the relative age of items in each cache should determine how much total memory each cache is assigned.  Onodes that are less than 60s old should for example be cached with a higher priority than OMAP data that is older than 60s, even if that might skew the ratios toward bluestore meta cache.  If an OSD is spending most of it's time serving RGW bucket index data, the rocksdb block cache may have significantly more recent data in it than bluestore's onode cache, in which case the ratios may skew the other direction.\r\n\r\nAnother direction for future work might be to move the cache balancer out of bluestore and make it general to the OSD itself.  Ultimately the user doesn't care what bluestore's cache size is.  They care about how much memory the OSD process uses overall.  Currently the amount of memory the OSD uses is extremely difficult to determine and is dependent on many additional factors beyond bluestore's cache size.  Future work may involve incorporating other buffers and caches into the autotuner (Pinned at PRI0) to determine how much memory each cache can be assigned.\r\n\r\nFinally, in the RGW scenario, all 3 caches tend to fill rapidly to the point where all are competing for resources.  This results in double caching of onodes in the bluestore meta cache and rocksdb block cache.  Specifically in the block cache, recently accessed onodes and OMAP data compete for resources despite the fact that those onodes are also likely cached in bluestore's cache.  It's possible that in addition to age-binned LRU caches, we may wish to create multiple LRU caches in rocksdb for different column families of data.  IE recent OMAP data and bluestore metadata may be cached at a higher priority than recent onodes cached in rocksdb's block cache that may also be cached in the bluestore metadata cache.\r\n\r\nSigned-off-by: Mark Nelson <mnelson@redhat.com>","created_at":"2018-05-15T14:15:50Z","updated_at":"2018-06-11T12:27:11Z","closed_at":"2018-06-11T12:27:11Z","merged_at":"2018-06-11T12:27:11Z","merge_commit_sha":"49009fd5ba38d0a6f994a3f952472eb479f592d2","assignee":null,"assignees":[],"requested_reviewers":[{"login":"liewegas","id":433031,"node_id":"MDQ6VXNlcjQzMzAzMQ==","avatar_url":"https://avatars1.githubusercontent.com/u/433031?v=4","gravatar_id":"","url":"https://api.github.com/users/liewegas","html_url":"https://github.com/liewegas","followers_url":"https://api.github.com/users/liewegas/followers","following_url":"https://api.github.com/users/liewegas/following{/other_user}","gists_url":"https://api.github.com/users/liewegas/gists{/gist_id}","starred_url":"https://api.github.com/users/liewegas/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/liewegas/subscriptions","organizations_url":"https://api.github.com/users/liewegas/orgs","repos_url":"https://api.github.com/users/liewegas/repos","events_url":"https://api.github.com/users/liewegas/events{/privacy}","received_events_url":"https://api.github.com/users/liewegas/received_events","type":"User","site_admin":false}],"requested_teams":[],"labels":[{"id":385981601,"node_id":"MDU6TGFiZWwzODU5ODE2MDE=","url":"https://api.github.com/repos/ceph/ceph/labels/bluestore","name":"bluestore","color":"0066ff","default":false},{"id":134774006,"node_id":"MDU6TGFiZWwxMzQ3NzQwMDY=","url":"https://api.github.com/repos/ceph/ceph/labels/feature","name":"feature","color":"00ff24","default":false},{"id":148431321,"node_id":"MDU6TGFiZWwxNDg0MzEzMjE=","url":"https://api.github.com/repos/ceph/ceph/labels/needs-qa","name":"needs-qa","color":"fad8c7","default":false},{"id":204137300,"node_id":"MDU6TGFiZWwyMDQxMzczMDA=","url":"https://api.github.com/repos/ceph/ceph/labels/performance","name":"performance","color":"bfdadc","default":false},{"id":731274994,"node_id":"MDU6TGFiZWw3MzEyNzQ5OTQ=","url":"https://api.github.com/repos/ceph/ceph/labels/wip-sage3-testing","name":"wip-sage3-testing","color":"5319e7","default":false}],"milestone":null,"commits_url":"https://api.github.com/repos/ceph/ceph/pulls/22009/commits","review_comments_url":"https://api.github.com/repos/ceph/ceph/pulls/22009/comments","review_comment_url":"https://api.github.com/repos/ceph/ceph/pulls/comments{/number}","comments_url":"https://api.github.com/repos/ceph/ceph/issues/22009/comments","statuses_url":"https://api.github.com/repos/ceph/ceph/statuses/1d51e764500f519018f572282ef0a269914edfd4","head":{"label":"markhpc:wip-bs-cache-autotune","ref":"wip-bs-cache-autotune","sha":"1d51e764500f519018f572282ef0a269914edfd4","user":{"login":"markhpc","id":1286295,"node_id":"MDQ6VXNlcjEyODYyOTU=","avatar_url":"https://avatars0.githubusercontent.com/u/1286295?v=4","gravatar_id":"","url":"https://api.github.com/users/markhpc","html_url":"https://github.com/markhpc","followers_url":"https://api.github.com/users/markhpc/followers","following_url":"https://api.github.com/users/markhpc/following{/other_user}","gists_url":"https://api.github.com/users/markhpc/gists{/gist_id}","starred_url":"https://api.github.com/users/markhpc/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/markhpc/subscriptions","organizations_url":"https://api.github.com/users/markhpc/orgs","repos_url":"https://api.github.com/users/markhpc/repos","events_url":"https://api.github.com/users/markhpc/events{/privacy}","received_events_url":"https://api.github.com/users/markhpc/received_events","type":"User","site_admin":false},"repo":{"id":53530961,"node_id":"MDEwOlJlcG9zaXRvcnk1MzUzMDk2MQ==","name":"ceph","full_name":"markhpc/ceph","owner":{"login":"markhpc","id":1286295,"node_id":"MDQ6VXNlcjEyODYyOTU=","avatar_url":"https://avatars0.githubusercontent.com/u/1286295?v=4","gravatar_id":"","url":"https://api.github.com/users/markhpc","html_url":"https://github.com/markhpc","followers_url":"https://api.github.com/users/markhpc/followers","following_url":"https://api.github.com/users/markhpc/following{/other_user}","gists_url":"https://api.github.com/users/markhpc/gists{/gist_id}","starred_url":"https://api.github.com/users/markhpc/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/markhpc/subscriptions","organizations_url":"https://api.github.com/users/markhpc/orgs","repos_url":"https://api.github.com/users/markhpc/repos","events_url":"https://api.github.com/users/markhpc/events{/privacy}","received_events_url":"https://api.github.com/users/markhpc/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/markhpc/ceph","description":"Ceph is a distributed object, block, and file storage platform ","fork":true,"url":"https://api.github.com/repos/markhpc/ceph","forks_url":"https://api.github.com/repos/markhpc/ceph/forks","keys_url":"https://api.github.com/repos/markhpc/ceph/keys{/key_id}","collaborators_url":"https://api.github.com/repos/markhpc/ceph/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/markhpc/ceph/teams","hooks_url":"https://api.github.com/repos/markhpc/ceph/hooks","issue_events_url":"https://api.github.com/repos/markhpc/ceph/issues/events{/number}","events_url":"https://api.github.com/repos/markhpc/ceph/events","assignees_url":"https://api.github.com/repos/markhpc/ceph/assignees{/user}","branches_url":"https://api.github.com/repos/markhpc/ceph/branches{/branch}","tags_url":"https://api.github.com/repos/markhpc/ceph/tags","blobs_url":"https://api.github.com/repos/markhpc/ceph/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/markhpc/ceph/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/markhpc/ceph/git/refs{/sha}","trees_url":"https://api.github.com/repos/markhpc/ceph/git/trees{/sha}","statuses_url":"https://api.github.com/repos/markhpc/ceph/statuses/{sha}","languages_url":"https://api.github.com/repos/markhpc/ceph/languages","stargazers_url":"https://api.github.com/repos/markhpc/ceph/stargazers","contributors_url":"https://api.github.com/repos/markhpc/ceph/contributors","subscribers_url":"https://api.github.com/repos/markhpc/ceph/subscribers","subscription_url":"https://api.github.com/repos/markhpc/ceph/subscription","commits_url":"https://api.github.com/repos/markhpc/ceph/commits{/sha}","git_commits_url":"https://api.github.com/repos/markhpc/ceph/git/commits{/sha}","comments_url":"https://api.github.com/repos/markhpc/ceph/comments{/number}","issue_comment_url":"https://api.github.com/repos/markhpc/ceph/issues/comments{/number}","contents_url":"https://api.github.com/repos/markhpc/ceph/contents/{+path}","compare_url":"https://api.github.com/repos/markhpc/ceph/compare/{base}...{head}","merges_url":"https://api.github.com/repos/markhpc/ceph/merges","archive_url":"https://api.github.com/repos/markhpc/ceph/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/markhpc/ceph/downloads","issues_url":"https://api.github.com/repos/markhpc/ceph/issues{/number}","pulls_url":"https://api.github.com/repos/markhpc/ceph/pulls{/number}","milestones_url":"https://api.github.com/repos/markhpc/ceph/milestones{/number}","notifications_url":"https://api.github.com/repos/markhpc/ceph/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/markhpc/ceph/labels{/name}","releases_url":"https://api.github.com/repos/markhpc/ceph/releases{/id}","deployments_url":"https://api.github.com/repos/markhpc/ceph/deployments","created_at":"2016-03-09T20:53:20Z","updated_at":"2016-03-09T20:53:39Z","pushed_at":"2018-06-05T18:01:16Z","git_url":"git://github.com/markhpc/ceph.git","ssh_url":"git@github.com:markhpc/ceph.git","clone_url":"https://github.com/markhpc/ceph.git","svn_url":"https://github.com/markhpc/ceph","homepage":"http://ceph.com","size":284912,"stargazers_count":0,"watchers_count":0,"language":"C++","has_issues":false,"has_projects":true,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":1,"mirror_url":null,"archived":false,"open_issues_count":0,"license":{"key":"other","name":"Other","spdx_id":null,"url":null,"node_id":"MDc6TGljZW5zZTA="},"forks":1,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"ceph:master","ref":"master","sha":"cbd53d1674ba5cec1f2ea7ffcebc555a99b37289","user":{"login":"ceph","id":1015767,"node_id":"MDEyOk9yZ2FuaXphdGlvbjEwMTU3Njc=","avatar_url":"https://avatars3.githubusercontent.com/u/1015767?v=4","gravatar_id":"","url":"https://api.github.com/users/ceph","html_url":"https://github.com/ceph","followers_url":"https://api.github.com/users/ceph/followers","following_url":"https://api.github.com/users/ceph/following{/other_user}","gists_url":"https://api.github.com/users/ceph/gists{/gist_id}","starred_url":"https://api.github.com/users/ceph/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ceph/subscriptions","organizations_url":"https://api.github.com/users/ceph/orgs","repos_url":"https://api.github.com/users/ceph/repos","events_url":"https://api.github.com/users/ceph/events{/privacy}","received_events_url":"https://api.github.com/users/ceph/received_events","type":"Organization","site_admin":false},"repo":{"id":2310495,"node_id":"MDEwOlJlcG9zaXRvcnkyMzEwNDk1","name":"ceph","full_name":"ceph/ceph","owner":{"login":"ceph","id":1015767,"node_id":"MDEyOk9yZ2FuaXphdGlvbjEwMTU3Njc=","avatar_url":"https://avatars3.githubusercontent.com/u/1015767?v=4","gravatar_id":"","url":"https://api.github.com/users/ceph","html_url":"https://github.com/ceph","followers_url":"https://api.github.com/users/ceph/followers","following_url":"https://api.github.com/users/ceph/following{/other_user}","gists_url":"https://api.github.com/users/ceph/gists{/gist_id}","starred_url":"https://api.github.com/users/ceph/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ceph/subscriptions","organizations_url":"https://api.github.com/users/ceph/orgs","repos_url":"https://api.github.com/users/ceph/repos","events_url":"https://api.github.com/users/ceph/events{/privacy}","received_events_url":"https://api.github.com/users/ceph/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/ceph/ceph","description":"Ceph is a distributed object, block, and file storage platform ","fork":false,"url":"https://api.github.com/repos/ceph/ceph","forks_url":"https://api.github.com/repos/ceph/ceph/forks","keys_url":"https://api.github.com/repos/ceph/ceph/keys{/key_id}","collaborators_url":"https://api.github.com/repos/ceph/ceph/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/ceph/ceph/teams","hooks_url":"https://api.github.com/repos/ceph/ceph/hooks","issue_events_url":"https://api.github.com/repos/ceph/ceph/issues/events{/number}","events_url":"https://api.github.com/repos/ceph/ceph/events","assignees_url":"https://api.github.com/repos/ceph/ceph/assignees{/user}","branches_url":"https://api.github.com/repos/ceph/ceph/branches{/branch}","tags_url":"https://api.github.com/repos/ceph/ceph/tags","blobs_url":"https://api.github.com/repos/ceph/ceph/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/ceph/ceph/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/ceph/ceph/git/refs{/sha}","trees_url":"https://api.github.com/repos/ceph/ceph/git/trees{/sha}","statuses_url":"https://api.github.com/repos/ceph/ceph/statuses/{sha}","languages_url":"https://api.github.com/repos/ceph/ceph/languages","stargazers_url":"https://api.github.com/repos/ceph/ceph/stargazers","contributors_url":"https://api.github.com/repos/ceph/ceph/contributors","subscribers_url":"https://api.github.com/repos/ceph/ceph/subscribers","subscription_url":"https://api.github.com/repos/ceph/ceph/subscription","commits_url":"https://api.github.com/repos/ceph/ceph/commits{/sha}","git_commits_url":"https://api.github.com/repos/ceph/ceph/git/commits{/sha}","comments_url":"https://api.github.com/repos/ceph/ceph/comments{/number}","issue_comment_url":"https://api.github.com/repos/ceph/ceph/issues/comments{/number}","contents_url":"https://api.github.com/repos/ceph/ceph/contents/{+path}","compare_url":"https://api.github.com/repos/ceph/ceph/compare/{base}...{head}","merges_url":"https://api.github.com/repos/ceph/ceph/merges","archive_url":"https://api.github.com/repos/ceph/ceph/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/ceph/ceph/downloads","issues_url":"https://api.github.com/repos/ceph/ceph/issues{/number}","pulls_url":"https://api.github.com/repos/ceph/ceph/pulls{/number}","milestones_url":"https://api.github.com/repos/ceph/ceph/milestones{/number}","notifications_url":"https://api.github.com/repos/ceph/ceph/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/ceph/ceph/labels{/name}","releases_url":"https://api.github.com/repos/ceph/ceph/releases{/id}","deployments_url":"https://api.github.com/repos/ceph/ceph/deployments","created_at":"2011-09-01T21:41:26Z","updated_at":"2018-06-11T10:42:27Z","pushed_at":"2018-06-11T12:27:11Z","git_url":"git://github.com/ceph/ceph.git","ssh_url":"git@github.com:ceph/ceph.git","clone_url":"https://github.com/ceph/ceph.git","svn_url":"https://github.com/ceph/ceph","homepage":"http://ceph.com","size":287471,"stargazers_count":4594,"watchers_count":4594,"language":"C++","has_issues":false,"has_projects":true,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":2477,"mirror_url":null,"archived":false,"open_issues_count":660,"license":{"key":"other","name":"Other","spdx_id":null,"url":null,"node_id":"MDc6TGljZW5zZTA="},"forks":2477,"open_issues":660,"watchers":4594,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/ceph/ceph/pulls/22009"},"html":{"href":"https://github.com/ceph/ceph/pull/22009"},"issue":{"href":"https://api.github.com/repos/ceph/ceph/issues/22009"},"comments":{"href":"https://api.github.com/repos/ceph/ceph/issues/22009/comments"},"review_comments":{"href":"https://api.github.com/repos/ceph/ceph/pulls/22009/comments"},"review_comment":{"href":"https://api.github.com/repos/ceph/ceph/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/ceph/ceph/pulls/22009/commits"},"statuses":{"href":"https://api.github.com/repos/ceph/ceph/statuses/1d51e764500f519018f572282ef0a269914edfd4"}},"author_association":"MEMBER","merged":true,"mergeable":null,"rebaseable":null,"mergeable_state":"unknown","merged_by":{"login":"liewegas","id":433031,"node_id":"MDQ6VXNlcjQzMzAzMQ==","avatar_url":"https://avatars1.githubusercontent.com/u/433031?v=4","gravatar_id":"","url":"https://api.github.com/users/liewegas","html_url":"https://github.com/liewegas","followers_url":"https://api.github.com/users/liewegas/followers","following_url":"https://api.github.com/users/liewegas/following{/other_user}","gists_url":"https://api.github.com/users/liewegas/gists{/gist_id}","starred_url":"https://api.github.com/users/liewegas/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/liewegas/subscriptions","organizations_url":"https://api.github.com/users/liewegas/orgs","repos_url":"https://api.github.com/users/liewegas/repos","events_url":"https://api.github.com/users/liewegas/events{/privacy}","received_events_url":"https://api.github.com/users/liewegas/received_events","type":"User","site_admin":false},"comments":23,"review_comments":0,"maintainer_can_modify":false,"commits":3,"additions":610,"deletions":152,"changed_files":11}}
{"actor":{"display_login":"liewegas"}}