{"action":"created","issue":{"url":"https://api.github.com/repos/cloudfoundry/bosh-vsphere-cpi-release/issues/174","repository_url":"https://api.github.com/repos/cloudfoundry/bosh-vsphere-cpi-release","labels_url":"https://api.github.com/repos/cloudfoundry/bosh-vsphere-cpi-release/issues/174/labels{/name}","comments_url":"https://api.github.com/repos/cloudfoundry/bosh-vsphere-cpi-release/issues/174/comments","events_url":"https://api.github.com/repos/cloudfoundry/bosh-vsphere-cpi-release/issues/174/events","html_url":"https://github.com/cloudfoundry/bosh-vsphere-cpi-release/issues/174","id":383881820,"node_id":"MDU6SXNzdWUzODM4ODE4MjA=","number":174,"title":"Datastore usage appears unbalanced after recent upgrades","user":{"login":"thelangley","id":11210817,"node_id":"MDQ6VXNlcjExMjEwODE3","avatar_url":"https://avatars3.githubusercontent.com/u/11210817?v=4","gravatar_id":"","url":"https://api.github.com/users/thelangley","html_url":"https://github.com/thelangley","followers_url":"https://api.github.com/users/thelangley/followers","following_url":"https://api.github.com/users/thelangley/following{/other_user}","gists_url":"https://api.github.com/users/thelangley/gists{/gist_id}","starred_url":"https://api.github.com/users/thelangley/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/thelangley/subscriptions","organizations_url":"https://api.github.com/users/thelangley/orgs","repos_url":"https://api.github.com/users/thelangley/repos","events_url":"https://api.github.com/users/thelangley/events{/privacy}","received_events_url":"https://api.github.com/users/thelangley/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":4,"created_at":"2018-11-23T16:35:27Z","updated_at":"2018-11-27T15:50:15Z","closed_at":"2018-11-27T13:32:37Z","author_association":"NONE","body":"**The (potential) Bug**\r\nWe've recently moved to vsphere-cpi v51.0.2 from v50 and upgraded our vCenters to 6.5 u2c\r\n\r\nWe're finding that when a new stemcell is being deployed, it has a tendency to have a \"favourite\" Datastore so much so that it tries to almost fill it to 100%, even if loads of the others have a fair amount free.\r\n\r\n```\r\n- type: replace\r\n  path: /instance_groups/name=bosh/properties/vcenter?\r\n  value: &vcenter\r\n    address: ((fqdn))\r\n    user: ((user))\r\n    password: ((theword))\r\n    datacenters:\r\n    - name: ((my_dc_name))\r\n      vm_folder: ((my_folder))\r\n      template_folder: ((my_template_folder))\r\n      datastore_pattern: ((my_datastore_pattern))\r\n      persistent_datastore_pattern: ((my_persistent_disk_datastore_pattern))\r\n      disk_path: ((my_amazing_path))\r\n      clusters:\r\n      - ((vsphere_cluster)): { ((poolies_info)) }\r\n```\r\n\r\n**To Reproduce**\r\nSteps to reproduce the behavior:\r\n1. Ensure you have many datastores in vCenter available for action\r\n2. Define them as a regex and use the pattern in your ops file\r\n3. Deploy\r\n4. Recreate the things\r\n5. Recreate the things\r\n\r\nOver time you will see one datastore gets more and more full\r\n\r\n**CPI Error Log**\r\nThis is no error in the logs but I can see the decisions being made.\r\n\r\nIt's using the regex nicely to find all the Datastores which match but then the roll of the dice usually selects a particular one.\r\n\r\n```\r\nD, [2018-11-23T12:29:16.747104 #31769] DEBUG -- [req_id cpi-212057]: All clusters provided: {\"my_amazing_cluster\"=>#<VSphereCloud::ClusterConfig:xxxxxx @name=\"my_amazing_cluster\", @config={\"resource_pool\"=>\"my_amazing_resource_pool\"}>}\r\nD, [2018-11-23T12:29:16.747207 #31769] DEBUG -- [req_id cpi-212057]: Running method 'FindByInventoryPath'...\r\nI, [2018-11-23T12:29:16.783249 #31769]  INFO -- [req_id cpi-212057]: Using global ephemeral disk datastore pattern: my_amazing_datastore_[0-9]{2}\r\nI, [2018-11-23T12:29:16.790485 #31769]  INFO -- [req_id cpi-212057]: Initiating VM Allocator with VM Placement Criteria: Disk Config: [Disk ==> Disk CID: Size:10240 Ephemeral:true Target-DS:my_amazing_datastore_[0-9]{2} Existing-DS:]  Req Memory: 4096  Mem Headroom: 128\r\nI, [2018-11-23T12:29:16.790552 #31769]  INFO -- [req_id cpi-212057]: Gathering vm placement resources for vm placement allocator pipeline\r\nD, [2018-11-23T12:29:16.823403 #31769] DEBUG -- [req_id cpi-212057]: Found requested resource pool: <[Vim.ResourcePool] resgroup-xxxx>\r\nD, [2018-11-23T12:29:16.833596 #31769] DEBUG -- [req_id cpi-212057]: Filter VM Placement Cluster: my_amazing_cluster Memory: 1907108 for free memory required: 4224\r\nD, [2018-11-23T12:29:16.833696 #31769] DEBUG -- [req_id cpi-212057]: Filter VM Placement Cluster : my_amazing_cluster hosts:  datastores: [<Datastore: / my_amazing_datastore_00 >, <Datastore: / my_amazing_datastore_01 >, <Datastore: / my_amazing_datastore_02>] for combination of DS satisfying disk configurations\r\nD, [2018-11-23T12:29:16.833755 #31769] DEBUG -- [req_id cpi-212057]: Trying to find placement for Disk ==> Disk CID: Size:10240 Ephemeral:true Target-DS: my_amazing_datastore_[0-9]{2} Existing-DS:\r\nI, [2018-11-23T12:29:16.833918 #31769]  INFO -- [req_id cpi-212057]: Initiating Disk Allocator with Storage Criteria: Space Req: 10240 Target Pattern: my_amazing_datastore_[0-9]{2}\r\n        Existing-DS Pattern:\r\nD, [2018-11-23T12:29:18.103354 #31769] DEBUG -- [req_id cpi-212057]: Found <Datastore: / my_amazing_datastore_01> for Disk ==> Disk CID: Size:10240 Ephemeral:true Target-DS:my_amazing_datastore_[0-9]{2} Existing-DS:\r\nD, [2018-11-23T12:29:18.103540 #31769] DEBUG -- [req_id cpi-212057]: Initial Storage Options for creating ephemeral disk from pattern: [\"my_amazing_datastore_01\"]\r\nD, [2018-11-23T12:29:18.103600 #31769] DEBUG -- [req_id cpi-212057]: Storage Options for creating ephemeral disk are: [\"my_amazing_datastore_01\"]\r\n```\r\n\r\nWe got down to less than 90Gb free on one datastore today and there's 7.9Tb free on another one?!?!\r\n\r\nWe have multiple boshes which use the same vCenter IaaS/same Datastores and all the \"upgraded\" vsphere-cpis appear to be making the same placement errors.\r\n\r\n**Expected behavior**\r\nIf one datastore is more empty than the others, the vsphere-cpi should make the decision to use it.\r\n\r\n**Release Version & Related Info:**\r\n - CPI Version: v51.0.2\r\n - BOSH Director Version:  265.11.0\r\n - Stemcell Name & Version: a range of Trusty and Xenial\r\n - vCenter Version: 6.5u2c\r\n - NSX(V) 6.3.6\r\n\r\nMany thanks"},"comment":{"url":"https://api.github.com/repos/cloudfoundry/bosh-vsphere-cpi-release/issues/comments/442108583","html_url":"https://github.com/cloudfoundry/bosh-vsphere-cpi-release/issues/174#issuecomment-442108583","issue_url":"https://api.github.com/repos/cloudfoundry/bosh-vsphere-cpi-release/issues/174","id":442108583,"node_id":"MDEyOklzc3VlQ29tbWVudDQ0MjEwODU4Mw==","user":{"login":"EleanorRigby","id":6512426,"node_id":"MDQ6VXNlcjY1MTI0MjY=","avatar_url":"https://avatars2.githubusercontent.com/u/6512426?v=4","gravatar_id":"","url":"https://api.github.com/users/EleanorRigby","html_url":"https://github.com/EleanorRigby","followers_url":"https://api.github.com/users/EleanorRigby/followers","following_url":"https://api.github.com/users/EleanorRigby/following{/other_user}","gists_url":"https://api.github.com/users/EleanorRigby/gists{/gist_id}","starred_url":"https://api.github.com/users/EleanorRigby/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/EleanorRigby/subscriptions","organizations_url":"https://api.github.com/users/EleanorRigby/orgs","repos_url":"https://api.github.com/users/EleanorRigby/repos","events_url":"https://api.github.com/users/EleanorRigby/events{/privacy}","received_events_url":"https://api.github.com/users/EleanorRigby/received_events","type":"User","site_admin":false},"created_at":"2018-11-27T15:50:15Z","updated_at":"2018-11-27T15:50:15Z","author_association":"MEMBER","body":"@thelangley : You're welcome. The new release is 51.0.4\r\n\r\n51.0.3 still has that skew. It might have re balanced because enough space might not have been there on filled datastore. I would encourage you to use [`51.0.4`](https://github.com/cloudfoundry/bosh-vsphere-cpi-release/releases/tag/v51.0.4)"}}
{"actor":{"display_login":"EleanorRigby"}}