{"action":"opened","issue":{"url":"https://api.github.com/repos/concourse/concourse/issues/2151","repository_url":"https://api.github.com/repos/concourse/concourse","labels_url":"https://api.github.com/repos/concourse/concourse/issues/2151/labels{/name}","comments_url":"https://api.github.com/repos/concourse/concourse/issues/2151/comments","events_url":"https://api.github.com/repos/concourse/concourse/issues/2151/events","html_url":"https://github.com/concourse/concourse/issues/2151","id":313990292,"number":2151,"title":"Timer Resources continue to be processed after the pipeline is deleted","user":{"login":"keithkroeger","id":10414523,"avatar_url":"https://avatars0.githubusercontent.com/u/10414523?v=4","gravatar_id":"","url":"https://api.github.com/users/keithkroeger","html_url":"https://github.com/keithkroeger","followers_url":"https://api.github.com/users/keithkroeger/followers","following_url":"https://api.github.com/users/keithkroeger/following{/other_user}","gists_url":"https://api.github.com/users/keithkroeger/gists{/gist_id}","starred_url":"https://api.github.com/users/keithkroeger/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/keithkroeger/subscriptions","organizations_url":"https://api.github.com/users/keithkroeger/orgs","repos_url":"https://api.github.com/users/keithkroeger/repos","events_url":"https://api.github.com/users/keithkroeger/events{/privacy}","received_events_url":"https://api.github.com/users/keithkroeger/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-04-13T07:01:06Z","updated_at":"2018-04-13T07:01:06Z","closed_at":null,"author_association":"NONE","body":"# Bug Report\r\n\r\n* Concourse version: 3.8.0\r\n* Deployment type (BOSH/Docker/binary): BOSH\r\n* Infrastructure/IaaS: Openstack\r\n\r\nWe have seen that it appears that resources on ATCs continue to be used/allocated once they've been used for a pipeline job even after the pipeline has been destroyed.\r\n\r\nCase:\r\n- Create several (we create 600 for exaggeration) pipelines.\r\n```yaml\r\nresources:\r\n- name: 5m\r\n  type: time\r\n  source: {interval: 5m}\r\n\r\n- name: 10m\r\n  type: time\r\n  source: {interval: 5m}\r\n\r\n- name: 15m\r\n  type: time\r\n  source: {interval: 5m}\r\n\r\n- name: repo\r\n  type: git\r\n  source:\r\n    branch: feature/ref-app-test\r\n    uri: https://***\r\n  check_every: 60m\r\n\r\n- name: build\r\n  type: git\r\n  source:\r\n    branch: develop\r\n    uri: https://***\r\n    private_key: ((vault-key-repo-private-key))\r\n  check_every: 60m\r\n\r\njobs:\r\n- name: mvn-build\r\n  interruptible: true\r\n  plan:\r\n  - aggregate:\r\n    - get: build\r\n    - get: repo \r\n    - get: 5m\r\n    - get: 10m\r\n    - get: 15m   \r\n  - task: build\r\n    file: build/tasks/maven/task.yml\r\n    params:\r\n      <<: *maven-params\r\n  - task: say-hello\r\n    config:\r\n      platform: linux\r\n      image_resource:\r\n        type: docker-image\r\n        source: {repository: ubuntu}\r\n      run:\r\n        path: /bin/bash\r\n        args:\r\n          - -exc\r\n          - |\r\n            rand=$((20 + RANDOM % 90))\r\n            sleep $rand\r\n            echo \"Hello world $rand\"\r\n```\r\n- allow these pipelines to run for a period of time\r\n- Destroy the pipelines\r\n\r\nOnce all pipelines are destroyed, we noticed a few things:\r\n1)  We continue to see log entries for builds that no longer exist (about 250/s/build)\r\n```\r\n14>1 2018-04-13T06:27:00.944373Z 2b3b1f06-86fb-4515-bf14-ec6a5f5a2a2e.web.default.sandbox.bosh atc - - [instance@47450 director=\"\" deployment=\"sandbox\" group=\"web\" az=\"z1\" id=\"2b3b1f06-86fb-4515-bf14-ec6a5f5a2a2e\"] {\"timestamp\":\"1523600820.146601915\",\"source\":\"atc\",\"message\":\"atc.syncer.test-4:scheduler.tick.scheduling.try-start-next-pending-build.scan.acquire.acquired\",\"log_level\":0,\"data\":{\"build-id\":17366,\"build-name\":\"28\",\"id\":[0,135],\"input\":\"10m\",\"resource\":\"10m\",\"session\":\"54.2034.42.3.2.5.44923\"}}\r\n<14>1 2018-04-13T06:27:00.944379Z 2b3b1f06-86fb-4515-bf14-ec6a5f5a2a2e.web.default.sandbox.bosh atc - - [instance@47450 director=\"\" deployment=\"sandbox\" group=\"web\" az=\"z1\" id=\"2b3b1f06-86fb-4515-bf14-ec6a5f5a2a2e\"] {\"timestamp\":\"1523600820.156314850\",\"source\":\"atc\",\"message\":\"atc.syncer.test-4:scheduler.tick.scheduling.try-start-next-pending-build.scan.release.released\",\"log_level\":0,\"data\":{\"build-id\":17366,\"build-name\":\"28\",\"id\":[0,135],\"input\":\"10m\",\"resource\":\"10m\",\"session\":\"54.2034.42.3.2.5.44924\"}}\r\n<14>1 2018-04-13T06:27:00.94438Z 2b3b1f06-86fb-4515-bf14-ec6a5f5a2a2e.web.default.sandbox.bosh atc - - [instance@47450 director=\"\" deployment=\"sandbox\" group=\"web\" az=\"z1\" id=\"2b3b1f06-86fb-4515-bf14-ec6a5f5a2a2e\"] {\"timestamp\":\"1523600820.156357050\",\"source\":\"atc\",\"message\":\"atc.syncer.test-4:scheduler.tick.scheduling.try-start-next-pending-build.scan.lock.did-not-get-lock\",\"log_level\":0,\"data\":{\"build-id\":17366,\"build-name\":\"28\",\"input\":\"10m\",\"resource\":\"10m\",\"session\":\"54.2034.42.3.2.5.1\"}}\r\n<14>1 2018-04-13T06:27:00.730347Z 825ff110-f889-403e-8482-6dfd1b2b130b.web.default.sandbox.bosh atc - - [instance@47450 director=\"\" deployment=\"sandbox\" group=\"web\" az=\"z1\" id=\"825ff110-f889-403e-8482-6dfd1b2b130b\"] {\"timestamp\":\"1523600819.934732914\",\"source\":\"atc\",\"message\":\"atc.create-job-build.trigger-immediately.try-start-next-pending-build.scan.acquire.not-acquired-already-held-locally\",\"log_level\":0,\"data\":{\"build-id\":17366,\"build-name\":\"28\",\"id\":[0,135],\"input\":\"15m\",\"job_name\":\"mvn-build\",\"resource\":\"15m\",\"session\":\"42259.1.1.6.40585\"}}\r\n```\r\n2) The number of web go routines does not recover and, in fact, never goes down (even 12h later).  There are over 2000 routines on each of our three ATCs.  These continue to remain until we recreate (bosh recreate -d sandbox web) all of the ATCs in the environment.\r\n\r\nWithout recreating our ATCs, the environment eventually will get to a state (if enough of these apparently lingering processes remain) where no jobs, or at least very few, will start and timers, rather than running every 5 minutes, run every 20-30 minutes."}}
{"actor":{"display_login":"keithkroeger"}}