{"action":"created","issue":{"url":"https://api.github.com/repos/heptio/ark/issues/836","repository_url":"https://api.github.com/repos/heptio/ark","labels_url":"https://api.github.com/repos/heptio/ark/issues/836/labels{/name}","comments_url":"https://api.github.com/repos/heptio/ark/issues/836/comments","events_url":"https://api.github.com/repos/heptio/ark/issues/836/events","html_url":"https://github.com/heptio/ark/issues/836","id":358835832,"node_id":"MDU6SXNzdWUzNTg4MzU4MzI=","number":836,"title":"Ark pod in STATUS: CrashLoopBackOff","user":{"login":"mehemken","id":16868431,"node_id":"MDQ6VXNlcjE2ODY4NDMx","avatar_url":"https://avatars2.githubusercontent.com/u/16868431?v=4","gravatar_id":"","url":"https://api.github.com/users/mehemken","html_url":"https://github.com/mehemken","followers_url":"https://api.github.com/users/mehemken/followers","following_url":"https://api.github.com/users/mehemken/following{/other_user}","gists_url":"https://api.github.com/users/mehemken/gists{/gist_id}","starred_url":"https://api.github.com/users/mehemken/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mehemken/subscriptions","organizations_url":"https://api.github.com/users/mehemken/orgs","repos_url":"https://api.github.com/users/mehemken/repos","events_url":"https://api.github.com/users/mehemken/events{/privacy}","received_events_url":"https://api.github.com/users/mehemken/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2018-09-10T23:53:24Z","updated_at":"2018-09-11T01:02:49Z","closed_at":null,"author_association":"NONE","body":"**What steps did you take and what happened:**\r\nI've followed the instructions in the documentation:\r\n- [Docs for AWS](https://heptio.github.io/ark/v0.9.0/aws-config)\r\n- [Getting Started](https://heptio.github.io/ark/v0.9.0/quickstart)\r\n\r\nHowever the ark pod goes into `CrashLoopBackOff`. I then run the nginx-example. When I take a backup it succeeds, however most of the fields in `ark backup describe nginx-backup` show up as `<none>`, `<n/a>`. Also, `ark` recognizes zero warnings and zero errors:\r\n\r\n    $ ark get backups\r\n    NAME           STATUS    CREATED                         EXPIRES   SELECTOR\r\n    nginx-backup   New       2018-09-10 15:42:09 -0700 PDT   29d       app=nginx\r\n    $ ark get restores\r\n    NAME                          BACKUP         STATUS    WARNINGS   ERRORS    CREATED                         SELECTOR\r\n    nginx-backup-20180910154449   nginx-backup   New       0          0         2018-09-10 15:44:50 -0700 PDT   <none>\r\n\r\nAfter an `ark restore`: Even though the restore succeeds, the namespace `nginx-test` does not exist.\r\n\r\nSo I check if the pod is running:\r\n\r\n    NAME                   READY     STATUS             RESTARTS   AGE\r\n    ark-4179056932-dgc9r   0/1       CrashLoopBackOff   9          26m\r\n\r\nOk so I check the logs:\r\n\r\n```\r\n...\r\nAn error occurred: object storage provider name must not be empty\r\n...\r\n```\r\n\r\nOk so I check the Google. The Google says \"**object storage provider name must not be empty**\" shows up in the GitHub repo, but a Ctrl-F search on the page [Google points to](https://github.com/heptio/ark/blob/master/pkg/cmd/server/server.go) says the phrase does not exist. 😞 No other results.\r\n\r\nI've checked that all the name fields in my configuration are set. Especially in the `Config` and `BackupStorageLocation` resources. I've set all the provider fields to aws.\r\n\r\n```\r\n---\r\napiVersion: ark.heptio.com/v1\r\nkind: Config\r\nmetadata:\r\n  namespace: heptio-ark-test\r\n  name: default\r\npersistentVolumeProvider:\r\n  name: aws\r\n  config:\r\n    region: us-west-2\r\n\r\n---\r\napiVersion: ark.heptio.com/v1\r\nkind: BackupStorageLocation\r\nmetadata:\r\n  name: default\r\n  namespace: heptio-ark-test\r\nspec:\r\n  provider: aws\r\n  objectStorage:\r\n    bucket: real-bucket-name\r\n    config:\r\n      region: us-west-2\r\n```\r\n\r\nThe AWS user I'm using for this has all the permissions specified in the documentation. The bucket name in the IAM JSON is the correct bucket name. The bucket name in the YAML is the correct bucket name.\r\n\r\nI've double checked that the namespace `heptio-ark-test` has been changed in all places. Including `ark client set namespace=heptio-ark-test`\r\n\r\n\r\n# How to proceed?\r\n\r\nFor the life of me I can't think of how else to troubleshoot this. The only error I get results in zero Google results. The documentation says nothing about this issue. Nobody else seems to have seen this error.\r\n\r\nHow do I proceed in troubleshooting this?\r\n\r\n# 🤦‍♂️\r\n\r\n**What did you expect to happen:**\r\n1. I should see something in the AWS Bucket\r\n1. The `ark restore` should restore the nginx-example\r\n\r\n**The output of the following commands will help us better understand what's going on**:\r\n(Pasting long output into a [GitHub gist](https://gist.github.com) or other pastebin is fine.)\r\n\r\n* `kubectl logs deployment/ark -n heptio-ark`\r\n```\r\ntime=\"2018-09-10T23:04:15Z\" level=info msg=\"setting log-level to INFO\"\r\ntime=\"2018-09-10T23:04:15Z\" level=info msg=\"Starting Ark server v0.9.4\" logSource=\"pkg/cmd/server/server.go:101\"\r\ntime=\"2018-09-10T23:04:15Z\" level=info msg=\"Checking existence of namespace\" logSource=\"pkg/cmd/server/server.go:264\" namespace=heptio-ark-test\r\ntime=\"2018-09-10T23:04:15Z\" level=info msg=\"Namespace exists\" logSource=\"pkg/cmd/server/server.go:270\" namespace=heptio-ark-test\r\ntime=\"2018-09-10T23:04:15Z\" level=info msg=\"Checking existence of Ark custom resource definitions\" logSource=\"pkg/cmd/server/server.go:299\"\r\ntime=\"2018-09-10T23:04:15Z\" level=info msg=\"All Ark custom resource definitions exist\" logSource=\"pkg/cmd/server/server.go:332\"\r\ntime=\"2018-09-10T23:04:15Z\" level=info msg=\"Retrieving Ark configuration\" logSource=\"pkg/cmd/server/server.go:337\"\r\ntime=\"2018-09-10T23:04:15Z\" level=info msg=\"Successfully retrieved Ark configuration\" logSource=\"pkg/cmd/server/server.go:355\"\r\ntime=\"2018-09-10T23:04:15Z\" level=info msg=\"Using default resource priorities\" logSource=\"pkg/cmd/server/server.go:405\" priorities=\"[namespaces persistentvolumes persistentvolumeclaims secrets configmaps serviceaccounts limitranges pods]\r\n\"\r\ntime=\"2018-09-10T23:04:15Z\" level=info msg=\"Configuring cloud provider for backup service\" logSource=\"pkg/cmd/server/server.go:454\"\r\nAn error occurred: object storage provider name must not be empty\r\n```\r\n\r\n* `ark backup describe <backupname>` or `kubectl get backup/<backupname> -n heptio-ark -o yaml`\r\n\r\n```\r\nName:         nginx-backup\r\nNamespace:    heptio-ark-test\r\nLabels:       <none>\r\nAnnotations:  <none>\r\n\r\nPhase:  New\r\n\r\nNamespaces:\r\n  Included:  *\r\n  Excluded:  <none>\r\n\r\nResources:\r\n  Included:        *\r\n  Excluded:        <none>\r\n  Cluster-scoped:  auto\r\n\r\nLabel selector:  app=nginx\r\n\r\nSnapshot PVs:  auto\r\n\r\nTTL:  720h0m0s\r\n\r\nHooks:  <none>\r\n\r\nBackup Format Version:  0\r\n\r\nStarted:    <n/a>\r\nCompleted:  <n/a>\r\n\r\nExpiration:  0001-01-01 00:00:00 +0000 UTC\r\n\r\nValidation errors:  <none>\r\n\r\nPersistent Volumes: <none included>\r\n```\r\n\r\n* `ark backup logs <backupname>`\r\n\r\n**hangs**\r\n\r\n* `ark restore describe <restorename>` or `kubectl get restore/<restorename> -n heptio-ark -o yaml`\r\n\r\n```\r\nName:         nginx-backup-20180910154449\r\nNamespace:    heptio-ark-test\r\nLabels:       <none>\r\nAnnotations:  <none>\r\n\r\nBackup:  nginx-backup\r\n\r\nNamespaces:\r\n  Included:  *\r\n  Excluded:  <none>\r\n\r\nResources:\r\n  Included:        *\r\n  Excluded:        <none>\r\n  Cluster-scoped:  auto\r\n\r\nNamespace mappings:  <none>\r\n\r\nLabel selector:  <none>\r\n\r\nRestore PVs:  auto\r\n\r\nPhase:\r\n\r\nValidation errors:  <none>\r\n\r\nWarnings:  <none>\r\nErrors:    <none>\r\n```\r\n\r\n* `ark restore logs <restorename>`\r\n`An error occurred: unable to retrieve logs because restore is not complete`\r\n\r\n**Anything else you would like to add:**\r\n[Miscellaneous information that will assist in solving the issue.]\r\nAfter writing the rest of this issue I noticed I could set the log-level to debug. All it told me was each custom resource it found... Something we already know from log-level INFO.\r\n\r\n```\r\ntime=\"2018-09-10T23:41:55Z\" level=info msg=\"setting log-level to DEBUG\"\r\ntime=\"2018-09-10T23:41:55Z\" level=debug msg=\"Found custom resource\" kind=PodVolumeRestore logSource=\"pkg/cmd/server/server.go:321\"\r\ntime=\"2018-09-10T23:41:55Z\" level=debug msg=\"Found custom resource\" kind=ResticRepository logSource=\"pkg/cmd/server/server.go:321\"\r\ntime=\"2018-09-10T23:41:55Z\" level=debug msg=\"Found custom resource\" kind=Backup logSource=\"pkg/cmd/server/server.go:321\"\r\ntime=\"2018-09-10T23:41:55Z\" level=debug msg=\"Found custom resource\" kind=Config logSource=\"pkg/cmd/server/server.go:321\"\r\ntime=\"2018-09-10T23:41:55Z\" level=debug msg=\"Found custom resource\" kind=DownloadRequest logSource=\"pkg/cmd/server/server.go:321\"\r\ntime=\"2018-09-10T23:41:55Z\" level=debug msg=\"Found custom resource\" kind=PodVolumeBackup logSource=\"pkg/cmd/server/server.go:321\"\r\ntime=\"2018-09-10T23:41:55Z\" level=debug msg=\"Found custom resource\" kind=Restore logSource=\"pkg/cmd/server/server.go:321\"\r\ntime=\"2018-09-10T23:41:55Z\" level=debug msg=\"Found custom resource\" kind=Schedule logSource=\"pkg/cmd/server/server.go:321\"\r\ntime=\"2018-09-10T23:41:55Z\" level=debug msg=\"Found custom resource\" kind=DeleteBackupRequest logSource=\"pkg/cmd/server/server.go:321\"\r\n```\r\n\r\n**Environment:**\r\n\r\n- Ark version (use `ark version`):  0.9.4\r\n- Kubernetes version (use `kubectl version`): Client 1.9 Server 1.7\r\n- Cloud provider or hardware configuration: AWS\r\n- OS (e.g. from `/etc/os-release`): Ubuntu 16.04\r\n"},"comment":{"url":"https://api.github.com/repos/heptio/ark/issues/comments/420110588","html_url":"https://github.com/heptio/ark/issues/836#issuecomment-420110588","issue_url":"https://api.github.com/repos/heptio/ark/issues/836","id":420110588,"node_id":"MDEyOklzc3VlQ29tbWVudDQyMDExMDU4OA==","user":{"login":"mehemken","id":16868431,"node_id":"MDQ6VXNlcjE2ODY4NDMx","avatar_url":"https://avatars2.githubusercontent.com/u/16868431?v=4","gravatar_id":"","url":"https://api.github.com/users/mehemken","html_url":"https://github.com/mehemken","followers_url":"https://api.github.com/users/mehemken/followers","following_url":"https://api.github.com/users/mehemken/following{/other_user}","gists_url":"https://api.github.com/users/mehemken/gists{/gist_id}","starred_url":"https://api.github.com/users/mehemken/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mehemken/subscriptions","organizations_url":"https://api.github.com/users/mehemken/orgs","repos_url":"https://api.github.com/users/mehemken/repos","events_url":"https://api.github.com/users/mehemken/events{/privacy}","received_events_url":"https://api.github.com/users/mehemken/received_events","type":"User","site_admin":false},"created_at":"2018-09-11T01:02:49Z","updated_at":"2018-09-11T01:02:49Z","author_association":"NONE","body":"@ncdc Hey Andy, thanks for that explanation! I'll give that a shot tomorrow."}}
{"actor":{"display_login":"mehemken"}}