{"action":"created","issue":{"url":"https://api.github.com/repos/gruntwork-io/terragrunt/issues/466","repository_url":"https://api.github.com/repos/gruntwork-io/terragrunt","labels_url":"https://api.github.com/repos/gruntwork-io/terragrunt/issues/466/labels{/name}","comments_url":"https://api.github.com/repos/gruntwork-io/terragrunt/issues/466/comments","events_url":"https://api.github.com/repos/gruntwork-io/terragrunt/issues/466/events","html_url":"https://github.com/gruntwork-io/terragrunt/issues/466","id":318131462,"number":466,"title":"Separate configuration file for Terragrunt?","user":{"login":"apparentlymart","id":20180,"avatar_url":"https://avatars1.githubusercontent.com/u/20180?v=4","gravatar_id":"","url":"https://api.github.com/users/apparentlymart","html_url":"https://github.com/apparentlymart","followers_url":"https://api.github.com/users/apparentlymart/followers","following_url":"https://api.github.com/users/apparentlymart/following{/other_user}","gists_url":"https://api.github.com/users/apparentlymart/gists{/gist_id}","starred_url":"https://api.github.com/users/apparentlymart/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/apparentlymart/subscriptions","organizations_url":"https://api.github.com/users/apparentlymart/orgs","repos_url":"https://api.github.com/users/apparentlymart/repos","events_url":"https://api.github.com/users/apparentlymart/events{/privacy}","received_events_url":"https://api.github.com/users/apparentlymart/received_events","type":"User","site_admin":false},"labels":[{"id":381296953,"url":"https://api.github.com/repos/gruntwork-io/terragrunt/labels/question","name":"question","color":"cc317c","default":true}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":19,"created_at":"2018-04-26T17:44:24Z","updated_at":"2018-05-11T20:03:03Z","closed_at":null,"author_association":"NONE","body":"Hi! I'm one of the engineers at HashiCorp who works on Terraform Core.\r\n\r\nAs you might be aware, we've been working for some time now on various improvements to the Terraform configuration language, to address a number of usability problems that have been reported over the years. The new changes are broadly compatible with existing configurations, and particularly those using the common idioms reflected in the Terraform documentation.\r\n\r\nUnfortunately resolving some of these problems required tightening up some things that were previously rather loose. Current and previous versions of Terraform ignore certain things that are considered invalid, often producing surprising results, and so to improve usability the new implementation has some more error checking in order to catch mistakes and produce error messages that are hopefully more actionable.\r\n\r\nWe're planning to include a migration helper tool in the next release in order to help detect and fix some of these rough edges. The details of what this tool does will be largely informed by real-world usage, and so we're planning to release an early preview of the new implementation as soon as it is in a usable state in the hope that we can gather as many examples as possible of patterns in the wild that the migration helper tool should be looking for.\r\n\r\nWith all of that context in mind, we know that there are lots of users wrapping Terraform with Terragrunt, and so we want to make sure there's a good upgrade path. I'm opening this issue to discuss a specific problem I've noticed, but I'd like to discuss any other potential hurdles too, in whatever forum makes sense.\r\n\r\n---\r\n\r\nTerragrunt currently uses an extension of the `terraform.tfvars` file as its own configuration file, relying on the fact that Terraform itself ignores definitions in that file that are not defined as variables, and then using a special variable name `terragrunt` whose value is not, by Terraform's rules, valid: it includes mixed-typed nested structures, has nested blocks, etc.\r\n\r\nThe new implementation has a few differences that will cause friction here if we proceed naively:\r\n* Terraform will consider the mixed-type nested block structures as invalid and will generate error messages about them during validation, since they are invalid by Terraform's rules.\r\n* The new configuration parser distinguishes more strongly between object expressions and nested blocks, in order to resolve various confusing behaviors that resulted from the ambiguity of that in the original HCL design.\r\n* The expression evaluator and the structural language were previously entirely separate (HIL vs. HCL), but they are combined in the new implementation, which means that interpolation sequences `${` are interpreted in all contexts, with the difference now being whether variables and functions are available, rather than whether interpolations are possible at all.\r\n\r\nLooking at [the \"live infrastructure\" example](https://github.com/gruntwork-io/terragrunt-infrastructure-live-example/blob/97be88a78e2e42c38307c778467f72ad43ed53c8/prod/terraform.tfvars), a naive syntax-level pass of migration would produce the following rewritten version:\r\n\r\n```hcl\r\n# Terragrunt is a thin wrapper for Terraform that provides extra tools for working with multiple Terraform modules,\r\n# remote state, and locking: https://github.com/gruntwork-io/terragrunt\r\nterragrunt = {\r\n  # Configure Terragrunt to automatically store tfstate files in an S3 bucket\r\n  remote_state = {\r\n    backend = \"s3\"\r\n    config = {\r\n      encrypt        = true\r\n      bucket         = \"terragrunt-example-prod-terraform-state\"\r\n      key            = \"$${path_relative_to_include()}/terraform.tfstate\"\r\n      region         = \"us-west-2\"\r\n      dynamodb_table = \"terraform-locks\"\r\n    }\r\n  }\r\n  # Configure root level variables that all resources can inherit\r\n  terraform = {\r\n    extra_arguments = {\r\n      # (this part will need to do some heuristic processing to decide whether to use\r\n      # direct objects or lists of objects, mimicking the behavior of the current hcl.Decode\r\n      # into an interface{}, so the details may not exactly match this in practice.)\r\n      bucket = {\r\n        commands = [\"$${get_terraform_commands_that_need_vars()}\"]\r\n        optional_var_files = [\r\n          \"$${get_tfvars_dir()}/$${find_in_parent_folders(\"account.tfvars\", \"ignore\")}\"\r\n        ]\r\n      }\r\n    }\r\n  }\r\n}\r\n```\r\n\r\nSince Terragrunt is using the HCL parser directly to process its configuration file today, it has the freedom to interpret the tree of HCL AST objects inside here as desired, and so I expect there are expectations here that are different than how Terraform generally deals with HCL structures it has no specific schema for. Also, due to the fact that interpolation syntax is now accepted in all quoted strings (variables and functions are just not available in some contexts), the naive migration tool will escape the `${` sequences under the assumption that the user intended them to be treated literally, since that was Terraform's own interpretation of a `.tfvars` file before this change.\r\n\r\nTherefore it feels to me like the most practical path to take here is for our migration tool to have a special case for a variable called `terragrunt` (possibly activated only if there is no `variable \"terragrunt\"` block in configuration) which can then have some light understanding of Terragrunt's configuration structure and make some allowances for it so that we can avoid unintentional breakage of the Terragrunt configuration.\r\n\r\nTerragrunt is currently relying on syntax features that are considered invalid in the new implementation, so leaving the configuration _untouched_ by the migration tool would inevitably lead to syntax errors during parsing.\r\n\r\n---\r\n\r\nIn order to avoid conflicts between Terraform's and Terragrunt's interpretation of the same file, I'd like to propose having Terragrunt look for a Terragrunt-specific configuration filename (e.g. `terragrunt.conf`) in preference to `terraform.tfvars`, and then Terraform would entirely ignore this file and leave Terragrunt's own parser to deal with it. Terragrunt could continue to use the original implementation of HCL, avoiding the need to deal with any of the new processing rules from the new implementation.\r\n\r\nIf we were to go down this path, we could build the configuration migration tool so that its special case for the `terragrunt` variable name is to migrate the contents out into a `terragrunt.conf` file that retains the block structure and original-HCL syntax that Terragrunt is expecting, and then deletes this value from the `terraform.tfvars` file altogether so that Terraform itself won't try to parse it.\r\n\r\nI understand that this is a pretty drastic shift in the design and usage of Terragrunt, so I wanted to start talking about this early so we have some time to think through the implications and any other alternative approaches. I hope that together we can find a reasonable path forward!\r\n"},"comment":{"url":"https://api.github.com/repos/gruntwork-io/terragrunt/issues/comments/388471317","html_url":"https://github.com/gruntwork-io/terragrunt/issues/466#issuecomment-388471317","issue_url":"https://api.github.com/repos/gruntwork-io/terragrunt/issues/466","id":388471317,"user":{"login":"lorengordon","id":8457307,"avatar_url":"https://avatars3.githubusercontent.com/u/8457307?v=4","gravatar_id":"","url":"https://api.github.com/users/lorengordon","html_url":"https://github.com/lorengordon","followers_url":"https://api.github.com/users/lorengordon/followers","following_url":"https://api.github.com/users/lorengordon/following{/other_user}","gists_url":"https://api.github.com/users/lorengordon/gists{/gist_id}","starred_url":"https://api.github.com/users/lorengordon/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lorengordon/subscriptions","organizations_url":"https://api.github.com/users/lorengordon/orgs","repos_url":"https://api.github.com/users/lorengordon/repos","events_url":"https://api.github.com/users/lorengordon/events{/privacy}","received_events_url":"https://api.github.com/users/lorengordon/received_events","type":"User","site_admin":false},"created_at":"2018-05-11T20:03:03Z","updated_at":"2018-05-11T20:03:03Z","author_association":"NONE","body":"@ebarault I'd love to see some interpolation in `.tfvars` files. Looking up ENV values, and many of the \"data\" resources would be killer. I'd especially like to be able to retrieve values from the SSM Parameter Store. It's quite a pain to expose these implementation-specific \"value lookup\" constructs in a module (especially if you publish it publicly). Getting this in terraform core would be awesome! 👍 👍 👍 "}}
{"actor":{"display_login":"lorengordon"}}