{"action":"created","issue":{"url":"https://api.github.com/repos/lark-parser/lark/issues/187","repository_url":"https://api.github.com/repos/lark-parser/lark","labels_url":"https://api.github.com/repos/lark-parser/lark/issues/187/labels{/name}","comments_url":"https://api.github.com/repos/lark-parser/lark/issues/187/comments","events_url":"https://api.github.com/repos/lark-parser/lark/issues/187/events","html_url":"https://github.com/lark-parser/lark/issues/187","id":341171794,"node_id":"MDU6SXNzdWUzNDExNzE3OTQ=","number":187,"title":"Parsing strings containing variable expansion","user":{"login":"oprypin","id":371383,"node_id":"MDQ6VXNlcjM3MTM4Mw==","avatar_url":"https://avatars3.githubusercontent.com/u/371383?v=4","gravatar_id":"","url":"https://api.github.com/users/oprypin","html_url":"https://github.com/oprypin","followers_url":"https://api.github.com/users/oprypin/followers","following_url":"https://api.github.com/users/oprypin/following{/other_user}","gists_url":"https://api.github.com/users/oprypin/gists{/gist_id}","starred_url":"https://api.github.com/users/oprypin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/oprypin/subscriptions","organizations_url":"https://api.github.com/users/oprypin/orgs","repos_url":"https://api.github.com/users/oprypin/repos","events_url":"https://api.github.com/users/oprypin/events{/privacy}","received_events_url":"https://api.github.com/users/oprypin/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":4,"created_at":"2018-07-13T22:00:27Z","updated_at":"2018-07-14T07:01:22Z","closed_at":null,"author_association":"NONE","body":"Hi,\r\nI'm having difficulties with parsing strings that contain variable expansions like in Bash. `\"Hello, $name!\"`.\r\n(So this is either a newbie question, or a feature request.)\r\n\r\nWhen I run this example, each character in the string is processed as a separate terminal. It makes sense why it happens but I don't want it to happen. It also seems to decimate performance on a larger input.\r\n\r\n```python\r\nimport lark\r\n\r\nparser = lark.Lark(r'''\r\nIDENTIFIER: /[A-Za-z_][A-Za-z_0-9]*/\r\nstring    : \"\\\"\" ( CHAR | ESCAPE | expansion ) * \"\\\"\"\r\nESCAPE    : \"\\\\\" ( \"$\" | \"\\\"\" | CHAR )\r\nexpansion : \"$\" IDENTIFIER\r\nCHAR      : /[^$\"\\n]/\r\n''', start='string')\r\n\r\ntree = parser.parse('\"Hello, $name! That will be \\$9.95\"')\r\nprint(tree.pretty())\r\n```\r\n```\r\nstring\r\n  H\r\n  e\r\n  l\r\n  l\r\n  o\r\n  ,\r\n\r\n  $name\r\n  !\r\n\r\n  T\r\n  h\r\n  a\r\n  t\r\n\r\n  w\r\n  i\r\n  l\r\n  l\r\n\r\n  b\r\n  e\r\n\r\n  \\$\r\n  9\r\n  .\r\n  9\r\n  5\r\n```\r\n\r\nI've tried changing `/[^$\"\\n]/` to `/[^$\"\\n]+/` but that causes an error.\r\n\r\nExisting suboptimal options I have:\r\n\r\n* Accept the poor performance and concatenate characters\r\n* Call a separate string parser recursively during traversal\r\n\r\nNote that this is the simplest possible scenario. What about expressions nested in a string, like Ruby's `\"Hello, #{arbitrary + expression}!\"` or Python's new `f'Hello, {arbitrary + expression}!'`? *python3.lark* doesn't seem to have these, anyway..."},"comment":{"url":"https://api.github.com/repos/lark-parser/lark/issues/comments/405004321","html_url":"https://github.com/lark-parser/lark/issues/187#issuecomment-405004321","issue_url":"https://api.github.com/repos/lark-parser/lark/issues/187","id":405004321,"node_id":"MDEyOklzc3VlQ29tbWVudDQwNTAwNDMyMQ==","user":{"login":"oprypin","id":371383,"node_id":"MDQ6VXNlcjM3MTM4Mw==","avatar_url":"https://avatars3.githubusercontent.com/u/371383?v=4","gravatar_id":"","url":"https://api.github.com/users/oprypin","html_url":"https://github.com/oprypin","followers_url":"https://api.github.com/users/oprypin/followers","following_url":"https://api.github.com/users/oprypin/following{/other_user}","gists_url":"https://api.github.com/users/oprypin/gists{/gist_id}","starred_url":"https://api.github.com/users/oprypin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/oprypin/subscriptions","organizations_url":"https://api.github.com/users/oprypin/orgs","repos_url":"https://api.github.com/users/oprypin/repos","events_url":"https://api.github.com/users/oprypin/events{/privacy}","received_events_url":"https://api.github.com/users/oprypin/received_events","type":"User","site_admin":false},"created_at":"2018-07-14T07:01:22Z","updated_at":"2018-07-14T07:01:22Z","author_association":"NONE","body":"I cannot just change everything to terminals, my actual rules are more complicated with expressions nested in them. (see also the bottom of my original post)\r\n\r\nAbout keeping as a rule:\r\n\r\n```python\r\nimport lark\r\n\r\nparser = lark.Lark(r'''\r\nIDENTIFIER: (\"A\" .. \"Z\" | \"a\" .. \"z\") (\"A\" .. \"Z\" | \"a\" .. \"z\" | \"0\" .. \"9\") *\r\nstring    : \"\\\"\" ( WORD | ESCAPE | expansion ) * \"\\\"\"\r\nESCAPE    : \"\\\\\" ( \"$\" | \"\\\"\" | CHAR )\r\nexpansion : \"$\" IDENTIFIER\r\nCHAR      : /[^$\"\\n]/\r\nWORD      : CHAR +\r\n''', start='string')\r\n\r\ntree = parser.parse('\"Hello, $name! That will be \\\\$9.95\"')\r\nprint(tree.pretty())\r\n```\r\n\r\n```\r\nlark.exceptions.UnexpectedCharacters: No terminal defined for '9' at line 1 col 31\r\n\r\n\"Hello, $name! That will be \\$9.95\"\r\n                              ^\r\n\r\nExpecting: {Terminal('IDENTIFIER')}\r\n```"}}
{"actor":{"display_login":"oprypin"}}