{"action":"created","issue":{"url":"https://api.github.com/repos/treasure-data/digdag/issues/768","repository_url":"https://api.github.com/repos/treasure-data/digdag","labels_url":"https://api.github.com/repos/treasure-data/digdag/issues/768/labels{/name}","comments_url":"https://api.github.com/repos/treasure-data/digdag/issues/768/comments","events_url":"https://api.github.com/repos/treasure-data/digdag/issues/768/events","html_url":"https://github.com/treasure-data/digdag/issues/768","id":317504211,"number":768,"title":"Output 'java.io.IOException: Argument list too long' using by sh operator.","user":{"login":"wakamen","id":3108692,"avatar_url":"https://avatars3.githubusercontent.com/u/3108692?v=4","gravatar_id":"","url":"https://api.github.com/users/wakamen","html_url":"https://github.com/wakamen","followers_url":"https://api.github.com/users/wakamen/followers","following_url":"https://api.github.com/users/wakamen/following{/other_user}","gists_url":"https://api.github.com/users/wakamen/gists{/gist_id}","starred_url":"https://api.github.com/users/wakamen/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/wakamen/subscriptions","organizations_url":"https://api.github.com/users/wakamen/orgs","repos_url":"https://api.github.com/users/wakamen/repos","events_url":"https://api.github.com/users/wakamen/events{/privacy}","received_events_url":"https://api.github.com/users/wakamen/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-04-25T07:14:18Z","updated_at":"2018-04-25T08:04:48Z","closed_at":null,"author_association":"NONE","body":"Hello.\r\nThis workflow is failed.\r\nDoes digdag pass environment variable automatically when using the sh operator?\r\n\r\nsample code\r\n- set_heavy_list_variable.dig\r\n```yml\r\n+set_some_variables:\r\n  _export:\r\n    rb:\r\n      require: variable_setter.rb\r\n  rb>: VariableSetter.set_heavy\r\n\r\n+exec_shell_command:\r\n  sh>: echo hello\r\n```\r\n\r\n- variable_setter.rb\r\n```ruby\r\nclass VariableSetter\r\n  def set_heavy\r\n    Digdag.env.store(nums: 1_000_000.times.map { |_i| 1 })\r\n  end\r\nend\r\n```\r\n\r\n- execution log\r\n```\r\n$ digdag run set_heavy_list_variable.dig \r\n2018-04-25 15:36:04 +0900: Digdag v0.9.24\r\n2018-04-25 15:36:07 +0900 [WARN] (main): Using a new session time 2018-04-25T00:00:00+00:00.\r\n2018-04-25 15:36:07 +0900 [INFO] (main): Using session /Users/xxx/dev/samples/digdag/.digdag/status/20180425T000000+0000.\r\n2018-04-25 15:36:07 +0900 [INFO] (main): Starting a new session project id=1 workflow name=set_heavy_list_variable session_time=2018-04-25T00:00:00+00:00\r\n2018-04-25 15:36:08 +0900 [INFO] (0018@[0:default]+set_heavy_list_variable+set_some_variables): rb>: VariableSetter.set_heavy\r\n2018-04-25 15:36:16 +0900 [INFO] (0018@[0:default]+set_heavy_list_variable+exec_shell_command): sh>: echo hello\r\n2018-04-25 15:36:16 +0900 [ERROR] (0018@[0:default]+set_heavy_list_variable+exec_shell_command): Task failed with unexpected error: java.io.IOException: Cannot run program \"/bin/sh\" (in directory \"/Users/xxx/dev/samples/digdag\"): error=7, Argument list too long\r\njava.lang.RuntimeException: java.io.IOException: Cannot run program \"/bin/sh\" (in directory \"/Users/xxx/dev/samples/digdag\"): error=7, Argument list too long\r\n\tat com.google.common.base.Throwables.propagate(Throwables.java:160)\r\n\tat io.digdag.standards.operator.ShOperatorFactory$ShOperator.runTask(ShOperatorFactory.java:139)\r\n\tat io.digdag.util.BaseOperator.run(BaseOperator.java:35)\r\n\tat io.digdag.core.agent.OperatorManager.callExecutor(OperatorManager.java:312)\r\n\tat io.digdag.cli.Run$OperatorManagerWithSkip.callExecutor(Run.java:694)\r\n\tat io.digdag.core.agent.OperatorManager.runWithWorkspace(OperatorManager.java:254)\r\n\tat io.digdag.core.agent.OperatorManager.lambda$runWithHeartbeat$2(OperatorManager.java:137)\r\n\tat io.digdag.core.agent.LocalWorkspaceManager.withExtractedArchive(LocalWorkspaceManager.java:25)\r\n\tat io.digdag.core.agent.OperatorManager.runWithHeartbeat(OperatorManager.java:135)\r\n\tat io.digdag.core.agent.OperatorManager.run(OperatorManager.java:119)\r\n\tat io.digdag.cli.Run$OperatorManagerWithSkip.run(Run.java:676)\r\n\tat io.digdag.core.agent.MultiThreadAgent.lambda$null$0(MultiThreadAgent.java:127)\r\n\tat java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)\r\n\tat java.util.concurrent.FutureTask.run(FutureTask.java:266)\r\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\r\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\r\n\tat java.lang.Thread.run(Thread.java:748)\r\nCaused by: java.io.IOException: Cannot run program \"/bin/sh\" (in directory \"/Users/xxx/dev/samples/digdag\"): error=7, Argument list too long\r\n\tat java.lang.ProcessBuilder.start(ProcessBuilder.java:1048)\r\n\tat io.digdag.standards.command.SimpleCommandExecutor.start(SimpleCommandExecutor.java:22)\r\n\tat io.digdag.standards.command.DockerCommandExecutor.start(DockerCommandExecutor.java:56)\r\n\tat io.digdag.standards.operator.ShOperatorFactory$ShOperator.runTask(ShOperatorFactory.java:126)\r\n\t... 15 common frames omitted\r\nCaused by: java.io.IOException: error=7, Argument list too long\r\n\tat java.lang.UNIXProcess.forkAndExec(Native Method)\r\n\tat java.lang.UNIXProcess.<init>(UNIXProcess.java:247)\r\n\tat java.lang.ProcessImpl.start(ProcessImpl.java:134)\r\n\tat java.lang.ProcessBuilder.start(ProcessBuilder.java:1029)\r\n\t... 18 common frames omitted\r\n2018-04-25 15:36:16 +0900 [INFO] (0018@[0:default]+set_heavy_list_variable^failure-alert): type: notify\r\nerror: \r\n  * +set_heavy_list_variable+exec_shell_command:\r\n    Cannot run program \"/bin/sh\" (in directory \"/Users/xxx/dev/samples/digdag\"): error=7, Argument list too long (runtime)\r\n\r\nTask state is saved at /Users/xxx/dev/samples/digdag/.digdag/status/20180425T000000+0000 directory.\r\n  * Use --session <daily | hourly | \"yyyy-MM-dd[ HH:mm:ss]\"> to not reuse the last session time.\r\n  * Use --rerun, --start +NAME, or --goal +NAME argument to rerun skipped tasks.\r\n```\r\n\r\nmy environment\r\n```\r\n$ digdag --version\r\n0.9.24\r\n\r\n$ java -version\r\njava version \"1.8.0_172\"\r\nJava(TM) SE Runtime Environment (build 1.8.0_172-b11)\r\nJava HotSpot(TM) 64-Bit Server VM (build 25.172-b11, mixed mode)\r\n```"},"comment":{"url":"https://api.github.com/repos/treasure-data/digdag/issues/comments/384197158","html_url":"https://github.com/treasure-data/digdag/issues/768#issuecomment-384197158","issue_url":"https://api.github.com/repos/treasure-data/digdag/issues/768","id":384197158,"user":{"login":"hiroyuki-sato","id":767650,"avatar_url":"https://avatars0.githubusercontent.com/u/767650?v=4","gravatar_id":"","url":"https://api.github.com/users/hiroyuki-sato","html_url":"https://github.com/hiroyuki-sato","followers_url":"https://api.github.com/users/hiroyuki-sato/followers","following_url":"https://api.github.com/users/hiroyuki-sato/following{/other_user}","gists_url":"https://api.github.com/users/hiroyuki-sato/gists{/gist_id}","starred_url":"https://api.github.com/users/hiroyuki-sato/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/hiroyuki-sato/subscriptions","organizations_url":"https://api.github.com/users/hiroyuki-sato/orgs","repos_url":"https://api.github.com/users/hiroyuki-sato/repos","events_url":"https://api.github.com/users/hiroyuki-sato/events{/privacy}","received_events_url":"https://api.github.com/users/hiroyuki-sato/received_events","type":"User","site_admin":false},"created_at":"2018-04-25T08:04:48Z","updated_at":"2018-04-25T08:04:48Z","author_association":"CONTRIBUTOR","body":"Hello, @wakamen \r\n\r\nYou can check environment variable easily with `env` command.\r\n\r\n```yaml\r\n+set_some_variables:\r\n  _export:\r\n    rb:\r\n      require: variable_setter.rb\r\n  rb>: VariableSetter.set_heavy\r\n\r\n+exec_shell_command:\r\n  sh>: env\r\n#  sh>: echo hello\r\n```\r\n\r\n```ruby\r\nclass VariableSetter\r\n  def set_heavy\r\n#    Digdag.env.store(nums: 1_000_000.times.map { |_i| 1 })\r\n    Digdag.env.store(nums: 10.times.map { |_i| 1 })\r\n  end\r\nend\r\n```\r\n\r\n```\r\n2018-04-25 17:00:11 +0900 [INFO] (0017@[0:default]+set_heavy_list_variable+exec_shell_command): sh>: env\r\n...\r\nnums=[1,1,1,1,1,1,1,1,1,1]\r\n```\r\n\r\n\r\nI think your issue is `/bin/sh` limitation, not digdag.\r\n`sh` doesn't allow to set long variable value like `nums=[1,1,1,1,1.......]` .\r\n\r\nYou can reproduce it without Digdag.\r\n\r\n```\r\nruby -e 'puts \"export TEST=\" + \"x\" * 10000000'  > /tmp/test.txt\r\n. /tmp/test.text\r\nls\r\n-bash: /bin/ls: Argument list too long\r\n```\r\n\r\nYou also can't set many environment variable too.\r\n\r\n```bash\r\n#!/bin/sh\r\n\r\n#for i in {1..1000000} ; do\r\nfor i in {1..50000} ; do\r\n  echo $i\r\n  export TEST$i=\"$i\"\r\ndone\r\n#env\r\nexec /bin/sh -c 'echo \"test\"'\r\n```\r\n\r\n```\r\ntest.sh: line 9: /bin/sh: Argument list too long\r\ntest.sh: line 9: /bin/sh: Undefined error: 0\r\n```"}}
{"actor":{"display_login":"hiroyuki-sato"}}