{"action":"opened","issue":{"url":"https://api.github.com/repos/broadinstitute/cromwell/issues/911","repository_url":"https://api.github.com/repos/broadinstitute/cromwell","labels_url":"https://api.github.com/repos/broadinstitute/cromwell/issues/911/labels{/name}","comments_url":"https://api.github.com/repos/broadinstitute/cromwell/issues/911/comments","events_url":"https://api.github.com/repos/broadinstitute/cromwell/issues/911/events","html_url":"https://github.com/broadinstitute/cromwell/issues/911","id":157737709,"number":911,"title":"Wide scatters fail in database persistence","user":{"login":"kcibul","id":1423491,"avatar_url":"https://avatars.githubusercontent.com/u/1423491?v=3","gravatar_id":"","url":"https://api.github.com/users/kcibul","html_url":"https://github.com/kcibul","followers_url":"https://api.github.com/users/kcibul/followers","following_url":"https://api.github.com/users/kcibul/following{/other_user}","gists_url":"https://api.github.com/users/kcibul/gists{/gist_id}","starred_url":"https://api.github.com/users/kcibul/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kcibul/subscriptions","organizations_url":"https://api.github.com/users/kcibul/orgs","repos_url":"https://api.github.com/users/kcibul/repos","events_url":"https://api.github.com/users/kcibul/events{/privacy}","received_events_url":"https://api.github.com/users/kcibul/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":0,"created_at":"2016-05-31T18:23:30Z","updated_at":"2016-05-31T18:23:30Z","closed_at":null,"body":"Running this WDL (with a wider scatter count... start with 250) fails with a database persistence error like:\r\n\r\n> The size of BLOB/TEXT data inserted in one transaction is greater than 10% of redo log size. Increase the redo log size using innodb_log_file_size.\r\n\r\nWhich we can't do with CloudSQL on Google.\r\n\r\n`\r\ntask FileSpam {\r\n  String sample_name = \"DeliciousFileSpam\"\r\n  Int index\r\n\r\n  command <<<\r\n    mkdir ${sample_name}_${index}\r\n    for i in `seq 0 900`\r\n    do\r\n      echo $i > ${sample_name}_${index}_$i.txt\r\n      echo $i > ${sample_name}_${index}/$i.txt\r\n    done\r\n  >>>\r\n  runtime {\r\n    docker: \"ubuntu:latest\"\r\n    memory: \"3 GB\"\r\n    cpu: \"1\"\r\n    disks: \"local-disk 5 HDD\"\r\n    preemptible: 3\r\n  }\r\n  output {\r\n    Array[File] outer = glob(\"${sample_name}_${index}_*\")\r\n    Array[File] inner = glob(\"${sample_name}_${index}/*\")\r\n  }\r\n}\r\n\r\ntask MatrixRotation {\r\n  Array[Array[String]] input_matrix\r\n\r\n    command <<<\r\n      python <<CODE\r\n      import csv\r\n      import sys\r\n      with open('${write_tsv(input_matrix)}') as tsv_in:\r\n        input_matrix = [line.strip().split('\\t') for line in tsv_in]\r\n        final_matrix = [[\"\" for x in range(len(input_matrix))] for y in range(len(input_matrix[0]))]\r\n        for x in range(len(input_matrix)):\r\n          for y in range(len(input_matrix[0])):\r\n            final_matrix[y][x] = input_matrix[x][y]\r\n      with open(\"output.tsv\", \"w\") as tsv_out:\r\n        # lineterminator is a workaround for a cromwell bug that doesnt allow for '\\r\\n' line endings which this outputs by default\r\n        writer = csv.writer(tsv_out, delimiter='\\t', lineterminator='\\n')\r\n        writer.writerows(final_matrix)\r\n      CODE\r\n      >>>\r\n\r\n    runtime {\r\n      docker: \"python:2.7\"\r\n      memory: \"1 GB\"\r\n      preemptible: 3\r\n    }\r\n\r\n    output {\r\n        File out = \"output.tsv\"\r\n        Array[Array[String]] output_matrix = read_tsv(out)\r\n    }\r\n}\r\n\r\nworkflow DeliciousFileSpam {\r\n  Array[Int] indexing_list = [ 1,  2,  3,  4,  5,  6,  7,  8,  9, 10,\r\n                              11, 12, 13, 14, 15, 16, 17, 18, 19, 20,\r\n                              21, 22, 23, 24, 25, 26, 27, 28, 29, 30,\r\n                              31, 32, 33, 34, 35, 36, 37, 38, 39, 40,\r\n                              41, 42, 43, 44, 45, 46, 47, 48, 49, 50,\r\n                              51, 52, 53, 54, 55, 56, 57, 58, 59, 60,\r\n                              61, 62, 63, 64, 65, 66, 67, 68, 69, 70,\r\n                              71, 72, 73, 74, 75, 76, 77, 78, 79, 80,\r\n                              81, 82, 83, 84, 85, 86, 87, 88, 89, 90,\r\n                              91, 92, 93, 94, 95, 96, 97, 98, 99, 100 ]\r\n\r\n  scatter (idx in indexing_list) {\r\n    call FileSpam { input: index = idx }\r\n  }\r\n\r\n  call MatrixRotation as rotateOuter { input: input_matrix=FileSpam.outer }\r\n  call MatrixRotation as rotateInner { input: input_matrix=FileSpam.inner }\r\n\r\n  output {\r\n    None\r\n  }\r\n}\r\n`"}}
{"actor":{"display_login":"kcibul"}}