{"action":"created","issue":{"url":"https://api.github.com/repos/PDAL/PDAL/issues/2105","repository_url":"https://api.github.com/repos/PDAL/PDAL","labels_url":"https://api.github.com/repos/PDAL/PDAL/issues/2105/labels{/name}","comments_url":"https://api.github.com/repos/PDAL/PDAL/issues/2105/comments","events_url":"https://api.github.com/repos/PDAL/PDAL/issues/2105/events","html_url":"https://github.com/PDAL/PDAL/issues/2105","id":344152167,"node_id":"MDU6SXNzdWUzNDQxNTIxNjc=","number":2105,"title":"filters.python add_dimension array not working","user":{"login":"devdylan","id":12073101,"node_id":"MDQ6VXNlcjEyMDczMTAx","avatar_url":"https://avatars0.githubusercontent.com/u/12073101?v=4","gravatar_id":"","url":"https://api.github.com/users/devdylan","html_url":"https://github.com/devdylan","followers_url":"https://api.github.com/users/devdylan/followers","following_url":"https://api.github.com/users/devdylan/following{/other_user}","gists_url":"https://api.github.com/users/devdylan/gists{/gist_id}","starred_url":"https://api.github.com/users/devdylan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/devdylan/subscriptions","organizations_url":"https://api.github.com/users/devdylan/orgs","repos_url":"https://api.github.com/users/devdylan/repos","events_url":"https://api.github.com/users/devdylan/events{/privacy}","received_events_url":"https://api.github.com/users/devdylan/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":12,"created_at":"2018-07-24T18:26:52Z","updated_at":"2018-07-25T18:01:37Z","closed_at":"2018-07-24T20:31:47Z","author_association":"CONTRIBUTOR","body":"Follow-up, from https://github.com/PDAL/PDAL/issues/2098\r\n\r\nWhen I use an array of dimensions in my python filter, the output model has the dimensions I added, but its values are all zero (as revealed by running `info --all`).\r\n```\r\n      {\r\n        \"average\": 0,\r\n        \"count\": 21178491,\r\n        \"maximum\": 0,\r\n        \"minimum\": 0,\r\n        \"name\": \"Red\",\r\n        \"position\": 12,\r\n        \"stddev\": 0,\r\n        \"variance\": 0\r\n      },\r\n      {\r\n        \"average\": 0,\r\n        \"count\": 21178491,\r\n        \"maximum\": 0,\r\n        \"minimum\": 0,\r\n        \"name\": \"Green\",\r\n        \"position\": 13,\r\n        \"stddev\": 0,\r\n        \"variance\": 0\r\n      },\r\n      {\r\n        \"average\": 0,\r\n        \"count\": 21178491,\r\n        \"maximum\": 0,\r\n        \"minimum\": 0,\r\n        \"name\": \"Blue\",\r\n        \"position\": 14,\r\n        \"stddev\": 0,\r\n        \"variance\": 0\r\n      },\r\n ```\r\n\r\nWhen I use a filters.ferry pre-step to add the dimensions, followed by the python filter (which has no add_dimension option), the output model has dimensions with the values I set in the python filter. Everything works as expected, but this adds considerable processing time to the pipeline.\r\n```\r\n      {\r\n        \"average\": 244.1093034,\r\n        \"count\": 21178491,\r\n        \"kurtosis\": 14.35534825,\r\n        \"maximum\": 255,\r\n        \"minimum\": 0,\r\n        \"name\": \"Red\",\r\n        \"position\": 12,\r\n        \"skewness\": -3.876008839,\r\n        \"stddev\": 42.26158408,\r\n        \"variance\": 1786.041489\r\n      },\r\n      {\r\n        \"average\": 49.4982151,\r\n        \"count\": 21178491,\r\n        \"kurtosis\": 1.284154791e+13,\r\n        \"maximum\": 255,\r\n        \"minimum\": 0,\r\n        \"name\": \"Green\",\r\n        \"position\": 13,\r\n        \"skewness\": 3.106217842e+12,\r\n        \"stddev\": 100.1434097,\r\n        \"variance\": 10028.70251\r\n      },\r\n      {\r\n        \"average\": 120.8368289,\r\n        \"count\": 21178491,\r\n        \"kurtosis\": -3.0158649e+11,\r\n        \"maximum\": 255,\r\n        \"minimum\": 0,\r\n        \"name\": \"Blue\",\r\n        \"position\": 14,\r\n        \"skewness\": 4.934173081e+11,\r\n        \"stddev\": 123.2736605,\r\n        \"variance\": 15196.39537\r\n      },\r\n```\r\n\r\n\r\nPipeline with filters.ferry pre-step to add dimensions:\r\n```\r\n{\r\n  \"pipeline\":[\r\n    \"/data/input/classified_lidar.las\",\r\n    {\r\n      \"type\": \"filters.ferry\",\r\n      \"dimensions\": \"=>Green,=>Red,=>Blue\"\r\n    },\r\n    {\r\n      \"type\":\"filters.python\",\r\n      \"script\":\"/data/scripts/pdal/prepare/colorize_classification.py\",\r\n      \"function\":\"colorize_classification\",\r\n      \"module\":\"anything\"\r\n    },\r\n    \"/data/output/colorized_classified_lidar.las\"\r\n  ]\r\n}\r\n```\r\n\r\n\r\nPipeline with array of dimensions in `add_dimension` (and no filters.ferry pre-step):\r\n```\r\n{\r\n  \"pipeline\":[\r\n    \"/data/input/point_cloud.las\",\r\n    {\r\n      \"type\":\"filters.python\",\r\n      \"script\":\"/data/scripts/pdal/prepare/colorize_classification.py\",\r\n      \"function\":\"colorize_classification\",\r\n      \"module\":\"anything\",\r\n      \"add_dimension\":[\"Red\", \"Green\", \"Blue\"]\r\n    },\r\n    \"/data/output/colorized_point_cloud.las\"\r\n  ]\r\n}\r\n```\r\n\r\n\r\n\r\nThis the gist of my filter:\r\n\r\n```\r\nimport numpy as np\r\n\r\ndef colorize_classification(ins, outs):\r\n    cls = ins['Classification']\r\n\r\n    red = np.ones(cls.size, dtype=np.uint8)\r\n    green = np.ones(cls.size, dtype=np.uint8)\r\n    blue = np.ones(cls.size, dtype=np.uint8)\r\n\r\n    it = np.nditer(cls, flags=['f_index'])\r\n    while not it.finished:\r\n\r\n        if (cls[it.index] ==1):\r\n            red[it.index] = 204\r\n            blue[it.index] = 20\r\n            green[it.index] = 153\r\n\r\n        elif (cls[it.index] ==2):\r\n            red[it.index] = 40\r\n            blue[it.index] = 70\r\n            green[it.index] = 255\r\n\r\n        ...\r\n            \r\n        else:\r\n            red[it.index] = 255\r\n            blue[it.index] = 255\r\n            green[it.index] = 255\r\n\r\n        it.iternext()\r\n\r\n    outs['Red'] = red.astype('float64')\r\n    outs['Blue'] = blue.astype('float64')\r\n    outs['Green'] = green.astype('float64')\r\n    return True\r\n```\r\n\r\nThanks for all your help."},"comment":{"url":"https://api.github.com/repos/PDAL/PDAL/issues/comments/407843354","html_url":"https://github.com/PDAL/PDAL/issues/2105#issuecomment-407843354","issue_url":"https://api.github.com/repos/PDAL/PDAL/issues/2105","id":407843354,"node_id":"MDEyOklzc3VlQ29tbWVudDQwNzg0MzM1NA==","user":{"login":"devdylan","id":12073101,"node_id":"MDQ6VXNlcjEyMDczMTAx","avatar_url":"https://avatars0.githubusercontent.com/u/12073101?v=4","gravatar_id":"","url":"https://api.github.com/users/devdylan","html_url":"https://github.com/devdylan","followers_url":"https://api.github.com/users/devdylan/followers","following_url":"https://api.github.com/users/devdylan/following{/other_user}","gists_url":"https://api.github.com/users/devdylan/gists{/gist_id}","starred_url":"https://api.github.com/users/devdylan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/devdylan/subscriptions","organizations_url":"https://api.github.com/users/devdylan/orgs","repos_url":"https://api.github.com/users/devdylan/repos","events_url":"https://api.github.com/users/devdylan/events{/privacy}","received_events_url":"https://api.github.com/users/devdylan/received_events","type":"User","site_admin":false},"created_at":"2018-07-25T18:01:37Z","updated_at":"2018-07-25T18:01:37Z","author_association":"CONTRIBUTOR","body":"I see what you're saying now - the python filter runs for each data source, including my \"empty\" data source (which just contains the new dimensions). Thus, I should handle this case in my python script:\r\n```\r\n    cls = ins['Classification']\r\n    if(cls.size < 1):\r\n        return True\r\n```\r\nThis provides slightly better performance than the ferry method, fyi. Thanks again!"}}
{"actor":{"display_login":"devdylan"}}