{"action":"created","issue":{"url":"https://api.github.com/repos/ibm-watson-data-lab/pixiedust/issues/644","repository_url":"https://api.github.com/repos/ibm-watson-data-lab/pixiedust","labels_url":"https://api.github.com/repos/ibm-watson-data-lab/pixiedust/issues/644/labels{/name}","comments_url":"https://api.github.com/repos/ibm-watson-data-lab/pixiedust/issues/644/comments","events_url":"https://api.github.com/repos/ibm-watson-data-lab/pixiedust/issues/644/events","html_url":"https://github.com/ibm-watson-data-lab/pixiedust/issues/644","id":305054137,"number":644,"title":"datetime64[ns] - unknown/unsupported","user":{"login":"rafalbigaj","id":824894,"avatar_url":"https://avatars0.githubusercontent.com/u/824894?v=4","gravatar_id":"","url":"https://api.github.com/users/rafalbigaj","html_url":"https://github.com/rafalbigaj","followers_url":"https://api.github.com/users/rafalbigaj/followers","following_url":"https://api.github.com/users/rafalbigaj/following{/other_user}","gists_url":"https://api.github.com/users/rafalbigaj/gists{/gist_id}","starred_url":"https://api.github.com/users/rafalbigaj/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rafalbigaj/subscriptions","organizations_url":"https://api.github.com/users/rafalbigaj/orgs","repos_url":"https://api.github.com/users/rafalbigaj/repos","events_url":"https://api.github.com/users/rafalbigaj/events{/privacy}","received_events_url":"https://api.github.com/users/rafalbigaj/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":{"login":"DTAIEB","id":4912140,"avatar_url":"https://avatars3.githubusercontent.com/u/4912140?v=4","gravatar_id":"","url":"https://api.github.com/users/DTAIEB","html_url":"https://github.com/DTAIEB","followers_url":"https://api.github.com/users/DTAIEB/followers","following_url":"https://api.github.com/users/DTAIEB/following{/other_user}","gists_url":"https://api.github.com/users/DTAIEB/gists{/gist_id}","starred_url":"https://api.github.com/users/DTAIEB/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/DTAIEB/subscriptions","organizations_url":"https://api.github.com/users/DTAIEB/orgs","repos_url":"https://api.github.com/users/DTAIEB/repos","events_url":"https://api.github.com/users/DTAIEB/events{/privacy}","received_events_url":"https://api.github.com/users/DTAIEB/received_events","type":"User","site_admin":false},"assignees":[{"login":"DTAIEB","id":4912140,"avatar_url":"https://avatars3.githubusercontent.com/u/4912140?v=4","gravatar_id":"","url":"https://api.github.com/users/DTAIEB","html_url":"https://github.com/DTAIEB","followers_url":"https://api.github.com/users/DTAIEB/followers","following_url":"https://api.github.com/users/DTAIEB/following{/other_user}","gists_url":"https://api.github.com/users/DTAIEB/gists{/gist_id}","starred_url":"https://api.github.com/users/DTAIEB/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/DTAIEB/subscriptions","organizations_url":"https://api.github.com/users/DTAIEB/orgs","repos_url":"https://api.github.com/users/DTAIEB/repos","events_url":"https://api.github.com/users/DTAIEB/events{/privacy}","received_events_url":"https://api.github.com/users/DTAIEB/received_events","type":"User","site_admin":false}],"milestone":{"url":"https://api.github.com/repos/ibm-watson-data-lab/pixiedust/milestones/24","html_url":"https://github.com/ibm-watson-data-lab/pixiedust/milestone/24","labels_url":"https://api.github.com/repos/ibm-watson-data-lab/pixiedust/milestones/24/labels","id":3188336,"number":24,"title":"1.1.9","description":null,"creator":{"login":"DTAIEB","id":4912140,"avatar_url":"https://avatars3.githubusercontent.com/u/4912140?v=4","gravatar_id":"","url":"https://api.github.com/users/DTAIEB","html_url":"https://github.com/DTAIEB","followers_url":"https://api.github.com/users/DTAIEB/followers","following_url":"https://api.github.com/users/DTAIEB/following{/other_user}","gists_url":"https://api.github.com/users/DTAIEB/gists{/gist_id}","starred_url":"https://api.github.com/users/DTAIEB/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/DTAIEB/subscriptions","organizations_url":"https://api.github.com/users/DTAIEB/orgs","repos_url":"https://api.github.com/users/DTAIEB/repos","events_url":"https://api.github.com/users/DTAIEB/events{/privacy}","received_events_url":"https://api.github.com/users/DTAIEB/received_events","type":"User","site_admin":false},"open_issues":0,"closed_issues":4,"state":"open","created_at":"2018-03-14T19:27:56Z","updated_at":"2018-03-14T21:21:29Z","due_on":null,"closed_at":null},"comments":4,"created_at":"2018-03-14T07:50:55Z","updated_at":"2018-03-15T06:59:23Z","closed_at":"2018-03-14T21:18:53Z","author_association":"NONE","body":"As a `data scientist`, I want to `use date time filed` so that `my data is visualised as a time series`\r\n\r\n### Expected behavior\r\n\r\nMy pyspark DF has schema:\r\n```\r\nroot\r\n |-- ts: timestamp (nullable = true)\r\n |-- metric: string (nullable = true)\r\n |-- usage: double (nullable = true)\r\n```\r\nI'd like to use it to visualise time series.\r\n\r\n### Actual behavior\r\n\r\nI'm not able to use `ts` filed which is indicated as:\r\n```\r\nts     unknown/unsupported\r\n```\r\n\r\nThe data frame is interpreted by pixiedust as:\r\n```\r\nField types:\r\nts: datetime64[ns]\r\nmetric: object\r\nusage: datetime64[ns]\r\n```\r\n\r\n![screen shot 2018-03-14 at 08 39 17](https://user-images.githubusercontent.com/824894/37389103-3da4615c-2763-11e8-96c1-a10456e86044.png)\r\n\r\n![screen shot 2018-03-14 at 08 40 04](https://user-images.githubusercontent.com/824894/37389122-55bbfeb2-2763-11e8-98aa-e8e6d5d6eb72.png)\r\n\r\n### Steps to reproduce the behavior\r\n\r\nThe original data set (parquet):\r\nhttps://ibm.box.com/shared/static/51kcmtifwjikzt2wkrch1s252os7tyy2.tgz\r\n\r\nSteps to prepare DF:\r\n```python\r\nfrom pyspark.sql.functions import to_utc_timestamp, avg\r\n\r\nmonitoringData = spark.read.parquet(\"wml-perf-data/wml-monitoring-metics.2018-03-13_12.14.14.parquet\")\r\n\r\nmetrics = monitoringData \\\r\n    .withColumn(\"ts\", to_utc_timestamp(monitoringData.timestamp, \"PST\")) \\\r\n    .groupBy(\"ts\", \"metric\").agg(avg(monitoringData.value).alias(\"usage\"))\r\n```"},"comment":{"url":"https://api.github.com/repos/ibm-watson-data-lab/pixiedust/issues/comments/373280345","html_url":"https://github.com/ibm-watson-data-lab/pixiedust/issues/644#issuecomment-373280345","issue_url":"https://api.github.com/repos/ibm-watson-data-lab/pixiedust/issues/644","id":373280345,"user":{"login":"rafalbigaj","id":824894,"avatar_url":"https://avatars0.githubusercontent.com/u/824894?v=4","gravatar_id":"","url":"https://api.github.com/users/rafalbigaj","html_url":"https://github.com/rafalbigaj","followers_url":"https://api.github.com/users/rafalbigaj/followers","following_url":"https://api.github.com/users/rafalbigaj/following{/other_user}","gists_url":"https://api.github.com/users/rafalbigaj/gists{/gist_id}","starred_url":"https://api.github.com/users/rafalbigaj/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rafalbigaj/subscriptions","organizations_url":"https://api.github.com/users/rafalbigaj/orgs","repos_url":"https://api.github.com/users/rafalbigaj/repos","events_url":"https://api.github.com/users/rafalbigaj/events{/privacy}","received_events_url":"https://api.github.com/users/rafalbigaj/received_events","type":"User","site_admin":false},"created_at":"2018-03-15T06:59:23Z","updated_at":"2018-03-15T06:59:23Z","author_association":"NONE","body":"Thanks!"}}
{"actor":{"display_login":"rafalbigaj"}}