{"action":"created","issue":{"url":"https://api.github.com/repos/rucio/rucio/issues/1730","repository_url":"https://api.github.com/repos/rucio/rucio","labels_url":"https://api.github.com/repos/rucio/rucio/issues/1730/labels{/name}","comments_url":"https://api.github.com/repos/rucio/rucio/issues/1730/comments","events_url":"https://api.github.com/repos/rucio/rucio/issues/1730/events","html_url":"https://github.com/rucio/rucio/issues/1730","id":376118077,"node_id":"MDU6SXNzdWUzNzYxMTgwNzc=","number":1730,"title":"REST API not returning full JSON","user":{"login":"ericvaandering","id":1672366,"node_id":"MDQ6VXNlcjE2NzIzNjY=","avatar_url":"https://avatars0.githubusercontent.com/u/1672366?v=4","gravatar_id":"","url":"https://api.github.com/users/ericvaandering","html_url":"https://github.com/ericvaandering","followers_url":"https://api.github.com/users/ericvaandering/followers","following_url":"https://api.github.com/users/ericvaandering/following{/other_user}","gists_url":"https://api.github.com/users/ericvaandering/gists{/gist_id}","starred_url":"https://api.github.com/users/ericvaandering/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ericvaandering/subscriptions","organizations_url":"https://api.github.com/users/ericvaandering/orgs","repos_url":"https://api.github.com/users/ericvaandering/repos","events_url":"https://api.github.com/users/ericvaandering/events{/privacy}","received_events_url":"https://api.github.com/users/ericvaandering/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":4,"created_at":"2018-10-31T19:07:14Z","updated_at":"2018-11-01T07:54:47Z","closed_at":null,"author_association":"CONTRIBUTOR","body":"Motivation\r\n----------\r\n\r\nIt seems like the Rucio REST API does not actually return true JSON. This is an e-mail from @vkuznet to me about this. The context is that CMS uses something called DAS to aggregate info across our data management system, data catalog, and other systems. It relies on REST APIs to do this in a reasonable fashion (not a client library for each system). Thoughts?\r\n\r\n\r\nIt seems Rucio APIs do not return proper JSON:\r\n\r\n```\r\ncurl -H \"X-Rucio-Auth-Token: $token\" http://cms-rucio-test.cern.ch/rses/\r\n\r\n{\"rse_type\": \"DISK\", \"id\": \"c7ed36b8906c4808b090b35976373f88\", \"region_code\": null, \"city\": null, \"updated_at\": \"Fri, 28 Sep 2018 19:11:56 UTC\", \"rse\": \"T2_US_Nebraska_Test\", \"created_at\": \"Fri, 28 Sep 2018 19:11:56 UTC\", \"ISP\": null, \"deleted\": false, \"time_zone\": null, \"longitude\": null, \"availability\": 7, \"deterministic\": true, \"volatile\": false, \"country_name\": null, \"latitude\": null, \"deleted_at\": null, \"continent\": null, \"ASN\": null, \"staging_area\": false}\r\n{\"rse_type\": \"DISK\", \"id\": \"1a8167d0f9c743749ecd3bf00d4fb04c\", \"region_code\": null, \"city\": null, \"updated_at\": \"Wed, 24 Oct 2018 12:06:07 UTC\", \"rse\": \"T3_US_NERSC\", \"created_at\": \"Wed, 24 Oct 2018 12:06:07 UTC\", \"ISP\": null, \"deleted\": false, \"time_zone\": null, \"longitude\": null, \"availability\": 7, \"deterministic\": true, \"volatile\": false, \"country_name\": null, \"latitude\": null, \"deleted_at\": null, \"continent\": null, \"ASN\": null, \"staging_area\": false}\r\n\r\n```\r\n\r\nand\r\n\r\n```\r\n\r\ncurl -H \"X-Rucio-Auth-Token: $token\" http://cms-rucio-test.cern.ch/accounts/\r\n{\"account\": \"das\", \"type\": \"USER\", \"email\": null}\r\n{\"account\": \"dciangot\", \"type\": \"USER\", \"email\": null}\r\n\r\n```\r\n\r\nYes, they return JSON dicts, but it is not single JSON document, and therefore\r\nit fails to be parsed. Here is example to prove it.\r\n\r\n```\r\ncurl -H \"X-Rucio-Auth-Token: $token\" http://cms-rucio-test.cern.ch/accounts/ > account.json\r\npython\r\nPython 2.7.5 (default, Jul 13 2018, 13:06:57)\r\n[GCC 4.8.5 20150623 (Red Hat 4.8.5-28)] on linux2\r\nType \"help\", \"copyright\", \"credits\" or \"license\" for more information.\r\nimport json\r\ndata = json.load(open(\"accounts.json\"))\r\nTraceback (most recent call last):\r\n File \"<stdin>\", line 1, in <module>\r\n File \"/usr/lib64/python2.7/json/__init__.py\", line 290, in load\r\n   **kw)\r\n File \"/usr/lib64/python2.7/json/__init__.py\", line 338, in loads\r\n   return _default_decoder.decode(s)\r\n File \"/usr/lib64/python2.7/json/decoder.py\", line 369, in decode\r\n   raise ValueError(errmsg(\"Extra data\", s, end, len(s)))\r\nValueError: Extra data: line 2 column 1 - line 9 column 1 (char 50 - 419)\r\n\r\n```\r\nI also tried to use HTTP header Content-Type: application/json and results are\r\nthe same.\r\n\r\nI don't want to crawl over strings (this is what APIs return now) and convert\r\nthem to JSON docs. Instead, I would [like]  a JSON document. For the record, the\r\nproper JSON would be something like (based on accounts API output):\r\n\r\n```\r\n[\r\n{\"account\": \"das\", \"type\": \"USER\", \"email\": null},\r\n{\"account\": \"dciangot\", \"type\": \"USER\", \"email\": null}\r\n]\r\n\r\n```\r\nIt is also desired that all APIs will use the same data-type, a list of dicts,\r\neven if one item in a list is present. This will allow to abstract parsing code\r\nmuch easily and make it API independent.\r\n\r\nBest,\r\nValentin.\r\n\r\nModification\r\n------------\r\n"},"comment":{"url":"https://api.github.com/repos/rucio/rucio/issues/comments/434961947","html_url":"https://github.com/rucio/rucio/issues/1730#issuecomment-434961947","issue_url":"https://api.github.com/repos/rucio/rucio/issues/1730","id":434961947,"node_id":"MDEyOklzc3VlQ29tbWVudDQzNDk2MTk0Nw==","user":{"login":"mlassnig","id":299743,"node_id":"MDQ6VXNlcjI5OTc0Mw==","avatar_url":"https://avatars2.githubusercontent.com/u/299743?v=4","gravatar_id":"","url":"https://api.github.com/users/mlassnig","html_url":"https://github.com/mlassnig","followers_url":"https://api.github.com/users/mlassnig/followers","following_url":"https://api.github.com/users/mlassnig/following{/other_user}","gists_url":"https://api.github.com/users/mlassnig/gists{/gist_id}","starred_url":"https://api.github.com/users/mlassnig/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mlassnig/subscriptions","organizations_url":"https://api.github.com/users/mlassnig/orgs","repos_url":"https://api.github.com/users/mlassnig/repos","events_url":"https://api.github.com/users/mlassnig/events{/privacy}","received_events_url":"https://api.github.com/users/mlassnig/received_events","type":"User","site_admin":false},"created_at":"2018-11-01T07:54:47Z","updated_at":"2018-11-01T07:54:47Z","author_association":"CONTRIBUTOR","body":"Hi,\r\nto explain, we stream the results from the database directly to the clients, that's why we do JSON streams.\r\n\r\nNow, if for a particular call you'd like to have a single JSON document, then we will have to add support for a client requesting MIME `application/json`. It's not really a problem, but the implications are of course that any wide-spread use of that will quickly exhaust the memory on our servers since the JSON has to be assembled there. For example, doing this for `list_replicas()` is a no go whereas for many other calls it would be ok."}}
{"actor":{"display_login":"mlassnig"}}