{"action":"closed","issue":{"url":"https://api.github.com/repos/os-data/eu-structural-funds/issues/112","repository_url":"https://api.github.com/repos/os-data/eu-structural-funds","labels_url":"https://api.github.com/repos/os-data/eu-structural-funds/issues/112/labels{/name}","comments_url":"https://api.github.com/repos/os-data/eu-structural-funds/issues/112/comments","events_url":"https://api.github.com/repos/os-data/eu-structural-funds/issues/112/events","html_url":"https://github.com/os-data/eu-structural-funds/issues/112","id":179239571,"number":112,"title":"Excel URL streamer","user":{"login":"cyberbikepunk","id":1629834,"avatar_url":"https://avatars.githubusercontent.com/u/1629834?v=3","gravatar_id":"","url":"https://api.github.com/users/cyberbikepunk","html_url":"https://github.com/cyberbikepunk","followers_url":"https://api.github.com/users/cyberbikepunk/followers","following_url":"https://api.github.com/users/cyberbikepunk/following{/other_user}","gists_url":"https://api.github.com/users/cyberbikepunk/gists{/gist_id}","starred_url":"https://api.github.com/users/cyberbikepunk/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cyberbikepunk/subscriptions","organizations_url":"https://api.github.com/users/cyberbikepunk/orgs","repos_url":"https://api.github.com/users/cyberbikepunk/repos","events_url":"https://api.github.com/users/cyberbikepunk/events{/privacy}","received_events_url":"https://api.github.com/users/cyberbikepunk/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/os-data/eu-structural-funds/labels/in%20progress","name":"in progress","color":"ededed"},{"url":"https://api.github.com/repos/os-data/eu-structural-funds/labels/pipeline","name":"pipeline","color":"c5def5"}],"state":"closed","locked":false,"assignee":{"login":"cyberbikepunk","id":1629834,"avatar_url":"https://avatars.githubusercontent.com/u/1629834?v=3","gravatar_id":"","url":"https://api.github.com/users/cyberbikepunk","html_url":"https://github.com/cyberbikepunk","followers_url":"https://api.github.com/users/cyberbikepunk/followers","following_url":"https://api.github.com/users/cyberbikepunk/following{/other_user}","gists_url":"https://api.github.com/users/cyberbikepunk/gists{/gist_id}","starred_url":"https://api.github.com/users/cyberbikepunk/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cyberbikepunk/subscriptions","organizations_url":"https://api.github.com/users/cyberbikepunk/orgs","repos_url":"https://api.github.com/users/cyberbikepunk/repos","events_url":"https://api.github.com/users/cyberbikepunk/events{/privacy}","received_events_url":"https://api.github.com/users/cyberbikepunk/received_events","type":"User","site_admin":false},"assignees":[{"login":"cyberbikepunk","id":1629834,"avatar_url":"https://avatars.githubusercontent.com/u/1629834?v=3","gravatar_id":"","url":"https://api.github.com/users/cyberbikepunk","html_url":"https://github.com/cyberbikepunk","followers_url":"https://api.github.com/users/cyberbikepunk/followers","following_url":"https://api.github.com/users/cyberbikepunk/following{/other_user}","gists_url":"https://api.github.com/users/cyberbikepunk/gists{/gist_id}","starred_url":"https://api.github.com/users/cyberbikepunk/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cyberbikepunk/subscriptions","organizations_url":"https://api.github.com/users/cyberbikepunk/orgs","repos_url":"https://api.github.com/users/cyberbikepunk/repos","events_url":"https://api.github.com/users/cyberbikepunk/events{/privacy}","received_events_url":"https://api.github.com/users/cyberbikepunk/received_events","type":"User","site_admin":false}],"milestone":null,"comments":12,"created_at":"2016-09-26T14:10:55Z","updated_at":"2016-09-29T12:34:18Z","closed_at":"2016-09-29T12:34:18Z","body":"Need to stream excel files into the pipeline from urls.\r\n\r\nSo @akariv : should i implement this as a service like so?\r\n \r\n> We start with the possibility to have URLs for files other than CSV.\r\n> e.g. https://raw.githubusercontent.com/openspending/os-fdp-adapters/master/os_fdp_adapters/excel-adapter/tests/test.xls\r\n> \r\n> Upon every URL, we hit the fdp-adapter endpoint:\r\n> http://next.openspending.org/fdp-adapter/convert?url=https://raw.githubusercontent.com/openspending/os-fdp-adapters/master/os_fdp_adapters/excel-adapter/tests/test.xls\r\n> \r\n>     If this file type is not supported, we get a 404 and we should proceed as before (assuming this is a csv/json file based on extension).\r\n\r\ncan't we just use the `iter` method of the `datapackage.Resource` class (cc @roll) ? \r\n\r\n```\r\n    def iter(self):\r\n        '''Lazily-iterates over rows in data.\r\n        This method is useful when you don't want to load all data in memory at\r\n        once.\r\n        Returns:\r\n            iter: An iterator that yields each row in this resource.\r\n        Raises:\r\n            ValueError: If the data isn't tabular, if the resource has\r\n                no data, or if its specified encoding is incorrect\r\n            IOError: If there was some problem opening the data file (e.g. it\r\n                doesn't exist or we don't have permissions to read it).\r\n        '''\r\n        result = None\r\n        inline_data = self.descriptor.get('data')\r\n        if self.local_data_path and os.path.isfile(self.local_data_path):\r\n            data_path_or_url = self.local_data_path\r\n        else:\r\n            data_path_or_url = self.remote_data_path\r\n\r\n        if inline_data:\r\n            inline_data = self._parse_inline_data()\r\n            result = iter(inline_data)\r\n        elif data_path_or_url:\r\n            dialect = self.descriptor.get('dialect', {})\r\n            parser_options = {}\r\n            if 'delimiter' in dialect:\r\n                parser_options['delimiter'] = dialect['delimiter']\r\n            if 'lineTerminator' in dialect:\r\n                parser_options['lineterminator'] = dialect['lineTerminator']\r\n            if len(dialect) > 0:\r\n                parser_options['constructor'] = tabulator.parsers.CSV\r\n\r\n            try:\r\n                table = tabulator.topen(data_path_or_url, headers='row1',\r\n                                        encoding=self.descriptor.get('encoding'),\r\n                                        parser_options=parser_options)\r\n                result = self._iter_from_tabulator(table, self.descriptor.get('schema'))\r\n            except tabulator.exceptions.TabulatorException as e:\r\n                msg = 'Data at \\'{0}\\' isn\\'t in a known tabular data format'\r\n                six.raise_from(ValueError(msg.format(data_path_or_url)), e)\r\n\r\n        if result is None:\r\n            if self.descriptor.get('path'):\r\n                # FIXME: This is a hack to throw an IOError when local data\r\n                # exists but couldn't be loaded for some reason. If \"path\"\r\n                # existed and there were no issues opening it, \"result\" would\r\n                # never be None.\r\n                raise IOError('Resource\\'s data couldn\\'t be loaded.')\r\n\r\n            raise ValueError('Resource has no data')\r\n\r\nreturn result\r\n```"}}
{"actor":{"display_login":"cyberbikepunk"}}