{"action":"opened","issue":{"url":"https://api.github.com/repos/simonmichael/hledger/issues/388","repository_url":"https://api.github.com/repos/simonmichael/hledger","labels_url":"https://api.github.com/repos/simonmichael/hledger/issues/388/labels{/name}","comments_url":"https://api.github.com/repos/simonmichael/hledger/issues/388/comments","events_url":"https://api.github.com/repos/simonmichael/hledger/issues/388/events","html_url":"https://github.com/simonmichael/hledger/issues/388","id":169804689,"number":388,"title":"easier import command for deduplicating CSV etc.","user":{"login":"simonmichael","id":10144,"avatar_url":"https://avatars.githubusercontent.com/u/10144?v=3","gravatar_id":"","url":"https://api.github.com/users/simonmichael","html_url":"https://github.com/simonmichael","followers_url":"https://api.github.com/users/simonmichael/followers","following_url":"https://api.github.com/users/simonmichael/following{/other_user}","gists_url":"https://api.github.com/users/simonmichael/gists{/gist_id}","starred_url":"https://api.github.com/users/simonmichael/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/simonmichael/subscriptions","organizations_url":"https://api.github.com/users/simonmichael/orgs","repos_url":"https://api.github.com/users/simonmichael/repos","events_url":"https://api.github.com/users/simonmichael/events{/privacy}","received_events_url":"https://api.github.com/users/simonmichael/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/simonmichael/hledger/labels/A%20WISH","name":"A WISH","color":"f9d0c4"},{"url":"https://api.github.com/repos/simonmichael/hledger/labels/csv","name":"csv","color":"bfd4f2"}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2016-08-07T16:41:48Z","updated_at":"2016-08-07T16:41:48Z","closed_at":null,"body":"From mail list:\r\n\r\nUsing a checksum for deduplication won't handle identical CSV records \r\nwell, right ? Those are unlikely with our usual banks but I think quite \r\npossible if you consider CSV data generally.\r\n\r\nHere's my recent plan for hledger. Let reading CSV files work as it \r\ncurrently does, but add a separate import command which does some \r\nadditional things: append the converted entries to the main journal \r\nfile, and save a position marker in CSVFILE.lastimport. Also when \r\nreading, if the marker file is found, skip any CSV records before the \r\nmarked position.\r\n\r\nThis much is generic and could be used with any data format, but I think \r\nit makes sense mainly for \"import\" data which you want to move into a \r\nmain journal, and which is somewhat sequential, eg CSV, OFX QIF. It's \r\nwould be harder and is probably not needed for eg journal or timeclock \r\nfiles.\r\n\r\nFor CSV specifically, I'm thinking the position marker will be the last \r\nCSV record processed. It could be elaborated later to consider \r\ntimestamps, checksums etc. if needed."}}
{"actor":{"display_login":"simonmichael"}}