{"action":"created","issue":{"url":"https://api.github.com/repos/younginnovations/aidstream/issues/61","repository_url":"https://api.github.com/repos/younginnovations/aidstream","labels_url":"https://api.github.com/repos/younginnovations/aidstream/issues/61/labels{/name}","comments_url":"https://api.github.com/repos/younginnovations/aidstream/issues/61/comments","events_url":"https://api.github.com/repos/younginnovations/aidstream/issues/61/events","html_url":"https://github.com/younginnovations/aidstream/issues/61","id":165019457,"number":61,"title":"Improved Activity CSV import.","user":{"login":"RajbanshiNeha","id":12490586,"avatar_url":"https://avatars.githubusercontent.com/u/12490586?v=3","gravatar_id":"","url":"https://api.github.com/users/RajbanshiNeha","html_url":"https://github.com/RajbanshiNeha","followers_url":"https://api.github.com/users/RajbanshiNeha/followers","following_url":"https://api.github.com/users/RajbanshiNeha/following{/other_user}","gists_url":"https://api.github.com/users/RajbanshiNeha/gists{/gist_id}","starred_url":"https://api.github.com/users/RajbanshiNeha/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/RajbanshiNeha/subscriptions","organizations_url":"https://api.github.com/users/RajbanshiNeha/orgs","repos_url":"https://api.github.com/users/RajbanshiNeha/repos","events_url":"https://api.github.com/users/RajbanshiNeha/events{/privacy}","received_events_url":"https://api.github.com/users/RajbanshiNeha/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/younginnovations/aidstream/labels/Feature","name":"Feature","color":"fef2c0"}],"state":"open","locked":false,"assignee":{"login":"shresthaboy","id":5945626,"avatar_url":"https://avatars.githubusercontent.com/u/5945626?v=3","gravatar_id":"","url":"https://api.github.com/users/shresthaboy","html_url":"https://github.com/shresthaboy","followers_url":"https://api.github.com/users/shresthaboy/followers","following_url":"https://api.github.com/users/shresthaboy/following{/other_user}","gists_url":"https://api.github.com/users/shresthaboy/gists{/gist_id}","starred_url":"https://api.github.com/users/shresthaboy/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/shresthaboy/subscriptions","organizations_url":"https://api.github.com/users/shresthaboy/orgs","repos_url":"https://api.github.com/users/shresthaboy/repos","events_url":"https://api.github.com/users/shresthaboy/events{/privacy}","received_events_url":"https://api.github.com/users/shresthaboy/received_events","type":"User","site_admin":false},"assignees":[{"login":"shresthaboy","id":5945626,"avatar_url":"https://avatars.githubusercontent.com/u/5945626?v=3","gravatar_id":"","url":"https://api.github.com/users/shresthaboy","html_url":"https://github.com/shresthaboy","followers_url":"https://api.github.com/users/shresthaboy/followers","following_url":"https://api.github.com/users/shresthaboy/following{/other_user}","gists_url":"https://api.github.com/users/shresthaboy/gists{/gist_id}","starred_url":"https://api.github.com/users/shresthaboy/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/shresthaboy/subscriptions","organizations_url":"https://api.github.com/users/shresthaboy/orgs","repos_url":"https://api.github.com/users/shresthaboy/repos","events_url":"https://api.github.com/users/shresthaboy/events{/privacy}","received_events_url":"https://api.github.com/users/shresthaboy/received_events","type":"User","site_admin":false},{"login":"sachit-singh","id":8382151,"avatar_url":"https://avatars.githubusercontent.com/u/8382151?v=3","gravatar_id":"","url":"https://api.github.com/users/sachit-singh","html_url":"https://github.com/sachit-singh","followers_url":"https://api.github.com/users/sachit-singh/followers","following_url":"https://api.github.com/users/sachit-singh/following{/other_user}","gists_url":"https://api.github.com/users/sachit-singh/gists{/gist_id}","starred_url":"https://api.github.com/users/sachit-singh/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sachit-singh/subscriptions","organizations_url":"https://api.github.com/users/sachit-singh/orgs","repos_url":"https://api.github.com/users/sachit-singh/repos","events_url":"https://api.github.com/users/sachit-singh/events{/privacy}","received_events_url":"https://api.github.com/users/sachit-singh/received_events","type":"User","site_admin":false}],"milestone":{"url":"https://api.github.com/repos/younginnovations/aidstream/milestones/3","html_url":"https://github.com/younginnovations/aidstream/milestones/Sprint-1","labels_url":"https://api.github.com/repos/younginnovations/aidstream/milestones/3/labels","id":1906547,"number":3,"title":"Sprint-1","description":null,"creator":{"login":"RajbanshiNeha","id":12490586,"avatar_url":"https://avatars.githubusercontent.com/u/12490586?v=3","gravatar_id":"","url":"https://api.github.com/users/RajbanshiNeha","html_url":"https://github.com/RajbanshiNeha","followers_url":"https://api.github.com/users/RajbanshiNeha/followers","following_url":"https://api.github.com/users/RajbanshiNeha/following{/other_user}","gists_url":"https://api.github.com/users/RajbanshiNeha/gists{/gist_id}","starred_url":"https://api.github.com/users/RajbanshiNeha/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/RajbanshiNeha/subscriptions","organizations_url":"https://api.github.com/users/RajbanshiNeha/orgs","repos_url":"https://api.github.com/users/RajbanshiNeha/repos","events_url":"https://api.github.com/users/RajbanshiNeha/events{/privacy}","received_events_url":"https://api.github.com/users/RajbanshiNeha/received_events","type":"User","site_admin":false},"open_issues":4,"closed_issues":0,"state":"open","created_at":"2016-07-27T07:21:17Z","updated_at":"2016-08-04T04:27:02Z","due_on":null,"closed_at":null},"comments":2,"created_at":"2016-07-12T08:17:15Z","updated_at":"2016-09-22T13:30:34Z","closed_at":null,"body":"The current Activity CSV import will be improved to mimic the current CSV converter (maintained by IATI secretariat), which will include number of major IATI elements. See the **Description** below.\r\n\r\nTasks\r\n=====\r\n- [x] 1. Design Templates and download the templates.\r\n- [x] 2. Validate the uploaded CSV file against their types.\r\n- [x] 3. Validate each activity.\r\n- [x] 4. Validate other elements.\r\n- [x] 5. Map CSV data into a database compatible format and write into a temporary json file.\r\n- [ ] 6. Validate duplications.\r\n- [x] 7. Importing CSV through a queue as a background process.\r\n- [ ] 8. Show lists of valid/invalid Results with options to select specific of them to import.\r\n- [x] 9. Import chosen valid activities into the database.\r\n- [ ] 10. Option to choose invalid rows, after which another CSV is generated through AidStream, with all chosen invalid rows.\r\n\r\nDescription\r\n========\r\n\r\nCurrent Activity CSV import feature is limited with limited number of fields, it doesn't include budget, transactions information. With the improved feature, we would like to add the provision for the importing of budget, transactions, and other fields depending upon the need of the users. \r\n\r\n## Template Structure\r\n\r\nTemplate will be custom based on the user's need. There will be 4 blocks that the users will be able to select. \r\n\r\n1. Basic fields (all mandatory fields like title, description, status, dates, sector, recipient country, recipient region, participating organisations)\r\n2. Other fields (which includes other fields than basic info - e.g. policy marker)\r\n3. Budget\r\n4. Transactions\r\n\r\nTemplate will also support multiple rows for the same project/activity. \r\n\r\nTemplate supports only one language for all the fields for the given activity. \r\n\r\n### Basic Fields\r\n\r\n* Title\r\n* General Description \r\n* Objectives\r\n* Target Groups\r\n* Participating Organisation Type - Multiple\r\n* Participating Organisation Name - Multiple\r\n* Participating Organisation Identifier - Multiple\r\n* Participating Organisation Role- Multiple\r\n* Activity Status\r\n* Planned Start Date\r\n* Planned End Date\r\n* Actual Start Date\r\n* Actual End Date\r\n* Recipient Country Code - Multiple\r\n* Recipient Country Percentage - Multiple\r\n* Recipient Region Code - Multiple\r\n* Recipient Region Percentage - Multiple\r\n* Sector Vocabulary - Multiple\r\n* Sector Code - Multiple\r\n* Sector Percentage - Multiple\r\n\r\n### Other fields\r\n\r\n* Policy Marker Vocabulary - Multiple\r\n* Policy Marker Code - Multiple\r\n* Policy Marker Significance - Multiple\r\n* Activity Scope\r\n\r\n### Budget\r\n\r\n* Budget Type\r\n* Budget Status\r\n* Budget Period Start\r\n* Budget Period End\r\n\r\n### Transaction\r\n\r\n* Transaction Internal Reference\r\n* Transaction Type\r\n* Transaction Date\r\n* Transaction Value\r\n* Transaction Value Date\r\n* Transaction Description\r\n* Transaction Provider Organisation Identifier\r\n* Transaction Provider Organisation Type\r\n* Transaction Provider Organisation Activity Identifier\r\n* Transaction Provider Organisation Description\r\n* Transaction Receiver Organisation Identifier\r\n* Transaction Receiver Organisation Type\r\n* Transaction Receiver Organisation Activity Identifier\r\n* Transaction Receiver Organisation Description\r\n* Transaction Sector Vocabulary\r\n* Transaction Sector Code\r\n* Transaction Recipient Country Code\r\n* Transaction Recipient Region Code\r\n\r\n\r\n## Process\r\n\r\n1. Download template\r\n2. Fill the data as per template\r\n3. Upload excel file\r\n4. \"Processing Interface\" before importing into database - shows all the activities rows, show validation error for each activity row (e.g. if the used codes are not valid for the given element say country, wrong data type - using string instead of numbers, handling duplicated activities) and will have options to select the valid activity for importing. \r\n5. Select the rows to be imported\r\n6. Import\r\n7. Data imported into the system and redirect to activities page\r\n\r\n\r\n### Excel Data Validation\r\n\r\nValidation will check the followings\r\n* at least minimum fields are filled (title, general description, activity-status, recipient-country or recipient-region)\r\n* whether the valid codelist is used or not for the fields which takes codes\r\n\r\nValidation will not check for other rules like if the % for recipient-country is used or not, whether all the % sum to 100 or not and other element level validation. These will be handled in the activity publishing workflow. \r\n\r\n### Handling duplicated activities\r\n\r\nThere might be activities in the excel which is already entered in the database (by comparing the iati-activity-identifier). In step 4, the uploader will see that the activity is already entered. There are 3 actions that could be taken by the uploader.\r\n\r\n1. Ignore the duplicate activity (doesn't take action)\r\n2. Delete the activity in the database and insert the excel activity row as a new activity\r\n3. Update the existing activity in the database with the updated fields values\r\n\r\n### Handling huge number of activities and huge sized files\r\n\r\nHuge size files will also have huge number of activities. The importing task will be asynchronous and once the file is uploaded, it will be sent to the csv-processing-queue. The queue will process each row and shows the results in the step 4 (Processing Interface) one at a time. Once the processing is complete, the users will be able to select the valid rows to import and submit. There will be number of workers which will be able to handle multiple files at the same time. \r\n\r\nAlso the processing time might be long. The user can close the window and when the user goes to the file upload interface, s/he can see the results of last file processed. In case the user uploads new file, then the last uploaded file processed rows will be lost without getting imported into the database. Proper instructions will be shown accordingly."},"comment":{"url":"https://api.github.com/repos/younginnovations/aidstream/issues/comments/248903739","html_url":"https://github.com/younginnovations/aidstream/issues/61#issuecomment-248903739","issue_url":"https://api.github.com/repos/younginnovations/aidstream/issues/61","id":248903739,"user":{"login":"SJ-bond","id":8158574,"avatar_url":"https://avatars.githubusercontent.com/u/8158574?v=3","gravatar_id":"","url":"https://api.github.com/users/SJ-bond","html_url":"https://github.com/SJ-bond","followers_url":"https://api.github.com/users/SJ-bond/followers","following_url":"https://api.github.com/users/SJ-bond/following{/other_user}","gists_url":"https://api.github.com/users/SJ-bond/gists{/gist_id}","starred_url":"https://api.github.com/users/SJ-bond/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/SJ-bond/subscriptions","organizations_url":"https://api.github.com/users/SJ-bond/orgs","repos_url":"https://api.github.com/users/SJ-bond/repos","events_url":"https://api.github.com/users/SJ-bond/events{/privacy}","received_events_url":"https://api.github.com/users/SJ-bond/received_events","type":"User","site_admin":false},"created_at":"2016-09-22T13:30:34Z","updated_at":"2016-09-22T13:30:34Z","body":"Organisations that we work with have also requested that the activity template include elements such as document link, website and contact information, as these are required by their donors. Are these elements included in the update?"}}
{"actor":{"display_login":"SJ-bond"}}