Imports: Can we add historical data to our nightly automation?



1 comment

  • Official comment
    Kristin Town

    Since it's historical data, it is static. Therefore, it essentially only needs to be processed once and then it's in the system. All files included in the automation profile - whatever data is included in the files on the SFTP - are processed nightly which means the larger the files or the more records each file contains, the longer the import process takes. This could negatively affect your site's performance during the automation window. This is one reason why we work with districts to import historical files separately. 

    The second reason is we don't include historical files as part of the nightly automation process is based on how the No Show records are managed. Most districts have enabled No Show settings in their nightly automation profile. This helps to make sure that inactive records are processed correctly and the current data shows accurately for your teachers and administrators. If you included the historical data one night just so that it would import and then removed that data from the files or stopped sending current data for some reason to the SFTP, the no show processing would take affect. Historical files would essentially be removed since you're not replacing them with another set of historical data.

    Say the task manager failed and didn't place current data on the SFTP. That would mean that all of your current data would be replaced with the historical data. It is essentially impossible to successfully import historical data nightly as part of the automation. 

    For this reason, we process historical data separately through manual imports. The files are placed in a separate folder on the SFTP and a ticket is submitted to to request that our data team process the import. 


    Please note, you don't have to create separate files for each historical academic year. It can all be in one file. Please let me know if you have any additional questions. Thank you, Electra!

    Comment actions Permalink

Please sign in to leave a comment.