Take the 2-minute tour ×

I'm the data guy at a small startup in Austin. All of the analysis that I do (thus far) is stored as a set of ad hoc scripts that I run from my laptop. This is a bad idea.

I'm going to sketch my plans here to deploy my analysis, and I'd like to know if there are any glaring holes that I've missed, or anything else that I should be considering. I think the outline I have keeps things atomic enough so that I can plug things in when and where I need to, but also allows me to run one script pretty easily. A secondary (longer-term) goal would be to put a simple web front-end up that allows a user (i.e., employee at my company) to invoke scripts one at a time, see here: Very simple web service: take inputs, email results .

I want to deploy the scripts to a server, and I'm thinking of organizing my scripts into a set of python modules. Then, I want my batch script to look like:

import analysis

batch_dict = analysis.build_batch_dict()
assert sorted(batch_dict.keys()) = ['Hourly', 'Monthly', 'Nightly', 'Weekly']

scripts_to_run = analysis.what_batch() # get from command line?
results_directory = analysis.make_results_directory()
failures = {}
for script in scripts_to_run:
    except Exception as e:


I'll rewrite my analyses so that they are handled by a class.

class AnalysisHandler(object):
    def analyze():
    def export_results(some_directory):
    def failed(exception):
    def run_with_non_default_args(*args, **kwargs):
    def something_else_im_missing_now():

All of the scripts will be handled by something that inherits from AnalysisHandler.

The directory structure on the server will look like:

    batch.py (see above)
share|improve this question
somewhere there is a question in here... –  nathan hayfield Feb 14 '13 at 17:10
"I'm going to sketch my plans here to deploy my analysis, and I'd like to know if there are any glaring holes that I've missed, or anything else that I should be considering." Do I need a question mark? –  BenDundee Feb 14 '13 at 17:12
Funky permissions issue may throw an exception on making a directory. From a testing background, I think you are better off wrapping your for loop in a 'with open(output) etc' and writing results as you go along, vs storing them. In the event that the server crashes you'd at least like to know what ran and whether there was a possibility one of your scripts got stuck in an infinite something and chewed up all the ram, etc. –  pyInTheSky Feb 14 '13 at 17:23
not sure the entire scope or complexity of your tests, but leveraging the unittest framework from 'import unittest', may give you some added power in sorting which tests to run, provide logging, etc. –  pyInTheSky Feb 14 '13 at 17:26
Would you recommend any threading/parallelization in the handler? Or is that just inviting trouble? –  BenDundee Feb 14 '13 at 17:41

1 Answer 1

up vote 1 down vote accepted

As a formal answer:

  • Watch out for and try to catch the exception when creating files (you may get permissions issues), especially when you add a front end and want to perhaps write in a directory an employee can't modify

  • With your current setup, I'd suggest wrapping the loop inside the 'with' statement, keeping an open file handle, and flushing results as they come in. This allows you to keep track of progress to some extent, and also lets you know if your server crashes, whether your tests were running, and possibly if one of them caused the crash

  • You seem to be developing quite a bit of framework. While the unittest module is meant to test python code, it can certainly be leveraged to replace a lot of your framework (ie sorting tests, specifying which tests to run, logging, etc). It will also provide you with an easy way of attaching an interface and then marking tests for expected failures and the like.

  • As important as output is to have, it's also important to make it useful. You may want it printed in clean text, but if it started in say, a python dictionary, and you flattened it, add a comma to your log file, and dump the dict as a string so you can scoop that thing right back into python and manipulate data if need be.

  • going off the last point, take a look at json.dumps and json.loads, especially for working w/ logs and a web UI. javascript is friendly w/ that format, and you can save yourself a whole lot of work by keeping everything in a 'python-happy' format

  • Threading the tests isn't hard to add if you need it. If you know you have one task that takes a long time to run, or is IO intensive, maybe let that go spin off on its own. If you do think you are going to go that route, plan on it from the beginning and be aware of variables you may need to protect, or push all your results onto a queue instead of a dictionary to head off the problem of race conditions

  • Be aware of the resolution of timestamps, especially if you are loading a dictionary that uses a timestamp as a key o_O ---> just don't do it

  • I noticed the config function and that it accepted args and kwargs. If you are going to allow configuring of tests, either via ui or file, especially if it's a file, use json friendly format, you can do kwargs = json.loads( open(configfile,'r').read() ) and be super happy that you didn't have to write a parser or regex, or modify your code when you add a parameter.

share|improve this answer
points[-1]: Nice. I also like the idea of taking inputs to scripts as json. In regards to the amount of framework---I thought that was relatively LITTLE overhead. Leveraging unittest is a nice idea, though. –  BenDundee Feb 14 '13 at 18:27
Indeed, it's not much, but I find with testing that it always starts out small and then grows rather quickly. If you are working on coding up a new test, you can keep it in the same directory and just mark as ignore. I've done both. I was very happy w/ my homebrew framework, especially since I did it before unittest added a lot of features in the python 2.7 release, but looking at the module now, I see the benefits of using a 'known' framework, especially if someone other than you will ever have to use it later on. –  pyInTheSky Feb 14 '13 at 18:51

Your Answer


