How to (almost) win Kaggle competitions


Tip 1: RTFM


Tip 2: Know your measure


Tip 3: Know your data


Tip 4: What before how


Tip 5: Do local validation


Tip 6: Make fewer submissions


Tip 7: Do your research


Tip 8: Apply the basics rigorously



Tip 9: The forum is your friend


Tip 10: Ensemble all the things



  1. Can you elaborate what you mean in Tip 5 by stating “The main scenarios when you should skip local validation is when the data is too small …”. What I experienced is that with too little observations, the leaderboard becomes very misleading, so my intuition would be to use more local validation for small datasets, not less.


    1. Good point. What I was referring to are scenarios where local validation is unreliable.

      For example, in the Arabic writer identification competition (, each of the 204 writers had only two training paragraphs (all containing the same text), while the test/leaderboard instances were a third paragraph with different content. I tried many forms of local validation but none of them yielded results that were consistent with the leaderboard, so I ended up relying on the leaderboard score.


      1. Ah, thanks, that clarifies what you meant. The (currently still running) Africa Soil Property contest ( seems a bit similar. I won’t put much more energy into that contest, but I am curious how it will work out in the end, and what things will have worked for the winners (maybe not much except pure luck).


  2. Could you provide some tips on #3(‘Getting to Know your data’) with respect to best practice visualisations to gain insights from data – especially considering the fact that datasets always have a large number of features. Plotting feature vs. label graphs do seem to be helpful, but for a large number of features will be impractical. So how should one go about data analysis via visualisation?


    1. It really depends on the dataset. For personal use, I don’t worry too much about pretty visualisations. Often just printing some summary statistics works well.

      Most text classification problems are hard to visualise. If, for example, you use bag of words (or n-grams) as your feature set, you could just print the top words for each label, or the top words that vary between labels. Another thing to look at would be commonalities between misclassified instances — these could be dependent on the content of the texts or their length.

      – In the Greek Media Monitoring competition (, I found that ‘Despite being manually annotated, the data isn’t very clean. Issues include identical texts that have different labels, empty articles, and articles with very few words. For example, the training set includes ten “articles” with a single word. Five of these articles have the word 68839, but each of these five was given a different label.’ — this was discovered by just printing some summary statistics and looking at misclassified instances
      – Looking into the raw data behind one of the widely-used sentiment analysis datasets, I found an issue that was overlooked by many other people who used the dataset: (look for the comment with my name — found four years after the original dataset was published)

      I hope this helps.


      1. Thanks a lot,
        So to summarize the following could be 3
        1. Using summary statistics such as means/stds/variance on the data, looking out for outliers,etc in the data
        2. Looking at misclassified instances during validation to find some sort of pattern in them
        3. Looking at label-specific raw data
        I apologize for the long overdue response, and thanks for these tips. This will surely be useful in my next Kaggle competition.


  3. I’m starting to dive into Kaggle competitions right now, and I’m having trouble with some of the simple practical considerations. For example, what IDE should I be using – IPython Notebooks? Where should I store the data? My personal computer surely doesn’t have 50GB+ space to spare. How long should I wait for a script to run before I deem it as “broken”?

    Any advice here would be greatly appreciated!


    1. Thanks for your comment, Derek. These are all good questions, but the answers really depend on the problem you’re working on and what you’re comfortable with.

      Personally, I find IPython Notebooks useful for playing around with the data and for documenting/storing throwaway code. However, once you have code that is more complex or code that you want to rerun, it’s better to save it in separate files. For editing these files, I use PyCharm.

      Not all problems require a large hard drive. If you are working on a large dataset, you can either purchase an external drive, or hire an instance from a cloud provider like AWS or DigitalOcean. The latter is generally cheaper than AWS, but they don’t offer GPUs. If you are working with a remote server, you can run IPython Notebook on the server and work from your browser.

      Regarding waiting for a script, for many models you can first build a simple version to test that everything works (e.g., build a random forest with just a few trees or train a neural network for a few iterations). If everything works well, you can run the full version. If the run time is long, it’s a good idea to take snapshots of the model and monitor performance on a hold-out set to ensure that you’re not wasting time overfitting.


      1. Thanks for the quick response! I think I’ll be using Sublime Text on EC2 with S3 in the short term, and possibly move onto Amazon ML with RedShift in the future. I’ll probably take snapshots by outputting results to the console or MatPlotLib every once in awhile, so that’s great advice as well.


Leave a Reply


You are commenting using your account. Log Out /  Change )

Google photo


Twitter picture


Facebook photo


Connecting to %s
