Most database research papers use synthetic data sets. That is, they use random-number generators to create their data on the fly. A popular generator is dbgen from the Transaction Processing Performance Council (TPC).
Why is that a problem?
- We
Most database research papers use synthetic data sets. That is, they use random-number generators to create their data on the fly. A popular generator is dbgen from the Transaction Processing Performance Council (TPC).
Why is that a problem?
- We end up working with simplistic models. If we consider the main table generated by dbgen, out of 17 columns, 7 have uniform distributions. This almost never happens with real data. Similarly, we often end up with attributes that are perfectly statistically independent. I believe that the randomly generated data is not be representative of the real data you find in real businesses.
- Because so many people use essentially the same data generators, we risk having solutions that are optimized for this particular synthetic data. This allows researchers to essentially cheat, sometimes unknowingly.
However, finding suitably large real data sets is difficult. My own research focus is on data warehousing. Most businesses are unwilling to share the data in their data warehouses. Even if they were willing to do so, sharing very large files is inconvenient.
Here are three moderately large data sets that I have used in my research:
- I found a table of wikileaks-related metadata on Google Fusion. By transforming it into a relational table, I was able to create a table with over a million rows. It is tiny by data warehousing standards, but the data is easily accessible.
- I like the Canadian Census from 1880. You have data about 4 million Canadians. The data is publicly available and convenient. See my paper in Information Sciences from 2011 for details on how we retrieved and