The ONS found that companies with more than 50 employees had significantly higher output per worker than smaller firms between 2008 and 2014.
It also found that only the biggest companies - those employing more than 250 employees - still have lower productivity than they did in 2008.
Low productivity has been partly blamed for the low wage growth since 2008.
Separate reports from the ONS have showed UK productivity is lagging considerably behind the other industrialised countries in the G7.
The analysis only covered non-financial businesses, which account for about two thirds of the UK economy.
Firms with fewer than 10 workers have seen the fastest productivity growth since 2009.
The ONS said the relatively fast growth among the smallest businesses may be due to, "changes to working patterns and practices or as a consequence of some unproductive firms ceasing trading, sometimes referred to as the 'cleansing' effect of a downturn".
Productivity is better in bigger firms, according to analysis from the Office for National Statistics (ONS).