non-degenerate energy levels where a bit flip requires an energy exchange with the bath, so it is typically much slower than phase flips, which only require an entropy exchange.

While the noise bias is one of many features which can colour a noise model, fault-tolerant protocols can be tailored to various other features. This research program thus naturally suggest an optimization cycle which combines

1.  Experimental noise characterization of device.

2.  Noise modeling.

3.  Fault tolerant protocol design tailored to model.

4.  Numerical benchmark of protocol.

The main message of this article is that the above optimization cycle is not viable, and that given access to a small quantum information processor, steps 1 and 4 could be combined into a single step: [*Experimental benchmark of fault-tolerant protocol*]{}. We are lead to this conclusion by three observations. First, experiments can only extract coarse information about the noise affecting the hardware. Second, the response of a fault-tolerant scheme depends strongly on the fine parameters of the noise. Third, the response of a fault-tolerant scheme to even simple noise models is computationally hard to predict. We will now elaborate on each of these observations.

A noise bias is but one of many features 