Title: On Fairness Measurement for Generative Models. Abstract: Deep generative models have made significant progress in improving the diversity and quality of generated data. Recently, there has been increased interest in fair generative models. Fairness in generative models is important, as some bias in the sensitive attributes of the generated samples could have severe effects in applications under high-stakes settings (e.g., criminal justice, healthcare). In this work, we conduct, for the first time, an in-depth study on fairness measurement, a critical component to gauge the research progress of fair generative models. Our work makes two contributions. As our first contribution, we reveal that there exist considerable errors in the existing fairness measurement framework. We attribute this to the lack of consideration for errors in the sensitive attribute classifiers. Contrary to prior assumptions, even highly accurate attribute classifiers can result in large errors in fairness measurement, e.g., a ResNet-18 for Gender with $\sim$97% accuracy could still lead to 4.98% estimation error when measuring the fairness of a StyleGAN2 trained on the CelebA-HQ. As our second (major) contribution, we address this error in the existing fairness measurement framework by proposing a CLassifier Error-Aware Measurement (CLEAM). CLEAM applies a statistical model to take into account the error in the attribute classifiers, leading to significant improvement in the accuracy of fairness measurement. Our experimental results on evaluating fairness of state-of-the-art GANs (StyleGAN2 and StyleSwin) show CLEAM is able to significantly  reduce fairness measurement errors, e.g., by 7.78% for StyleGAN2 (8.68%$\rightarrow$0.90%), and by 7.16% for StyleSwin (8.23%$\rightarrow$1.07%)} when targeting the  Gender attribute. Furthermore, our proposed CLEAM has minimal additional overhead when compared to the existing baseline. Code and instructions to reproduce the results are included in Supplementary.