I don't know the quality of your spoken French, oldgit_gittinolder, but if it's not great, it's possible the clerk didn't understand your pronunciation of "moyenne" (it's actually "moyen" since café is masculine).  And perhaps the helpful woman behind you suggested you speak in your native English - which was understood.