AbstractA number of ballast water compliance monitoring devices (CMDs) have been made commercially available to verify the efficacy of ballast water management systems by quantifying the living organisms for both plankton size classes (≥50 μm and ≥10–<50 μm). This study aimed to examine whether new CMDs can provide a reliable indication of compliance regarding Regulation D-2 and to evaluate their performance for indicative analysis of organisms by assessing their accuracy (comparison to microscopy) and precision (comparison within measurement). Challenge fresh water samples were collected in four locations of Lake Ontario, Canada, whereas marine challenge water samples were collected around the Bay of Fundy, New Brunswick, Canada. Ballast water samples were collected from ships visiting several ports across Canada. Overall, accuracy was higher (>80%) in estimating organisms from prepared-challenge water (Ballast Eye and BallastWISE) than from ballast water samples (>70%) (B-QUA only). The sensitivity ranged from 50 to 100% for the ≥50 μm organism size class, whereas for the ≥10–<50 μm organism size class, it was higher for freshwater samples (>75%) than for marine samples (>50%). The performance of CMDs should be assessed under real-world conditions for a better understanding and to improve their use.