Background: Without reference annotation, statistical metrics such as sensitivity and positive predictive value (PPV) cannot be calculated. Annotating a large ECG database may not be feasible, hence, the interest in developing an evaluation tool that does not require reference annotation. Methods: We developed a tool for evaluating key performance attributes (KPA) including arrhythmia detection, heart-rate, ST value, and noise tolerance. The tool has three layers of KPA graphics. The top layer includes interactive distribution graphs of the KPA values for an overview of aggregated results for the entire database. From this top layer the user can select an individual record to launch interactive trending graphs that display the KPA values, or their discrepancies, for a time span on that particular record. From this second layer the user can identify any KPA value of interest (e.g., a specific arrhythmia label) to view the underlying ECG waveform. Navigating through these three layers, the user is able to quickly confirm the validity of KPA reported by the algorithm. Results: We modified the noise tolerance of an exercise ECG arrhythmia algorithm. Then used this tool to visually verify the resulting improvement on the Telemetric and Holter ECG Warehouse (THEW) stress database E-OTH-12-0927-015 (n=927). To confirm the visual verification, we annotated the 20 records in this database for which the algorithms had reported the most PVCs. The corresponding PPV for these records had indeed increased from 70% to 90%, while maintaining 95% sensitivity, mainly due to higher tolerance of motion artifact.
Conclusions: Our graphical tool can be used to accurately evaluate the performance of algorithms on an unannotated database with a fraction of effort needed for annotating the database. It could also be a handy clinical research tool for quick evaluation of a live growing database where reference annotation is not yet obtained.