ECG annotation and arrhythmia algorithm evaluation: beat and rhythm labels, PhysioNet reference databases and ANSI/AAMI EC57 reporting

The short answer

To evaluate an arrhythmia algorithm the way device makers and FDA reviewers expect, you compare its output, beat by beat and episode by episode, against expert reference annotations on standard ECG databases. You then report the results using the method in ANSI/AAMI EC57, which FDA recognizes. The standard databases are the MIT-BIH Arrhythmia, AHA, European ST-T, Noise Stress Test and CU databases. All except the AHA Database are free on PhysioNet; the AHA Database is available only from ECRI. The reference software for the beat-by-beat comparison is bxb, part of the free WFDB toolkit.

EC57 calls itself a recommended practice, not a performance standard. It tells you how to test and report, not what score to hit. Its own scope says database results are “uncertain indicators of performance in clinical settings.” That makes the quality of your own reference labels just as important as the benchmark.

Sources: ANSI/AAMI EC57:2012/(R)2020 (AAMI catalog page); ANSI/AAMI EC57:2012/(R)2020 (publisher preview pages), section 1.1; FDA recognized consensus standard 3-118; PhysioNet AHA Database page; bxb man page. This page is a summary for planning, not regulatory advice for a specific device.

Beat labels and the five AAMI classes

PhysioNet databases store reference labels as WFDB annotation files. Each annotation has a time (sample number), a type code and optional fields: subtype, channel, number and an auxiliary text string. Beat-type codes come from the MIT-BIH convention. For evaluation, they are collapsed into five classes:

The mapping above is the one coded in bxb. If SVEB statistics are not requested, the S codes are counted as N. A comment in the WFDB source notes that the AAMI recommended practice excludes records containing paced beats from its reporting requirements, and ecgeval ships record lists that leave them out. In the MIT-BIH database, four records include paced beats: 102, 104, 107 and 217.

Sources: PhysioBank annotations reference; WFDB source code (bxb.c); ecgeval; MIT-BIH Arrhythmia Database Directory.

Rhythm and non-beat labels

Rhythms are not labeled beat by beat. A rhythm change annotation (+) carries an auxiliary string that marks the start of the new rhythm, for example (N normal sinus rhythm, (AFIB atrial fibrillation, (AFL atrial flutter, (VT ventricular tachycardia, (SVTA supraventricular tachyarrhythmia, (B ventricular bigeminy and (VFL ventricular flutter. Ventricular fibrillation or flutter episodes can also be bracketed with [ and ]. Other non-beat codes mark signal quality changes (~), isolated QRS-like artifacts (|), ST changes (s), T-wave changes (T), non-conducted P waves (x) and free-text comments (").

EC57-style evaluation needs only beat annotations from the algorithm under test. Rhythm, VF, AF and ST annotations are needed only if you claim those features.

Sources: PhysioBank annotations reference; Evaluating ECG Analyzers (WFDB Applications Guide).

The reference databases

The WFDB guide lists the AHA, MIT-BIH, European ST-T, Noise Stress Test and CU databases as the ones EC38 and EC57 require. The supraventricular and AF databases are useful supplements for SVEB and AF claims. The AHA Database was built at Washington University in St. Louis with AHA sponsorship and NHLBI funding, and completed in 1985. PhysioNet's page states that it “has not been made available by ECRI for download from PhysioNet or any other site; it may be obtained only from ECRI.” PhysioNet hosts one sample record that was excluded from the database, plus converters that translate ECRI's formats into WFDB format.

Sources: MIT-BIH Arrhythmia Database (PhysioNet); PhysioNet AHA Database page; WFDB a2m man page; European ST-T Database; MIT-BIH Noise Stress Test Database; CU Ventricular Tachyarrhythmia Database; MIT-BIH Supraventricular Arrhythmia Database; MIT-BIH Atrial Fibrillation Database; Evaluating ECG Analyzers: databases.

What EC57 performance reporting looks like

Some rules are built into the reference tools:

• Matching window: a test beat matches a reference beat if their times are within 0.15 seconds (the bxb default).
• Learning period: by default, comparison starts 5 minutes into each record.
• No custom settings: the bxb options that change the start time or match window are flagged as a “non-standard comparison” that EC38 and EC57 do not permit.
• Reproducibility: the algorithm under test must write its own annotation file through an automated, disclosed procedure. The WFDB guide treats that file as the interface between developer and evaluator.
• Results by record: EC57 emphasizes record-by-record presentation as well as totals, and bxb writes its line-format reports in the layout of EC57's example tables.

The bxb man page says it is the reference implementation of the EC38 and EC57 beat-by-beat comparison and “must be used to obtain the beat-by-beat performance statistics cited in EC38 and EC57 in order to be in compliance with these standards.” The rxr tool handles runs, epicmp handles VF, AF and ST episodes, and ecgeval runs the full set and summarizes it.

Sources: bxb man page; Evaluating ECG Analyzers (WFDB Applications Guide); ANSI/AAMI EC57:2012/(R)2020 (publisher preview pages), section 1.1. The man pages cite the 1998 editions of EC38 and EC57; we have not checked the clause numbers against the 2012 text.

Noise stress testing

The Noise Stress Test Database adds calibrated recorded noise to two clean MIT-BIH records (118 and 119). After the first 5 minutes, two-minute noisy segments alternate with two-minute clean ones. Signal-to-noise ratios are 24, 18, 12, 6, 0 and −6 dB. Electrode motion artifact is used because it “can mimic the appearance of ectopic beats and cannot be removed easily by simple filters.” The reference labels are copies of the clean originals, so every extra or missed beat is the algorithm's error.

The WFDB guide notes that “EC38 specifies that performance on these records must be reported, although no specific performance levels are required.” The nst tool can generate more noisy records from any ECG record. This is useful for wearable signals, which carry more motion artifact than Holter tapes.

Sources: MIT-BIH Noise Stress Test Database; Evaluating ECG Analyzers: noise stress testing.

What FDA expects

FDA recognizes ANSI/AAMI EC57:2012 as consensus standard 3-118 (complete recognition, entered January 30, 2014). The recognition covers arrhythmia detectors and alarms, ST-segment monitors, cardiac monitors, electrocardiographs and ambulatory ECGs with analysis algorithms (21 CFR 870.1025, 870.2300, 870.2340 and 870.2800). The current AAMI edition is EC57:2012, reaffirmed in 2020.

For arrhythmia detectors, the special control under 21 CFR 870.1025 is FDA's 2003 guidance. It still cites EC57:1998 and recommends:

• testing the beat-detection algorithm according to EC57;
• evaluating with “at least two databases”;
• never using the development database to test and verify the device;
• having rhythms annotated “by at least three qualified cardiologists”;
• using real patient recordings, with simulated data only for arrhythmias too rare to collect;
• supplying validation information for any database not in EC57.

It also says a device tested only for specific arrhythmias should be labeled only for those.

Public 510(k) summaries show how this appears in practice. The summary for K151269, an ECG patch arrhythmia detector, states that its algorithm “has been tested in accordance with the requirements of the FDA recognized standard ANSI/AAMI EC57: 2012.” The summary for K173461, a wearable ECG recorder, reports validation on MIT-BIH, AHA and MIT-BIH AF data plus a private database “annotated using experienced medical professionals, using prospectively defined guidelines.” Search similar devices in the FDA 510(k) database.

Sources: FDA recognized consensus standard 3-118; FDA Class II Special Controls Guidance: Arrhythmia Detector and Alarm (2003); 21 CFR 870.1025; K151269; K173461; AAMI EC57 catalog page.

Limits and common pitfalls

• Benchmarks are not the real world. EC57 notes that the databases do not contain a representative sample of nonventricular arrhythmias, paced patients or typical artifacts, and that they have limited bandwidth. The WFDB guide warns it is “risky to extrapolate” to real-world performance, especially if the evaluation data were also used in development.
• Subject leakage. MIT-BIH records 201 and 202 come from the same subject. Keep them in the same partition in any subject-independent split.
• Prevalence. MIT-BIH was enriched for uncommon arrhythmias. PhysioNet says pooled results should not be read as prevalence estimates, and positive predictivity will not carry over to a screening population.
• Selective reporting. PhysioNet asks users to state which records were used, any exclusions and the class mapping. Unexplained record dropping or custom AAMI mappings make results impossible to compare.
• Labels that are not ground truth. Some PhysioNet sets ship automated beat labels: AFDB's beat files are uncorrected detector output, and CUDB's beat labels are described as “in no sense definitive.” Check before you use them for beat-level metrics.
• Wearable signals differ. Most of the standard records are two-channel Holter tapes. Single-lead patch, watch and ring ECGs have different leads, sampling rates and noise, so you will usually need your own annotated data.

Sources: ANSI/AAMI EC57:2012/(R)2020 (publisher preview pages), section 1.1; Evaluating ECG Analyzers: databases; MIT-BIH Arrhythmia Database; MIT-BIH Atrial Fibrillation Database; CU Ventricular Tachyarrhythmia Database.

Labeling your own ECG data: a checklist

• Write the label set down first. Use the MIT-BIH beat codes and rhythm strings, or map yours to them, so the data can be scored with the AAMI classes and the WFDB tools.
• Store labels as WFDB annotation files, or export to them, with the sample-level beat times.
• Use independent expert annotators plus adjudication. MIT-BIH used two cardiologists with consensus, and the European ST-T Database used two cardiologists with a coordinating center. FDA's 2003 guidance recommends at least three qualified cardiologists for rhythm annotation.
• Pre-label with a detector if you like, as MIT-BIH did, but have experts correct every beat. Record which labels were reviewed.
• Measure inter-rater agreement before adjudication, and report it.
• Split by subject. Keep development and test data separate, and never tune on the test set.
• Run bxb, rxr and epicmp with standard settings. Report per-record and gross results, with exclusions stated.

Sources: MIT-BIH Arrhythmia Database; European ST-T Database; FDA arrhythmia detector guidance; bxb man page. The checklist is our practice summary.

Key references

• Moody & Mark, IEEE Eng Med Biol Mag 2001;20(3):45–50: history and design of the MIT-BIH database.
• Evaluating ECG Analyzers (WFDB Applications Guide): G. Moody's step-by-step guide to EC38 and EC57 evaluations with WFDB.
• ANSI/AAMI EC57:2012/(R)2020 (AAMI catalog page): the standard itself (paid).

Related resources

• Medical data labeling
• Expert clinical labeling
• SaMD regulatory consulting
• Inter-rater agreement for medical data labeling
• Medical data annotation guideline template
• EEG and wearable data formats for annotation
• Sensor accuracy validation