physionet-logo-kp6The PhysioNet/CinC Challenge

The annual PhysioNet/CinC Challenge – 2021:
Will Two Do? Varying Dimensions in Electrocardiography

2021 Challenge Summary

The electrocardiogram (ECG) is a non-invasive representation of the electrical activity of the heart. Although the twelve-lead ECG is the standard diagnostic screening system for many cardiological issues, the limited accessibility of twelve-lead ECG devices provides a rationale for smaller, lower-cost, and easier to use devices. While single-lead ECGs are limiting, reduced-lead ECG systems hold promise, with evidence that subsets of the standard twelve leads can capture useful information, and even be comparable to twelve-lead ECGs in some limited contexts. In 2017 we challenged the public to classify AF from a single-lead ECG, and in 2020 we challenged the public to diagnose a much larger number of cardiac problems using twelve-lead recordings. However, there is limited evidence to demonstrate the utility of reduced-lead ECGs for capturing a wide range of diagnostic information.

In this year’s Challenge, we ask the following question: ‘Will two do?’ This year’s Challenge builds on last year’s Challenge, which asked participants to classify cardiac abnormalities from twelve-lead ECGs. We are asking you to build an algorithm that can classify cardiac abnormalities from twelve-lead, six-lead, four-lead, three-lead, and two-lead ECGs. We will test each algorithm on databases of these reduced-lead ECGs, and the differences in performances of the algorithms on these databases will reveal the utility of reduced-lead ECGs in comparison to standard twelve-lead EGCs.

Competition Background

While planning CinC 2000, local hosts Roger Mark and George Moody proposed to organize an activity that would make effective use of their newly-established PhysioNet web site to stimulate rapid progress on an unsolved problem of practical clinical significance. A timely contribution of data made it possible to create the first PhysioNet/CinC Challenge, which attracted the attention of more than a dozen teams to the subject of detecting sleep apnea from the ECG. Their efforts were broadly successful, they discussed their findings at CinC 2000, and an annual tradition was born.

PhysioNet offers free access via the web to large and growing collections of recorded physiologic signals and related open-source software. Originally established under the auspices of the NIH’s National Center for Research Resources, PhysioNet has been funded since September 2007 under a cooperative agreement with the NIH’s National Institute of Biomedical Imaging and Bioengineering (NIBIB), and with the NIH’s National Institute of General Medical Sciences (NIGMS).

In complementary ways, PhysioNet and Computing in Cardiology catalyze and support scientific communication and collaboration between basic and clinical scientists. The annual meetings of CinC are gatherings of researchers from many nations and disciplines, bridging the geographic and specialty chasms that separate understanding from practice, while PhysioNet provides on-line data and software resources that support collaborations of basic and clinical researchers throughout the year. The annual PhysioNet/CinC Challenges seek to provide stimulating yet friendly competitions, while at the same time offering both specialists and non-specialists alike opportunities to make progress on significant open problems whose solutions may be of profound clinical value. The use of shared data provided via PhysioNet makes it possible for participants to work independently toward a common objective. At CinC, participants can make meaningful results-based comparisons of their methods; lively and well-informed discussions are the norm at scientific sessions dedicated to these challenges. Discovery of the complementary strengths of diverse approaches to a problem when coupled with deep understanding of that problem frequently sparks new collaborations and opportunities for further study.

A new challenge topic is announced each year on the Challenge page at PhysioNet. The PhysioNet team assembles and posts the raw materials needed to begin work. Usually, these raw materials consist of a collection of data to be analyzed; the analyses are provided for a subset of the data (the “learning set”) in each case, and the challenge is to analyze the remaining data (the “test set”).

What will be the topic of the next challenge? It might be image analysis, or simulation, or forecasting…. An ideal challenge problem is interesting, clinically important, and possible to study using available materials that have not been widely circulated previously. Moreover, there must be an objective way to evaluate the quality of a challenge entry (for an analysis problem, this usually means there must be a known set of correct analyses of the data, i.e., a “gold standard” against which entries can be compared). You are invited to submit your ideas for future challenge topics to the Challenge organizer, Gari Clifford; suggestions accompanying relevant data are particularly welcome.

PhysioNet/CinC Challenge Awards

Challenges are open to all. An important milestone for participants is the deadline for submitting abstracts for CinC, which is 15 April each year. Those wishing to qualify as official entrants, with eligibility for awards, must submit an abstract describing their work as well as an entry for scoring by about one week before the abstract deadline. A limited number of revised entries may be submitted between the abstract deadline and the final challenge deadline in early September. Eligibility for awards also requires participants to present their work in a scientific session of CinC. See the Challenge pages on PhysioNet for deadlines and rules for this year’s competition.

Most Challenges are presented as two events (often a narrowly-defined question and a more general one), or sometimes more.

An award is offered for the best solution obtained by any eligible participant.  To qualify for this prize, the solution must include computer code in open-source format.  Each entry should contain a file that defines the specific open-source license under which the software is available. See the Challenge Web page for more details on the problem and the submission process for each year’s challenge.  The winners receive their awards during the final plenary session on Wednesday afternoon. Follow the links below for details about previous challenges.

PhysioNet/CinC Challenge Award Winners

2021: Varying Dimensions in Electrocardiography

Overall Challenge Competition:

  • To be announced

Hackathon Competition:

  • N/A

Best Challenge Oral:

  • To be announced

Best Challenge Poster:

  • To be announced

2020: Classification of 12-lead ECGs 

Overall Challenge Competition:

  • 1st Place: Annamalai Natarajan, Yale Chang, Sara Mariani, Asif Rahman, Gregory Boverman, Shruti Vij, Jonathan Rubin, A Wide and Deep Transformer Neural Network for 12-Lead ECG Classification
  • 2nd Place: Zhibin Zhao, Hui Fang, Samuel D. Relton, Ruqiang Yan, Yuhong Liu, Zhijing Li, Jing Qin, David C. Wong, Adaptive lead weighted ResNet trained with different duration signals for classifying 12-lead ECGs 
  • 3rd Place: Zhaowei Zhu, Han Wang, Tingting Zhao, Yangming Guo, Zhuoyang Xu, Zhuo Liu, Siqi Liu, Xiang Lan, Xingzhi Sun, Mengling Feng, Classification of Cardiac Abnormalities From ECG Signals Using SE-ResNet

Hackathon Competition:

  • 1st Place: N/A
  • 2nd Place: N/A
  • 3rd Place: N/A

Best Challenge Oral:

  • 1st Place: David Assaraf, Jeremy Levy, Janmajay Singh, Armand Chocron and Joachim A. Behar, Classification of 12-lead ECGs using digital biomarkers and representation learning

Best Challenge Poster:

  • 1st Place: Guadalupe García Isla, Rita Laureanti, Valentina Corino and Luca Mainardi, ECG Morphological Decomposition for Automatic Rhythm Identification

2019: Early Prediction of Sepsis from Clinical Data:

Overall Challenge Competition:

  • 1st Place: James Morrill, Andrey Kormilitzin, Alejo Nevado-Holgado, Sumanth Swaminathan, Sam Howison, Terry Lyons, The Signature-based Model for Early Detection of Sepsis from Electronic Health Records in the Intensive Care Unit
  • 2nd Place: John Anda Du, Nadi Sadr, Philip de Chazal, A Comparison of Neural Network Approaches for Sepsis Prediction 
  • 3rd Place: Morteza Zabihi, Sepsis Prediction in Intensive Care Unit Using Ensemble of XGboost Models

Hackathon Competition:

  • 1st Place: Meicheng Yang, Hongxiang Gao, Xingyao Wang, Yuwen Li, Xing Liu, Jianqing Li, Chengyu Liu
  • 2nd Place: Jonathan Rubin, Yale Chang, Saman Parvaneh, Gregory Boverman
  • 3rd Place: John Anda Du, Miquel Alfaras, Naoki Nonaka, Inès Krissaane, Edwar Hernando Macias Toro, Matthieu Scherpf

Best Challenge Oral:

  • 1st Place: Marcus Vollmer, Christian F. Luz, Philipp Sodmann, Bhanu Sinha, Sven-Olaf Kuhn,Time-specific Metalearners for the Early Prediction of Sepsis

Best Challenge Poster:

  • 1st Place: Chloé Pou-Prom, Zhen Yang, Maitreyee Sidhaye, David Dai, Development of a Sepsis Early Warning Indicator

2018: You Snooze, You win:

Official results, as well as a paper describing the Challenge, are available on the Physionet pages. Top scores were achieved by:

  • Matthew Howe-Patterson, Bahareh Pourbabaee, and Frederic Benard (0.54) 
  • Guðni Fannar Kristjansson, Heiðar Már Þráinsson, Hanna Ragnarsdóttir, Bragi Marinósson, Eysteinn Gunnlaugsson, Eysteinn Finnsson, Sigurður Ægir Jónsson, Halla Helgadóttir, and Jón Skírnir Ágústsson (0.45) 
  • Runnan He, Kuanquan Wang, Yang Liu, Na Zhao, Yongfeng Yuan, Qince Li, and Henggui Zhang (0.43)

#An unofficial entry from Hongyang Li and Yuanfang Guan (who unfortunately missed the deadline to submit an abstract) achieved a score of 0.55.

2017:AF classification from a short single lead ECG recording

Morteza Zabihi, Ali Bahrami Rad, Aggelos K. Katsaggelos, Serkan Kiranyaz, Susanna Narkilahti, Moncef GabboujDetection of Atrial Fibrillation in ECG Hand-held Devices Using a Random Forest Classifier

Tomás Teijeiro, Constantino A. García, Paulo Félix, Daniel CastroArrhythmia Classification from the Abductive Interpretation of Short Single-lead ECG Records

Shreyasi Datta, Chetanya Puri, Ayan Mukherjee, Rohan Banerjee, Anirban Dutta Choudhury, Arijit Ukil, Soma Bandyopadhyay, Rituraj Singh, Arpan Pal, Sundeep
KhandelwalA Robust AF Classifier using Time and Frequency Features from Single Lead ECG Signal

Shenda Hong, Meng Wu, Yuxi Zhou, Qingyun Wang, Junyuan Shang, Hongyan Li, Junqing Xie
ENCASE: an ENsemble ClASsifiEr for ECG Classification Using Expert Features and
Deep Neural Networks

2016: Classification of Normal/Abnormal Heart Sound Recordings

1. Cristhian Potes*, Saman Parvaneh, Asif Rahman, Bryan Conroy, Daniel Schulman and John Ames
Hybrid Feature Aggregation for Detection of Abnormal Heart Sound
Philips Research, Cambridge, MA, USA

2. Morteza Zabihi*1, Ali Bahrami Rad2, Serkan Kiranyaz3, Moncef Gabbouj1 and Aggelos K. Katsaggelos4  PhysioNet/CinC Challenge: Normal/Abnormal PCG Classification using an Ensemble of Support Vector Machines
1Tampere University of Technology, Tampere, Finland 2University of Stavanger, Stavanger, Norway 3Qatar University, Doha, Qatar 4Northwestern University, Evanston, USA

3. Edmund Kay* and Anurag Agarwal Katsaggelos  Neural Networks and the Continuous Wavelet Transform for Classifying Heart Sounds
University of Cambridge, UK

2015: Reducing False Arrhythmia Alarms in the ICU

Overall Winners
Event 1. Filip Plešinger, Petr Klimeš, Josef Halámek, & Pavel Jurák from the Institute of Scientific Instruments of the CAS, Brno, Czech Republic
Event 2. Sibylle Fallet, Sasan Yazdani, and Jean-Marc Vesin from the Applied Signal Processing Group, Swiss Federal Institute of Technology, Lausanne, Switzerland
Event 1 Results
1. Filip Plešinger, Petr Klimeš, Josef Halámek, & Pavel Jurák
2.Vignesh Kalidas and Lakshman Tamil
3.Paula Couto, Ruben Ramalho and Rui Rodrigues
Event 2 Results
1.Sibylle Fallet, Sasan Yazdani, and Jean-Marc Vesin
2.Filip Plešinger, Petr Klimeš, Josef Halámek, & Pavel Jurák
3.Vignesh Kalidas and Lakshman Tamil

2014: Robust Detection of Heart Beats in Multimodal Data

Phase I Results
1. Marcus Vollmer, University of Greifswald, Greifswald, Germany
2. Urska Pangerc and Franc Jager, Faculty of Computer and Information Science, Ljubljana, Slovenia
3. Lars Johannesen, Jose Vicente, Christopher Scully, Loriano Galeotti and David Strauss, CDRH, US FDA, Silver Spring, MD, USA United States
4. Quan Ding, Yong Bai, Yusuf Erol, Rebeca Salas-Boni, Xiaorong Zhang, Lei Li and Xiao Hu, University of California, San Francisco, CA, USA
Phase II Results
1. Thomas De Cooman, Griet Goovaerts, Carolina Varon, Devy Widjaja and Sabine Van Huffel, KU Leuven, Department of Electrical Engineering-ESAT, STADIUS Center for Dynamical Systems, Signal Processing and Data Analytics, Belgium + iMinds, Medical IT Department, Belgium
2. Marcus Vollmer, University of Greifswald, Greifswald, Germany
3. Urska Pangerc and Franc Jager, Faculty of Computer and Information Science, Ljubljana, Slovenia
4. Filip Plesinger, Juraj Jurco, Josef Halamek and Pavel Jurak, Institute of Scientific Instruments of the ASCR, Brno, Czech Republic
Phase III Results
1. Alistair E W Johnson, Joachim Behar, Fernando Andreotti, Gari D Clifford and Julien Oster, University of Oxford, England, United Kingdom
2. Soo-Kng Teo, Bo Yang, Bart Hoeben, Dong Huang, Monterola Christopher and Yi Su, Institute of High Performance Computing, A*STAR, Singapore
3. Thomas De Cooman, Griet Goovaerts, Carolina Varon, Devy Widjaja and Sabine Van Huffel, Center for Dynamical Systems, Signal Processing and Data Analytics, Belgium + iMinds, Medical IT Department, Belgium
4. Jan Jakub Gierałtowski, Kamil Ciuchciński, Iga Grzegorczyk, Katarzyna Kośna, Mateusz Soliński and Piotr Podziemski, Warsaw University of Technology, Poland

2013: Noninvasive Fetal ECG

Events 1 and 2:
Maurizio Varanini, Gennaro Tartarisco, Lucia Billeci, Alberto Macerata, Giovanni Pioggia, and Rita Balocchi, CNR ICP, Pisa, Italy (first place; also third place in events 4 and 5)
Rui Rodrigues, Universidade Nova de Lisboa (third place, event 1)
Luigi Yuri Di Marco, Alberto Marzo, and Alejandro Frangi, University of Sheffield (third place, event 2)
Event 3:
Piotr Podziemski and Jan Gierałtowski, Warsaw University of Technology, Warsaw, Poland (first place; also second place in events 1 and 2)
Events 4 and 5:
Fernando Andreotti, Maik Riedl, Tilo Himmelsbach, Daniel Wedekind, Sebastian Zaunseder, Niels Wessel, and Hagen Malberg, Dresden University of Technology, Dresden, Germany (first place)
Jukka A Lipponen and Mika P Tarvainen, University of Eastern Finland (second place)

Joachim Behar, Julien Oster, and Gari Clifford, University of Oxford, UK, obtained the top scores in events 1 and 2 while participating unofficially.

2012: Predicting Mortality of ICU Patients

Event 1:
Alistair Johnson, Nic Dunkley, Louis Mayaud, Athanasios Tsanas, Andrew Kramer and Gari Clifford, University of Oxford, UK
Event 2:
Luca Citi and Riccardo Barbieri, MIT, Cambridge, Massachusetts, USA, and Massachusetts General Hospital, Boston, Massachusetts, USA

2011: Improving the Quality of ECGs Collected using Mobile Phones

Event 1:
Henian Xia, Gabriel Garcia, Joseph McBride, Adam Sullivan, Thibaut De Bock, Jujhar Bains, Dale Wortham, and Xiaopeng Zhao, University of Tennessee, Knoxville, Tennessee, USA (first place)
Gari Clifford, Daniel Lopez, Qiao Li, and Iead Rezek, University of Oxford, UK (second place)
Event 2:
Henian Xia, Gabriel Garcia, Joseph McBride, Adam Sullivan, Thibaut De Bock, Jujhar Bains, Dale Wortham, and Xiaopeng Zhao, University of Tennessee, Knoxville, Tennessee, USA (first place)
Benjamin Moody, MIT, Cambridge, Massachusetts (second place)
Event 3:
Dieter Hayn, Bernhard Jammerbund, Günter Schreier, AIT, Graz, Austria (first place)
Václav Chudáček, Lukáš Zach, Jakub Kužílek, Jiří Spilka and Lenka Lhotská, Czech Technical University, Prague, Czech Republic (second place)

2010: Mind the Gap!

Event 1 (general and open-source):
Rui Rodrigues, Universidade Nova de Lisboa, Portugal
Event 2 (general and open-source):
The team of Adam Sullivan, Henian Xia, Joseph McBride, and Xiaopeng Zhao, University of Tennessee, Knoxville, Tennessee, USA

2009: Predicting Acute Hypotensive Episodes

Event 1 (general and open-source):
Xiaoxiao Chen, Michigan State University, Lansing, Michigan, USA
Event 2 (general and open-source):
Jorge Henriques and Teresa Rocha, University of Coimbra, Coimbra, Portugal

2008: Detecting and Quantifying T-Wave Alternans

Giovanni Bortolan, ISIB-CNR, Padova, Italy, and
Ivaylo Christov, Bulgarian Academy of Sciences, Sofia, Bulgaria
Alexander Khaustov, Incart, St. Petersburg, Russia

Jubair Saieed, Bangladesh University of Engineering and Technology, Dhaka, Bangladesh, obtained the top score overall while participating unofficially in the open-source division.

2007: Electrocardiographic Imaging of Myocardial Infarction

Event 1:
Hamid SadAbadi, KN Toosi University, Teheran, Iran
Event 2:
Mohamed Mneimneh, Marquette University, Milwaukee, Wisconsin, USA
Event 3 (tie):
Masood Ghasemi, KN Toosi University, Teheran, Iran
Mohamed Mneimneh, Marquette University, Milwaukee, Wisconsin, USA
Hamid SadAbadi, KN Toosi University, Teheran, Iran
Event 4 (tie):
Mohamed Mneimneh, Marquette University, Milwaukee, Wisconsin, USA
Hamid SadAbadi, KN Toosi University, Teheran, Iran

2006: QT Interval Measurement

Division 1 (manual/semi-automated):
Mariano Llamedo Soria, Universidad Tecnologica Nacional FRBA, Buenos Aires, Argentina
Division 2 (automated, general):
Dieter Hayn, ARC Seibersdorf Research GmbH, Graz, Austria
Division 3 (automated, open-source):
Yuri Chesnokov, Cambridge University, Cambridge, UK

2005: The First Five Challenges Revisited

Richard Povinelli, Marquette University, Milwaukee, Wisconsin, USA
C Raab, University of Potsdam, Potsdam, Germany

2004: Spontaneous Termination of Atrial Fibrillation

Event A (general):
Dieter Hayn, ARC Seibersdorf Research GmbH, Graz, Austria
Event A (open-source):
Maurizio Varanini, CNR-ICP, Pisa, Italy
Event B (general and open-source):
Federico Cantini, CNR-ICP, Pisa, Italy

Simona Petrutiu, a member of the research group at Northwestern University that contributed the data, obtained the top scores in both events while participating unofficially.

2003: Distinguishing Ischemic from Non-Ischemic ST Changes

Philip Langley, University of Newcastle upon Tyne, Newcastle, UK

2002: RR Interval Time Series Modeling

Event 1:
DC Lin, Ryerson University, Toronto, Canada
Event 2:
Albert C-C Yang, National Yang-Ming University, Taipei, Taiwan

2001: Predicting Paroxysmal Atrial Fibrillation

Event 1:
Günther Schreier, ARC Seibersdorf, Graz, Austria
Event 2:
Günther Schreier, ARC Seibersdorf, Graz, Austria

Wei Zong, a member of the PhysioNet team at MIT, obtained the top score in event 2 while participating unofficially.

2000: Detecting Sleep Apnea from the ECG

Event 1:
Murray Jarvis, Caltech, Pasadena, California, USA
Event 2:
James McNames, Portland State University, Portland, Oregon, USA