The multi-label classification of electrocardiography (ECG) is to distinguish multiple concurrent abnormalities in one record, which is helpful for the diagnosis of heart disease. In this paper, we proposed a deep multi-branch neural network for multi-label classification of ECG abnormalities. The algorithm consists of three branches. In the first branch, to classify morphological abnormalities such as ST-segment depression (STD) and ST-segment elevation (STE), we adopted a deep residual network with squeeze and excitation layers to learn the morphological features of 12-lead ECG records. In the second branch, to detect the RR-interval related abnormalities such as premature atrial contraction (PAC) and premature ventricular contractions (PVC), the QRS location sequence of each record was detected by the Pan-Tompkins algorithm and fed into a long short-term memory (LSTM) based network to learn the rhythm features. In the third branch, information such as sex and age was used. All features of the three branches were then concatenated and fed into a fully connected layer to predict the probability of each ECG abnormality. ECG abnormalities with a probability exceeding 0.5 are predicted to be present. The algorithm was trained and evaluated on the dataset of the 2020 Physionet Challenge and yielded a general class-weighted F-score (F_2 score) of 80% and a generalization of the Jaccard measure (G_2 score) of 59.5%. The result shows that the proposed method is effective for multi-label classification of ECG abnormalities.