Introduction: Heart disease is the leading cause of death in the world, hence the fast and accurate classification of cardiac arrhythmias is vital. In this work, our team named "SharifHeart" developed a new method based on deep neural networks, to tackle the problem of the cardiac abnormality classification on ECG signals with either twelve, six, three, or two leads. Methods: The main network consists of different modules, each of which is rich enough to classify the ECG signals solely. However, they have diverse structures that offer them different capabilities and specialization to deal with different types of features and signal resolutions. The main factors that vary across modules are the size and the number of convolution kernels, the number of layers, and the place where skip connections or max-pooling layers are applied. The final network is an ensemble of these specialized modules which can be trained in an end-to-end manner. We further use module-dropout to reduce the strong dependency of the network to a specific module and encourage all of the modules to cooperate in the classification task. To increase the robustness of modules, we also perform different augmentation procedures such as random ECG pad and crop, adding low-frequency artifacts, and lead-dropout. Result: Our main method achieved a score of 0.64 on the 5-fold cross-validation of the training data but it failed to get a score on official validation data. Another naive approach got an average score of -0.18 on official validation data, placing us around 121 out of 143 entries in the unofficial phase of the challenge. Conclusion: Using an ensemble of different modules in the proposed model, provides specialized feature extractors that potentially can improve the classification accuracy.