Aim: In this study, we proposed an end-to-end deep learning scheme for detecting atrial fibrillation in real-time patient monitoring. Methods: The scheme was constructed with a multi-layer CNN and a multi-layer LSTM. The multi-layer CNN was used for capturing ECG’s multi-granularity morphological features, while the LSTM was used for extracting the long-term temporal relationship. The output sequences of the LSTM layers were weighted by a self-attention module, and then processed by a dense network to obtain detection results. Thirty seconds multi-lead ECG data were sequentially fed into this model every five seconds for atrial fibrillation detection. The sequential detection results were smoothed to improve accuracy while dealing with continuous ECG waveform data. Results: The model was trained and tested on multiple open access ECG databases annotated by cardiologists. Training dataset consisted of ECG data segments with the same length of thirty seconds extracted from the original databases, while the testing dataset consisted of long term ECG data records. Episode and duration accuracies were calculated referring to the performance evaluation method of atrial fibrillation detection defined in the EC57 standard. With the proposed scheme, we got an episode F1 score of 85.73%, and a duration F1 score of 95.47% on the testing dataset. Conclusion: Our proposed scheme achieved a competitive performance for atrial fibrillation detection, and could be used in real-time patient monitoring.