Endoscopic images are generally corrupted by various artefacts, such as blur caused by the irregular motion of endoscopists or organ peristalsis, preventing endoscopists from performing efficient and high-accuracy diagnosis and limiting the utility of computer-aided diagnosis in endoscopy. As a result, an endoscopy artefact detection algorithm that is both robust and reliable is critical in endoscopic imaging. This study proposes an endoscopy artefact detection method based on an improved Faster Region-based Convolutional Neural Network (Faster R-CNN) detector, simply named Faster RFD3-CNN. We introduce deformable convolutions and a feature pyramid network (FPN) to strengthen the model's adaptability and feature extraction ability. Furthermore, class-aware non-maximum suppression (NMS) and false positive elimination are utilized to filter out some bounding boxes with low confidence or overlap. The experimental results on the public access EndoCV2020 data set show that the mean average precision (mAP) can reach 42.66% and the intersection of union (IoU) is 31.61%. The mAP is increased by nearly 15% compared with the existing methods. Compared with the basic Faster R-CNN, mAP and IoU are improved by 7.39% and 11.19%, respectively. To sum up, the proposed method in this paper has superior performance in endoscopy artefact detection.