Clinical Documents include clinical narratives, images, tables, charts, etc., that have substantial value to researchers in the medical domain. Nevertheless, this information is intermingled with Protected Health Information (PHI), posing confidentiality risks. This work offers an end-to-end de-identification framework for autonomously removing PHI from Scanned Clinical Document Images like discharge summaries, radiology reports, laboratory reports, etc. A synthetic dataset was created for experimental purposes containing 550 clinical document images collected from various patients of different hospitals. The de-identification framework includes two phases: (1) Annotation of Dataset and Training of Yolov3- DLA Deep learning model, which comprises of labeling of PHI regions in the clinical document Images using nine pre-defined categories and training Yolov3-DLA model. (2) De-identification identifies the regions of the Clinical Document Image using the Yolov3-Document Layout Analysis (Yolov3-DLA) trained model and verifies whether each region contains PHI fields that will be removed for anonymization. The proposed model, which uses Document layout analysis to identify the regions based on the structure of the contents and helps ease the identification process of PHI regions, achieved an F1 score of 97.21%. The proposed end-to-end framework is a robust de-identification solution for clinical narrative corpora. Any form of clinical narrative documents may be readily adapted to use this approach.