Dielectric properties (DPs), as a physical property, are widely used in real-time identification of normal or tumor tissue in vivo. Open-ended coaxial probe method is one of the most common modalities in measuring DPs. This technique secures its promising potential in early-stage skin cancer diagnosis for its non-invasive merit. Although various studies have been reported, systematic assessment is lacking to advance to clinical applications. Parameter interactions and detecting limitations remain unclear. In this study, we examine the minimum detectable size of tumor by using a three-layer skin model by simulation. DP changes under different tumor dimension, probe size, skin thickness, and cancer type were discussed. The smallest detecting size of BCC inside the skin is 0.5 mm radius × 0.1 mm; for SCC inside the skin is 1.4 mm × 1.3 mm in radius and height; the smallest distinguishing size of BCC is 0.6 mm × 0.7 mm in radius and height; for SCC is 1.0 mm × 1.0 mm in radius and height; for MM is 0.7 mm × 0.4 mm in radius and height. We also provide a detailed systematic evaluation of the parameters employed in the method for further applications.