Dental cavity represent one of the widespread illness of the tooth. Method for treating of the tooth is to drill the cavity and to fill the hole with suitable material. Measurements show that during drilling the tooth vibrates with increasing mass that causes unpleasant feeling for patient. The aim of the paper is to give the theoretical explanation for this phenomena and to give suggestion for vibration elimination. During drilling, mass of the tooth is decreasing and the so called ‘reactive force’ occurs. Drilling and reactive force cause tooth vibration. The system is modeled as a nonlinear time variable system. An analytical procedure for solving of the equation of vibration is developed. The solution is assumed in the form of the generalized trigonometric function with time variable amplitude and phase. It is obtained that not only the amplitude but also the frequency of tooth vibration in drilling are increased. In addition to reactive force the drilling velocity, diameter of the drill tool and spindle speed affect the vibration level. The appropriate values of these parameters would eliminate or decrease the patient bad feeling.