Lithium niobate exhibits slow refractive index drift after temperature excursions typical in device manufacturing. Such drift can degrade performance in certain devices. We analyze the birefringence changes in a range of temperatures to measure the magnitude and rate of change. Experiments were conducted by first annealing congruently grown, magnesium doped, and lithium-enriched crystals at a temperatures between 95oC and 215oC. Subsequent optical evaluation was performed between 95oC and 122oC using a table-top apparatus. The sample was illuminated with incandescent polarized light and transmitted light was collected with a compact spectrometer after passing through an analyzer so fringes could be recorded. Careful fringe analysis allows a precise estimate of birefringence changes at a reference wavelength with precision < 10−6. The observed birefringence changes can be explained by assuming that existing lithium vacancies re-arrange around positively charged point defects via lithium vacancy migration. Computer simulations for a simple model reproduce the major observations such as magnitude of change, activation energy for relaxation and the stretched exponential nature of the change. Similar effects are expected in lithium tantalate and the results suggest ways to minimize the influence on device operation.