Spin qubits with long dephasing times are an essential requirement for the development of new quantum technologies and have many potential applications ranging from quantum information processing to quantum memories and quantum networking. Here we report a theoretical study and the calculation of the spin dephasing time of defect color centers for the negatively charged nitrogen vacancy center in diamond. We employ ab initio density functional theory to compute the electronic structure, and extract the dephasing time using a cumulant expansion approach. We find that phonon-induced dephasing is a limiting factor for T_2 at low temperatures, in agreement with recent experiments that use dynamical decoupling techniques. This approach can be generalized to other spin defects in semiconductors, molecular systems, and other band gapped materials.