Since spring 2020, the human world seems to be exceptionally silent due to mobility reduction caused by COVID-19 pandemic. To better measure real-time decline of human mobility and changes of socio-economic activities in a timely manner, we constructed a silent index (SI) based on Google’s mobility data. We systematically investigated the relations between SI, new COVID-19 cases, government policy, and the level of economic development. Results showed a drastic impact of the COVID-19 pandemic on increasing SI. The impact of COVID-19 on human mobility varied significantly by country and places. Bi-directional causality between SI and the new COVID-19 cases was detected, with a lagging period of one to two weeks. The travel restriction and social policies could immediately affect SI in one week; however, could not effectively sustain in the long run. Underdeveloped countries are more affected by the COVID-19 pandemic.