Water conservation is critical because it is one of the most significant factors for all species' survival. According to the Central Ground Water Board's estimates, underground water will be scarce in 15 Indian states by 2025 . However, as per the report from Ministry of Environment and Forest, only 10 to 20% rainwater is harvested. In urban areas, it is seen that nowadays concrete structures are constructed everywhere, so it is not possible to recharge the water table by natural process of raining. All the rainwater which is one of the sources for fresh water goes to drain. This causes the depletion of groundwater level which may be fatal in the future. Therefore, a rainwater harvesting system is required in Indian household to conserve water. On the other hand, the urban cities are having high pollutant levels which in turn affecting the quality of water. The government has defined certain parameters for quality checking of water. With the aid of the suitable sensors and IoT boards, quality checking, monitoring and water segregation can be easily achieved. In this work, an efficient and cost effective technique of rainwater harvesting has been presented. Some real case studies have been done at Indian houses in order to validate the effectiveness of system in terms of its implementation and cost analysis. As a result, the proposed system becomes one of the smart solutions to address the modern problem of rainwater harvesting with water quality management in low cost.