Fabric surface flaw inspection is essential for textile quality control, and it is demanding to replace human inspectors with the automatic machine vision-based flaw inspection system. To alleviate the time-consuming problem of sparse coding in detecting phase, this work presents a real time fabric flaw inspection method by using grouped sparse dictionary. Firstly, the over-complete sparse dictionary is learned from normal fabric images; Secondly, the learned sparse dictionary is grouped into several sub-dictionaries by evaluating reconstruction error. Finally, the grouped dictionary is used to represent image and identify flaw regions as they cannot be represented well, leading to large reconstruction error. In addition, a non-maximum suppression algorithm is also proposed to reduce false inspection further. Experiments on various fabric flaws and real-time implementation on the proposed vision-based hardware system are conducted to evaluate the performance of proposed method. In comparison with other dictionary learning methods, the experimental results demonstrate that the proposed method can reduce the running time significantly and achieve a decent performance, which is capable of meeting the real-time inspection requirement without compromising inspection accuracy.