Most asteroids with a diameter larger than ∼ 300 m are rubble piles i.e. consisting of more than one solid object. All asteroids are rotating but almost all asteroids larger than ∼ 300 m rotate with a period longer than 2.3 hours, which is the critical period where the centrifugal force equals the gravitational force. This indicates that there are nearly no adhesive interaction forces between the asteroid fragments. We show that this is due to the surface roughness of the asteroid particles which reduces the van der Waals interaction between the particles by a factor of 100 for micrometer sized particles and even more for larger particles. We show that surface roughness results in an interaction force which is independent of the size of the particles, in contrast to the linear size dependency expected for particles with smooth surfaces. Thus, two stone fragments of size 100 nm attract each other with the same non-gravitational force as two fragments of size 10 m.