This paper presents a leader-follower formation control of multiple mobile robots by position-based method using a fisheye camera. A fisheye camera has a wide field of view and recognizes a wide range of objects. In this paper, the fisheye camera is first modeled on spherical coordinates and a position estimation technique is proposed by using an AR marker. This paper furthermore presents a method for estimating the velocity of a leader robot based on a disturbance observer using the obtained position information. The proposed techniques are combined with a formation control based on the virtual structure. The effectiveness of the proposed method is demonstrated by simulation and experiments using two real mobile robots.