In this paper deferred correction method is applied to derive explicitly different arbitrary high order finite difference (FD) formulae for the numerical differentiation of analytic functions. With this approach, various first and second order FD formulae are given with error terms explicitly expanded as Taylor series of the analytic function. These lower order approximations are successively improved by one or two to give FD formulae of arbitrary high order. The new approach efficiently recovers all the existing FD formulae on uniformly spaced grid points and the standard backward, forward and central FD formulae which are usually only given heuristically in terms of formal power series of FD operators. Examples of new FD formulae suited for timestepping schemes are built, and a stable self-starting time-stepping scheme of order four is provided to illustrate the application of the new FD approach to time-stepping method. The new approach extends to unequally spaced grid points and allows to Gaussian quadratures with exact error terms while it can also be used to correct high order Newton-Côte quadratures to quadratures with weights all positive. Contrary to existing quadrature rules, the new approach allows to a better approximation of the integral of a function f on an interval [a,b] using arbitrary given interpolation nodes within this interval.