Low rank representation is a very effective method, which is often used in unsupervised learning and semi-supervised classification. Due to the presence of noise and nonlinearity structure in real data, low rank representation fails to obtain the global information of the data well, so it can not be able to get enough discrimination information. For this motivation, this paper proposes a novel nonlinear low rank representation by estimating optimal transformations (EOTLRR) which can obtain a better global structure of the dataset by increasing the correlation among samples that may belong to the same category. The proposed method establishes an objective function that maximizes the largest q-ky Fan norm of the covariance matrix of data samples through estimating optimal transformations. Alternative Conditional Expectation method is employed to maximize the conditional expectation of samples, which can increase the correlation among the samples belonged to the same category. Experimental results on five real datasets and a synthetic dataset show that our method has superior performance compared with other latest methods.