Dimensionality reduction has been explored to address the curse of dimensionality in high dimensional datasets of modern pattern recognition applications. In pattern recognition tasks, it is important to quantify how distinct two data samples are. Unsupervised metric learning serves for this purpose. In dimensionality reduction, a more adequate metric for a given dataset is implicitly learned. Principal Component Analysis is still the most used dimensionality reduction algorithm. Several modifications of this method have already been proposed as other algorithms belonging to the nonlinear class as well. However, all of them somehow rely on the Euclidean norm, which is known to fail in high dimensions and to be sensitive to outliers. So, in this paper, a new entropic approach was proposed, where the neighborhood of a data sample was mapped to an entropic space, where a stochastic divergence replaces the Euclidean. This approach was adopted to compute a new entropic covariance matrix that does not use inner product to estimate correlation between two features. A data sample neighborhood was mapped into an univariate Gaussian distribution and the statistical distance used was the Cauchy-Schwarz divergence. This new matrix was supplied to Principal Component Analysis classic algorithm. We compared the new method with existing linear and nonlinear algorithms. Using several real datasets, the comparison was made under two perspectives: cluster analysis and classification. Using a statistical test, it was possible to conclude that the new approach led to significant better results in both perspectives in comparison to all other algorithms considered.