divergence

In information geometry, a divergence is a kind of statistical distance: a binary function which establishes the separation from one probability distribution to another on a statistical manifold. The simplest divergence is squared Euclidean distance (SED), and divergences can be viewed as generalizations of SED. The other most important divergence is relative entropy (also called Kullback–Leibler divergence), which is central to information theory. There are numerous other specific divergences and classes of divergences, notably f-divergences and Bregman divergences (see ). Definition Given a differentiable manifold of dimension , a divergence on is a -function satisfying: for all (non-negativity), if and only if (positivity), At every point , is a positive-definite quadratic form for infinitesimal displacements from . In applications to statistics, the manifold is typically the space of parameters of a parametric family of probability distributions. Condition 3 means that defines an inner product on the tangent space for every . Since is on , this defines a Riemannian metric on .