Kullback–Leibler divergence of a prior from the truth
"Another interpretation of KL divergence is this: suppose a number X is about to be drawn randomly from a discrete set with probability distribution p(x). If Alice knows the true distribution p(x), while Bob believes (has a prior) that the distribution is q(x), then Bob will be more surprised than Alice, on average, upon seeing the value of X. The KL divergence is the (objective) expected value of Bob's (subjective) surprisal minus Alice's surprisal, measured in bits if the log is in base 2. In this way, the extent to which Bob's prior is "wrong" can be quantified in terms of how "unnecessarily surprised" it's expected to make him [Português: a medida em que a prévia de Bob está "errada" pode ser quantificada em termos do quão "desnecessariamente surpreso" espera-se fazê-lo]" (Wikipedia, 2014).
Claude Shannon. Available from < http://en.wikipedia.org/wiki/Claude_Shannon >. access on 19 October 2014.
Information theory. Available from < http://en.wikipedia.org/wiki/Information_theory >. Português: < http://translate.google.com/translate?hl=pt-BR&sl=en&tl=pt&u=http%3A%2F%2Fen.wikipedia.org%2Fwiki%2FInformation_theory >. access on 19 October 2014.
http://www.google.com.br/search?hl=pt-BR&q=orientation+community+%22information+theory%22&btnG=Pesquisar