View source for Physics/A/Shannon entropy

{{Subpages}}

==Simple information entropy==
{| class="wikitable floatright"
|+ <math>2^S=\Omega\leftrightarrow S =\log_2 \Omega</math>
!# coins <br><math>S</math><br>Entropy
!# messages <br><math>\Omega</math><br>Omega
|-
|<div class="center"><math>2</math></div>
|<math>4=2^2=2\cdot 2</math>
|-
|<div class="center"><math>3</math></div>
|<math>8=2^3=2\cdot 2\cdot 2</math>
|-
|<div class="center"><math>4</math></div>
|<math>16=2^4</math>
|-
|<div class="center"><math>5</math></div>
|<math>32=2^5</math>
|-
| colspan="2" |'''Table 1:''' ''More than two coins''<br><math>S</math>= entropy;&nbsp;<math>\Omega</math>= the number <br>of possible messages
|- 
|}
[[File:Entropy_flip_2_coins.jpg|left|thumb|180px|'''Fig. 1''': Entropy of 2 coins]][[File:Shannon entropy 5 coin illustration.svg|'''Figure 2'''. Elephant with 5 bits of entropy can give 3 bits to Bird and 2 bits to Rat.|thumb|left|175px]][[File:Shannon entropy coins and base 2.svg|'''Figure 3:''' Rat could display the same information using a single 4-sided [[wikt:special:permalink/63429864#Etymology_2|die.]]|thumb|250px]]
'''Figure 1'''  illustrates the fact that flipping two coins can yield four get four different outcomes (represented as four pairs of flipped coins.) Using ({{math|H}},{{math|T}}) to represent (heads,tails) we can list these four outcomes,<!--
--><ref>Figure 1 is currently being used by Wikipedias in ten other languages, and a brief visit to some of them serves to remind reader that humans communicate using a variety of alphabets: [[w:ar:Special:Permalink/53023917#تعريف|الصفحة_الرئيسية (Arabic)]]{{spaces|2}}&sdot;{{spaces|2}}[[w:cs:Special:Permalink/20018752|Česká (Chech)]]{{spaces|2}}&sdot;{{spaces|2}}s[[w:da:Special:Permalink/10657898#Møntkast|Dansk (Danish)]]{{spaces|2}}&sdot;{{spaces|2}}[[w:Special:Permalink/1033726547|English]]{{spaces|2}}&sdot;{{spaces|2}}[[w:eu:Special:Permalink/7845025|euskarazko (Basque)]]{{spaces|2}}&sdot;{{spaces|2}}[[w:fa:Special:Permalink/32775201|ویژه:آمار (Persian)]]{{spaces|2}}&sdot;{{spaces|2}}[[w:ko:Special:Permalink/29813671|코리안(Korean)]]{{spaces|2}}&sdot;{{spaces|2}}[[w:pa:Special:Permalink/527912|ਪੰਜਾਬੀ (Punjabi)]]{{spaces|2}}&sdot;{{spaces|2}}[[w:sr:Special:Permalink/23595094|језик (Serbian)]]{{spaces|2}}&sdot;{{spaces|2}}[[w:uk:Special:Permalink/32794321|Українці (Ukrainian)]]{{spaces|2}}&sdot;{{spaces|2}}[[w:zh:Special:Permalink/66791361|中文 (Chinese)]]</ref>

:{{spaces|3}} {{math|HH}}, {{math|HT}}, {{math|TH}}, and {{math|TT}},

We shall use the symbol {{math|&Omega;}} ([[w:Omega|Omega]]) to denote the number of messages that can be sent by these two coins:

:{{spaces|3}} {{math|&Omega; {{=}} 4 (outcomes.)}}

The popularity of this inspired me to seek other ways to use coins to understand what is known as [[w:Entropy (information theory)|Shannon's entropy]].  Table 1 extends Figure 1 to include the relationship between entropy <math>S</math> and the number of messages <math>\Omega</math> (called [[w:Omega|Omega]]) one might generate by displaying {{math|S}} coins as either "heads" or "tails".  But neither the table, the figure, nor even the formula,  

:{{spaces|3}}<math>S=log_2\Omega,</math>   

captures the full complexity associated with Shannon's information entropy.  A hint at this complexity can be seen in the following question:  

''If entropy is equivalent to the number of coins used to convey information, how should one deal with a "one-sided coin?''

Such a question must be viewed outside the scope of mathematics.  The fact that one rarely hears [[w:Shouting fire in a crowded theater|"fire!" in a crowded theater]] does not remove that word from the lexicon of how the audience should behave in a theater.  If coins<rev>i.e. a base-1 "alphabet"<nowiki></ref></nowiki> Shannon's entropy, <math>H,</math> was defined so coins contribute nothing to the entropy  

''Define S as simple entropy and note that it is the number of "coins" used.  Shanon's entropy, H, defined so that it also depends on the probability with which a coin is used.  Connect probability to frequency.''

<math>H\le S =N</math>  
===Additive property I===
'''Figure&nbsp;2''' illustrations how Elephant communicates with the animal kingdom every morning by displaying 5 coins, using his trunk to indicate where the sequence begins. As shown in '''Figure&nbsp;3''', Elephant asks Crow to display the first 3 coins and Rat to display the last two coins.  When acting on behalf of Elephant, Crow and Rat each point to the first coin in their message, and the animal kingdom understands that Crow is to be read first.  In this case Crow and Rat are <u>not</u> sending <u>independent signals</u>, but are instead following Elephant's instructions.  On the other hand, if Crow and Rat are act independently, Crow controls 3 bits of entropy, while Rat controls 2 bits.  The relationship between acting dependently and independently can be summarized as:

''Elephants total, or net, entropy <math>S_\text{Tot}</math> equals the sum of the entropies controlled by Crow and Rat, but the {{math|Total}} number of messages Elephant can send equals the product of what Crow and Rat could send if they act independently:''  

{{spaces|3}}<math>S_\text{Tot}=S_\text{Crow}+S_\text{Rat}</math> {{spaces|2}} <math>\Leftrightarrow</math> {{spaces|2}}<math>\Omega_\text{Tot}=\Omega_\text{Crow}\cdot\Omega_\text{Rat}</math>

===Entropy is independent of the "alphabet" or "language" used===
Neither our simple entropy <math>S</math>, nor the Shannon entropy <math>H</math> do does not distinguish between the "language" or "alphabets" one might use when sending messages.  Rat could equivalently send the messages by displaying two coins.<ref>...with the understanding that order matters: HT and TH are different messages.</ref>  Or messages could be expressed using binary numbers,<ref>00, 01, 10, 11 (with perhaps the understanding that 0 means "tails" and 1 denotes "heads"</ref> or even four Arabic numbers using the four-sided die shown in '''Figure&nbsp;3'''.

== First derivation of Shannon entropy ==
<math>H = \underbrace{\sum p_j log_2\left(p_j^{-1}\right)}_{\text{all possible messages}}\equiv -\underbrace{\sum p_j log_2\left(p_j\right)}_{\text{all possible messages}}</math>

===Expectation value reviewed===
* [[w:Expected_value]] [[w:simple:Weighted average]]

The expectation, or "average" value of the four integers: <math>\{1, 2, 2, 5\}</math>

<math>E(x)=\frac{1+2+2+5}{4}= 2.5 =\frac 1 4\cdot 1 + \frac 1 2\cdot 2 + \frac 1 4\cdot 5
</math>

The fractions  {{Math|1/4}},{{Math|1/2}}, and {{Math|1/4}}, refer to the probabilities of {{math|x}} being {{math|1}}, {{math|2}}, and {{math|5}}, respectively.  This permits us to write the expectation value as a sum involving all probabilities:

<math>E(x)= 1\cdot p_{(x=1)}+2\cdot p_{(x=2)}+5\cdot p_{(x=5)}\rightarrow \sum_x p_x\cdot x </math>

===Additive property revisited===
<math>-H_T= \sum_{k=1}^{N_T}p_k\log p_k,</math> where <math>N_T=N_1N_2,</math> and <math>p_k=p_\alpha p_\beta,</math>

where the logarithm's base is two: <math>\log p_k\equiv\log_2(p_k).</math>

<math>-H_T=\sum_{i=1}^{N_1}\sum_{i=j}^{N_2}(p_ip_k)\log (p_ip_k)</math>
<math>= \sum_{i=1}^{N_1}\sum_{i=j}^{N_2}p_ip_k\cdot\underbrace{\left(\log p_i+\log p_k\right)}_{\log p_ip_j}</math>
<math>= \sum_{i=1}^{N_1}\sum_{i=j}^{N_2}\left(p_ip_j\log p_i + p_ip_j\log p_j\right)</math>
<math> = \underbrace{\left(\sum_{i=j}^{N_2}p_j\right)}_{=1}\sum_{i=1}^{N_1}p_i\log p_i 
                    +\underbrace{\left(\sum_{i=1}^{N_1}p_i\right)}_{=1}\sum_{j=1}^{N_2}p_j\log p_j</math>
<math> = \sum_{i=1}^{N_1}p_i\log p_i +  \sum_{j=1}^{N_2}p_j\log p_j</math>
<math> = -H_1-H_2</math>

====log2====
<math>-H_T= \sum_{k=1}^{N_T}p_k\log_2 p_k,</math> where <math>N_T=N_1N_2,</math> and <math>p_k=p_\alpha p_\beta,</math>

where the logarithm's base is two: <math>\log_2 p_k\equiv\log_2(p_k).</math>

<math>-H_T=\sum_{i=1}^{N_1}\sum_{i=j}^{N_2}(p_ip_k)\log_2 (p_ip_k)</math>
<math>= \sum_{i=1}^{N_1}\sum_{i=j}^{N_2}p_ip_k\cdot\underbrace{\left(\log_2 p_i+\log_2 p_k\right)}_{\log_2 p_ip_j}</math>
<math>= \sum_{i=1}^{N_1}\sum_{i=j}^{N_2}\left(p_ip_j\log_2 p_i + p_ip_j\log_2 p_j\right)</math>
<math> = \underbrace{\left(\sum_{i=j}^{N_2}p_j\right)}_{=1}\sum_{i=1}^{N_1}p_i\log_2 p_i 
                    +\underbrace{\left(\sum_{i=1}^{N_1}p_i\right)}_{=1}\sum_{j=1}^{N_2}p_j\log_2 p_j</math>
<math> = \sum_{i=1}^{N_1}p_i\log_2 p_i +  \sum_{j=1}^{N_2}p_j\log_2 p_j</math>
<math> = -H_1-H_2</math>

===Wheel of fortune===
[[File:Entropy wheel of fortune.svg|thumb|Shannon's entropy for unfair coin|180px]]

<math>S_T=\log(n_C+n_R)</math><math>=  H_{CR} + \overbrace{\frac{n_C}{n_C+n_R}}^{p_C}\log n_C+ \overbrace{\frac{n_R}{n_C+n_R}}^{p_R}\log n_R</math>

<math>H_{CR} = \log(n_C+n_R)-\overbrace{\frac{n_C}{n_C+n_R}}^{p_C}\log n_C- \overbrace{\frac{n_R}{n_C+n_R}}^{p_R}\log n_R</math><math>= \log(n_C+n_R)-p_C\log n_C- p_R\log n_R</math>

Use fact that, <math>p_C+p_R = 1,</math> to write this as.

<math>H_{CR} </math><math>= (p_C+p_R)\log(n_C+n_R)-p_C\log n_C- p_R\log n_R</math>
<math> = p_C\left(\underbrace{\log(n_C+n_R)-\log n_C}_{\log(1/p_C)}\right)-
p_R\left(\underbrace{\log(n_C+n_R)-\log n_R}_{\log(1/p_R)}\right)</math>
<math> = p_C log(1/p_C)+p_R log(1/p_R)</math>

===Another path to Shannon entropy===
====Move====
Moved to [[Draft:Information theory/Permutations and combinations]]: <math>\text{If unlabeled: }\Omega = 5! = 120\,\text{outcomes}</math>, <math>\text{If labeled: }\Omega =\frac{(2+3)!}{2!\, 3!}=10\,\text{outcomes}</math>



[[File:5-take-3 Crows-A.svg|thumb|180px|center|5 choose 3 versus permutation of ordered set of five objects.]]

====Don't move====
Employ fact that <math>\Omega=2^S</math> is often a very large number, since {{math|S}} is often a large number.  A simplified version of Sterling's formula, <math>n!\approx n\ln n -n</math> becomes for logarithms to base two:

<math>log (n!)\approx n\log n - \frac{n}{\ln 2} </math> <math>log (n!)\approx n\log n - n/\ln 2 </math>

<math>n=n_C+n_R</math>

<math>\widetilde S = \log\left[\frac{(n_C+n_R)!}{n_C!n_R!}\right]</math>

<math>\approx (n_C+n_R)\log (n_C+n_R) -n_C\log n_C -n_R\log n_R</math>

<math>=  n_C\Bigl[ \log (n_C+n_R) -\log n_C\Bigr] + n_R\Bigl[ \log (n_C+n_R) - \log n_R\Bigr] </math>

<math>\widetilde S =  n_C\log\left(\frac{n}{n_C}\right)+n_R\log\left(\frac{n}{n_R}\right)\quad\text{if }n>>1 </math>

====table====
[[File:Entropy per coin fixed ratio 2 to 1.png|thumb|left|Information entropy for collection of one-sided coins to simulate random events]]
{| class="wikitable floatright"
|+ Entropy per coin <math>(n_2=2n_1)</math>
! align="center" style="background:#f0f0f0;"|<math>n_1+n_2</math>
! align="center" style="background:#f0f0f0;"|Excel
! align="center" style="background:#f0f0f0;"|Sterling
|-
| 3||0.5283||0.5739
|-
| 6||0.6511||0.6628
|-
| 12||0.7459||0.7489
|-
| 24||0.8120||0.8127
|-
| 48||0.8549||0.8551
|-
| 96||0.8814||0.8815
|-
| 192||0.8973||0.8973
|-
| 384||0.9065||0.9065
|-
| 768||0.9117||0.9117
|-
| 1536||||0.9147
|-
| 3072||||0.9163
|-
| 6144||||0.9172
|-
| 12288||||0.9177
|-
| 36864||||0.9181
|-
| 147456||||0.9182
|-
|
|
|0.9183
|}

====gamma from wikipedia====
[[w:special:permalink/1040737805#Versions_suitable_for_calculators]]

:<math>n! = \Gamma(n + 1),</math>

:<math> \sqrt{\pi} \left(\frac{x}{e}\right)^x \left( 8x^3 + 4x^2 + x + \frac{1}{100} \right)^{1/6} < \Gamma(1+x) <  \sqrt{\pi} \left(\frac{x}{e}\right)^x \left( 8x^3 + 4x^2 + x + \frac{1}{30} \right)^{1/6}.</math>

====See also====
*[[Permutation notation]] by [[user:Watchduck]]
*[[w:Permutation]]
*[[w:Combination]]
*[[w:Combinatorial proof]]
*[[w:Mathematical_proof]]
*[[b:Proof_by_Induction]]


<references />

==EVERYTHING BELOW THIS IS SCRAPBOOK==

===Images used in this draft===
<gallery>
Shannon entropy coins and base 2.svg
Entropy_flip_2_coins.jpg
Shannon entropy 5 coin illustration.svg
File:5-take-3 Crows-A.svg
File:5-take-3 Crows-B.svg
File:5-take-3 Crows-C.svg
File:Entropy wheel of fortune.svg
Entropy per coin fixed ratio 2 to 1.png
</gallery>
[[File:5-take-3 Crows-A.svg|thumb|right|190px|{{EquationRef|Fcute}}: I hope I can use this here.  There is no other page that might need it..]]
===Two coin examples===
<math>\begin{matrix}
\,      & TT                      &TH                        &HT                        &HH                 \\
H=       &p_1 \log_2(1/p_1)       &+p_2 \log_2(1/p_2)        &+p_3 \log_2(1/p_3)        &+p_4 \log_2(1/p_4)   \\
1=       &0                       &+0                        &+\frac 12\log_2 (2)       &+\frac 12\log_2 (2)   \\
1.918...=&\frac 13\frac 12\log_2(6)&+\frac 13\frac 12\log_2(6)&+\frac 23\frac 12\log_2(3)&+\frac 23\frac 12\log_2(3)\\ 
1.836...=&\frac 13\frac 13\log_2(9)&+\frac 13\frac 23\log_2(9/2)&+\frac 23\frac 13\log_2(9/2)                                                                                       
                                                                                       &+\frac 23\frac 23\log_2(9/4)\\
\end{matrix} </math>   

===additive property of shannon entropy===

<math>-\Sigma H =-(H_1+H_2)= 
\sum_{\alpha=1}^{N_1}p_\alpha\log p_\alpha +
\sum_{\beta=1}^{N_2}p_\beta\log p_\beta.</math>

In our example, the crow's entropy is <math>H_1</math>, and <math>N_1=8</math>.  The eight outcomes for the crow's three coins have probabilities:

<math>p_\alpha\in\{ p_1, p_2,p_3,p_4,p_5,p_6,p_7,p_8\}</math> for the outcomes <math>(000 ,001 , 010, 011, 100, 101, 111),</math> respectively.  Similarly, the set of Rat's four possible outcomes <math>p_\beta\in\{ p_1, p_2,p_3,p_4\}</math>.<ref>Here <math>\in</math> denotes that <math>p</math> is an element of a given set.  The use <math>\alpha</math> and <math>\beta</math> helps permits us to estabish that <math>p_\alpha=p_4</math> for Crow needs not be numerically equal to <math>p_\beta=p_4</math> for Rat.</ref>
====short====
<math>-H_T= \sum_{k=1}^{N_T}p_k\log p_k,</math> where <math>N_T=N_1N_2,</math> and <math>p_k=p_\alpha p_\beta.</math>
<math>H_T=\sum_{i=1}^{N_1}\sum_{i=j}^{N_2}(p_ip_k)\log (p_ip_k)</math>
<math>= \sum_{i=1}^{N_1}\sum_{i=j}^{N_2}p_ip_k\cdot\left(\log p_i+\log p_k)\right)</math>
<math>= \sum_{i=1}^{N_1}\sum_{i=j}^{N_2}\left(p_ip_j\log p_i + p_ip_j\log p_j\right)</math>
<math> = \sum_{i=j}^{N_2}p_j\sum_{i=1}^{N_1}p_i\log p_i +  \sum_{i=1}^{N_1}p_i\sum_{j=1}^{N_2}p_j\log p_j</math>
=== Compression image ===
[[File:Compression_Ephesos.jpg|center|80x80px|Students can explore image compression using Microsoft Paint. To a sequence of such images, copy an image into Microsoft Paint and successively reduce it's size by 50% N times. Then return each image back to its original size. In this example a coin from the jpeg Ephesos_620-600_BC.jpg was reduced a total of 6 times.]]

=== Image gallery ===
All images from the same randomized version of [[:file:Image entropy with Inkscape First 01.svg]]<gallery mode="packed" heights="40px" style="text-align:center">
Image entropy with Inkscape First 03.svg|svg
Image entropy with Inkscape First 05.png|png 
Image entropy with Inkscape First 02.svg|svg &times; 4
Image entropy with Inkscape First 04.png|png &times; 4
</gallery>{{multiple image|width=60|image1=Image entropy with Inkscape First 05.png|image2=Image entropy with Inkscape First 04.png|footer=64&times;64 randomized grid (100% and 25%)}}

=== blurbs ===
I thought of calling it "information", but the word was overly used, so I decided to call it "uncertainty". [...] Von Neumann told me, "You should call it entropy, for two reasons. In the first place your uncertainty function has been used in statistical mechanics under that name, so it already has a name. In the second place, and more important, nobody knows what entropy really is, so in a debate you will always have the advantage."


<math>
\overbrace{00}^{{}^{\prime\prime\,}\!1^{\,\prime\prime}}\;
\overbrace{00}^{{}^{\prime\prime\,}\!2^{\,\prime\prime}}\;
\overbrace{00}^{{}^{\prime\prime\,}\!3^{\,\prime\prime}}\;
\overbrace{00}^{{}^{\prime\prime\,}\!4^{\,\prime\prime}}\;
</math>

=== Rules are needed to keep entropy "simple" ===
Crows messages are {000,001,010,011,100,101,111}, which corresponds to the integers 0-7 in base.  The Crow could send more messages by choosing whether to include leading zeros, and the sole reason for forbidding such signals  is to keep the system as simple as possible.  For example, we "declare" the single sided coin to be incapable of conveying information because <math>log(1)=0</math>, and not because it is impossible to convey a message by not flipping a coin.  Later, this convention that silence never conveys information will lead us to an alternative path to Shannon's formula for entropy.

== ''H'' is Shannon entropy: ''H&#x2264;S'' ==
Shannon entropy <math>H</math> is a generalization of the simple formula,  of the simple formula, <math>S=\log_2\Omega</math>, that can be used to adjust for situations where some messages are either never sent, or are less likely to be sent.  It involves the probability of certain messages be sent or not sent, and the formula can be used regardless of the mechanism responsible the messages are selected.

The vagueness associated with whether these probabilities are estimated by observation of many signals, or determined by some unknown means does not prevent Shannon's entropy from being useful. For example:

* [[wikipedia:Gadsby (novel)|''Gadsby'']] is a 50,000 word novel that does not contain the letter "e".  Any casual analysis leads to the conclusion that the omission of the letter is deliberate.
* Those engaged in espionage can make inferences about a signal in which the "alphabet" appears to be a uniformly random sequence of letters can is likely to be encrypted.<ref>When hiding information, steps are sometimes taken to ensure that the signals are not too random. Criminals selling of illegal drugs might try to make the messages seem innocent by pretending to be making deliveries of pizza and beverages.</ref>
* In written English, the letters "q" and "k" have the same pronunciation, which suggests that our text documents could be made smaller by removing the letter "q", for example, spelling "quick" as "quik".  This is not much of a concern for text documents, but images and movies can be more easily stored or transmitted if [[wikipedia:Image compression|compression]] is used.

The entropy of Rat's message does not depend on whether the binary system of two coins is used, or whether Rat displays the information using the four sided die shown in '''Figure&nbsp;3.'''

== Math ==
 <math>\{HH,HT,TH,TT\}\Leftrightarrow \{00,01,10,11\}\Leftrightarrow \{\text{one},\text{two},\text{three},\text{four}\}</math>

  Letting, {{math|H}} = "heads" and {{math|T}}= "tails", these 4 outcomes can be described in a number of different ways.  Four such "outcomes" can also be counted in other ways, for example:
<math>\{HH,HT,TH,TT\}\Leftrightarrow \{00,01,10,11\}\Leftrightarrow \{\text{one},\text{two},\text{three},\text{four}\}</math>

This is the simplest example of the relationship between the

REWRITE: Figure&nbsp;1 is currently being used by Wikipedias in five different languages.<ref>4 wikis</ref>  It illustrates the fact that the value of Shannon Entropy <math>S</math> is the number of coins flipped:

, if the coins are “fair”, i.e. <math>S=N.</math>.  The simplest example of an “unfair” coin would be one with either two “head” or two “tails”.  Any reasonable measure of information content would ignore that coin and yield an entropy of <math>N-1.</math>  The scope of informal peek into the entropy is limited:

Figure 2:mThe elephant with 5 coins can send, <math>2^5=32,</math> different messages, which is associated with 5 bits of information (if the coins are "fair").  If three coins are given the the crow and two coins to the rat, the crow can send, <math>2^3=8,</math> different messages, while the rat can send only, <math>2^2=4,</math> different messages.  Even though the rat an  crow can independently send only <math>8+4=12</math> different messages, together, they possess the same number of bits of information, <math>2+3=5,</math> as the elephant had.  This illustrates the additive nature of entropy.

=== Assume all outcomes are equally probable ===
 I find Crow the more illuminating, in part due to confusions 4=2&#xD7;2 versus 4=2+2.  
 it is often to count not coins but outcomes.  For rat equivalent to four sided coin.  The "alphabet" used to send information is not important.  Coins has advantage and disadvantage.  DELETE
<math>
\underbrace{
\overbrace{00}^{{}^{\prime\prime\,}\!1^{\,\prime\prime}}\;
\overbrace{00}^{{}^{\prime\prime\,}\!2^{\,\prime\prime}}\;
\overbrace{00}^{{}^{\prime\prime\,}\!3^{\,\prime\prime}}
}_{\text{equally probable}}
\underbrace{
\overbrace{00}^{{}^{\prime\prime\,}\!4^{\,\prime\prime}}\;
}_{P}
</math>

<math>p_1=p_2=p_3=\frac{1-P}{3}\implies </math>

<math>p_1+p_2+p_3</math>

==Appendix==
===Reminders===
[[File:Quizbank logo.svg|thumb|QB|50px]]
* '''[[Information theory/Shannon entropy]]''' Reminders:
** Include links from [[Information theory]] and [[Introduction_to_Information_Theory]]  
-----

===decisions===
# '''REFERENCE''' Wikipedia Permalink section '''\#Characterization''' for [https://en.wikipedia.org/w/index.php?title=Entropy_(information_theory)&oldid=1033726547#Characterization two rigorous paths] to Shannon's formula

====Characterization====
:{| class="toccolours collapsible collapsed" width="80%" style="text-align:left"
!From '''[https://en.wikipedia.org/w/index.php?title=Entropy_(information_theory)&oldid=1033726547#Characterization w:Entropy (Information theory)]'''
|-'''Bold text'''
|Let <math display="inline">\operatorname{I}</math> be the information function which one assumes to be twice continuously differentiable, one has:

:<math>\begin{align}
& \operatorname{I}(p_1 p_2) &=\ & \operatorname{I}(p_1) +  \operatorname{I}(p_2) && \quad \text{Starting from property 4} \\
& p_2 \operatorname{I}'(p_1 p_2) &=\ & \operatorname{I}'(p_1) && \quad \text{taking the derivative w.r.t}\ p_1 \\
& \operatorname{I}'(p_1 p_2) + p_1 p_2 \operatorname{I}''(p_1 p_2) &=\ & 0 && \quad \text{taking the derivative w.r.t}\ p_2 \\
& \operatorname{I}'(u) + u \operatorname{I}''(u) &=\ & 0 && \quad \text{introducing}\, u = p_1 p_2 \\
& (u \mapsto u \operatorname{I}'(u))' &=\ & 0
\end{align}</math>

This [[differential equation]] leads to the solution <math>\operatorname{I}(u) = k \log u</math> for any <math>k \in \mathbb{R}</math>. Property 2 leads to <math>k < 0</math>. Properties 1 and 3 then hold also.
|}

* What is the entropy S when one of the “unfair” in that the two outcomes occur with unequal probabilities.<ref>For example, what if one coin is weighted so that, <math>p=1/4</math> and  <math>q=1-p= p=1/4\text{?.}</math></ref>
*How can this definition be extended to include entities of more than two outcomes?<ref>For example, a pair of six sided dice is used to generate outcomes.</ref>
*What does it mean to say t

*hat [https://en.wikipedia.org/w/index.php?title=Entropy_in_thermodynamics_and_information_theory&oldid=1020584746#Theoretical_relationship entropy is an “extensive property”?]
*How can this essay contribute to higher education?

===Expected number of bits and entropy per symbol===
[[w:Variable-length_code#Uniquely_decodable_codes]] helped me with [[w:Shannon's source coding theorem]]

===Number of messages===
This has a trivial resolution if we note that the exponential function, <math>y=log_2(x),</math> and logarithmic function, <math>x=log_2(y),</math>  are mutually inverse functions:<ref>An easy way to remember this is that if you see <math>y=2^x</math>, remember that '''the logarithm is the exponent'''.</ref>

:<math>y=2^x \text{ if and only if } x=\log_2(y)\;\ldots (\text{ provided } y>0.)</math>

[[File:D6 6.svg|30px]] For example, the <math>\Omega = 6</math> messages associated with the throw of a six-sided die has an entropy of <math>S=log_2(6)\approx 2.5585</math>

===micro/macro state grand ensemble/Gibbs entropy===
*These words are used differently and should be avoided
*[[w:Microstate (statistical mechanics)]]
*http://pillowlab.princeton.edu/teaching/statneuro2018/slides/notes08_infotheory.pdf uses Lagrangian multiplier to establish extremal See (8.19), or equivalently 8.2.2.  See also efficiency at (8.22)

-----
Since this essay is about the [[w:Entropy|entropy]] of [[w:Entropy (information theory)|''information'']],

===large numbers===
[[w:Observable_universe#Matter_content—number_of_atoms]] 10^80

# no attempt to explain.  This introduction the formula is not rigorous.  purpose familiarize properties.   slowly build, begin w info result of randon generator flipping N coins, ignoring surpisal.  N evolves to shannon entropy. 
#The elephant shown to right uses 5 coins to send messages.  Sometimes he gives three coins to the crow and two coins to the rat so they can also send messages.  The number of possible messages available to each animal depends on how many coins are used.  

#kilobytes is a smallfile
##https://en.wikipedia.org/wiki/Byte
##https://en.wikipedia.org/wiki/Kilobyte

===Links===
*[[b:An Intuitive Guide to the Concept of Entropy Arising in Various Sectors of Science]] Uses Stirling approximation to show the p log p arises from large amounts of entropy.  
*[https://towardsdatascience.com/the-intuition-behind-shannons-entropy-e74820fe9800 towardsdatascience.com] pages can be accessed for free at first, but later it seems you need to pay.
*https://planetcalc.com/2476/
*[[w:Stirling's approximation]]
* https://math.stackexchange.com/questions/331103/intuitive-explanation-of-entropy
* https://en.wikipedia.org/w/index.php?title=Entropy_in_thermodynamics_and_information_theory&oldid=1020584746#See_also
**Interesting links, mostly to WP
* https://en.wikipedia.org/w/index.php?title=Entropy_(information_theory)&oldid=1033726547#Characterization
** 2 derivations of PlogP stuff. 
**I=-log p is information content (low p means big I)
**already referenced!
*https://en.wikipedia.org/wiki/Entropy_(statistical_thermodynamics)
** Uses log n! = n log n to justify PlogP stuff
*[[w:Binary entropy function]] graphs <math>\operatorname H(X) = \operatorname H_\text{b}(p) = -p \log_2 p - (1 - p) \log_2 (1 - p)</math>,
====Boltzmann or Gibbs?====
*[[w:Boltzmann's entropy formula]]


<references />

[[Category:User:Guy vandegrift/Drafts]]
[[Category:Entropy]]