Solution to problem 87-6* : The entropy of a Poisson

distribution
Citation for published version (APA):
Boersma, J. (1988). Solution to problem 87-6* : The entropy of a Poisson distribution. SIAM Review, 30(2), 314-
317. https://doi.org/10.1137/1030059

DOI:
10.1137/1030059

Document status and date:

Published: 01/01/1988

Document Version:
Publisher’s PDF, also known as Version of Record (includes final page, issue and volume numbers)

Please check the document version of this publication:

• A submitted manuscript is the version of the article upon submission and before peer-review. There can be
important differences between the submitted version and the official published version of record. People
interested in the research are advised to contact the author for the final version of the publication, or visit the
DOI to the publisher's website.
• The final author version and the galley proof are versions of the publication after peer review.
• The final published version features the final layout of the paper including the volume, issue and page
numbers.
Link to publication

General rights
Copyright and moral rights for the publications made accessible in the public portal are retained by the authors and/or other copyright owners
and it is a condition of accessing publications that users recognise and abide by the legal requirements associated with these rights.

• Users may download and print one copy of any publication from the public portal for the purpose of private study or research.
• You may not further distribute the material or use it for any profit-making activity or commercial gain
• You may freely distribute the URL identifying the publication in the public portal.

If the publication is distributed under the terms of Article 25fa of the Dutch Copyright Act, indicated by the “Taverne” license above, please
follow below link for the End User Agreement:
www.tue.nl/taverne

Take down policy

If you believe that this document breaches copyright please contact us at:
openaccess@tue.nl
providing details and we will investigate your claim.

Download date: 16. Sep. 2020

314 PROBLEMS AND SOLUTIONS

thesimpleunconstrained
forx from(1 1),we endup considering
Finally,substituting
scalaroptimizationproblem
.vT T7~~~~~lFz~~2( ViT7 ~ l)1
(15) max 1+ 1.+ 1 + 4( 2 2
z z L ? V4 - Z4 - 2(12/h2)Z2
412(l + hz) + +
8/hz}
(12), namely
Alsorecallthatz liesin therange(14) and theoptimalz mustsatisfy
(16) z2 1-+2 4_ Z4 - 2(1 + 2gh2)z2+ 8ghz_hz-Z2.
Finally,theoptimalfluidlevelx is foundbysubstituting theoptimalz in (13).
The problemparameters are A= 0.1132 and h = 3.4375.For theseparameters
thequarticequationassociatedwith(14) hasa uniquerealpositivesolutionz = 0.914
and therefore (14) yieldstherange0 < z _ z = 0.914.Furthermore, themaximumin
(15) is thenattainedat theboundaryz ofthez-range, i.e., z* = z = 0.914 and thus
x* = l/z = 1.094.Finally,note that(12) is thensatisfiedsince 0.914< 1.094< 2.523.
The insightgainedabove affords us thepossibility of ascertaining therangeof
validity (in parameter space)of our solution.In particular, small,
ifA is sufficiently
thenz > 1,x* = l/z< 1 and theleft-hand partof( 12) is violated.Hence,thesmallest
possiblevalue of the parameter A forour analysisto be validis thatwhichmakes
= 1 theuniquereal,positive solutionofthequarticsolution
z4+ 2(1+ 2gh2)z2- 8ghz- 4 = 0.
In other words ,u_ = (I/4h(h - 2)). Furthermore,it is required that h> h 2.
Hence, the parameterspace (A,h) is partitionedby the curve 8(h) = (1/4h(h- 2)),
h >)h= 2.

ofa PoissonDistribution
The Entropy
Problem87-6*,byC. ROBERT APPLEDORN (IndianaUniversity).
The following problemaroseduringa studyof data compression schemesfor
encodedradiographic
digitally Huffman
images.Specifically, optimum(minimum)
codesare employedto reducethestoragerequirements forhighresolution
gray-level
images,forexample,2048 x 2048 pixelsby 10 bitsper pixelfora singleimage.
Entropy estimatesprovidea lowerboundfortheaveragestoragein termsofbitsper
pixelthatcan be achievedusingthesecodingmethods.
Due to thenatureoftheproblem, thePoissonprobabilitydensity with
function
meanvalueparameter interest:
m is ofparticular
mke
k- -m
p(k) =

H associatedwiththisprobability
The entropy density is givenby
function
00
H=- i p(k) logp(k).
k=O

a closed-
calculationis to determine
The problemthatdevelopsduringtheentropy
formsolutionto thefollowing
summation:
00
ke-rn
E log (k!)
k=O k!
 PROBLEMS AND SOLUTIONS 315
It is conjectured
bytheauthorthatforlargem (say,m > 10),theentropy
H can
be approximated by
H=2 log(2-rem).
Proveor disprove.
SolutionbyRONALD J. EVANS (University ofCalifornia,
San Diego).
Let SP(x)= logF(x + 1). Bydefinition
oftheentropyH,
+ m- k logm) = m(1- logm) + (P.,
H= , p(k)(QP(k)
k=O

where
sm=Ep(k)(P(k).
k=O

By [2, Chap. 3, Entry10], [3, Entry10], as (real) m-- oo,

( ) + m'P(2)(m)+mP3)(m)+ m2P4)(m)+ O(m-2)

2 6 8
(Severalminorerrors in [2,Chap.3,?10]and[3,?10]arecorrectedin [4].)Substituting
in the well-known asymptotic (P()(m) [1, (6.1.41),
expansionsforthe derivatives
(6.3.18),(6.4.11)],we deducethat,as m-> oo,

H= log(27rem)- 2+1
2 12m

REFERENCES

[1] M. ABRAMOWITZ AND I. STEGUN, eds., HandbookofMathematical

Functions,
Dover, New York,
1965.
[2] B. C. BERNDT, Ramanujan'sNotebooks, PartI, Springer-Verlag,
NewYork,1985.
[3] B. C. BERNDT, R. J.EVANS, AND B. M. WILSON, Chapter 3 ofRamanujan'sSecondNotebook, Adv.
in Math., 49 (1983), pp. 123-169.
[4] R. J. EVANS, Ramanujan'sSecondNotebook: Asymptoticexpansionsforhypergeometricseriesand
relatedfunctions,in Proc. Ramanujan CentenaryConference,Academic Press,New York, 1988.

Solutionby J. BOERSMA (EindhovenUniversity of Technology,Eindhoven,the

Netherlands).
It is easilyseenthattheentropy
H associatedwiththePoissonprobability
density
function is givenby
00 00
~~~~~~~mk,e-m
H=- p(k) logp(k) = m-m logm+ log(k!)"k
k=O k=O

In thelatterseriesreplacelog(k!) byMalmsten'srepresentation
[1,Formula1.9(1)]

log(k!)=J k- 1_ -t1 dt;

thenit is foundthat

H=m-mlogm+ {m- 1ex[m(- e )} tdt

[
I e-ms ds
= m- mloggm- (m- 1 s)
by the substitution1 - e-t = s.
316 PROBLEMS AND SOLUTIONS

in (*) is decomposedinto
The finalintegral

-mf, [ ds+ m
(_-e ds +,| - ds

sj[og (I -s) s 2d- (s -log(I S) s -]d

wherebythe successive integralsare shortlydenoted by I1, I2, I5. By back
s = 1 - e-titis foundthat
substitution

J0
(-it - e-t) d=

where-ydenotesEuler'sconstant(cf.[1, Formula1.7(19)]).Throughan integration

bypartsthesecondintegral
reducesto
1e-ms M eI- 1 edt
I2 =+mf dsl-em-m+m d
s 0 o s o t
= 1-e-m-m+m[El(m)+log m+,y],
in whichEl (m) = f e-tt-udt standsforthe exponential (cf. [1, Formulae
integral
9.7(1),(5)]). In thesamemannerwe have

I3 = 1(1 e dt= -[E1(m) + logm+-y]

In the fourthintegralwe substitutes = 1 - e-t,whichyields

1
I4= ( - et')2
JI4 -0 dt;
- e-t ]t;
) - 14
next,I4 is decomposedinto

The latterintegrals
can be evaluateddirectly
and by use of [1, Formulae1.9(3),
1.7(19)],viz.

J I- t t2 ]
J 2
t o 22t

ie-t t 2 t dt 21og(2 ),
f ( - + 1
-tlt- - 1 dt
 PROBLEMS AND SOLUTIONS 317
Thuswe find
I4 =2+2log (21r) -2

Byinserting
thepreviousresultsinto(*), we obtaintherepresentation

H= 2log(27rem)+ m+ E(m)-e-m)- f I)+- --] ds.

It does notseempossibleto evaluatethelatterintegral in closedform;however,an

evaluationby numericalintegration is certainlyfeasible.The asymptotics
of the
integral,as m-> oo,is readilyestablishedbyuse ofWatson'slemma.To thatendwe
inserttheTaylorseries
_1 1~ ~~~~~~0
_ _
0 _

1 IsI<1
~~~~akSk,
log(1 -s) s 2 k=1
al =
withcoefficients , a2 = 1, a3 = 79, etc., and integratetermby term.Then the
completeasymptotic
expansionoftheentropy
H is foundas
I
H~- log (2irem)- E00 ak(k- )!m-k (m >oo)
2 k=1

apartfromexponentiallysmall termsof orderO(e-m).

REFERENCE

[1] A. ERDtLYI, W. MAGNUS, F. OBERHETTINGER, AND F. G. TRICOMI, Higher Transcendental

Functions,
Vols. I, II, McGraw-Hill, New York, 1953.

A similarsolutionwas obtainedbyPETER SMITH (Keele University,

U.K.).
N. M. BLACHMAN (GTE GovernmentSystemsCorp.,MountainView,California)
notesthatsince the Poisson distributionhas variancem and becomes approximately
normal forlarge m, its entropybecomes that part of the N(m, n) distribution,viz.,
log v2e.
A. A. JAGERS (UniversiteitTwente,Enschede,the Netherlands)showsthat
log(27rm)<H< log(27rm)+1.

AlsosolvedbyW. B. JORDAN(Scotia,NewYork),J.VANKAN, P. SONNENVELD(Delft

Universityof Technology), T. KAUFFMAN (Houston, Texas), S. L. PAVERI-
FONTANA (VirginiaPolytechnicInstituteand State University,Blacksburg,Vir-
ginia), T. J. OTT (Bell CommunicationsResearch,Morristown,New Jersey),P.
R. PUDAITE (student,University of Illinois,Urbana),M. RENARDY (Virginia
PolytechnicInstituteand StateUniversity, Blacksburg,Virginia),and P. WAGNER
(Universityof Innsbruck,Austria).

An Identity
Problem87-8,byJOHNW. MOON (University
ofAlberta).
Showthat
X 56n2+33n-8f21
.nS. (n +, 2)(n J, 1)