Scribd Logo
Upload

Welcome to Scribd!

  • 77 views

    hw3 Pset

    Uploaded by

    Kushagra Singhal
    This document outlines homework assignments for an information theory course. It includes 6 exercises covering topics like divergence of order statistics, maximum entropy, finiteness of entropy, information radius vs diameter, entropy of binomial distributions, and capacity of a binary symmetric channel. The exercises involve proving theorems, finding necessary and sufficient conditions, and deriving asymptotic bounds on entropy and capacity. References include the textbook "Elements of Information Theory".

    Copyright:

    © All Rights Reserved
    Download as PDF, TXT or read online from Scribd

    hw3 Pset

    Uploaded by

    Kushagra Singhal
    77 views2 pages
    This document outlines homework assignments for an information theory course. It includes 6 exercises covering topics like divergence of order statistics, maximum entropy, finiteness of entropy, information radius vs diameter, entropy of binomial distributions, and capacity of a binary symmetric channel. The exercises involve proving theorems, finding necessary and sufficient conditions, and deriving asymptotic bounds on entropy and capacity. References include the textbook "Elements of Information Theory".

    Original Description:

    fcbgdfgdfgdfgdf

    Original Title

    hw3_pset

    Copyright

    © © All Rights Reserved

    Available Formats

    PDF, TXT or read online from Scribd

    Did you find this document useful?

    Is this content inappropriate?

    This document outlines homework assignments for an information theory course. It includes 6 exercises covering topics like divergence of order statistics, maximum entropy, finiteness of entropy, information radius vs diameter, entropy of binomial distributions, and capacity of a binary symmetric channel. The exercises involve proving theorems, finding necessary and sufficient conditions, and deriving asymptotic bounds on entropy and capacity. References include the textbook "Elements of Information Theory".

    Copyright:

    © All Rights Reserved
    Download as PDF, TXT or read online from Scribd
    Download as pdf or txt
    77 views2 pages

    hw3 Pset

    Uploaded by

    Kushagra Singhal
    This document outlines homework assignments for an information theory course. It includes 6 exercises covering topics like divergence of order statistics, maximum entropy, finiteness of entropy, information radius vs diameter, entropy of binomial distributions, and capacity of a binary symmetric channel. The exercises involve proving theorems, finding necessary and sufficient conditions, and deriving asymptotic bounds on entropy and capacity. References include the textbook "Elements of Information Theory".

    Copyright:

    © All Rights Reserved
    Download as PDF, TXT or read online from Scribd
    Download as pdf or txt
    of 2

    Fall 2014

    ECE563 - Information Theory


    Homework 3
    Due: Sep. 25, 2013
    Prof. Yihong Wu
    1 Reading
    1. Read [1, Chapter 4]
    2 Exercises
    1. (Divergence of order statistics) Given x
    n
    = (x
    1
    , . . . , x
    n
    ) R
    n
    , let x
    (1)
    . . . x
    (n)
    denote the
    ordered entries. Let P, Q be distributions on R and P
    X
    n = P
    n
    , Q
    X
    n = Q
    n
    .
    (a) Prove that
    D(P
    X
    (1)
    ,...,X
    (n)
    ||Q
    X
    (1)
    ,...,X
    (n)
    ) = nD(P||Q). (1)
    (b) Show that
    D(Binom(n, p)||Binom(n, q)) = nd(p||q).
    2. (Maximum entropy.) Prove that for any X taking values on N = {1, 2, . . .} such that EX < ,
    H(X) EXh

    1
    EX

    .
    Find the necessary and sucient condition for equality. Hint: Find an appropriate Q such that
    RHS - LHS = D(P
    X
    ||Q).
    3. (Finiteness of entropy) A consequence of Exercise 2 is the following: For any N-valued random
    variable X, EX < ensures that H(X) < . Next let us improve this result.
    (a) Show that Elog X < H(X) < .
    Moreover, show that the condition of X being integer-valued is not superuous by giving
    a counterexample.
    (b) Show that if k P
    X
    (k) is a decreasing sequence, then H(X) < Elog X < .
    Moreover, show that the monotonicity of pmf is not superuous by giving a counterexample.
    4. (Information radius v.s. diameter.) Let {P
    Y |X=x
    : x X} be a set of distributions. Prove that
    sup
    P
    X
    I(X; Y ) C sup
    x,x

    X
    D(P
    Y |X=x
    ||P
    Y |X=x
    )
    Comment: This is the information-theoretic version of radius diameter.
    5. Let S Binom(n, p). Prove that
    H(S) =
    1
    2
    log n +O(1),
    where O(1) is uniform in 0 p 1.
    1
    Bonus: Prove that
    H(S) =
    1
    2
    log(2enp(1 p)) +O

    1
    n

    ,
    where O(
    1
    n
    ) is uniform in 0 p 1.
    Hint: Stirlings approximation:
    e
    1
    12n+1

    n!

    2n(n/e)
    n
    e
    1
    12n
    , n 1
    6. Consider the conditional distribution P
    Y
    m
    |X
    : [0, 1] {0, 1}
    m
    , where given x [0, 1], Y
    m
    is
    i.i.d. Bern(x). Dene the capacity:
    C(m) max
    P
    X
    I(X; Y
    m
    ).
    Our goal is to show that
    C(m) =
    1
    2
    log m+O(1).
    (a) Let S =

    n
    i=1
    Y
    i
    . Prove that
    C(m) = max
    P
    X
    I(X; S).
    (b) Show that
    C(m) = min
    Q
    S
    sup
    0x1
    D(Binom(n, x)||Q
    S
    ).
    where Q
    S
    is a distribution on {0, . . . , m}. Hint: Use the minimax representation (Theorem
    4.5). Why is the condition of the theorem satised?
    (c) Choosing a uniform Q
    S
    and show that
    C(m)
    1
    2
    log m+O(1) , m .
    (d) Choose a uniform P
    X
    and show that
    C(m)
    1
    2
    log m+O(1) , m .
    Hint: For the last two parts Ex. 5 might be useful.
    References
    [1] T. Cover and J. Thomas, Elements of Information Theory, Second Edition, Wiley, 2006
    2

    You might also like

    pFad - Phonifier reborn

    Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

    Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.


    Alternative Proxies:

    Alternative Proxy

    pFad Proxy

    pFad v3 Proxy

    pFad v4 Proxy