\documentclass[11pt]{article}
\usepackage{handout}
\usepackage{amsmath,amssymb,amsthm,epsfig}
\usepackage[ruled,vlined]{algorithm2e}
\begin{document}
\input{courseMacros}
\newcommand{\junk}[1]{}
\newtheorem{Theorem}{Theorem}
\newtheorem{Corollary}{Corollary}
\newtheorem{Lemma}{Lemma}
\newtheorem{Claim}{Claim}
\theoremstyle{definition}
\newtheorem{Definition}{Definition}
\newtheorem{problem}{Problem}
\newenvironment{Proof}{\noindent{\bf Proof: }}{\qed}
%\theoremstyle{remark}
\newtheorem{Remark}{Remark}

\thispagestyle{empty} \thispagestyle{empty}
\setlength{\parindent}{0pt} \setlength{\parskip}{1.8ex}


%%%  EDIT THE FOLLOWING TO UPDATE DATE AND LECTURE NUMBER

\lecturehead{CS G399}{Algorithmic Power Tools I}{Fall 2007}{2
November 2007}{Emrah Bayraktaroglu}

\newcommand{\p}{\mathcal{P}}
\newcommand{\U}{\mathcal{U}}
\newcommand{\coll}{\mathcal{S}}
\newcommand{\xpy}{x + \lambda y}
\newcommand{\Alg}{\mathcal{A}}
\newcommand{\opt}{\mbox{OPT}}
\newcommand{\eps}{\varepsilon}


\noindent {\bf Lecture Outline:}


%%% Itemized outline of the lecture
\begin{itemize}
  \item Introduction to Information Theory (Discrete Memoryless Channels)
  \item Zero-Error Capacity of a Communication Channel
  \item Pentagon
\end{itemize}

In this lecture, we will give an introduction to Shannon's
mathematical model of information and a notion of the capacity of a
channel.  We will also analyze the zero-error capacity of the Pentagon
graph, which is arguably the first problem that introduced
semi-definite programming to computing.

\section{Introduction to Information Theory}
Let $\overline{x}$ be a vector of symbols that represents the input
and $\overline{y}$ be the output vector we get when we send
$\overline{x}$ over the channel C. It is highly possible that the
channel would alter the input and cause $\overline{y}$ to be
different that $\overline{x}$. That would lead to cases such as:

YOU WILL NOT ATTACK $\rightarrow$ YOU WILL NOW ATTACK

At this point, Shannon introduced DMCs (Discrete Memoryless Channels)
to model the channels and information sent over them.

\subsection{Discrete Memoryless Channel}
In a discrete memoryless channel, we assume that messages are composed
of symbols taken from a discrete domain and each message sent on the
channel is independent of previous messages, thus making the channel
memoryless. The information theory we will discuss in the class today
is based on DMCs.  We model information probabilistically, i.e., the
amount of information according to this model is a function of the
number of distinct messages that can be sent out.  Formally, let
$\overline{x}$ be a probability distribution over symbols and let
$P_x$ be the probability that $\overline{x} = x$.  One can define
\emph{the surprisal} of symbol $x$ as
\[\log{(\frac{1}{P_x})}\].
The \emph{entropy}, or \emph{uncertainty}, of $\overline{x}$, given by
$H(\overline{x})$, is defined in terms of the surprisal of each symbol
$x$ as follows.
\[H(\overline{x})=\sum_{x}{P_x\log(\frac{1}{P_x})}\]
Intuitively, we are defining the amount of information in
$\overline{x}$ in terms of the different values it can take; the
\emph{entropy} of a symbol space gives us the average amount of
information we can get from a symbol.

Now it's time to define the information over the channel and capacity
of a channel. We define it as the average amount of information we get
about the input by observing the output. Let $X$ be a random variable
representing the symbol space for input and $Y$ be the output random
variable.  We define the {\em mutual information}\/ as
\[I(X,Y)=H(X)-H(X|Y)\]
Here $H(X)$ gives the uncertainty in the input and $H(X|Y)$ gives the
uncertainty in the input, given the output.  The difference between
the two gives the information that has been provided.

And the capacity of a channel is the maximum information we can send
over the channel. Capacity $C$ is given as;

\begin{equation}\label{cap1}
    C=max_{x}{I(X,Y)}
\end{equation}

\begin{Theorem}\textbf{(Shannon's Noisy Channel Theorem)}

Every DMC's capacity C given by (\ref{cap1}) is attainable
asymptotically error-free.
\end{Theorem}

By attainable, we mean that there exist encoding and decoding
procedures such that capacity $C$ is achieved and the resulting bit
error rate can be made arbitrarily close to zero.

Around the same time that Shannon developed information theory,
Hamming had also developed models for studying the communication of
information.  Instead of using a probabilistic model for errors, he
considered bounds on number of errors in a message (e.g., at most $d$
bits of a message are flipped), which perhaps made it more difficult
to obtain general results.

\junk{Let's examine this theorem in two cases;

\textbf{\emph{Binary Case}}: This is the case where we have only two
possible inputs. The capacity in this case is:
\[\frac{C}{1-H(P_b)}\]
where $H(P_b)$ is the \emph{binary entropy function}.
}

\section{Zero-Error Capacity}
Shannon's Noisy Channel Theorem says that capacity $C$ can be achieved
with error that can made arbitrarily close to zero.  It does not
guarantee zero error however.  Indeed, if every input symbol can be
transformed by the channel to any other output symbol with non-zero
probability, it is impossible to attain any zero error capacity.  In
his seminal paper, Shannon also introduced the concept of zero-error
capacity.

Let $G=(V,E)$ be a graph, $V$ set of vertices, which represent input
symbols. Given $a,b\in V$, $(a,b)\in E$ if the symbols $a$ and $b$
cannot be confused with each other, i.e., the set of possible output
symbols for $a$ is disjoint from the set of possible output symbols
for $b$. Any clique of $G$ corresponds to a set of symbols that can be
used simultaneously with zero-error.  Thus, if we send one symbol at a
time, we can attain a capacity of
\[\log_{2}{\omega(G)}\]
with zero error.  

But we can do better by considering blocks of symbols.  Suppose we
send messages out as blocks of $n$ symbols.  Then, we can attain a
capacity of $\frac{\log_2{\omega(G^n)}}{n}$, where the product $G^n =
G\times G\times\ldots\times G$ ($n$ times) is defined as follows.

$G' = G \times H = (V', E')$

$V'=\{(g_1 h_1)|g_1\in V_g,h_1 \in V_h\}$

$E'=\{<(g_1 h_1),(g_2 h_2)>|<g_1,g_2> \in E_g \vee <h_1,h_2> \in
E_h\}$

Zero-error capacity, also known as Shannon capacity is defined as:
\[SC(G^n) = \sup_n{\frac{1}{n}\log_2{\omega(G^n)}}.
\]
It can be shown that $\omega(G^n)$ is supermultiplicative; that is,
$\omega(G^{m\cdot n}) \ge \omega(G^m) \cdot \omega(G^n)$.  This allows
us to replace the supremum by a limit.
\[SC(G^n)=\lim_{n\rightarrow \infty}{\frac{1}{n}\log_2{\omega(G^n)}} =
\lim_{n\rightarrow \infty}{\log_2(\sqrt[n]{\omega(G^n)})}
\]
For convenience, the Shannon capacity is often defined as
$\sqrt[n]{\omega(G^n)}$, which is what we will adopt in the remainder.

We do not know of any decidable procedure for computing the Shannon
capacity of an arbitrary graph.  In the following, we will try to
obtain upper and lower bounds and calculate the capacity for a special
graph, the pentagon.  One simple upper bound on the Shannon capacity
of $G$ is $\log_2 |V(G)|$.

\subsection{The Pentagon}
This section will introduce the pentagon which will be used for
capacity upper bound discussions in the next lecture. Pentagon is a
graph $C_5$ that has five vertices each of which represent a symbol
in the symbol domain connected in the shape of a pentagon. The
figures \ref{fig:pent} and \ref{fig:pentsqr} are representations for
$C_5$ and $(C_5^2)$, which yields $(C_5)$ $\omega(C_5)=2$ and
$\omega(C_5^2)=5$.

\begin{figure}
 \center \epsfig{file=pent.eps, width =100pt}
\caption{Representation of the pentagon ($C_5$)} \label{fig:pent}
\end{figure}
\begin{figure}
 \center \epsfig{file=pentsqr.eps, width =100pt}
\caption{Representation for $C_5^2$} \label{fig:pentsqr}
\end{figure}



%% ...
\end{document}
