\documentclass[11pt]{article}
\usepackage{handout}
\usepackage{amsmath,amssymb,amsthm,epsfig}
\begin{document}
\input{courseMacros}

\newtheorem{Theorem}{Theorem}
\newtheorem{Corollary}{Corollary}
\newtheorem{Lemma}{Lemma}
\newtheorem{Claim}{Claim}
\theoremstyle{definition}
\newtheorem{Definition}{Definition}
\newtheorem{problem}{Problem}
\newenvironment{Proof}{\noindent{\bf Proof: }}{\qed}
%\theoremstyle{remark}
\newtheorem{Remark}{Remark}

\thispagestyle{empty}
\thispagestyle{empty}
\setlength{\parindent}{0pt}
\setlength{\parskip}{1.8ex}

\lecturehead{CS G399}{Algorithmic Power Tools I}{Fall 2007}{14
September 2007}{Eric Robinson}


%\newcommand{\p}{\mathcal{P}}
%\newcommand{\U}{\mathcal{U}}
%\newcommand{\coll}{\mathcal{S}}
%\newcommand{\xpy}{x + \lambda y}
%\newcommand{\Alg}{\mathcal{A}}
%\newcommand{\opt}{\mbox{OPT}}
%\newcommand{\eps}{\varepsilon}

\noindent {\bf Lecture Outline:}
\begin{itemize}
\item
Linear Programming: Vertex Definitions
\item
Linear Programming: Optimal at a Vertex
\item
LP: Duality
\item
LP: Duality: Weak Duality
\item
LP: Duality: Set Cover
\end{itemize}

This lecture gives an introduction to the geometry of linear
programming.  We discuss three equivalent definitions of a vertex of a
polytope, and then show that there exists an optimal solution to any
bounded feasible LP lies at a vertex of the underlying polytope.  We
then introduce the important concept of LP duality and present an
alternative analysis of the greedy set cover algorithm using the idea
of dual fitting.  References for the material covered in this lecture
are~\cite{goemans:lp,karloff:lp,vazirani:approximateBook}.

\section{Linear Programming}

Linear Programming was described in the previous class.  Recall the
standard form:
\[
\begin{array}{llll}
\min         & c^T x &     &  \\
\mbox{s.t.}  & Ax    & \ge & b \\
             & x     & \ge & 0 \\
\end{array}
\]

\subsection{Vertex Definitions}

As was discussed the previous class, three definitions of a vertex,
$x$, for a convex polytope, $P$, can be used:
\begin{itemize}
\item $\neg \exists y \not = 0 \mbox{ s.t. } x + y, x - y \in P$
\item $\neg \exists y ,z \in P, \alpha \in (0,1) \mbox{ s.t. } \alpha y + (1-\alpha)z = x$
\item $\exists n$ linearly independant equations that are tight at $x$
\end{itemize}

The proof that definitions 1 and 2 are equivalent shall be left to the
reader.  Here we will prove that claims 1 and 3 are equivalent.

\subsubsection{1 implies 3}

Suppose $\neg 3$.  Let $A'$ be the submatrix corresponding to the
tighe inequalities, let b' be the corresponding right hand sides.
This leads to the inequality $A'x = b'$, where there are remaining
equations $A''$ and right hand sides, $b - b''$.  The rank of $A'$ is
strictly less than $n$.  This implies that $\exists y \not = 0 \mbox{
s.t. } A'y = 0$.

Consider $x + \lambda y$, $A'(x+\lambda y) = A'x + \lambda A' y = b'.
Find \lambda \ge 0 \mbox{ s.t. } x + \lambda y, x - \lambda \in P$.
Now consider any one of the non-tight inequalities, $j$.  $A_j x >
b_j$.  Apply $\lambda$, getting $A_j(x+ \lambda y) = A_j x + \lambda
A_j y$.  It is known that $A_j x > b_j$.  Because $A_j x = b_j$ is not
tight, $\lambda A_j y$ must be feasible in one direction for a short
distance, and the other direction infinitely.  We shall select a
distance $\lambda_j$ equal to that short distance.  Finally, we shall
select $\lambda = \min_j | \lambda_j |$.  Because the associated
constraints are not tight, $\lambda$ cannot be zero.  This directly
contradicts 1.

\subsubsection{3 implies 1}

Let $A'$ denote the submatrix for tight inequalities, as was done
above.  This implies the rank of $A' \ge n$.  Suppose $neg 1$.  This
implies $\exists y \ne 0 \mbox{ s.t. } x + y, x - y \in P$.  Consider
the fact that $A'(x+y) \ge b', A'(x-y) \ge b'$.  This implies that
$A'y = 0$, which implies that the rank of $A'$ is less than $n$.  This
contradicts the assertion made earlier.

\subsection{Optimality at a Vertex}

The claim was made in the previous class that an optimal solution must
occur at a vertex.  More specifically, if the linear program is
feasible and bounded, then $\exists v \in Vertices(P) \mbox{ s.t. }
\forall x \in P, C^T v \le C^T x$.  This statement, along with
fundamentally stating that an optimal solution must occur at a vertex,
also shows the decidability of solving for the system, as one can
simply check all the vertices.  A proof follows.

\subsubsection{Proof}

The proof shall proceed by picking a non-vertex point and showing that
there exists as good or better of a solution with at least one less
non-tight equation.  Hence progress is made towards a vertex by
applying this process iteratively.  To find a vertex that is optimal
this iteration will have to be done at most $n+m$ times.

Suppose $x \in P$ and $x \not \subset Vertices(P)$.  This implies
$\exists y \ne 0 \mbox{ s.t. } x + y, x - y \in P$.  This in turn
implies $(x + y, x - y \ge 0) \rightarrow (x_i = 0 \rightarrow y_i =
0)$.  Consider $v = x + \lambda y$.  $C^T v = C^T x + \lambda C^T y$.
Assume without loss of generality that $C^T y \le 0$.  If this was not
the case, the other direction could simply have been selected ($x -
y$).

Now suppose that $A'$ and $b'$ give the set of tight variables, as
before.  We know that $A'(x + y) \ge b'$.  This implies $A' y = 0$,
and that $A'(x+\lambda y) = b'$.  Once again, we will find the
$\lambda_j$ values for the non-tight equations.  There are two
possibilities:
\begin{itemize}
\item $\exists \lambda_j \ge 0$: In this case, we selecte the smallest
such $\lambda_j$ value, $\lambda$.  The point $x + \lambda y$ now
makes one additional inequality tight while satisfying the property
that it keeps the previously tight inequalities tight.  And we have
made progress towards a solution.
\item $\forall_j \lambda_j < 0$: Here, either $C^T y < 0$, in which
case the solution is unbounded, or $C^T y < 0$ in which case we use
the go to the $\exists \lambda_j \ge 0$ case, as our cost function
will remain the same no matter which direction we move in.
\end{itemize}

\subsection{Duality}

One can view any minimization linear program as a maximization.
Consider the following linear system:
\[
\begin{array}{llll}
\min         & 3x_1 + 2x_2 + 8x_3  &     & \\
\mbox{s.t.}  & x_1 - x_2 + 2x_3    & \ge & 5 \\
             & x_1 + 2x_2 + 4x_3   & \ge & 10 \\
             & x_1, x_2, x_3       & \ge & 0 \\
\end{array}
\]

Where $Z^*$ is OPT, we know $Z^* = 3x^*_1 + 2x^*_2 + 8x^*_3$, for some
$x^*_1, x^*_2, x^*_3 \in P$.  By adding two of the inequalities, we
arrive at $2x_1 + x_2 + x_3 \ge 15$.  Since $x^*_1, x^*_2, x^*_3 \ge
0$, we know that $Z^* \ge 15$.  But we aren't limited to addition,
multiplication is another way the equations can be combined.  So how
is this new formulation bouned?  This is done by using the dual
formulation, $D$ of the minimization, which for this problem is:
\[
\begin{array}{llll}
\max         & 5y_1 + 10y_2        &     & \\
\mbox{s.t.}  & y_1 + y_2           & \le & 3 \\
             & -y_1 + 2y_2         & \le & 2 \\
             & 2y_1 + 4y_2         & \le & 8 \\
             & y_1, y_2            & \ge & 0 \\
\end{array}
\]

The theory of LP duality (sometimes referred to as the Strong Duality
Theorem) says that if the primal LP $P$ is bounded and feasible, then
the value of the primal LP equals the value of the dual LP.  

\subsubsection{Weak Duality}

Weak duality makes only the claim that the value of the primal LP is
at least the value of the dual LP.  Consider the primal $P$ and its
dual $D$:
\[
\begin{array}{llll|llll}
P            &       &     &   & D \\
\min         & c^T x &     &   & \max         & b^T y \\
\mbox{s.t.}  & Ax    & \ge & b & \mbox{s.t.}  & A^T y    & \le & c \\
             & x     & \ge & 0 &              & y        & \ge & 0 \\
\end{array}
\]

Suppose that $x^*$ is an optimal solution to $P$ and $y^*$ is an
optimal solution to $D$.  We need only show that $c^T x^* \ge b^T y^*$.
\begin{eqnarray*}
c^T x^* & \ge & (A^T y^*)^T x^* \\ 
& = & {y^*}^T A x^*\\
b^T y^* & \le & {x^*}^T A^T y^* \\
& = & ({y^*}^T A x^*)^T
\end{eqnarray*}
Noting that the last equation of each of these comparison are
identical (since the transpose of a scalar is the scalar itself) leads
to the desired conclusion.

\subsubsection{Set cover analysis using dual fitting}

The set cover linear programming problem, $P$, can be recast into its
dual framework, $D$:
\[
\begin{array}{lllll|lllll}
P            &                    &     &   &                 & D \\
\min         & c_s x_s            &     &   &                 & \max        & \sum_{e \in U} y_e \\
\mbox{s.t.}  & \sum_{S \ni e} x_s & \ge & 1 & \forall e \in U & \mbox{s.t.} & \sum_{e \in s} y_e & \le & c_s & \forall s \in S \\
             & x_s                & \ge & 0 & \forall s \in S &             & y_e        & \ge & 0 & \forall e \in U \\
\end{array}
\]

\paragraph{Analysis}

We seek to construct a dual-feasible solution with a cost close to the
integer solution.  We shall look at $y_e$, or $price(e)$, the cost of
an element when it is first covered.  This corresponds to
$\frac{cost(s)}{\mbox{number of uncovered elements covered}}$ when $e$
is first covered.  The cost of the greedy solution is $\sum_{e \in U}
y_e$.

The problem with this is that the constraints for the dual are not
covered because the greedy solution adds in an additional cost for
previously covered elements.  So, we will uniformly decrease the
$y_e$'s, $y_e = \frac{price(e)}{\alpha}$.  The claim is that $y$ is
dual feasible if $\alpha = H_n$, where $n$ is the number of elements
in $U$.

To prove this, we need to show $\sum_{e \in s} y_e \le c_s$.  If we
considered the elements ordered again, and pick any set $s \in S$,
we'll see that a portion of the elements have been covered, and a
portion have not.  This means that:
\begin{eqnarray*}
{y_e}_1   & \le    & \frac{c_s}{k} \times \frac{1}{H_n} \\
{y_e}_2   & \le    & \frac{c_s}{k-1} \times \frac{1}{H_n} \\
\vdots  & \vdots & \vdots \\
\end{eqnarray*}
Or,
\begin{eqnarray*}
\sum_{e \in S} y_e & \le & \frac{c_s}{H_n} \times ( \frac{1}{k} + \frac{1}{k-1} + \cdots + 1 ) \\
                   & =   & \frac{c_s}{H_n} \times H_k \\
                   & \le   & c_s \\
\end{eqnarray*}

So, from this, we can show that the greedy solution is an $H_n$
approximation:
\[
\begin{array}{rclr}
\mbox{Cost of greedy algorithm} & =   & H_n \times cost(y) & \\
             & \le & H_n \times OPT(D)  & \mbox{proof above} \\
             & \le & H_n \times OPT(P)  & \mbox{weak duality} \\
             & \le & H_n \times OPT     & \mbox{fractional to integral} \\
\end{array}
\]

\bibliographystyle{alpha}
\bibliography{/home/rraj/tex/bibtex/refs}

\end{document}
