\documentstyle[handout,11pt]{article}
\begin{document}
\input{/home/rraj/Courses/3391/F99/Lectures/macros/lecmacros}
\thispagestyle{empty}
\setlength{\parindent}{0pt}
\setlength{\parskip}{1.8ex}

\lecturehead{3391}{Advanced Algorithms}{10 November 1999}{7}{Trevor Mendez}

\begin{center}
\Large{\bf Linear Programming: The Simplex Algorithm and Relaxations}
\end{center}

The material covered in this lecture is based on the presentation in
Chapter~2 of the text on linear programming by Howard
Karloff~\cite{karloff:lp}.
\section{Simplex Algorithm}
\label{sec:lec7.simplex}

The {\it Simplex Algorithm} is based on the fact that the optimal solution
to a feasible LP (linear program) can be found at one of the vertices of
the polytope defined by the set of constraints.  The algorithm starts from
an arbitrary vertex represented by a {\it basic feasible solution} (bfs),
and at each iteration uses a technique called pivoting to search for an
adjacent vertex with an improved cost to the solution to move to.  If no
adjacent vertex has an improved cost, then it can be proved that the current
vertex represents the optimal solution.  The algorithm must search through a
set of potentially exponentially many vertices, and as a result is not
polynomial in the worst-case.  Even so, it performs very well in practice,
and was the algorithm of choice for several decades.

(Throughout this document, all LP's are given by:
minimize $c^T x$ subject to $Ax = b$ and $x \geq 0$, unless explicitly
defined otherwise.)

\subsection{Pivoting}

Pivoting is the mechanism used to manipulate the basis corresponding to a
vertex $v$'s bfs, to find the basis that corresponds to an adjacent vertex.
To perform a pivot step we replace one of the $m$ linearly independent columns
of the basis corresponding to $v$ with one of the nonbasic columns from $A$
in such a way that we still have $m$ linearly independent columns.  This
produces the basis of an adjacent vertex, $v'$.  The Simplex Algorithm
uses pivoting to examine the neighbors of $v$ until a neighbor is found
that corresponds to a feasible (non-negative) solution which is an
improvement over the bfs for $v$. We illustrate how pivoting is done with an
example.

\begin{Example}
\label{ex:one}
{\em
Assume that in $LP_1$,
\[A = \left[ \begin{array}{ccccc}
 5 & 2 & -3 & 16 & 4 \\
 2 & 3 &  1 &  3 & 1 \\
 1 & 7 &  6 & -1 & 2
\end{array} \right] \]
\[b = \left[ \begin{array}{c}
8 \\ 8 \\ 25
\end{array}  \right] \]
and we have determined a basis,
\[B = \{1, 3, 5\}.\]
($B$ is specified here by identifying the {\it basic} columns.  In other words,
 $x_2 = x_4 = 0$.)
This gives us the following three linearly independent equations,
which can be solved to find the corresponding bfs:
\begin{eqnarray*}
5 x_1 - 3x_2 + 4 x_5 & = & 8 \\
2 x_1 + x_2 + x_5 & = & 8 \\
x_1 + 6 x_3 + 2 x_5 & = & 25
\end{eqnarray*}
The solution, $x_1 = 1$, $x_3 = 3$, $x_5 = 3$; can also be written
as:
\[bfs = \left[ \begin{array}{c}
1 \\ 0 \\ 3 \\ 0 \\ 3
\end{array}  \right]. \]

We now perform a pivot to find another basis, by adding a nonbasic column
to $B$, and removing one of the basic columns from $B$.
Assume that we choose to add column $A_4$.  We can express $A_4$
as a linear combination of the columns that are in $B$ as follows:
\begin{eqnarray*}
A_4 & = & \alpha A_1 + \beta A_3 + \gamma A_5 \\
16 & = & 5 \alpha - 3 \beta + 4 \gamma \\
3 & = & 2 \alpha + \beta + \gamma \\
-1 & = & \alpha + 6 \beta + 2 \gamma
\end{eqnarray*}
The solution is: $\alpha = 1$, $\beta = -1$, $\gamma = 2$.  Therefore,
$A_4 = A_1 - A_3 + 2A_5$.  Since we want to find a new basis, $B'$, with
a feasible solution, we must find a basic column to remove, such that:
\begin{itemize}
\item $A_{B'} x_{B'} = b$,
\item $x_{B'} \geq 0$,
\item and $x_2 = 0$.
\end{itemize}
where $A_{B'}$ refers to the basic columns of $A$.  We have already solved
\begin{eqnarray*}
A_1 x_1 + A_3 x_3 + A_5 x_5 & = & b
\end{eqnarray*}
and want to modify that solution to include $A_4$ in the basis so that:
\begin{eqnarray*}
A_1 {x'}_1 + A_3 {x'}_3 + A_4 \Theta + A_5 {x'}_5 & = & b \\
{x'}_1, {x'}_3, {x'}_5 & \geq & 0
\end{eqnarray*}
Since we know that $A_4 = A_1 - A_3 + 2A_5$, we simply need to find
$\Theta \geq 0$ such that,
\begin{eqnarray*}
A_1 ({x'}_1 + \Theta) + A_3 ({x'}_3 - \Theta) + A_5 ({x'}_5 + 2 \Theta)
& = & b.
\end{eqnarray*}
But we know that:
\begin{eqnarray*}
A_1 x_1 + A_3 x_3 + A_5 x_5 & = & b
\end{eqnarray*}
so we have:
\[ \begin{array}{cccccc}
{x}_1 & = & {x'}_1 + \Theta  = & 1, & \Rightarrow & \Theta \leq 1 \\
{x}_3 & = & {x'}_3 - \Theta  = & 3, & \Rightarrow & \Theta \geq -3 \\
{x}_5 & = & {x'}_5 + 2 \Theta  = & 3, & \Rightarrow & \Theta \leq \frac{3}{2}
\end{array} \]
We simply choose the smallest non-negative $\Theta$
that satisfies these conditions, namely
$\Theta = 1 \Rightarrow x_1 = 0$.  Thus our new basis is
$B' = \{4, 3, 5\}$, and our bfs has ${x'}_4 = 1, {x'}_3 = 4, {x'}_5 = 1$, and
we have completed a pivot. \qed
}
\end{Example}

Let us review the steps involved in pivoting.  The idea is to start from a
basis, $B = \{1, 2, \ldots , m\}$ and then find a suitable 
nonbasic column $j \in \{1, 2, \ldots , m\}$ to replace a basic column
$l \not\in \{1, 2, \ldots , m\}$ in order to arrive at an adjacent basis, $B'$.
This is achieved using the following steps:
\begin{enumerate}
\item Assume that the starting basis $B = \{1, 2, \ldots , m\}$, is known.

      $B$ contains $m$ linearly independent columns of the $n$ columns in $A$.
      We have already found the bfs for $B$, ${bfs}_B$ which satisfies:
\begin{eqnarray}
\label{eqn:one}
\sum_{i=1}^{m} A_i x_i & = & b \\
\label{eqn:two}
x & \geq & 0
\end{eqnarray}

\item Choose a nonbasic column $j \not\in \{1, 2, \ldots , m\}$ to add to $B$.

(In the Simplex Algorithm, each $j \not\in \{1, 2, \ldots , m\}$ can be tried
in turn until one is found that results in a bfs with an {\em improved} cost.)

\item Find the linear dependence of column $j$ on the basic columns, by
      finding $\alpha_{i,j}$ such that:
\begin{eqnarray}
\label{eqn:three}
A_j & = & \sum_{i=1}^{m} \alpha_{i,j} A_i.
\end{eqnarray}

\item Find our options for ${x'}_j = \Theta$, so that
\begin{eqnarray}
\label{eqn:four}
\sum_{i=1}^{m} A_i {x'}_i + A_j {x'}_j & = & b
\end{eqnarray}
We can find the feasible range of values for $\Theta$ as follows:
\[ \begin{array}{lccc}
{\displaystyle \sum_{i=1}^{m} A_i x_i} + (A_j {x'}_j - A_j {x'}_j) & = & b &
\mbox{(based on Equation \ref{eqn:one})} \\
{\displaystyle \sum_{i=1}^{m} A_i x_i} + A_j {x'}_j -
\Theta \sum_{i=1}^{m} \alpha_{i,j} A_i \\
& = & b & \mbox{(based on Equation \ref{eqn:three})} \\
{\displaystyle \sum_{i=1}^{m} A_i (x_i - \Theta \alpha_{i,j})}
 + A_j {x'}_j & = & b
\end{array} \]
So ${x'}_i = x_i - \Theta \alpha_{i,j}$, and in order for
${x'}_i \geq 0$, we must have $\Theta \leq \frac{x_i}{\alpha_{i,j}}$.

\item Choose the basic column to $l \in \{1, 2, \ldots , m\}$ to remove from
      $B$.

We want to choose the smallest value of $\Theta$ that will result in a
feasible solution, so we take $\Theta$ to be the
\[\underbrace{\min}_{\forall \alpha_{i,j} > 0} \frac{x_i}{\alpha_{i,j}}.\]
$l$ is the value of $i$ associated with our choice of $\Theta$.

(N.B. If $\forall \alpha_{i,j} \leq 0$, then the solution is unbounded.)
\end{enumerate}

\subsection{Tableau Method}

The {\em Tableau Method} provides an efficient means of performing the
pivots required for the Simplex Algorithm.  The idea is to manipulate
$A$ and $b$ so that the basic columns of $A$ become the identity matrix.
When this happens, Step 3 of the pivoting process (above)---finding
the $\alpha_{i,j}$ coefficients for $A_j$---becomes trivial, because
(based on Equation \ref{eqn:three}) $\alpha_{i,j} = a_{i,j}$
(where $a_{i,j}$ is the $(i,j)$ entry of $A$---the member in the $ith$ row of
$A_j$).

\begin{Example}
\label{ex:two}
{\em
Consider $LP_2$ defined by
\begin{eqnarray*}
6 & = & x_1 + x_2 + 4 x_6 \\
14 & = & -2 x_1 + 2 x_2 + x_3 - x_4 + x_6 \\
-11 & = & x_1 - 2 x_2 + x_4 + 2 x_6 \\
7 & = & x_1 - 3 x_4 + x_5 - 5 x_6
\end{eqnarray*}
for which we have determined that a basis, $B = \{2, 3, 4, 5\}$.
We can represent $LP_2$ by an $m \times (n + 1)$ matrix, a {\it tableau},
as follows:
\[ \begin{array}{|c|c|c|c|c|c|c|} \hline
b & A_1 & A_2 & A_3 & A_4 & A_5 & A_6 \\ \hline
6    &  1 & 1  & 0 &  0 & 0 & 4 \\ \hline
14   & -2 & 2  & 1 & -1 & 0 & 1 \\ \hline
-11  &  1 & -2 & 0 &  1 & 0 & 2 \\ \hline
7    &  1 & 0  & 0 & -3 & 1 & -5 \\ \hline
\end{array} \]
By performing row operations on the tableau, we can transfrom the columns
that represent $B$ into the identity matrix, without affecting the solution
space.  This transformation yields,
\[ \begin{array}{|c|c|c|c|c|c|c|} \hline
b & A_1 & A_2 & A_3 & A_4 & A_5 & A_6 \\ \hline
6  &  1 & 1  & 0 &  0 & 0 & 4 \\ \hline
3  & -1 & 0  & 1 &  0 & 0 & 3 \\ \hline
1  &  3 & 0  & 0 &  1 & 0 & 10 \\ \hline
10 & 10 & 0  & 0 &  0 & 1 & 25 \\ \hline
\end{array} \]
There are two possible nonbasic columns that we may consider adding to
our basis as part of a pivot.  Assume that we choose column 1, so that
$j = 1$.  Then we have,
\begin{eqnarray*}
A_j = A_1 & = & 1 A_2 - 1 A_3 + 3 A_4 + 10 A_5
\end{eqnarray*}
$\Theta$ is determined by
\[ \Theta = \underbrace{\min}_{\forall \alpha_{i,1} > 0} \frac{x_i}{\alpha_{i,j}}
= \frac{1}{3}.\]
(since $\alpha_{i,j} = a_{i,j}$), so that $l = 4$, and the new basis, $B'$,
is $\{2, 3, 1, 5\}$.

Row operations are once again performed in order to convert $B'$ to the
identity matrix (and update $b$ to the new bfs $b'$), yielding
\[ \begin{array}{|c|c|c|c|c|c|c|} \hline
b' & {A'}_1 & {A'}_2 & {A'}_3 & {A'}_4 & {A'}_5 & {A'}_6 \\ \hline
\frac{17}{3}  &  0 & 1  & 0 &  \frac{-1}{3} & 0 & \frac{2}{3} \\ \hline
\frac{10}{3}  &  0 & 0  & 1 &  \frac{1}{3} & 0 & \frac{19}{3} \\ \hline
\frac{1}{3}  &   1 & 0  & 0 &  \frac{1}{3} & 0 & \frac{10}{3} \\ \hline
\frac{20}{3} &  0 & 0  & 0 &  \frac{-10}{3} & 1 & \frac{-25}{3} \\ \hline
\end{array} \]
The cost function, $c^T x$, is what actually determines which of the nonbasic
columns gets chosen to include in the new basis---we must find one which
results in decrease in the cost.  For any proposed replacement of some
column $l$ by some column $j$, we must simply check that the cost of the
new solution is less that the cost of the previous solution. 
The cost of new solution is given by:
\begin{eqnarray*}
\mbox{new cost} & = & \sum_{i=1}^{m} {x'}_i c_i + {x'}_j c_j \\
&= & \sum_{i=1}^{m} (x_i - \Theta \alpha_{i,j}) + \Theta c_j \\
&= & \underbrace{\sum_{i=1}^{m} x_i c_i}_{\mbox{old cost}} + \Theta 
\underbrace{[c_j - \sum_{i=1}^{m} \alpha_{i,j} c_i]}
_{\mbox{modified cost, $\bar{c}_j$}}
\end{eqnarray*}
So if the modified cost, $\bar{c}_j$, is negative,, then the new cost will
be reduced. \qed
}
\end{Example}

Let us review how pivoting is performed using the Tableau Method.
A tableau, constructed as follows, is used to perform pivots:
\[ \begin{array}{|c|c|c|c|c|} \hline
\ast   & \bar{c}_1 & \bar{c}_2 & \ldots & \bar{c}_n \\ \hline
b_1    & A_{1,1} & A_{1,2} & \ldots & A_{1,n} \\ \hline
b_2    & A_{2,1} & A_{2,2} & \ldots & A_{2,n} \\ \hline
\vdots &  \vdots &  \vdots & \vdots & \vdots  \\ \hline
b_m    & A_{m,1} & A_{m,2} & \ldots & A_{m,n} \\ \hline
\end{array} \]
in which the basic columns of $A$ form the identity matrix, and
the each nonbasic column, $j$, contains the coefficients, $\alpha_{i,j}$.
(An additional row, row 0, is used to store the modified costs.)

A pivot is made with the following steps:
\begin{enumerate}
\item Find a column $j$ such that $\bar{c}_j < 0$, by examining row 0.

(If no such column exists, we have found the optimal solution.)

\item Find column $l$ with $\alpha_{l,j} > 0$, that minimizes $\Theta$ 

(If all $\alpha_{i,j} \leq 0$ then either the solution is unbounded--unless
we didn't have a basis to begin with.)

\item Replace $l$ by $j$
\end{enumerate}

We have yet to specify how the starting basis is found an LP.
Suppose we are given $LP_A$: minimize $c^T x$ subject to $Ax \leq b$,
and $x \geq 0$.  Then we can add one {\it surplus}
variable per constraint, such that:
\begin{eqnarray*}
\sum_j a_{i,j} x_j + s_i = b_i, & i \in \{1, 2, \ldots, m\}
\end{eqnarray*}
Then our bfs is: \{$x_j = 0, s_i = b_i\}$.
If instead of $Ax \leq b$ we have $Ax = b$, then we can similarly
add {\it artificial} variables, such that:
\begin{eqnarray*}
\sum_j a_{i,j} x_j + y_i = b_i  i \in \{1, 2, \ldots, m\}
\end{eqnarray*}
to form $LP_B$ (for which we can easily construct a basis)
for which we minimize $\sum y_i$, for $y_i \geq 0$.
$LP_A$ will be feasible iff the optimal solution for $LP_B$ is 0.
To get a bfs for $LP_A$ we solve $LP_B$ using the Simplex Algorithm by
starting with the bfs, $\{x_j = 0, y_i = b_i\}$, and if the optimal solution
for $LP_B$ is 0, then all $y_i's$ are 0 and the $x_i'$ form a bfs for $LP_A$.

\section{LP Relaxations}

Integer optimization problems can often be approximated using a technique
known as LP Relaxation.  Given an Integer Program (IP), relax the integers-only
conditions and find an optimal solution to the resulting Linear Program.
Then use ``rounding'' to construct an integers-only solution.
In subsequent lectures we shall examine the application of LP
Relaxation-techniques to approximate several NPC problems.
We demonstrate this approach by showing how it can be used to solve the
Weighted Set Cover problem.

\section{LP Relaxation Approach to Weighted Set Cover}

\begin{Definition}[Weighted Set Cover]
\label{def:WSC}
\ \\{\em INSTANCE:} A finite set ${\cal U} = \{1, 2, \ldots, n\}$;
a collection of subsets of ${\cal U}$, ${\cal F} = \{S_1, S_2, \ldots, S_m\}$;
a weight, $w_i$ for each $S_i \in {\cal F}$.

{\em PROBLEM:} To find a subset of ${\cal F}$, $C$, such that every element
in ${\cal U}$ is covered by at least one set in $C$, and for which the total
weight, $\sum_{S_i \in C} w_i$, is minimized.
\end{Definition}

\subsection*{IP for Weighted Set Cover}
\[ x_i = \left\{ \begin{array}{ll}
        1 & \mbox{if $S_i \in C$} \\
        0 & \mbox{otherwise}
\end{array} \right. \]
\[ \begin{array}{lll}
\mbox{minimize} & {\displaystyle \sum_{i=1}^{m} w_i x_i }\\
\mbox{subject to} & {\displaystyle \sum_{S_i: j \in S_i} x_i \geq 1,  \forall j \in {\cal U}}\\
& x_i \in {0, 1} \end{array} \]

Let $W$ be the optimal solution to the IP.

\subsection*{LP Relaxation for Weighted Set Cover}
\[ \begin{array}{lll}
\mbox{minimize} & {\displaystyle \sum_{i=1}^{m} w_i x_i} \\
\mbox{subject to} & {\displaystyle \sum_{S_i: j \in S_i} x_i \geq 1, \forall j \in {\cal U}}\\
& 0 \leq x_i \geq 1, \forall i \end{array} \]

Let $\hat{W}$ be the optimal solution to the LP.  Then $\hat{W} \leq W$.

\subsection*{Randomized Rounding of the LP Solution}
The optimal solution to the LP, $\hat{W}$, describes the solution by
specifying $\hat{x}_i$.  We use Randomized Rounding to find a solution for
the IP as follows:  Select the set $S_i$ with probability $\hat{x}_i$,
independent of what is done with any other set.

There are two obvious concerns to using such an approach:
\begin{itemize}
\item Feasibility.  Will rounding produce a set cover?

Consider an element $j \in {\cal U}$.
Suppose $j \in \{S_1, S_2, \ldots, S_k\}.$
Then
\begin{eqnarray*}
\Pr[\mbox{not covering $j$}] & = & (1 - \hat{x}_1) (1 - \hat{x}_2) \ldots (1 - \hat{x}_k)
\end{eqnarray*}
And since the LP had a solution, one of these $\hat{x}_i$'s must be
non-zero, so that we have a probability less than 1.  Since the
geometric mean of $1 - \hat{x}_1$, $1 - \hat{x}_2$, \ldots, $1 -
\hat{x}_k$ is at most their arithmetic mean, we have:
\begin{eqnarray*}
\left((1 - \hat{x}_1) (1 - \hat{x}_2) \ldots (1 - \hat{x}_k)\right)^{1/k} \le (1 - \sum \frac{\hat{x}_i}{k}).
\end{eqnarray*}
This implies that
\begin{eqnarray*}
\Pr[\mbox{not covering $j$}] & \leq & \left( 1 - \sum \frac{\hat{x}_i}{k} \right)^k
\leq \frac{1}{e}
\end{eqnarray*}

\item Cost.  Will rounding blow up the value of the solution?

The expected cost of the solution is:
\[ \begin{array}{llllll}
E[\mbox{cost}] & = & \sum w_i \hat{x}_i & = & \hat{W} & \leq W
\end{array} \]
\end{itemize}

So, after having solved the LP optimally, we can simply repeat the
Randomized Rounding step $t$ times and include any set, $S_i$ that is
selected during any of the $t$ trials.  We then have:
\[ \begin{array}{lll}
\Pr[\mbox{$j$ not covered after 1 trial}] & \leq & \frac{1}{e}\\
\Pr[\mbox{$j$ not covered after $t$ trials}] & \leq & \frac{1}{e^t}
\end{array} \]

If we select $t = 2 \ln \ n$, then we have:
\[ \begin{array}{lllll}
\Pr[\mbox{$j$ not covered after $t$ trials}] & \leq & \frac{1}{e^t} & =
\frac{1}{e^{2 \ln \ n}} & = \frac{1}{n^2}
\end{array} \]
And since Boole's inequality states that
\[ \begin{array}{lll}
\Pr[\bigcup_{i} E_i] & \leq & \sum \Pr[E_i]
\end{array} \]
we have:
\[ \begin{array}{lll}
\Pr[\mbox{$\exists j$ not covered}] & \leq & \frac{1}{n}.
\end{array} \]

If $\hat{W}_i$ is the cost of experiment (trial) $i$, then we have:
\[ \begin{array}{lllll}
E[\hat{W}_i] & \leq & W \\
E[\hat{W}] & = & \sum_{i=1}^{t} \hat{W}_i & \leq & t W \\
E[\hat{W}] & \leq & t W & \leq & (2 \ln \ n) W
\end{array} \]
and with $\Pr \leq 1 - \frac{1}{n}$ we will have a feasible solution with
$cost \leq (2 \ln \ n) W.$

\begin{Definition}[Markov's Inequality]
\label{def:MI}
\ \\
Let $X$ be a non-negative random variable.  Then,
\[ \begin{array}{lllll}
\Pr[X \geq c] & \leq & \frac{E[X]}{c}, & c > 0.\\
E[X] & = & \sum_{\alpha} \alpha \Pr[X = \alpha]\\
     & \geq & \sum_{\alpha \geq c} \alpha \Pr[X = \alpha]\\
     & \geq & \sum_{\alpha \geq c} c \Pr[X = \alpha]\\
     & = & c \Pr[X \geq c]
\end{array} \]
\end{Definition}

By Markov's Inequality, we have:
\[ \begin{array}{lll}
\Pr[\hat{W} \geq 2 c \ \ln \ n \ W] & \leq & \frac{1}{c}
\end{array} \]

\begin{Theorem}
\label{thm:one}
For $c = 4$, the Randomized Rounding Algorithm returns a feasible solution with
probability $\geq 1 - \frac{1}{n}$; and has $cost \leq 8 \ln W$, with
probability $\geq \frac{3}{4}.$
\end{Theorem}

Note, the algorithm runs in $O(\log \ n)$ time.

\bibliographystyle{alpha}
\bibliography{/home/rraj/tex/bibtex/refs}

\end{document}

