\documentclass[11pt]{article}
\usepackage{handout}
\usepackage{amsmath,amssymb,amsthm,epsfig}
\input{epsf}
\begin{document}
\newtheorem{Theorem}{Theorem}
\newtheorem{Corollary}{Corollary}
\newtheorem{Lemma}{Lemma}
\newtheorem{Claim}{Claim}
\theoremstyle{definition}
\newtheorem{Definition}{Definition}
\newtheorem{problem}{Problem}
\newenvironment{Proof}{\noindent{\bf Proof: }}{\qed}
%\theoremstyle{remark}
\newtheorem{Remark}{Remark}
\newtheorem{Example}{Example}
\newcommand{\example}[1]{
\begin{Example}
{\em #1} \comqed
\end{Example}
}


\thispagestyle{empty}
\thispagestyle{empty}
\setlength{\parindent}{0pt}
\setlength{\parskip}{1.8ex}

\newcommand{\Opt}{\mbox{OPT}}

\newcommand{\p}{\mathcal{P}}
\newcommand{\U}{\mathcal{U}}
\newcommand{\coll}{\mathcal{S}}
\newcommand{\xpy}{x + \lambda y}
\newcommand{\Alg}{\mathcal{A}}
\newcommand{\opt}{\mbox{OPT}}
\newcommand{\eps}{\varepsilon}

\newcommand{\lecturehead}[3]{
{\noindent College of Computer \& Information Science \hfill #3}\\
{\noindent Northeastern University \hfill #1: #2}}


\lecturehead{CS7800}{Advanced Algorithms}{Fall 2014}

%\newcommand{\p}{\mathcal{P}}
%\newcommand{\U}{\mathcal{U}}
%\newcommand{\coll}{\mathcal{S}}
%\newcommand{\xpy}{x + \lambda y}
%\newcommand{\Alg}{\mathcal{A}}
%\newcommand{\opt}{\mbox{OPT}}
%\newcommand{\eps}{\varepsilon}

\noindent {\bf Lecture Outline:}
\begin{itemize}
\item
Introduction
\item
Geometry \& Duality
\item
The Simplex Algorithm
\end{itemize}

Two excellent reference sources (among many others) are Michel
Goemans's lecture notes on linear programming~\cite{goemans:lp}, and
Howard Karloff's text on linear programming~\cite{karloff:lp}.  Some
of the material has been drawn from these two sources.  Thanks also to
Clifford Bryant, Jr., Mohsen Ghassemi, Trevor Mendez, Eric Robinson,
and San Tan, who all contributed to these notes when they were
graduate students here.

\section{Introduction}
Many linear programming formulations arise from situations where a decision 
maker wants to minimize the cost of meeting a set of requirements. In the diet 
problem we would like to develop a diet using $n$ food items such that it 
satisfies the daily vitamin requirements. Let the food items be numbered
1 through $n$ and let the $m$ vitamin mineral requirements be given by
$b_1, \ldots, b_m$. 

From food item $j$, you get $a_{ij}$ units of mineral $i$ per unit of $j$.
If you decide to have $x_{j}$ units of item $j$, you get $x_{j}a_{ij}$
units of mineral $i$.

\begin{eqnarray*}
\sum_{j} x_j a_{ij} \geq  b_{i} \ \ \ \forall i \\
\end{eqnarray*}
If the cost of food item $j$ is $c_{j}$ per unit. Then the total cost
incurred = $\sum_{j} c_{j} x_{j}$.

We would like to \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ {minimize} $\sum_{j} c_j \ x_j$ \\
\ \ subject \ to: 
\begin{eqnarray*}
\ \ \sum_{j=1}^{n} x_i a_{i,j} \ \geq \ b_{i} \ \ i=1,..,m \\
\end{eqnarray*}
Alternatively, one can write the above optimization problem as: 
\[ minimize \left[ \begin{array}{llll} c_{1} & c_{2} \ldots c_{n}
\end{array} \right]
\left[ \begin{array}{c}
x_{1} \\ x_{2} \\ \vdots \\ x_{n}
\end{array} \right] \]
\[ \ \ subject \ to: \ \left[ \begin{array}{ll}
a_{11} & a_{12} \ldots a_{1n} \\
a_{21} & a_{22} \ldots a_{2n} \\
a_{m1} & a_{m2} \ldots a_{mn}
\end{array} \right] \left[ \begin{array}{c}
x_{1} \\ x_{2} \\ \vdots \\ x_{n} 
\end{array} \right] \geq 
\left[ \begin{array}{ll}
b_{1} \\ b_{2} \\ \vdots \\ b_{n}
\end{array} \right] \]

The above problem is of the form min $c^Tx$ {subject} {to:} $Ax \geq b$. Such 
that a problem is referenced to as a {\it linear program}. This is because
the objective function as well as the constraints are linear combination
of the variables.

We now show how optimization problems can often be written down as
linear programs.

\subsection{Knapsack Problem}

In the knapsack problem we are given $n$ items of sizes $w_{1}$
through $w_{n}$ and profits $p_{1}$ through $p_{n}$. The goal is to
build the items with total size $\leq$ $B$ such that the total profit
is maximized.

The LP representation of the problem can be shown as:

\[ \begin{array}{lll}
maximize & \sum x_j p_j \\
subject \ to & \sum_{j=1}^{n} x_j w_j \leq B \\
& x_j \in \{0,1\} 
\end{array} \]
\[ c^T = \left[ \begin{array}{c}
p_{1} \\ \vdots \\ p_{n} 
\end{array} \right] \]
\[ A = \left[ \begin{array}{ll} 
w_{1} & w_{2} \ldots w_{n} 
\end{array} \right] \]
\[ b = \left[ \begin{array}{ll} 
B
\end{array} \right] \]
Note that the constraint $x_j \in$ \{0,1\} is not a linear constraint so
that the above program is an integer linear program. 
\subsection{Minimum Spanning Tree Problem}
Recall that in the minimum spanning tree problem, the set of instances
is the set of all weighted undirected graphs. For a given instance
graph $G$, the set $S(G)$ of solutions for $G$ is the set of all trees
that span every vertex of $G$.  Finally, the value of a solution tree
$T$ is simply the sum of the weights of the edges of $T$.

One LP representation of the problem is: 

\[ \begin{array}{lll}
minimize & \sum_{e} w_e x_e \\
subject \ to & \sum_{e \ \in \ c} x_e \geq 1  & \ for \ all \ cut \ c  \\
& x_e = {0 \ or \ 1} \ & \ for \ all \ edges \ e 
\end{array} \]
Note that since there are an exponential number of cuts, the number of 
constraints is exponential.

\subsection{Weighted Vertex Cover Problem}
A vertex cover of undirected graph $G = (V.E)$ is a subset ${V}'
\subseteq {V}$ such that if (${u,v}) \in {E}$, then ${u} \in {V}'$ or
${v} \in {V}'$ (or both). That is, each vertex covers its incident
edges, and a vertex cover for ${G}$ is a set of vertices that covers
all the edges in ${E}$.

The LP representation of the problem is: 

\[ \begin{array}{lll}
minimize & \sum_{u} x_u w_u \\
subject \ to & x_u + x_v \geq 1 & \forall ({u},{v}) \in E \\
& x_u, x_v = {0 \ or \ 1} 
\end{array} \]

\section{Equivalent Forms}
\label{sec:form}
We define the linear programming (LP) problem of minimizing a linear
function subject to linear inequality constraints. The general,
standard, and canonical forms of the linear programming problem are
given in summation and matrix form.  A formal definition is given for
a vertex of a polytope or polyhedron, and it is proved that an LP
always attains its optimum at a vertex.  Then it is proved that the
set of vectors corresponding to the current basis of an LP are
linearly independent if and only if the basic feasible solution is a
vertex point.


\subsection{General Form}
\label{sec:form:gen}

This is the general form of the linear programming problem.
The $c_{i}$'s can be interpreted as costs.
In this case, the objective is to minimize the total cost
subject to the linear constraints.
\begin{equation}
\mathrm{Minimize} \ \sum_{i=1}^{n} c_{i} x_{i} \qquad
\mathrm{(objective \ function)} \label{eqn:gen:sum}
\end{equation}
\begin{displaymath} \begin{array}{lrcrll} \mathrm{subject \ to} & \sum_{j=1}^{n}
a_{ij} x_{j} & \geq & b_{i} & , i = 1, \ldots, m_{1} &
\mathrm{(inequality)} \\
 & \sum_{j=1}^{n} a_{ij} x_{j} & = & b_{i} & , i = m_{1} + 1, \ldots, m_{1} + m_{2} & \mathrm{(equality)} \\
 & x_{j} & \geq & 0 & , j = 1, \ldots, n_{1} & \mathrm{(non-negativity)} \\
 & x_{j} & \lesseqgtr & 0 & , j = n_{1} + 1, \ldots, n & \mathrm{(unconstrained)}
\end{array} \end{displaymath}

The general form of the LP can be written more compactly in matrix notation.
\begin{equation}
\mathrm{Minimize} \ \mathbf{c}^{\mathit{T}} \mathbf{x}
\label{eqn:gen:mat}
\end{equation}
\[ \begin{array}{lrcl}
\mathrm{subject \ to} & \mathbf{Ax_{\mathrm{1}}}                    & \geq & \mathbf{b} \\
                      & \mathbf{A}\mathit{'}\mathbf{x_{\mathrm{2}}} & =    & \mathbf{b}\mathit{'} \\
                      & \mathbf{x}_{\mathrm{1}}                     & \geq & 0 \\
                      & \mathbf{x}_{\mathrm{2}}                     & <=>  & 0
\end{array} \]
where
\begin{description}
\item $\mathbf{c}$ and $\mathbf{x}$ are $n$ x $1$ vectors,
\item $\mathbf{A}$ is an $m_{1}$ x $n$ matrix,
\item $\mathbf{A}\mathit{'}$ is an $m_{2}$ x $n$ matrix,
\item $\mathbf{b}$ is an $m_{1}$ x $1$ vector,
\item $\mathbf{b}\mathit{'}$ is an $m_{2}$ x $1$ vector,
\item $\mathbf{x_{\mathrm{1}}}$ is an $n_{1}$ x $1$ vector,
\item $\mathbf{x_{\mathrm{2}}}$ is an $n_{2}$ x $1$ vector, and
\item $ \mathbf{x} = \left\{ \begin{array}{cc} \mathbf{x_{\mathrm{1}}} \\ \mathbf{x_{\mathrm{2}}} \end{array} \right\} $.
\end{description}

\subsection{Standard Form}
\label{sec:form:std}

This is the standard form of the linear programming problem.
Here the constraints take the form of linear equality constraints,
plus non-negativity constraints on the independent variables.
\begin{equation}
\mathrm{Minimize} \ \sum_{i=1}^{n} c_{i} x_{i} \qquad
\mathrm{(objective \ function)} \label{eqn:std:sum}
\end{equation}
\[ \begin{array}{lrcrll}
\mathrm{subject \ to} & \sum_{j=1}^{n} a_{ij} x_{j} & =    & b_{i} & , i = 1, \ldots, m & \mathrm{(equality)} \\
                      & x_{j}                       & \geq & 0     & , j = 1, \ldots, n & \mathrm{(non-negativity)}
\end{array} \]

The standard form of the LP can also be written more compactly in matrix notation.
\begin{equation}
\mathrm{Minimize} \ \mathbf{c}^{\mathit{T}} \mathbf{x}
\label{eqn:std:mat}
\end{equation}
\[ \begin{array}{lrcl}
\mathrm{subject \ to} & \mathbf{Ax} & =    & \mathbf{b} \\
                      & \mathbf{x}  & \geq & 0
\end{array} \]
where
\begin{description}
\item $\mathbf{c}$ and $\mathbf{x}$ are $n$ x $1$ vectors,
\item $\mathbf{A}$ is an $m$ x $n$ matrix, and
\item $\mathbf{b}$ is an $m$ x $1$ vector.
\end{description}

\begin{Definition}
\label{def:feasible} If $x$ satisfies $\mathbf{Ax} = \mathbf{b}, \
\mathbf{x} \geq \mathbf{0}$, then $x$ is \textbf{feasible}.
\end{Definition}

\subsection{Canonical Form}
\label{sec:form:can}

This is the canonical form of the linear programming problem.
Here the constraints take the form of linear inequality constraints,
plus non-negativity constraints on the independent variables.
\begin{equation}
\mathrm{Minimize} \ \sum_{i=1}^{n} c_{i} x_{i} \qquad
\mathrm{(objective \ function)} \label{eqn:can:sum}
\end{equation}
\[ \begin{array}{lrcrll}
\mathrm{subject \ to} & \sum_{j=1}^{n} a_{ij} x_{j} & \geq & b_{i} & , i = 1, \ldots, m & \mathrm{(inequality)} \\
                      & x_{j}                       & \geq & 0     & , j = 1, \ldots, n & \mathrm{(non-negativity)}
\end{array} \]

Once more, the canonical form of the LP can be written more compactly in matrix notation.
\begin{equation}
\mathrm{Minimize} \ \mathbf{c}^{\mathit{T}} \mathbf{x}
\label{eqn:can:mat}
\end{equation}
\[ \begin{array}{lrcl}
\mathrm{subject \ to} & \mathbf{Ax} & \geq & \mathbf{b} \\
                      & \mathbf{x}  & \geq & 0
\end{array} \]
where
\begin{description}
\item $\mathbf{c}$ and $\mathbf{x}$ are $n$ x $1$ vectors,
\item $\mathbf{A}$ is an $m$ x $n$ matrix, and
\item $\mathbf{b}$ is an $m$ x $1$ vector.
\end{description}

\subsection{Constraint Conversion}

It is possible to convert between equality and inequality
constraints by the following method.
\begin{enumerate}
\item To convert a less than or equal inequality constraint,
\[ Ax \leq b\]
to an equality constraint, add the vector of slack variables, $s$.
\[ Ax + s = b, \ s \geq 0\]
\item To convert a greater than or equal inequality constraint,
\[ Ax \geq b\]
to an equality constraint, add the vector of surplus variables,
$t$.
\[ Ax - t = b, \ t \geq 0\]
\item Finally, to convert an equality constraint, \[ Ax = b \] to
an inequality constraint, add two inequality constraints.
\[ Ax \leq b, \ -Ax \leq -b \]
\end{enumerate}

\section{Example}
\label{sec:example}

\begin{Example}
\label{ex:one}

Consider the following linear program:
\[ \begin{array}{lrcrcr}
\mathrm{Minimize}     &        &   &  x_{2} &      &   \\
\mathrm{subject \ to} &  x_{1} &   &        & \geq & 2 \\
                      & 3x_{1} & - &  x_{2} & \geq & 0 \\
                      &  x_{1} & + &  x_{2} & \geq & 6 \\
                      & -x_{1} & + & 2x_{2} & \geq & 0 \\
                      &  x_{1} &   &        & \geq & 0 \\
                      &        &   &  x_{2} & \geq & 0
\end{array} \]
The optimal solution is (4, 2) of cost 2 (See
Figure~\ref{fig:inequal}). If we were maximizing $x_{2}$, instead
of minimizing under the same feasible region, the resulting linear
program would be unbounded, since $x_{2}$ can increase
arbitrarily. From this picture, the reader should be convinced
that, for any objective function for which the linear program is
bounded, there exists an optimal solution which is a ``corner'' of
the feasible region. This notion will be formalized in the next
section.
\begin{figure}
\begin{center}
\epsfig{file=inequal.eps} \caption{Feasible Region}
\label{fig:inequal}
\end{center}
\end{figure}
\end{Example}

\section{Solving LPs}

Below lists the three main linear programming algorithms listed historically:
\begin{description}
\item[Simplex] - Invented by George Dantzig in 1947 to solve linear
programming problems, this technique was fast for most practical
applications.  However, it is non-polynomial in worst case scenarios,
and later examples were provided where simplex failed to perform
efficiently.
\item[Ellipsoid] - Introduced by Naum Z. Shor, Arkady Nemirovsky, and
David B. Yudin in 1972, and shown to be polynomial by Leonid
Khachiyan, this technique, while polynomial, is in practice typically
much slower than the simplex algorithm and was therefor rarely used.
It was useful for showing that general linear programming was in P
however.
\item[Interior Point] - Introduced and developed by Narendra Karmarkar
in 1984, this mehtod has the advantage of being polynomial like the
ellipsoid algorithm, but also fast in practice, like the simplex
algorithm.  For this reason it was used in practice for a long time.
Now, however, hybrids and other variations are used in many cases.
\end{description}

\section{The Geometry of LP}
Consider a linear program with $n$ variables and $m$ linear
constraints.  The set of possible values for the $n$ variables is a
subset of $\Re^n$.  Each constraint corresponds to a half-space of
$\Re^n$ as defined by an associated hyperplane.  The body enclosed
by the set of these hyperplanes is referred to as {\em polytope}.

For any linear program, exactly one of the following conditions holds:
\begin{itemize}
\item The solution is unbounded (infinite)
\item The solution is infeasible (no polytope exists)
\item An optimal solution occurs on a vertex of the polytope
\end{itemize}

A polytope is always convex, by which we mean that 
\begin{itemize}
\item A line between 2 points in the polytope remains in the polytope
\item If $x, y \in P$, $\alpha x + (1-\alpha)y \in P$ for $\alpha \in [ 0, 1 ]$
\end{itemize}

A vertex, $x$, of a convex polytope $P$, is a ``corner point'' in the
polytope and can be defined by any one of these three equivalent
statements, as we show next.
\begin{itemize}
\item $\neg \exists y \not = 0 \mbox{ s.t. } x + y, x - y \in P$
\item $\neg \exists y, z \in P \mbox{ s.t. } x = \alpha y + (1-\alpha)z, \alpha \in ( 0, 1 )$
\item $\exists n$ linear inequalities of $P$ that are tight at $x$
\end{itemize} 

The proof that definitions 1 and 2 are equivalent shall be left to the
reader.  Here we will prove that claims 1 and 3 are equivalent.

\smallskip
\noindent{1 $\implies$ 3}: Suppose $\neg 3$.  Let $A'$ be the
submatrix corresponding to the tight inequalities, let b' be the
corresponding right hand sides.  This leads to the inequality $A'x =
b'$.  The rank of $A'$ is strictly less than $n$.  This implies that
$\exists y \not = 0 \mbox{ s.t. } A'y = 0$.

Consider $x + \lambda y$, $A'(x+\lambda y) = A'x + \lambda A' y = b'$.
Find $\lambda \ge 0 \mbox{ s.t. } x + \lambda y, x - \lambda y\in P$.
Now consider any one of the non-tight inequalities, $j$.  $A_j x >
b_j$.  Apply $\lambda$, getting $A_j(x+ \lambda y) = A_j x + \lambda
A_j y$.  It is known that $A_j x > b_j$.  Because $A_j x = b_j$ is not
tight, $\lambda A_j y$ must be feasible in one direction for a short
distance, and the other direction infinitely.  We shall select a
distance $\lambda_j$ equal to that short distance.  Finally, we shall
select $\lambda = \min_j | \lambda_j |$.  Because the associated
constraints are not tight, $\lambda$ cannot be zero.  This directly
contradicts 1.

\smallskip
\noindent{3 $\implies$ 1:} Let $A'$ denote the submatrix for tight
inequalities, as was done above.  This implies the rank of $A' \ge n$.
Suppose $\neg 1$.  This implies $\exists y \ne 0 \mbox{ s.t. } x + y,
x - y \in P$.  Consider the fact that $A'(x+y) \ge b', A'(x-y) \ge
b'$.  This implies that $A'y = 0$, which implies that the rank of $A'$
is less than $n$.  This contradicts the assertion made earlier.

\subsection{Optimality at a Vertex}

We now show that whenever an LP is feasible and bounded, an optimal
solution must occur at a vertex.  More specifically, if the linear
program is feasible and bounded, then $\exists v \in Vertices(P)
\mbox{ s.t. } \forall x \in P, C^T v \le C^T x$.  This statement,
along with fundamentally stating that an optimal solution must occur
at a vertex, also shows the decidability of solving for the system, as
one can simply check all the vertices.

The proof shall proceed by picking a non-vertex point and showing that
there exists as good or better of a solution with at least one less
non-tight equation.  Hence progress is made towards a vertex by
applying this process iteratively.  To find a vertex that is optimal
this iteration will have to be done at most $n$ times.

Suppose $x \in P$ and $x \notin Vertices(P)$.  This implies
$\exists y \ne 0 \mbox{ s.t. } x + y, x - y \in P$.  This in turn
implies $(x + y, x - y \ge 0) \rightarrow (x_i = 0 \rightarrow y_i =
0)$.  Consider $v = x + \lambda y$.  $C^T v = C^T x + \lambda C^T y$.
Assume without loss of generality that $C^T y \le 0$.  If this was not
the case, the other direction could simply have been selected ($x -
y$).

Now suppose that $A'$ and $b'$ give the set of tight variables, as
before.  We know that $A'(x + y) \ge b'$.  This implies $A' y = 0$,
and that $A'(x+\lambda y) = b'$.  Once again, we will find the
$\lambda_j$ values for the non-tight equations.  There are two
possibilities:
\begin{itemize}
\item $\exists \lambda_j \ge 0$: In this case, we select the smallest
such $\lambda_j$ value, $\lambda$.  The point $x + \lambda y$ now
makes one additional inequality tight while satisfying the property
that it keeps the previously tight inequalities tight.  And we have
made progress towards a solution.
\item $\forall_j \lambda_j < 0$: Here, either $C^T y < 0$, in which
  case the solution is unbounded, or $C^T y = 0$ in which case we set
  $\lambda$ equal to $\min_j |\lambda_j|$.
\end{itemize}


\subsection{Duality}
One can view any minimization linear program as a maximization.
Consider the following linear system:
\[
\begin{array}{llll}
\min         & 3x_1 + 2x_2 + 8x_3  &     & \\
\mbox{s.t.}  & x_1 - x_2 + 2x_3    & \ge & 5 \\
             & x_1 + 2x_2 + 4x_3   & \ge & 10 \\
             & x_1, x_2, x_3       & \ge & 0 \\
\end{array}
\]

Where $Z^*$ is OPT, we know $Z^* = 3x^*_1 + 2x^*_2 + 8x^*_3$, for some
$x^*_1, x^*_2, x^*_3 \in P$.  By adding two of the inequalities, we
arrive at $2x_1 + x_2 + x_3 \ge 15$.  Since $x^*_1, x^*_2, x^*_3 \ge
0$, we know that $Z^* \ge 15$.  But we aren't limited to addition,
multiplication is another way the equations can be combined.  So how
is this new formulation bouned?  This is done by using the dual
formulation, $D$ of the minimization, which for this problem is:
\[
\begin{array}{llll}
\max         & 5y_1 + 10y_2        &     & \\
\mbox{s.t.}  & y_1 + y_2           & \le & 3 \\
             & -y_1 + 2y_2         & \le & 2 \\
             & 2y_1 + 4y_2         & \le & 8 \\
             & y_1, y_2            & \ge & 0 \\
\end{array}
\]

The theory of LP duality (sometimes referred to as the Strong Duality
Theorem) says that if the primal LP $P$ is bounded and feasible, then
the value of the primal LP equals the value of the dual LP.  

\subsubsection{Weak Duality}

Weak duality makes only the claim that the value of the primal LP is
at least the value of the dual LP.  Consider the primal $P$ and its
dual $D$:
\[
\begin{array}{llll|llll}
P            &       &     &   & D \\
\min         & c^T x &     &   & \max         & b^T y \\
\mbox{s.t.}  & Ax    & \ge & b & \mbox{s.t.}  & A^T y    & \le & c \\
             & x     & \ge & 0 &              & y        & \ge & 0 \\
\end{array}
\]

Suppose that $x^*$ is an optimal solution to $P$ and $y^*$ is an
optimal solution to $D$.  We need only show that $c^T x^* \ge b^T y^*$.
\begin{eqnarray*}
c^T x^* & \ge & (A^T y^*)^T x^* \\ 
& = & {y^*}^T A x^*\\
b^T y^* & \le & {x^*}^T A^T y^* \\
& = & ({y^*}^T A x^*)^T
\end{eqnarray*}
Noting that the last equation of each of these comparison are
identical (since the transpose of a scalar is the scalar itself) leads
to the desired conclusion.

\section{Simplex Algorithm}
\label{sec:lec7.simplex}

The {\it Simplex Algorithm} is based on the fact that the optimal solution
to a feasible LP (linear program) can be found at one of the vertices of
the polytope defined by the set of constraints.  The algorithm starts from
an arbitrary vertex represented by a {\it basic feasible solution} (bfs),
and at each iteration uses a technique called pivoting to search for an
adjacent vertex with an improved cost to the solution to move to.  If no
adjacent vertex has an improved cost, then it can be proved that the current
vertex represents the optimal solution.  The algorithm must search through a
set of potentially exponentially many vertices, and as a result is not
polynomial in the worst-case.  Even so, it performs very well in practice,
and was the algorithm of choice for several decades.

(Throughout this document, all LP's are given by:
minimize $c^T x$ subject to $Ax = b$ and $x \geq 0$, unless explicitly
defined otherwise.)

\subsection{Pivoting}

Pivoting is the mechanism used to manipulate the basis corresponding to a
vertex $v$'s bfs, to find the basis that corresponds to an adjacent vertex.
To perform a pivot step we replace one of the $m$ linearly independent columns
of the basis corresponding to $v$ with one of the nonbasic columns from $A$
in such a way that we still have $m$ linearly independent columns.  This
produces the basis of an adjacent vertex, $v'$.  The Simplex Algorithm
uses pivoting to examine the neighbors of $v$ until a neighbor is found
that corresponds to a feasible (non-negative) solution which is an
improvement over the bfs for $v$. We illustrate how pivoting is done with an
example.

\begin{Example}
\label{ex:three}
{\em
Assume that in $LP_1$,
\[A = \left[ \begin{array}{ccccc}
 5 & 2 & -3 & 16 & 4 \\
 2 & 3 &  1 &  3 & 1 \\
 1 & 7 &  6 & -1 & 2
\end{array} \right] \]
\[b = \left[ \begin{array}{c}
8 \\ 8 \\ 25
\end{array}  \right] \]
and we have determined a basis,
\[B = \{1, 3, 5\}.\]
($B$ is specified here by identifying the {\it basic} columns.  In other words,
 $x_2 = x_4 = 0$.)
This gives us the following three linearly independent equations,
which can be solved to find the corresponding bfs:
\begin{eqnarray*}
5 x_1 - 3x_2 + 4 x_5 & = & 8 \\
2 x_1 + x_2 + x_5 & = & 8 \\
x_1 + 6 x_3 + 2 x_5 & = & 25
\end{eqnarray*}
The solution, $x_1 = 1$, $x_3 = 3$, $x_5 = 3$; can also be written
as:
\[bfs = \left[ \begin{array}{c}
1 \\ 0 \\ 3 \\ 0 \\ 3
\end{array}  \right]. \]

We now perform a pivot to find another basis, by adding a nonbasic column
to $B$, and removing one of the basic columns from $B$.
Assume that we choose to add column $A_4$.  We can express $A_4$
as a linear combination of the columns that are in $B$ as follows:
\begin{eqnarray*}
A_4 & = & \alpha A_1 + \beta A_3 + \gamma A_5 \\
16 & = & 5 \alpha - 3 \beta + 4 \gamma \\
3 & = & 2 \alpha + \beta + \gamma \\
-1 & = & \alpha + 6 \beta + 2 \gamma
\end{eqnarray*}
The solution is: $\alpha = 1$, $\beta = -1$, $\gamma = 2$.  Therefore,
$A_4 = A_1 - A_3 + 2A_5$.  Since we want to find a new basis, $B'$, with
a feasible solution, we must find a basic column to remove, such that:
\begin{itemize}
\item $A_{B'} x_{B'} = b$,
\item $x_{B'} \geq 0$,
\item and $x_2 = 0$.
\end{itemize}
where $A_{B'}$ refers to the basic columns of $A$.  We have already solved
\begin{eqnarray*}
A_1 x_1 + A_3 x_3 + A_5 x_5 & = & b
\end{eqnarray*}
and want to modify that solution to include $A_4$ in the basis so that:
\begin{eqnarray*}
A_1 {x'}_1 + A_3 {x'}_3 + A_4 \Theta + A_5 {x'}_5 & = & b \\
{x'}_1, {x'}_3, {x'}_5 & \geq & 0
\end{eqnarray*}
Since we know that $A_4 = A_1 - A_3 + 2A_5$, we simply need to find
$\Theta \geq 0$ such that,
\begin{eqnarray*}
A_1 ({x'}_1 + \Theta) + A_3 ({x'}_3 - \Theta) + A_5 ({x'}_5 + 2 \Theta)
& = & b.
\end{eqnarray*}
But we know that:
\begin{eqnarray*}
A_1 x_1 + A_3 x_3 + A_5 x_5 & = & b
\end{eqnarray*}
so we have:
\[ \begin{array}{cccccc}
{x}_1 & = & {x'}_1 + \Theta  = & 1, & \Rightarrow & \Theta \leq 1 \\
{x}_3 & = & {x'}_3 - \Theta  = & 3, & \Rightarrow & \Theta \geq -3 \\
{x}_5 & = & {x'}_5 + 2 \Theta  = & 3, & \Rightarrow & \Theta \leq \frac{3}{2}
\end{array} \]
We simply choose the smallest non-negative $\Theta$
that satisfies these conditions, namely
$\Theta = 1 \Rightarrow x_1 = 0$.  Thus our new basis is
$B' = \{4, 3, 5\}$, and our bfs has ${x'}_4 = 1, {x'}_3 = 4, {x'}_5 = 1$, and
we have completed a pivot. \qed
}
\end{Example}

Let us review the steps involved in pivoting.  The idea is to start from a
basis, $B = \{1, 2, \ldots , m\}$ and then find a suitable 
nonbasic column $j \in \{1, 2, \ldots , m\}$ to replace a basic column
$l \not\in \{1, 2, \ldots , m\}$ in order to arrive at an adjacent basis, $B'$.
This is achieved using the following steps:
\begin{enumerate}
\item Assume that the starting basis $B = \{1, 2, \ldots , m\}$, is known.

      $B$ contains $m$ linearly independent columns of the $n$ columns in $A$.
      We have already found the bfs for $B$, ${bfs}_B$ which satisfies:
\begin{eqnarray}
\label{eqn:one}
\sum_{i=1}^{m} A_i x_i & = & b \\
\label{eqn:two}
x & \geq & 0
\end{eqnarray}

\item Choose a nonbasic column $j \not\in \{1, 2, \ldots , m\}$ to add to $B$.

(In the Simplex Algorithm, each $j \not\in \{1, 2, \ldots , m\}$ can be tried
in turn until one is found that results in a bfs with an {\em improved} cost.)

\item Find the linear dependence of column $j$ on the basic columns, by
      finding $\alpha_{i,j}$ such that:
\begin{eqnarray}
\label{eqn:three}
A_j & = & \sum_{i=1}^{m} \alpha_{i,j} A_i.
\end{eqnarray}

\item Find our options for ${x'}_j = \Theta$, so that
\begin{eqnarray}
\label{eqn:four}
\sum_{i=1}^{m} A_i {x'}_i + A_j {x'}_j & = & b
\end{eqnarray}
We can find the feasible range of values for $\Theta$ as follows:
\[ \begin{array}{lccc}
{\displaystyle \sum_{i=1}^{m} A_i x_i} + (A_j {x'}_j - A_j {x'}_j) & = & b &
\mbox{(based on Equation \ref{eqn:one})} \\
{\displaystyle \sum_{i=1}^{m} A_i x_i} + A_j {x'}_j -
\Theta \sum_{i=1}^{m} \alpha_{i,j} A_i \\
& = & b & \mbox{(based on Equation \ref{eqn:three})} \\
{\displaystyle \sum_{i=1}^{m} A_i (x_i - \Theta \alpha_{i,j})}
 + A_j {x'}_j & = & b
\end{array} \]
So ${x'}_i = x_i - \Theta \alpha_{i,j}$, and in order for
${x'}_i \geq 0$, we must have $\Theta \leq \frac{x_i}{\alpha_{i,j}}$.

\item Choose the basic column to $l \in \{1, 2, \ldots , m\}$ to remove from
      $B$.

We want to choose the smallest value of $\Theta$ that will result in a
feasible solution, so we take $\Theta$ to be the
\[\underbrace{\min}_{\forall \alpha_{i,j} > 0} \frac{x_i}{\alpha_{i,j}}.\]
$l$ is the value of $i$ associated with our choice of $\Theta$.

(N.B. If $\forall \alpha_{i,j} \leq 0$, then the solution is unbounded.)
\end{enumerate}

\subsection{Tableau Method}

The {\em Tableau Method} provides an efficient means of performing the
pivots required for the Simplex Algorithm.  The idea is to manipulate
$A$ and $b$ so that the basic columns of $A$ become the identity matrix.
When this happens, Step 3 of the pivoting process (above)---finding
the $\alpha_{i,j}$ coefficients for $A_j$---becomes trivial, because
(based on Equation \ref{eqn:three}) $\alpha_{i,j} = a_{i,j}$
(where $a_{i,j}$ is the $(i,j)$ entry of $A$---the member in the $ith$ row of
$A_j$).

\begin{Example}
\label{ex:two}
{\em
Consider $LP_2$ defined by
\begin{eqnarray*}
6 & = & x_1 + x_2 + 4 x_6 \\
14 & = & -2 x_1 + 2 x_2 + x_3 - x_4 + x_6 \\
-11 & = & x_1 - 2 x_2 + x_4 + 2 x_6 \\
7 & = & x_1 - 3 x_4 + x_5 - 5 x_6
\end{eqnarray*}
for which we have determined that a basis, $B = \{2, 3, 4, 5\}$.
We can represent $LP_2$ by an $m \times (n + 1)$ matrix, a {\it tableau},
as follows:
\[ \begin{array}{|c|c|c|c|c|c|c|} \hline
b & A_1 & A_2 & A_3 & A_4 & A_5 & A_6 \\ \hline
6    &  1 & 1  & 0 &  0 & 0 & 4 \\ \hline
14   & -2 & 2  & 1 & -1 & 0 & 1 \\ \hline
-11  &  1 & -2 & 0 &  1 & 0 & 2 \\ \hline
7    &  1 & 0  & 0 & -3 & 1 & -5 \\ \hline
\end{array} \]
By performing row operations on the tableau, we can transfrom the columns
that represent $B$ into the identity matrix, without affecting the solution
space.  This transformation yields,
\[ \begin{array}{|c|c|c|c|c|c|c|} \hline
b & A_1 & A_2 & A_3 & A_4 & A_5 & A_6 \\ \hline
6  &  1 & 1  & 0 &  0 & 0 & 4 \\ \hline
3  & -1 & 0  & 1 &  0 & 0 & 3 \\ \hline
1  &  3 & 0  & 0 &  1 & 0 & 10 \\ \hline
10 & 10 & 0  & 0 &  0 & 1 & 25 \\ \hline
\end{array} \]
There are two possible nonbasic columns that we may consider adding to
our basis as part of a pivot.  Assume that we choose column 1, so that
$j = 1$.  Then we have,
\begin{eqnarray*}
A_j = A_1 & = & 1 A_2 - 1 A_3 + 3 A_4 + 10 A_5
\end{eqnarray*}
$\Theta$ is determined by
\[ \Theta = \underbrace{\min}_{\forall \alpha_{i,1} > 0} \frac{x_i}{\alpha_{i,j}}
= \frac{1}{3}.\]
(since $\alpha_{i,j} = a_{i,j}$), so that $l = 4$, and the new basis, $B'$,
is $\{2, 3, 1, 5\}$.

Row operations are once again performed in order to convert $B'$ to the
identity matrix (and update $b$ to the new bfs $b'$), yielding
\[ \begin{array}{|c|c|c|c|c|c|c|} \hline
b' & {A'}_1 & {A'}_2 & {A'}_3 & {A'}_4 & {A'}_5 & {A'}_6 \\ \hline
\frac{17}{3}  &  0 & 1  & 0 &  \frac{-1}{3} & 0 & \frac{2}{3} \\ \hline
\frac{10}{3}  &  0 & 0  & 1 &  \frac{1}{3} & 0 & \frac{19}{3} \\ \hline
\frac{1}{3}  &   1 & 0  & 0 &  \frac{1}{3} & 0 & \frac{10}{3} \\ \hline
\frac{20}{3} &  0 & 0  & 0 &  \frac{-10}{3} & 1 & \frac{-25}{3} \\ \hline
\end{array} \]
The cost function, $c^T x$, is what actually determines which of the nonbasic
columns gets chosen to include in the new basis---we must find one which
results in decrease in the cost.  For any proposed replacement of some
column $l$ by some column $j$, we must simply check that the cost of the
new solution is less that the cost of the previous solution. 
The cost of new solution is given by:
\begin{eqnarray*}
\mbox{new cost} & = & \sum_{i=1}^{m} {x'}_i c_i + {x'}_j c_j \\
&= & \sum_{i=1}^{m} (x_i - \Theta \alpha_{i,j}) + \Theta c_j \\
&= & \underbrace{\sum_{i=1}^{m} x_i c_i}_{\mbox{old cost}} + \Theta 
\underbrace{[c_j - \sum_{i=1}^{m} \alpha_{i,j} c_i]}
_{\mbox{modified cost, $\bar{c}_j$}}
\end{eqnarray*}
So if the modified cost, $\bar{c}_j$, is negative,, then the new cost will
be reduced. \qed
}
\end{Example}

Let us review how pivoting is performed using the Tableau Method.
A tableau, constructed as follows, is used to perform pivots:
\[ \begin{array}{|c|c|c|c|c|} \hline
\ast   & \bar{c}_1 & \bar{c}_2 & \ldots & \bar{c}_n \\ \hline
b_1    & A_{1,1} & A_{1,2} & \ldots & A_{1,n} \\ \hline
b_2    & A_{2,1} & A_{2,2} & \ldots & A_{2,n} \\ \hline
\vdots &  \vdots &  \vdots & \vdots & \vdots  \\ \hline
b_m    & A_{m,1} & A_{m,2} & \ldots & A_{m,n} \\ \hline
\end{array} \]
in which the basic columns of $A$ form the identity matrix, and
the each nonbasic column, $j$, contains the coefficients, $\alpha_{i,j}$.
(An additional row, row 0, is used to store the modified costs.)

A pivot is made with the following steps:
\begin{enumerate}
\item Find a column $j$ such that $\bar{c}_j < 0$, by examining row 0.

(If no such column exists, we have found the optimal solution.)

\item Find column $l$ with $\alpha_{l,j} > 0$, that minimizes $\Theta$ 

(If all $\alpha_{i,j} \leq 0$ then either the solution is unbounded--unless
we didn't have a basis to begin with.)

\item Replace $l$ by $j$
\end{enumerate}

We have yet to specify how the starting basis is found an LP.
Suppose we are given $LP_A$: minimize $c^T x$ subject to $Ax \leq b$,
and $x \geq 0$.  Then we can add one {\it surplus}
variable per constraint, such that:
\begin{eqnarray*}
\sum_j a_{i,j} x_j + s_i = b_i, & i \in \{1, 2, \ldots, m\}
\end{eqnarray*}
Then our bfs is: \{$x_j = 0, s_i = b_i\}$.
If instead of $Ax \leq b$ we have $Ax = b$, then we can similarly
add {\it artificial} variables, such that:
\begin{eqnarray*}
\sum_j a_{i,j} x_j + y_i = b_i  i \in \{1, 2, \ldots, m\}
\end{eqnarray*}
to form $LP_B$ (for which we can easily construct a basis)
for which we minimize $\sum y_i$, for $y_i \geq 0$.
$LP_A$ will be feasible iff the optimal solution for $LP_B$ is 0.
To get a bfs for $LP_A$ we solve $LP_B$ using the Simplex Algorithm by
starting with the bfs, $\{x_j = 0, y_i = b_i\}$, and if the optimal solution
for $LP_B$ is 0, then all $y_i's$ are 0 and the $x_i'$ form a bfs for $LP_A$.


\bibliographystyle{alpha}
\bibliography{refs}

\end{document}
