\documentclass[11pt]{article}
\usepackage{handout}
\usepackage{amsmath,amssymb,amsthm,epsfig}
\begin{document}
\input{courseMacros}

\thispagestyle{empty}
\thispagestyle{empty}
\setlength{\parindent}{0pt}
\setlength{\parskip}{1.8ex}
%% COM3391 Lecture 6 (1st half), 11/3/99
%% Scribed by Clifford M. Bryant, Jr.


\head{CS G713}{Advanced Algorithms}{Fall 2003}{19 November 2003}{22}

\begin{center}
\Large{\bf Introduction To Linear Programming}
\end{center}
The following notes are adapted from lecture notes of an Advanced
Algorithms course I taught in Fall 1999; these notes were scribed by 3
students Mohsen Ghassemi, Clifford Bryant, Jr., and Trevor Mendez.
The material covered is based on Chapter 29 of the text and two other
sources: Michel Goemans's lecture notes on linear
programming~\cite{goemans:lp}, and Howard Karloff's text on linear
programming~\cite{karloff:lp}.

\section{Introduction}
We define the linear programming (LP) problem of minimizing a linear
function subject to linear inequality constraints.  We begin with a
simple example of linear programming.  Then, the general, standard,
and canonical forms of the linear programming problem are given in
summation and matrix form.  A formal definition is given for a vertex
of a polytope or polyhedron, and it is proved that an LP always
attains its optimum at a vertex.  Then it is proved that the set of
vectors corresponding to the current basis of an LP are linearly
independent if and only if the basic feasible solution is a vertex
point.  Finally, we cover the simplex algorithm in detail.

\section{The Diet Problem}
\label{sec:lec.lp}

Many linear programming formulations arise from situations where a decision 
maker wants to minimize the cost of meeting a set of requirements. In the diet 
problem we would like to develop a diet using $n$ food items such that it 
satisfies the daily vitamin requirements. Let the food items be numbered
1 through $n$ and let the $m$ vitamin mineral requirements be given by
$b_1, \ldots, b_m$. 

From food item $j$, you get $a_{ij}$ units of mineral $i$ per unit of $j$.
If you decide to have $x_{j}$ units of item $j$, you get $x_{j}a_{ij}$
units of mineral $i$.

\begin{eqnarray*}
\sum_{j} x_j a_{ij} \geq  b_{i} \ \ \ \forall i \\
\end{eqnarray*}
If the cost of food item $j$ is $c_{j}$ per unit. Then the total cost
incurred = $\sum_{j} c_{j} x_{j}$.

We would like to \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ {minimize} $\sum_{j} c_j \ x_j$ \\
\ \ subject \ to: 
\begin{eqnarray*}
\ \ \sum_{j=1}^{n} x_i a_{i,j} \ \geq \ b_{i} \ \ i=1,..,m \\
\end{eqnarray*}
Alternatively, one can write the above optimization problem as: 
\[ minimize \left[ \begin{array}{llll} c_{1} & c_{2} \ldots c_{n}
\end{array} \right]
\left[ \begin{array}{c}
x_{1} \\ x_{2} \\ \vdots \\ x_{n}
\end{array} \right] \]
\[ \ \ subject \ to: \ \left[ \begin{array}{ll}
a_{11} & a_{12} \ldots a_{1n} \\
a_{21} & a_{22} \ldots a_{2n} \\
a_{m1} & a_{m2} \ldots a_{mn}
\end{array} \right] \left[ \begin{array}{c}
x_{1} \\ x_{2} \\ \vdots \\ x_{n} 
\end{array} \right] \geq 
\left[ \begin{array}{ll}
b_{1} \\ b_{2} \\ \vdots \\ b_{n}
\end{array} \right] \]

The above problem is of the form min $c^Tx$ {subject} {to:} $Ax \geq b$. Such 
that a problem is referenced to as a {\it linear program}. This is because
the objective function as well as the constraints are linear combination
of the variables.

We now consider some optimization problems that we have studied and see
weather we can write them down as a linear program.

\subsection{Knapsack Problem}

In the knapsack problem we are given $n$ items of sizes $w_{1}$ 
through $w_{n}$ and profits $p_{1}$ through $p_{n}$. The goal is to build the 
items with total size $\leq$ $B$ such that the total profit is maximized. 

The LP representation of the problem can be shown as:

\[ \begin{array}{lll}
maximize & \sum x_j p_j \\
subject \ to & \sum_{j=1}^{n} x_j w_j \leq B \\
& x_j \in \{0,1\} 
\end{array} \]
\[ c^T = \left[ \begin{array}{c}
p_{1} \\ \vdots \\ p_{n} 
\end{array} \right] \]
\[ A = \left[ \begin{array}{ll} 
w_{1} & w_{2} \ldots w_{n} 
\end{array} \right] \]
\[ b = \left[ \begin{array}{ll} 
B
\end{array} \right] \]
Note that the constraint $x_j \in$ \{0,1\} is not a linear constraint so
that the above program is an integer linear program. 
\subsection{Minimum Spanning Tree Problem}
In the minimum spanning tree roblem, the set of instances is the set of all 
weighted undirected graphs. For a given instance graph $G$, the set $S(G)$ of 
solutions for $G$ is the set of all trees that span every vertex of $G$. 
Finally, the value of a solution tree $T$ is simply the sum of the weights 
of the edges of $T$.

One LP representation of the problem is: 

\[ \begin{array}{lll}
minimize & \sum_{e} w_e x_e \\
subject \ to & \sum_{e \ \in \ c} x_e \geq 1  & \ for \ all \ cut \ c  \\
& x_e = {0 \ or \ 1} \ & \ for \ all \ edges \ e 
\end{array} \]
Note that since there are an exponential number of cuts, the number of 
constraints is exponential.

\setlength{\oddsidemargin}{0in} \setlength{\evensidemargin}{0in}
\setlength{\textwidth}{6.5in} \setlength{\topmargin}{1in}
\setlength{\headheight}{0in} \setlength{\headsep}{0in}
\setlength{\footskip}{.5in} \setlength{\textheight}{8.5in}
%\setlength{\voffset}{1in}

\newcommand{\comqed}{\hfill \rule{7pt}{7pt}}
\newtheorem{Theorem}{Theorem}
\newtheorem{Corollary}{Corollary}
\newtheorem{Lemma}{Lemma}
\newtheorem{Claim}{Claim}
\theoremstyle{definition}
\newtheorem{Definition}{Definition}
\newenvironment{Proof}{\noindent{\bf Proof: }}{\comqed}
%\theoremstyle{remark}
\newtheorem{Remark}{Remark}
\newtheorem{Example}{Example}
\newcommand{\example}[1]{
\begin{Example}
{\em #1} \comqed
\end{Example}
}

\newcommand{\p}{\mathcal{P}}
\newcommand{\xpy}{x + \lambda y}

\section{Equivalent Forms}
\label{sec:form}

\subsection{General Form}
\label{sec:form:gen}

This is the general form of the linear programming problem.
The $c_{i}$'s can be interpreted as costs.
In this case, the objective is to minimize the total cost
subject to the linear constraints.
\begin{equation}
\mathrm{Minimize} \ \sum_{i=1}^{n} c_{i} x_{i} \qquad
\mathrm{(objective \ function)} \label{eqn:gen:sum}
\end{equation}
\begin{displaymath} \begin{array}{lrcrll} \mathrm{subject \ to} & \sum_{j=1}^{n}
a_{ij} x_{j} & \geq & b_{i} & , i = 1, \ldots, m_{1} &
\mathrm{(inequality)} \\
 & \sum_{j=1}^{n} a_{ij} x_{j} & = & b_{i} & , i = m_{1} + 1, \ldots, m_{1} + m_{2} & \mathrm{(equality)} \\
 & x_{j} & \geq & 0 & , j = 1, \ldots, n_{1} & \mathrm{(non-negativity)} \\
 & x_{j} & \lesseqgtr & 0 & , j = n_{1} + 1, \ldots, n & \mathrm{(unconstrained)}
\end{array} \end{displaymath}

The general form of the LP can be written more compactly in matrix notation.
\begin{equation}
\mathrm{Minimize} \ \mathbf{c}^{\mathit{T}} \mathbf{x}
\label{eqn:gen:mat}
\end{equation}
\[ \begin{array}{lrcl}
\mathrm{subject \ to} & \mathbf{Ax_{\mathrm{1}}}                    & \geq & \mathbf{b} \\
                      & \mathbf{A}\mathit{'}\mathbf{x_{\mathrm{2}}} & =    & \mathbf{b}\mathit{'} \\
                      & \mathbf{x}_{\mathrm{1}}                     & \geq & 0 \\
                      & \mathbf{x}_{\mathrm{2}}                     & <=>  & 0
\end{array} \]
where
\begin{description}
\item $\mathbf{c}$ and $\mathbf{x}$ are $n$ x $1$ vectors,
\item $\mathbf{A}$ is an $m_{1}$ x $n$ matrix,
\item $\mathbf{A}\mathit{'}$ is an $m_{2}$ x $n$ matrix,
\item $\mathbf{b}$ is an $m_{1}$ x $1$ vector,
\item $\mathbf{b}\mathit{'}$ is an $m_{2}$ x $1$ vector,
\item $\mathbf{x_{\mathrm{1}}}$ is an $n_{1}$ x $1$ vector,
\item $\mathbf{x_{\mathrm{2}}}$ is an $n_{2}$ x $1$ vector, and
\item $ \mathbf{x} = \left\{ \begin{array}{cc} \mathbf{x_{\mathrm{1}}} \\ \mathbf{x_{\mathrm{2}}} \end{array} \right\} $.
\end{description}

\subsection{Standard Form}
\label{sec:form:can}

This is the standard form of the linear programming problem.
Here the constraints take the form of linear inequality constraints,
plus non-negativity constraints on the independent variables.
\begin{equation}
\mathrm{Minimize} \ \sum_{i=1}^{n} c_{i} x_{i} \qquad
\mathrm{(objective \ function)} \label{eqn:can:sum}
\end{equation}
\[ \begin{array}{lrcrll}
\mathrm{subject \ to} & \sum_{j=1}^{n} a_{ij} x_{j} & \geq & b_{i} & , i = 1, \ldots, m & \mathrm{(inequality)} \\
                      & x_{j}                       & \geq & 0     & , j = 1, \ldots, n & \mathrm{(non-negativity)}
\end{array} \]

Once more, the canonical form of the LP can be written more compactly in matrix notation.
\begin{equation}
\mathrm{Minimize} \ \mathbf{c}^{\mathit{T}} \mathbf{x}
\label{eqn:can:mat}
\end{equation}
\[ \begin{array}{lrcl}
\mathrm{subject \ to} & \mathbf{Ax} & \geq & \mathbf{b} \\
                      & \mathbf{x}  & \geq & 0
\end{array} \]
where
\begin{description}
\item $\mathbf{c}$ and $\mathbf{x}$ are $n$ x $1$ vectors,
\item $\mathbf{A}$ is an $m$ x $n$ matrix, and
\item $\mathbf{b}$ is an $m$ x $1$ vector.
\end{description}

\subsection{Slack Form}
\label{sec:form:std}

This is the slack form of the linear programming problem.
Here the constraints take the form of linear equality constraints,
plus non-negativity constraints on the independent variables.
\begin{equation}
\mathrm{Minimize} \ \sum_{i=1}^{n} c_{i} x_{i} \qquad
\mathrm{(objective \ function)} \label{eqn:std:sum}
\end{equation}
\[ \begin{array}{lrcrll}
\mathrm{subject \ to} & \sum_{j=1}^{n} a_{ij} x_{j} & =    & b_{i} & , i = 1, \ldots, m & \mathrm{(equality)} \\
                      & x_{j}                       & \geq & 0     & , j = 1, \ldots, n & \mathrm{(non-negativity)}
\end{array} \]

The standard form of the LP can also be written more compactly in matrix notation.
\begin{equation}
\mathrm{Minimize} \ \mathbf{c}^{\mathit{T}} \mathbf{x}
\label{eqn:std:mat}
\end{equation}
\[ \begin{array}{lrcl}
\mathrm{subject \ to} & \mathbf{Ax} & =    & \mathbf{b} \\
                      & \mathbf{x}  & \geq & 0
\end{array} \]
where
\begin{description}
\item $\mathbf{c}$ and $\mathbf{x}$ are $n$ x $1$ vectors,
\item $\mathbf{A}$ is an $m$ x $n$ matrix, and
\item $\mathbf{b}$ is an $m$ x $1$ vector.
\end{description}

\begin{Definition}
\label{def:feasible} If $x$ satisfies $\mathbf{Ax} = \mathbf{b}, \
\mathbf{x} \geq \mathbf{0}$, then $x$ is \textbf{feasible}.
\end{Definition}

\subsection{Constraint Conversion}

It is possible to convert between equality and inequality
constraints by the following method.
\begin{enumerate}
\item To convert a less than or equal inequality constraint,
\[ Ax \leq b\]
to an equality constraint, add the vector of slack variables, $s$.
\[ Ax + s = b, \ s \geq 0\]
\item To convert a greater than or equal inequality constraint,
\[ Ax \geq b\]
to an equality constraint, add the vector of surplus variables,
$t$.
\[ Ax - t = b, \ t \geq 0\]
\item Finally, to convert an equality constraint, \[ Ax = b \] to
an inequality constraint, add two inequality constraints.
\[ Ax \leq b, \ -Ax \leq -b \]
\end{enumerate}

\section{Example}
\label{sec:example}

\begin{Example}
\label{ex:one}

Consider the following linear program:
\[ \begin{array}{lrcrcr}
\mathrm{Minimize}     &        &   &  x_{2} &      &   \\
\mathrm{subject \ to} &  x_{1} &   &        & \geq & 2 \\
                      & 3x_{1} & - &  x_{2} & \geq & 0 \\
                      &  x_{1} & + &  x_{2} & \geq & 6 \\
                      & -x_{1} & + & 2x_{2} & \geq & 0 \\
                      &  x_{1} &   &        & \geq & 0 \\
                      &        &   &  x_{2} & \geq & 0
\end{array} \]
The optimal solution is (4, 2) of cost 2 (See
Figure~\ref{fig:inequal}). If we were maximizing $x_{2}$, instead
of minimizing under the same feasible region, the resulting linear
program would be unbounded, since $x_{2}$ can increase
arbitrarily. From this picture, the reader should be convinced
that, for any objective function for which the linear program is
bounded, there exists an optimal solution which is a ``corner'' of
the feasible region. This notion will be formalized in the next
section.
\begin{figure}
\begin{center}
\epsfig{file=inequal.eps} \caption{Feasible Region}
\label{fig:inequal}
\end{center}
\end{figure}
\end{Example}

\section{The Geometry of LP}
\label{sec:geometry}

\begin{Definition}
\label{def:polytope} A \textbf{polytope} is the analogue in
$n$-dimensional space of point, segment, polygon, and polyhedron
in spaces of dimensions 0, 1, 2, and 3. A \textbf{convex polytope}
in $n$-space is the convex span of a finite set of points that do
not all lie in the same hyperplane; thus a convex polytope is a
bounded convex subset enclosed by a finite number of hyperplanes.
\end{Definition}
\begin{Definition}
\label{def:convex} A set is \textbf{convex} if it contains the
line segment joining any two of its points; in a \emph{vector
space}, a set such that $rx + (1 - r)y$ is in the set for $0 < r <
1$, if $x$ and $y$ are in the set.
\end{Definition}
\begin{Definition}
\label{def:vertex} A point $x$ is a \textbf{vertex} of $\p$ if
$\not\exists y \neq 0$ such that $x + y, \ x - y \in \p$.
\end{Definition}

\begin{Theorem}
\label{thm:vertex} If the optimal solution to an LP in canonical
form is bounded (from below), then given any $x \in \p$, $\exists$
a vertex $v$ such that $c^{T}v \leq c^{T}x$.
\end{Theorem}

\begin{Corollary}
If the LP is bounded, then $\exists$ a vertex that yields the
optimal solution.
\end{Corollary}

\begin{Proof} (of theorem) Given $x \in \p$. If $x$ is a vertex,
we are done. Otherwise, $\exists y \neq 0$ such that $x + y \in
\p$ and $x - y \in \p$. By feasibility of $x + y$ and $x - y$,
$A(x + y) = b, \ A(x - y) = b$. But this implies that $Ay  = 0$.
Furthermore, since $x + y \ge 0$ and $x - y \ge 0$, $y_j = 0$ whenever
$x_j = 0$.

Without loss of generality, choose $y$ such that $c^{T} y \leq 0$.
Then $c^{T}(\xpy) = c^{T}x + \lambda c^{T}y \leq c^{T}x$.
\begin{description}
\item Case 1. $\exists j$ such that $y_{j} < 0$. \\
We need to ensure that $\xpy \geq 0$ to maintain feasibility.  We know
that whenever $x_i = 0$, we have $y_i = 0$.  Hence set $\lambda =
\min_{\{j:y_{j}<0\}}\{\frac{x_{j}}{-y_{j}}\}$. Then it is clear
that $\xpy \geq 0$. Then $\xpy$ has at least one more zero
component than $x$.
\item Case 2. $y_{j} \geq 0, \ \forall j$. \\
In this case, if $c^Ty = 0$, then we can negate every component of $y$
and switch to case 1 (note that $y \neq 0$).  It remains to consider
the subcase $c^{T}y < 0$. Clearly, $\xpy \geq 0$ for
$\lambda > 0$. Hence, $c^{T}(\xpy) = c^{T}x + \lambda c^{T}y$.
Then $\lambda$ can be chosen arbitrarily large, and the objective
function is unbounded. Contradiction.
\end{description}
Case 1 can happen at most $n$ times, since $x$ has $n$ components.
By induction on the number of non-zero components of $x$, we
obtain a vertex $v$.
\end{Proof}

\begin{Remark}
Linear Programming was developed during World War II to solve
logistics problems for military resources. The simplex method for
solving linear programming problems was developed in 1947 by
George Dantzig, an American mathematician. Although the simplex
method performs well in practice, in 1972 Klee and Minty
demonstrated an example in which the simplex method takes an
exponential number of steps.  In 1979, L. G. Khachian produced the
ellipsoidal method, which was shown to solve any linear program in
polynomial time. However, when implemented, his method was not
competitive with the simplex method, and lost favor.  Then, in
1984, Narinder K. Karmarkar presented a method based on the
projective transformation of a simplex. Karmarkar showed his
method solves any linear program in time that is a polynomial
function of the data of the problem.  In contrast to Khachian's
method, implementations of various generalizations of Karmarkar's
method are very competitive with the simplex method, and generally
outperform it for large problems.
\end{Remark}

\begin{Theorem}
Let $x$ be a point in $\p$. The submatrix $A_{x} = \{$ columns
corresponding to $x_{j} > 0 \}$ is linearly independent if and
only if $x$ is a vertex.
\end{Theorem}
\begin{Proof}
\begin{description}
\item ($\Leftarrow$) Suppose $A_{x}$ has linearly dependent
columns. Then $\exists y$ such that $A_{x}y = 0, \ y \neq 0$.
Extend $y$ to $\mathbb{R}^{n}$ by adding zero-valued components.
Then $\exists y \in \mathbb{R}^{n}$ such that $Ay = 0, \ y \neq
0$. Consider $\xpy$ for small $\lambda > 0$. Then we can chose
$\lambda$ such that $x + \lambda y, \ x - \lambda y \in \p$, by an
argument analogous to that in Case 1 of the proof of
Theorem~\ref{thm:vertex}, above. Hence, $x$ is not a vertex.
\item ($\Rightarrow$) Assume $x$ is not a vertex. Then, by
definition, $\exists y \neq 0$ such that $x + y, \ x - y \in \p$.
Let $A_{y}$ be the submatrix corresponding to the non-zero
components of $y$. As in the proof of Theorem~\ref{thm:vertex},
\[ \left. \begin{array}{c}
Ax + Ay = b \\ Ax - Ay = b
\end{array} \right\} \Rightarrow Ay = 0. \]
Therefore, $A_{y}$ has dependent columns, since $y \neq 0$.
Moreover,
\[ \left. \begin{array}{c}
x + y \geq 0 \\ x - y \geq 0
\end{array} \right\} \Rightarrow y_{j} = 0 \text{ whenever } x_{j} = 0. \]
Therefore, $A_{y}$ is a submatrix of $A_{x}$. Since $A_{y}$ is a
submatrix of $A_{x}$, $A_{x}$ has linearly dependent columns.
\end{description}
\end{Proof}

\section{Simplex Algorithm}
\label{sec:lec7.simplex}

The {\it Simplex Algorithm} is based on the fact that the optimal solution
to a feasible LP (linear program) can be found at one of the vertices of
the polytope defined by the set of constraints.  The algorithm starts from
an arbitrary vertex represented by a {\it basic feasible solution} (bfs),
and at each iteration uses a technique called pivoting to search for an
adjacent vertex with an improved cost to the solution to move to.  If no
adjacent vertex has an improved cost, then it can be proved that the current
vertex represents the optimal solution.  The algorithm must search through a
set of potentially exponentially many vertices, and as a result is not
polynomial in the worst-case.  Even so, it performs very well in practice,
and was the algorithm of choice for several decades.

(Throughout this section, all LP's are given by: minimize $c^T x$
subject to $Ax = b$ and $x \geq 0$, unless explicitly defined
otherwise.)

\subsection{Pivoting}

Pivoting is the mechanism used to manipulate the basis corresponding to a
vertex $v$'s bfs, to find the basis that corresponds to an adjacent vertex.
To perform a pivot step we replace one of the $m$ linearly independent columns
of the basis corresponding to $v$ with one of the nonbasic columns from $A$
in such a way that we still have $m$ linearly independent columns.  This
produces the basis of an adjacent vertex, $v'$.  The Simplex Algorithm
uses pivoting to examine the neighbors of $v$ until a neighbor is found
that corresponds to a feasible (non-negative) solution which is an
improvement over the bfs for $v$. We illustrate how pivoting is done with an
example.

\begin{Example}
\label{ex:two}
{\em
Assume that in $LP_1$,
\[A = \left[ \begin{array}{ccccc}
 5 & 2 & -3 & 16 & 4 \\
 2 & 3 &  1 &  3 & 1 \\
 1 & 7 &  6 & -1 & 2
\end{array} \right] \]
\[b = \left[ \begin{array}{c}
8 \\ 8 \\ 25
\end{array}  \right] \]
and we have determined a basis,
\[B = \{1, 3, 5\}.\]
($B$ is specified here by identifying the {\it basic} columns.  In other words,
 $x_2 = x_4 = 0$.)
This gives us the following three linearly independent equations,
which can be solved to find the corresponding bfs:
\begin{eqnarray*}
5 x_1 - 3x_2 + 4 x_5 & = & 8 \\
2 x_1 + x_2 + x_5 & = & 8 \\
x_1 + 6 x_3 + 2 x_5 & = & 25
\end{eqnarray*}
The solution, $x_1 = 1$, $x_3 = 3$, $x_5 = 3$; can also be written
as:
\[bfs = \left[ \begin{array}{c}
1 \\ 0 \\ 3 \\ 0 \\ 3
\end{array}  \right]. \]

We now perform a pivot to find another basis, by adding a nonbasic column
to $B$, and removing one of the basic columns from $B$.
Assume that we choose to add column $A_4$.  We can express $A_4$
as a linear combination of the columns that are in $B$ as follows:
\begin{eqnarray*}
A_4 & = & \alpha A_1 + \beta A_3 + \gamma A_5 \\
16 & = & 5 \alpha - 3 \beta + 4 \gamma \\
3 & = & 2 \alpha + \beta + \gamma \\
-1 & = & \alpha + 6 \beta + 2 \gamma
\end{eqnarray*}
The solution is: $\alpha = 1$, $\beta = -1$, $\gamma = 2$.  Therefore,
$A_4 = A_1 - A_3 + 2A_5$.  Since we want to find a new basis, $B'$, with
a feasible solution, we must find a basic column to remove, such that:
\begin{itemize}
\item $A_{B'} x_{B'} = b$,
\item $x_{B'} \geq 0$,
\item and $x_2 = 0$.
\end{itemize}
where $A_{B'}$ refers to the basic columns of $A$.  We have already solved
\begin{eqnarray*}
A_1 x_1 + A_3 x_3 + A_5 x_5 & = & b
\end{eqnarray*}
and want to modify that solution to include $A_4$ in the basis so that:
\begin{eqnarray*}
A_1 {x'}_1 + A_3 {x'}_3 + A_4 \Theta + A_5 {x'}_5 & = & b \\
{x'}_1, {x'}_3, {x'}_5 & \geq & 0
\end{eqnarray*}
Since we know that $A_4 = A_1 - A_3 + 2A_5$, we simply need to find
$\Theta \geq 0$ such that,
\begin{eqnarray*}
A_1 ({x'}_1 + \Theta) + A_3 ({x'}_3 - \Theta) + A_5 ({x'}_5 + 2 \Theta)
& = & b.
\end{eqnarray*}
But we know that:
\begin{eqnarray*}
A_1 x_1 + A_3 x_3 + A_5 x_5 & = & b
\end{eqnarray*}
so we have:
\[ \begin{array}{cccccc}
{x}_1 & = & {x'}_1 + \Theta  = & 1, & \Rightarrow & \Theta \leq 1 \\
{x}_3 & = & {x'}_3 - \Theta  = & 3, & \Rightarrow & \Theta \geq -3 \\
{x}_5 & = & {x'}_5 + 2 \Theta  = & 3, & \Rightarrow & \Theta \leq \frac{3}{2}
\end{array} \]
We simply choose the smallest non-negative $\Theta$
that satisfies these conditions, namely
$\Theta = 1 \Rightarrow x_1 = 0$.  Thus our new basis is
$B' = \{4, 3, 5\}$, and our bfs has ${x'}_4 = 1, {x'}_3 = 4, {x'}_5 = 1$, and
we have completed a pivot. \qed
}
\end{Example}

Let us review the steps involved in pivoting.  The idea is to start from a
basis, $B = \{1, 2, \ldots , m\}$ and then find a suitable 
nonbasic column $j \in \{1, 2, \ldots , m\}$ to replace a basic column
$l \not\in \{1, 2, \ldots , m\}$ in order to arrive at an adjacent basis, $B'$.
This is achieved using the following steps:
\begin{enumerate}
\item Assume that the starting basis $B = \{1, 2, \ldots , m\}$, is known.

      $B$ contains $m$ linearly independent columns of the $n$ columns in $A$.
      We have already found the bfs for $B$, ${bfs}_B$ which satisfies:
\begin{eqnarray}
\label{eqn:one}
\sum_{i=1}^{m} A_i x_i & = & b \\
\label{eqn:two}
x & \geq & 0
\end{eqnarray}

\item Choose a nonbasic column $j \not\in \{1, 2, \ldots , m\}$ to add to $B$.

(In the Simplex Algorithm, each $j \not\in \{1, 2, \ldots , m\}$ can be tried
in turn until one is found that results in a bfs with an {\em improved} cost.)

\item Find the linear dependence of column $j$ on the basic columns, by
      finding $\alpha_{i,j}$ such that:
\begin{eqnarray}
\label{eqn:three}
A_j & = & \sum_{i=1}^{m} \alpha_{i,j} A_i.
\end{eqnarray}

\item Find our options for ${x'}_j = \Theta$, so that
\begin{eqnarray}
\label{eqn:four}
\sum_{i=1}^{m} A_i {x'}_i + A_j {x'}_j & = & b
\end{eqnarray}
We can find the feasible range of values for $\Theta$ as follows:
\[ \begin{array}{lccc}
{\displaystyle \sum_{i=1}^{m} A_i x_i} + (A_j {x'}_j - A_j {x'}_j) & = & b &
\mbox{(based on Equation \ref{eqn:one})} \\
{\displaystyle \sum_{i=1}^{m} A_i x_i} + A_j {x'}_j -
\Theta \sum_{i=1}^{m} \alpha_{i,j} A_i \\
& = & b & \mbox{(based on Equation \ref{eqn:three})} \\
{\displaystyle \sum_{i=1}^{m} A_i (x_i - \Theta \alpha_{i,j})}
 + A_j {x'}_j & = & b
\end{array} \]
So ${x'}_i = x_i - \Theta \alpha_{i,j}$, and in order for
${x'}_i \geq 0$, we must have $\Theta \leq \frac{x_i}{\alpha_{i,j}}$.

\item Choose the basic column to $l \in \{1, 2, \ldots , m\}$ to remove from
      $B$.

We want to choose the smallest value of $\Theta$ that will result in a
feasible solution, so we take $\Theta$ to be the
\[\underbrace{\min}_{\forall \alpha_{i,j} > 0} \frac{x_i}{\alpha_{i,j}}.\]
$l$ is the value of $i$ associated with our choice of $\Theta$.

(N.B. If $\forall \alpha_{i,j} \leq 0$, then the solution is unbounded.)
\end{enumerate}

\subsection{Tableau Method}

The {\em Tableau Method} provides an efficient means of performing the
pivots required for the Simplex Algorithm.  The idea is to manipulate
$A$ and $b$ so that the basic columns of $A$ become the identity matrix.
When this happens, Step 3 of the pivoting process (above)---finding
the $\alpha_{i,j}$ coefficients for $A_j$---becomes trivial, because
(based on Equation \ref{eqn:three}) $\alpha_{i,j} = a_{i,j}$
(where $a_{i,j}$ is the $(i,j)$ entry of $A$---the member in the $ith$ row of
$A_j$).

\begin{Example}
\label{ex:three}
{\em
Consider $LP_2$ defined by
\begin{eqnarray*}
6 & = & x_1 + x_2 + 4 x_6 \\
14 & = & -2 x_1 + 2 x_2 + x_3 - x_4 + x_6 \\
-11 & = & x_1 - 2 x_2 + x_4 + 2 x_6 \\
7 & = & x_1 - 3 x_4 + x_5 - 5 x_6
\end{eqnarray*}
for which we have determined that a basis, $B = \{2, 3, 4, 5\}$.
We can represent $LP_2$ by an $m \times (n + 1)$ matrix, a {\it tableau},
as follows:
\[ \begin{array}{|c|c|c|c|c|c|c|} \hline
b & A_1 & A_2 & A_3 & A_4 & A_5 & A_6 \\ \hline
6    &  1 & 1  & 0 &  0 & 0 & 4 \\ \hline
14   & -2 & 2  & 1 & -1 & 0 & 1 \\ \hline
-11  &  1 & -2 & 0 &  1 & 0 & 2 \\ \hline
7    &  1 & 0  & 0 & -3 & 1 & -5 \\ \hline
\end{array} \]
By performing row operations on the tableau, we can transfrom the columns
that represent $B$ into the identity matrix, without affecting the solution
space.  This transformation yields,
\[ \begin{array}{|c|c|c|c|c|c|c|} \hline
b & A_1 & A_2 & A_3 & A_4 & A_5 & A_6 \\ \hline
6  &  1 & 1  & 0 &  0 & 0 & 4 \\ \hline
3  & -1 & 0  & 1 &  0 & 0 & 3 \\ \hline
1  &  3 & 0  & 0 &  1 & 0 & 10 \\ \hline
10 & 10 & 0  & 0 &  0 & 1 & 25 \\ \hline
\end{array} \]
There are two possible nonbasic columns that we may consider adding to
our basis as part of a pivot.  Assume that we choose column 1, so that
$j = 1$.  Then we have,
\begin{eqnarray*}
A_j = A_1 & = & 1 A_2 - 1 A_3 + 3 A_4 + 10 A_5
\end{eqnarray*}
$\Theta$ is determined by
\[ \Theta = \underbrace{\min}_{\forall \alpha_{i,1} > 0} \frac{x_i}{\alpha_{i,j}}
= \frac{1}{3}.\]
(since $\alpha_{i,j} = a_{i,j}$), so that $l = 4$, and the new basis, $B'$,
is $\{2, 3, 1, 5\}$.

Row operations are once again performed in order to convert $B'$ to the
identity matrix (and update $b$ to the new bfs $b'$), yielding
\[ \begin{array}{|c|c|c|c|c|c|c|} \hline
b' & {A'}_1 & {A'}_2 & {A'}_3 & {A'}_4 & {A'}_5 & {A'}_6 \\ \hline
\frac{17}{3}  &  0 & 1  & 0 &  \frac{-1}{3} & 0 & \frac{2}{3} \\ \hline
\frac{10}{3}  &  0 & 0  & 1 &  \frac{1}{3} & 0 & \frac{19}{3} \\ \hline
\frac{1}{3}  &   1 & 0  & 0 &  \frac{1}{3} & 0 & \frac{10}{3} \\ \hline
\frac{20}{3} &  0 & 0  & 0 &  \frac{-10}{3} & 1 & \frac{-25}{3} \\ \hline
\end{array} \]
The cost function, $c^T x$, is what actually determines which of the nonbasic
columns gets chosen to include in the new basis---we must find one which
results in decrease in the cost.  For any proposed replacement of some
column $l$ by some column $j$, we must simply check that the cost of the
new solution is less that the cost of the previous solution. 
The cost of new solution is given by:
\begin{eqnarray*}
\mbox{new cost} & = & \sum_{i=1}^{m} {x'}_i c_i + {x'}_j c_j \\
&= & \sum_{i=1}^{m} (x_i - \Theta \alpha_{i,j}) + \Theta c_j \\
&= & \underbrace{\sum_{i=1}^{m} x_i c_i}_{\mbox{old cost}} + \Theta 
\underbrace{[c_j - \sum_{i=1}^{m} \alpha_{i,j} c_i]}
_{\mbox{modified cost, $\bar{c}_j$}}
\end{eqnarray*}
So if the modified cost, $\bar{c}_j$, is negative, then the new cost will
be reduced. \qed
}
\end{Example}

Let us review how pivoting is performed using the Tableau Method.
A tableau, constructed as follows, is used to perform pivots:
\[ \begin{array}{|c|c|c|c|c|} \hline
\ast   & \bar{c}_1 & \bar{c}_2 & \ldots & \bar{c}_n \\ \hline
b_1    & A_{1,1} & A_{1,2} & \ldots & A_{1,n} \\ \hline
b_2    & A_{2,1} & A_{2,2} & \ldots & A_{2,n} \\ \hline
\vdots &  \vdots &  \vdots & \vdots & \vdots  \\ \hline
b_m    & A_{m,1} & A_{m,2} & \ldots & A_{m,n} \\ \hline
\end{array} \]
in which the basic columns of $A$ form the identity matrix, and
the each nonbasic column, $j$, contains the coefficients, $\alpha_{i,j}$.
(An additional row, row 0, is used to store the modified costs.)

A pivot is made with the following steps:
\begin{enumerate}
\item Find a column $j$ such that $\bar{c}_j < 0$, by examining row 0.

(If no such column exists, we have found the optimal solution.)

\item Find column $l$ with $\alpha_{l,j} > 0$, that minimizes $\Theta$ 

(If all $\alpha_{i,j} \leq 0$ then either the solution is unbounded--unless
we didn't have a basis to begin with.)

\item Replace $l$ by $j$
\end{enumerate}

We have yet to specify how the starting basis is found for an LP.
Suppose we are given $LP_A$: minimize $c^T x$ subject to $Ax \leq b$,
and $x \geq 0$.  Then we can add one {\it surplus}
variable per constraint, such that:
\begin{eqnarray*}
\sum_j a_{i,j} x_j + s_i = b_i, & i \in \{1, 2, \ldots, m\}
\end{eqnarray*}
Then our bfs is: \{$x_j = 0, s_i = b_i\}$.
If instead of $Ax \leq b$ we have $Ax = b$, then we can similarly
add {\it artificial} variables, such that:
\begin{eqnarray*}
\sum_j a_{i,j} x_j + y_i = b_i  i \in \{1, 2, \ldots, m\}
\end{eqnarray*}
to form $LP_B$ (for which we can easily construct a basis)
for which we minimize $\sum y_i$, for $y_i \geq 0$.
$LP_A$ will be feasible iff the optimal solution for $LP_B$ is 0.
To get a bfs for $LP_A$ we solve $LP_B$ using the Simplex Algorithm by
starting with the bfs, $\{x_j = 0, y_i = b_i\}$, and if the optimal solution
for $LP_B$ is 0, then all $y_i's$ are 0 and the $x_i'$ form a bfs for $LP_A$.

\bibliographystyle{alpha}
\bibliography{/home/rraj/tex/bibtex/refs}

\end{document}




