College of Computer & Information Science Fall 2014 Northeastern University CS7800: Advanced Algorithms

Lecture Outline:

• Introduction

• Geometry & Duality

• The Simplex Algorithm

Two excellent reference sources (among many others) are Michel Goemans’s lecture notes on linear programming [Goe94], and Howard Karloff’s text on linear programming [Kar91]. Some of the material has been drawn from these two sources. Thanks also to Clifford Bryant, Jr., Mohsen Ghassemi, Trevor Mendez, Eric Robinson, and San Tan, who all contributed to these notes when they were graduate students here.

1 Introduction

Many linear programming formulations arise from situations where a decision maker wants to minimize the cost of meeting a set of requirements. In the diet problem we would like to develop a diet using n food items such that it satisfies the daily vitamin requirements. Let the food items be numbered 1 through n and let the m vitamin mineral requirements be given by b_{1}, …, b_{m}.

From food item j, you get a_{ij} units of mineral i per unit of j. If you decide to have x_{j} units of item j, you get x_{j}a_{ij} units of mineral i.

∑_{j} x_{j}a_{ij} ≥ b_{i} ∀i

If the cost of food item j is c_{j} per unit. Then the total cost incurred = ∑_{j} c_{j}x_{j}.

We would like to minimize ∑_{j} c_{j} x_{j} subject to:

∑_{j=1}^{n} x_{i}a_{i,j} ≥ b_{i} i = 1, .., m

Alternatively, one can write the above optimization problem as:

minimize [ c_{1} c_{2} … c_{n} ] [ x_{1} ; x_{2} ; ⋮ ; x_{n} ]

subject to : [ \begin{matrix}a_{11} & a_{12} … a_{1n} \\ a_{21} & a_{22} … a_{2n} \\ a_{m1} & a_{m2} … a_{mn} \end{matrix} ] [ x_{1} ; x_{2} ; ⋮ ; x_{n} ] ≥ [ b_{1} ; b_{2} ; ⋮ ; b_{n} ]

The above problem is of the form min c^{T} x subject to: Ax ≥ b. Such that a problem is referenced to as a linear program. This is because the objective function as well as the constraints are linear combination of the variables.

We now show how optimization problems can often be written down as linear programs.

1.1 Knapsack Problem

In the knapsack problem we are given n items of sizes w_{1} through w_{n} and profits p_{1} through p_{n}. The goal is to build the items with total size ≤ B such that the total profit is maximized.

The LP representation of the problem can be shown as:

maximize ∑ x_{j}p_{j}
subject to ∑_{j=1}^{n} x_{j}w_{j} ≤ B
x_{j} ∈ {0, 1}

c^{T} = [ p_{1} ; ⋮ ; p_{n} ]

A = [ w_{1} w_{2} … w_{n} ]

b = [ B ]

Note that the constraint x_{j} ∈ {0,1} is not a linear constraint so that the above program is an integer linear program.

1.2 Minimum Spanning Tree Problem

Recall that in the minimum spanning tree problem, the set of instances is the set of all weighted undirected graphs. For a given instance graph G, the set S(G) of solutions for G is the set of all trees that span every vertex of G. Finally, the value of a solution tree T is simply the sum of the weights of the edges of T.

One LP representation of the problem is:

minimize ∑_{e} w_{e}x_{e}
subject to ∑_{e ∈ c} x_{e} ≥ 1 for all cut c
x_{e} = 0 or 1  for all edges e

Note that since there are an exponential number of cuts, the number of constraints is exponential.

1.3 Weighted Vertex Cover Problem

A vertex cover of undirected graph G = (V.E) is a subset V′ ⊆ V such that if (u, v) ∈ E, then u ∈ V′ or v ∈ V′ (or both). That is, each vertex covers its incident edges, and a vertex cover for G is a set of vertices that covers all the edges in E.

The LP representation of the problem is:

minimize ∑_{u} x_{u}w_{u}
subject to x_{u} + x_{v} ≥ 1  ∀(u, v) ∈ E
x_{u}, x_{v} = 0 or 1

2 Equivalent Forms

We define the linear programming (LP) problem of minimizing a linear function subject to linear inequality constraints. The general, standard, and canonical forms of the linear programming problem are given in summation and matrix form. A formal definition is given for a vertex of a polytope or polyhedron, and it is proved that an LP always attains its optimum at a vertex. Then it is proved that the set of vectors corresponding to the current basis of an LP are linearly independent if and only if the basic feasible solution is a vertex point.

2.1 General Form

This is the general form of the linear programming problem. The c_{i}’s can be interpreted as costs. In this case, the objective is to minimize the total cost subject to the linear constraints.

Minimize ∑_{i=1}^{n} c_{i}x_{i}  (objective function)  (1)

subject to ∑_{j=1}^{n} a_{ij}x_{j} ≥ b_{i} , i = 1, …, m_{1}  (inequality)
∑_{j=1}^{n} a_{ij}x_{j} = b_{i} , i = m_{1} + 1, …, m_{1} + m_{2} (equality)
x_{j} ≥ 0 , j = 1, …, n_{1}  (non − negativity)
x_{j} ⋚ 0 , j = n_{1} + 1, …, n  (unconstrained)

The general form of the LP can be written more compactly in matrix notation.

Minimize c^{T} x  (2)
subject to Ax_{1} ≥ b
A′x_{2} = b′
x_{1} ≥ 0
x_{2} <=> 0

where

c and x are n x 1 vectors, A is an m_{1} x n matrix, A′ is an m_{2} x n matrix, b is an m_{1} x 1 vector, b′ is an m_{2} x 1 vector, x_{1} is an n_{1} x 1 vector, x_{2} is an n_{2} x 1 vector, and

x = \{ \begin{matrix}x_{1} \\ x_{2} \end{matrix} \}.

2.2 Standard Form

This is the standard form of the linear programming problem. Here the constraints take the form of linear equality constraints, plus non-negativity constraints on the independent variables.

Minimize ∑_{i=1}^{n} c_{i}x_{i}  (objective function)  (3)
subject to ∑_{j=1}^{n} a_{ij}x_{j} = b_{i} , i = 1, …, m (equality)
x_{j} ≥ 0 , j = 1, …, n (non − negativity)

The standard form of the LP can also be written more compactly in matrix notation.

Minimize c^{T} x  (4)
subject to Ax = b
x ≥ 0

where

c and x are n x 1 vectors, A is an m x n matrix, and b is an m x 1 vector. Definition 1. If x satisfies Ax = b, x ≥ 0, then x is feasible.

2.3 Canonical Form

This is the canonical form of the linear programming problem. Here the constraints take the form of linear inequality constraints, plus non-negativity constraints on the independent variables.

Minimize ∑_{i=1}^{n} c_{i}x_{i}  (objective function)  (5)

subject to ∑_{j=1}^{n} a_{ij}x_{j} ≥ b_{i} , i = 1, …, m (inequality)
x_{j} ≥ 0 , j = 1, …, n (non − negativity)

Once more, the canonical form of the LP can be written more compactly in matrix notation.

Minimize c^{T} x  (6)
subject to Ax ≥ b
x ≥ 0

where c and x are n x 1 vectors, A is an m x n matrix, and b is an m x 1 vector.

2.4 Constraint Conversion

It is possible to convert between equality and inequality constraints by the following method.

1. To convert a less than or equal inequality constraint,

Ax ≤ b

to an equality constraint, add the vector of slack variables, s.

Ax + s = b, s ≥ 0

2. To convert a greater than or equal inequality constraint,

Ax ≥ b

to an equality constraint, add the vector of surplus variables, t.

Ax − t = b, t ≥ 0

3. Finally, to convert an equality constraint,

Ax = b

to an inequality constraint, add two inequality constraints. Ax ≤ b, −Ax ≤ −b

Figure 1: Feasible Region

3 Example

Example 1. Consider the following linear program:

Minimize  x_{2}
subject to x_{1}  ≥ 2
3x_{1} − x_{2} ≥ 0
x_{1} + x_{2} ≥ 6
−x_{1} + 2x_{2} ≥ 0
x_{1}  ≥ 0
x_{2} ≥ 0

The optimal solution is (4, 2) of cost 2 (See Figure 1). If we were maximizing x_{2}, instead of minimizing under the same feasible region, the resulting linear program would be unbounded, since x_{2} can increase arbitrarily. From this picture, the reader should be convinced that, for any objective function for which the linear program is bounded, there exists an optimal solution which is a “corner” of the feasible region. This notion will be formalized in the next section.

4 Solving LPs

Below lists the three main linear programming algorithms listed historically:

Simplex - Invented by George Dantzig in 1947 to solve linear programming problems, this technique was fast for most practical applications. However, it is non-polynomial in worst case scenarios, and later examples were provided where simplex failed to perform efficiently.

Ellipsoid - Introduced by Naum Z. Shor, Arkady Nemirovsky, and David B. Yudin in 1972, and shown to be polynomial by Leonid Khachiyan, this technique, while polynomial, is in practice typically much slower than the simplex algorithm and was therefor rarely used. It was useful for showing that general linear programming was in P however.

Interior Point - Introduced and developed by Narendra Karmarkar in 1984, this mehtod has the advantage of being polynomial like the ellipsoid algorithm, but also fast in practice, like the simplex algorithm. For this reason it was used in practice for a long time. Now, however, hybrids and other variations are used in many cases.

5 The Geometry of LP

Consider a linear program with n variables and m linear constraints. The set of possible values for the n variables is a subset of ℜ^{n}. Each constraint corresponds to a half-space of ℜ^{n} as defined by an associated hyperplane. The body enclosed by the set of these hyperplanes is referred to as polytope.

For any linear program, exactly one of the following conditions holds:

• The solution is unbounded (infinite)

• The solution is infeasible (no polytope exists)

• An optimal solution occurs on a vertex of the polytope

A polytope is always convex, by which we mean that

• A line between 2 points in the polytope remains in the polytope

• If x, y ∈ P, αx + (1 − α)y ∈ P for α ∈ [0, 1]

A vertex, x, of a convex polytope P, is a “corner point” in the polytope and can be defined by any one of these three equivalent statements, as we show next.

• ¬∃y ≠ 0 s.t. x + y, x − y ∈ P

• ¬∃y, z ∈ P s.t. x = αy + (1 − α)z, α ∈ (0, 1)

• ∃n linear inequalities of P that are tight at x

The proof that definitions 1 and 2 are equivalent shall be left to the reader. Here we will prove that claims 1 and 3 are equivalent.

1 ⟹ 3: Suppose ¬3. Let A′ be the submatrix corresponding to the tight inequalities, let b’ be the corresponding right hand sides. This leads to the inequality A′x = b′. The rank of A′ is strictly less than n. This implies that ∃y ≠ 0 s.t. A′y = 0.

Consider x + λy, A′(x + λy) = A′x + λA′y = b′. Find λ ≥ 0 s.t. x + λy, x − λy ∈ P. Now consider any one of the non-tight inequalities, j. A_{j}x > b_{j}. Apply λ, getting A_{j}(x + λy) = A_{j}x + λA_{j}y. It is known that A_{j}x > b_{j}. Because A_{j}x = b_{j} is not tight, λA_{j}y must be feasible in one direction for a short distance, and the other direction infinitely. We shall select a distance λ_{j} equal to that short distance. Finally, we shall select λ = min_{j} |λ_{j}|. Because the associated constraints are not tight, λ cannot be zero. This directly contradicts 1.

3 ⟹ 1: Let A′ denote the submatrix for tight inequalities, as was done above. This implies the rank of A′ ≥ n. Suppose ¬1. This implies ∃y ≠ 0 s.t. x + y, x − y ∈ P. Consider the fact that A′(x + y) ≥ b′, A′(x − y) ≥ b′. This implies that A′y = 0, which implies that the rank of A′ is less than n. This contradicts the assertion made earlier.

5.1 Optimality at a Vertex

We now show that whenever an LP is feasible and bounded, an optimal solution must occur at a vertex. More specifically, if the linear program is feasible and bounded, then ∃v ∈ V ertices(P) s.t. ∀x ∈ P, C^{T} v ≤ C^{T} x. This statement, along with fundamentally stating that an optimal solution must occur at a vertex, also shows the decidability of solving for the system, as one can simply check all the vertices.

The proof shall proceed by picking a non-vertex point and showing that there exists as good or better of a solution with at least one less non-tight equation. Hence progress is made towards a vertex by applying this process iteratively. To find a vertex that is optimal this iteration will have to be done at most n times.

Suppose x ∈ P and x ∉ V ertices(P). This implies ∃y ≠ 0 s.t. x + y, x − y ∈ P. This in turn implies (x + y, x − y ≥ 0) → (x_{i} = 0 → y_{i} = 0). Consider v = x + λy. C^{T} v = C^{T} x + λC^{T} y. Assume without loss of generality that C^{T} y ≤ 0. If this was not the case, the other direction could simply have been selected (x − y).

Now suppose that A′ and b′ give the set of tight variables, as before. We know that A′(x + y) ≥ b′. This implies A′y = 0, and that A′(x + λy) = b′. Once again, we will find the λ_{j} values for the non-tight equations. There are two possibilities:

• ∃λ_{j} ≥ 0: In this case, we select the smallest such λ_{j} value, λ. The point x + λy now makes one additional inequality tight while satisfying the property that it keeps the previously tight inequalities tight. And we have made progress towards a solution.

• ∀_{j}λ_{j} < 0: Here, either C^{T} y < 0, in which case the solution is unbounded, or C^{T} y = 0 in which case we set λ equal to min_{j} |λ_{j}|.

5.2 Duality

One can view any minimization linear program as a maximization. Consider the following linear system:

min 3x_{1} + 2x_{2} + 8x_{3}
s.t. x_{1} − x_{2} + 2x_{3} ≥ 5
x_{1} + 2x_{2} + 4x_{3} ≥ 10
x_{1}, x_{2}, x_{3}  ≥ 0

Where Z^{∗} is OPT, we know Z^{∗} = 3x_{1}^{∗} + 2x_{2}^{∗} + 8x_{3}^{∗}, for some x_{1}^{∗}, x_{2}^{∗}, x_{3}^{∗} ∈ P. By adding two of the inequalities, we arrive at 2x_{1} + x_{2} + x_{3} ≥ 15. Since x_{1}^{∗}, x_{2}^{∗}, x_{3}^{∗} ≥ 0, we know that Z^{∗} ≥ 15. But we aren’t limited to addition, multiplication is another way the equations can be combined. So how is this new formulation bouned? This is done by using the dual formulation, D of the minimization, which for this problem is:

max 5y_{1} + 10y_{2}
s.t. y_{1} + y_{2}  ≤ 3
−y_{1} + 2y_{2} ≤ 2
2y_{1} + 4y_{2} ≤ 8
y_{1}, y_{2}  ≥ 0

The theory of LP duality (sometimes referred to as the Strong Duality Theorem) says that if the primal LP P is bounded and feasible, then the value of the primal LP equals the value of the dual LP.

5.2.1 Weak Duality

Weak duality makes only the claim that the value of the primal LP is at least the value of the dual LP. Consider the primal P and its dual D:

P  D
min c^{T} x  max b^{T} y
s.t. Ax ≥ b s.t. A^{T} y ≤ c
x ≥ 0  y  ≥ 0

Suppose that x^{∗} is an optimal solution to P and y^{∗} is an optimal solution to D. We need only show that c^{T} x^{∗} ≥ b^{T} y^{∗}.

c^{T} x^{∗} ≥ (A^{T} y^{∗})^{T} x^{∗}
= y^{∗}^{T} Ax^{∗}
b^{T} y^{∗} ≤ x^{∗}^{T} A^{T} y^{∗}
= (y^{∗}^{T} Ax^{∗})^{T}

Noting that the last equation of each of these comparison are identical (since the transpose of a scalar is the scalar itself) leads to the desired conclusion.

6 Simplex Algorithm

The Simplex Algorithm is based on the fact that the optimal solution to a feasible LP (linear program) can be found at one of the vertices of the polytope defined by the set of constraints. The algorithm starts from an arbitrary vertex represented by a basic feasible solution (bfs), and at each iteration uses a technique called pivoting to search for an adjacent vertex with an improved cost to the solution to move to. If no adjacent vertex has an improved cost, then it can be proved that the current vertex represents the optimal solution. The algorithm must search through a set of potentially exponentially many vertices, and as a result is not polynomial in the worst-case. Even so, it performs very well in practice, and was the algorithm of choice for several decades.

(Throughout this document, all LP’s are given by: minimize c^{T} x subject to Ax = b and x ≥ 0, unless explicitly defined otherwise.)

6.1 Pivoting

Pivoting is the mechanism used to manipulate the basis corresponding to a vertex v’s bfs, to find the basis that corresponds to an adjacent vertex. To perform a pivot step we replace one of the m linearly independent columns of the basis corresponding to v with one of the nonbasic columns from A in such a way that we still have m linearly independent columns. This produces the basis of an adjacent vertex, v′. The Simplex Algorithm uses pivoting to examine the neighbors of v until a neighbor is found that corresponds to a feasible (non-negative) solution which is an improvement over the bfs for v. We illustrate how pivoting is done with an example.

Example 2. Assume that in LP_{1},

A = [ \begin{matrix}5 & 2 & −3 & 16 & 4 \\ 2 & 3 & 1 & 3 & 1 \\ 1 & 7 & 6 & −1 & 2 \end{matrix} ]

b = [ \begin{matrix}8 \\ 8 \\ 25 \end{matrix} ]

and we have determined a basis,

B = {1, 3, 5}.

(B is specified here by identifying the basic columns. In other words, x_{2} = x_{4} = 0.) This gives us the following three linearly independent equations, which can be solved to find the corresponding bfs:

5x_{1} − 3x_{2} + 4x_{5} = 8
2x_{1} + x_{2} + x_{5} = 8
x_{1} + 6x_{3} + 2x_{5} = 25

The solution, x_{1} = 1, x_{3} = 3, x_{5} = 3; can also be written as:

bfs = [ \begin{matrix}1 \\ 0 \\ 3 \\ 0 \\ 3 \end{matrix} ] .

We now perform a pivot to find another basis, by adding a nonbasic column to B, and removing one of the basic columns from B. Assume that we choose to add column A_{4}. We can express A_{4} as a linear combination of the columns that are in B as follows:

A_{4} = αA_{1} + βA_{3} + γA_{5}
16 = 5α − 3β + 4γ
3 = 2α + β + γ
−1 = α + 6β + 2γ

The solution is: α = 1, β = −1, γ = 2. Therefore, A_{4} = A_{1} − A_{3} + 2A_{5}. Since we want to find a new basis, B′, with a feasible solution, we must find a basic column to remove, such that:

• A_{B′}x_{B′} = b,

• x_{B′} ≥ 0,

• and x_{2} = 0.

where A_{B′} refers to the basic columns of A. We have already solved

A_{1}x_{1} + A_{3}x_{3} + A_{5}x_{5} = b

and want to modify that solution to include A_{4} in the basis so that:

A_{1}x′_{1} + A_{3}x′_{3} + A_{4}Θ + A_{5}x′_{5} = b
x′_{1}, x′_{3}, x′_{5} ≥ 0

Since we know that A_{4} = A_{1} − A_{3} + 2A_{5}, we simply need to find Θ ≥ 0 such that,

A_{1}(x′_{1} + Θ) + A_{3}(x′_{3} − Θ) + A_{5}(x′_{5} + 2Θ) = b.

But we know that:

A_{1}x_{1} + A_{3}x_{3} + A_{5}x_{5} = b

so we have:

\begin{aligned}x_{1} & = & x′_{1} + Θ = & 1, & ⇒ & Θ ≤ 1 \\ x_{3} & = & x′_{3} − Θ = & 3, & ⇒ & Θ ≥ −3 \\ x_{5} & = & x′_{5} + 2Θ = & 3, & ⇒ & Θ ≤ \frac{3}{2} \end{aligned}

We simply choose the smallest non-negative Θ that satisfies these conditions, namely Θ = 1 ⇒ x_{1} = 0. Thus our new basis is B′ = {4, 3, 5}, and our bfs has x′_{4} = 1, x′_{3} = 4, x′_{5} = 1, and we have completed a pivot.

Let us review the steps involved in pivoting. The idea is to start from a basis, B = {1, 2, …, m} and then find a suitable nonbasic column j ∈ {1, 2, …, m} to replace a basic column l ∉ {1, 2, …, m} in order to arrive at an adjacent basis, B′. This is achieved using the following steps:

1. Assume that the starting basis B = {1, 2, …, m}, is known.

B contains m linearly independent columns of the n columns in A. We have already found the bfs for B, bfs_{B} which satisfies:

∑_{i=1}^{m} A_{i}x_{i} = b  (7)
x ≥ 0  (8)

2. Choose a nonbasic column j ∉ {1, 2, …, m} to add to B.

(In the Simplex Algorithm, each j ∉ {1, 2, …, m} can be tried in turn until one is found that results in a bfs with an improved cost.)

3. Find the linear dependence of column j on the basic columns, by finding α_{i,j} such that:

A_{j} = ∑_{i=1}^{m} α_{i,j}A_{i}.  (9)

4. Find our options for x′_{j} = Θ, so that

∑_{i=1}^{m} A_{i}x′_{i} + A_{j}x′_{j} = b  (10)

We can find the feasible range of values for Θ as follows:

∑_{i=1}^{m} A_{i}x_{i} + (A_{j}x′_{j} − A_{j}x′_{j})  = b (based on Equation 7)

∑_{i=1}^{m} A_{i}x_{i} + A_{j}x′_{j} − Θ ∑_{i=1}^{m} α_{i,j}A_{i}

= b (based on Equation 9)
∑_{i=1}^{m} A_{i}(x_{i} − Θα_{i,j}) + A_{j}x′_{j}  = b

So x′_{i} = x_{i} − Θα_{i,j}, and in order for x′_{i} ≥ 0, we must have Θ ≤ \frac{x_{i}}{α_{i,j}}.

5. Choose the basic column to l ∈ {1, 2, …, m} to remove from B.

We want to choose the smallest value of Θ that will result in a feasible solution, so we take Θ to be the

\underbrace{min}_{∀α_{i,j}>0} \frac{x_{i}}{α_{i,j}} .

l is the value of i associated with our choice of Θ.

(N.B. If ∀α_{i,j} ≤ 0, then the solution is unbounded.)

6.2 Tableau Method

The Tableau Method provides an efficient means of performing the pivots required for the Simplex Algorithm. The idea is to manipulate A and b so that the basic columns of A become the identity matrix. When this happens, Step 3 of the pivoting process (above)—finding the α_{i,j} coefficients for A_{j}—becomes trivial, because (based on Equation 9) α_{i,j} = a_{i,j} (where a_{i,j} is the (i, j) entry of A—the member in the ith row of A_{j}).

Example 3. Consider LP_{2} defined by

6 = x_{1} + x_{2} + 4x_{6}
14 = −2x_{1} + 2x_{2} + x_{3} − x_{4} + x_{6}
−11 = x_{1} − 2x_{2} + x_{4} + 2x_{6}
7 = x_{1} − 3x_{4} + x_{5} − 5x_{6}

for which we have determined that a basis, B = {2, 3, 4, 5}. We can represent LP_{2} by an m×(n+1) matrix, a tableau, as follows:

\begin{matrix}b & A_{1} & A_{2} & A_{3} & A_{4} & A_{5} & A_{6} \\ 6 & 1 & 1 & 0 & 0 & 0 & 4 \\ 14 & −2 & 2 & 1 & −1 & 0 & 1 \\ −11 & 1 & −2 & 0 & 1 & 0 & 2 \\ 7 & 1 & 0 & 0 & −3 & 1 & −5 \end{matrix}

By performing row operations on the tableau, we can transfrom the columns that represent B into the identity matrix, without affecting the solution space. This transformation yields,

\begin{matrix}b & A_{1} & A_{2} & A_{3} & A_{4} & A_{5} & A_{6} \\ 6 & 1 & 1 & 0 & 0 & 0 & 4 \\ 3 & −1 & 0 & 1 & 0 & 0 & 3 \\ 1 & 3 & 0 & 0 & 1 & 0 & 10 \\ 10 & 10 & 0 & 0 & 0 & 1 & 25 \end{matrix}

There are two possible nonbasic columns that we may consider adding to our basis as part of a pivot. Assume that we choose column 1, so that j = 1. Then we have,

A_{j} = A_{1} = 1A_{2} − 1A_{3} + 3A_{4} + 10A_{5}

Θ is determined by

Θ = \underbrace{min}_{∀α_{i,1}>0} \frac{x_{i}}{α_{i,j}} = \frac{1}{3}.

(since α_{i,j} = a_{i,j}), so that l = 4, and the new basis, B′, is {2, 3, 1, 5}.

Row operations are once again performed in order to convert B′ to the identity matrix (and update b to the new bfs b′), yielding

\begin{matrix}b′ & A′_{1} & A′_{2} & A′_{3} & A′_{4} & A′_{5} & A′_{6} \\ \frac{17}{3} & 0 & 1 & 0 & \frac{−1}{3} & 0 & \frac{2}{3} \\ \frac{10}{3} & 0 & 0 & 1 & \frac{1}{3} & 0 & \frac{19}{3} \\ \frac{1}{3} & 1 & 0 & 0 & \frac{1}{3} & 0 & \frac{10}{3} \\ \frac{20}{3} & 0 & 0 & 0 & \frac{−10}{3} & 1 & \frac{−25}{3} \end{matrix}

The cost function, c^{T} x, is what actually determines which of the nonbasic columns gets chosen to include in the new basis—we must find one which results in decrease in the cost. For any proposed replacement of some column l by some column j, we must simply check that the cost of the new solution is less that the cost of the previous solution. The cost of new solution is given by:

new cost = ∑_{i=1}^{m} x′_{i}c_{i} + x′_{j}c_{j}

= ∑_{i=1}^{m}(x_{i} − Θα_{i,j}) + Θc_{j}

= \underbrace{∑_{i=1}^{m} x_{i}c_{i}}_{old cost} +Θ \underbrace{[c_{j} − ∑_{i=1}^{m} α_{i,j}c_{i}]}_{modified cost, c̄_{j}}

So if the modified cost, c̄_{j}, is negative,, then the new cost will be reduced.

Let us review how pivoting is performed using the Tableau Method. A tableau, constructed as follows, is used to perform pivots:

\begin{matrix}∗ & c̄_{1} & c̄_{2} & … & c̄_{n} \\ b_{1} & A_{1,1} & A_{1,2} & … & A_{1,n} \\ b_{2} & A_{2,1} & A_{2,2} & … & A_{2,n} \\ ⋮ & ⋮ & ⋮ & ⋮ & ⋮ \\ b_{m} & A_{m,1} & A_{m,2} & … & A_{m,n} \end{matrix}

in which the basic columns of A form the identity matrix, and the each nonbasic column, j, contains the coefficients, α_{i,j}. (An additional row, row 0, is used to store the modified costs.)

A pivot is made with the following steps:

1. Find a column j such that c̄_{j} < 0, by examining row 0. (If no such column exists, we have found the optimal solution.)

2. Find column l with α_{l,j} > 0, that minimizes Θ (If all α_{i,j} ≤ 0 then either the solution is unbounded–unless we didn’t have a basis to begin with.)

3. Replace l by j

We have yet to specify how the starting basis is found an LP. Suppose we are given LP_{A}: minimize c^{T} x subject to Ax ≤ b, and x ≥ 0. Then we can add one surplus variable per constraint, such that:

∑_{j} a_{i,j}x_{j} + s_{i} = b_{i}, i ∈ {1, 2, …, m}

Then our bfs is: {x_{j} = 0, s_{i} = b_{i}}. If instead of Ax ≤ b we have Ax = b, then we can similarly add artificial variables, such that:

∑_{j} a_{i,j}x_{j} + y_{i} = b_{i}i ∈ {1, 2, …, m}

to form LP_{B} (for which we can easily construct a basis) for which we minimize ∑ y_{i}, for y_{i} ≥ 0. LP_{A} will be feasible iff the optimal solution for LP_{B} is 0. To get a bfs for LP_{A} we solve LP_{B} using the Simplex Algorithm by starting with the bfs, {x_{j} = 0, y_{i} = b_{i}}, and if the optimal solution for LP_{B} is 0, then all y′_{i}s are 0 and the x′_{i} form a bfs for LP_{A}.

References

[Goe94] M. Goemans. Introduction to Linear Programming. Lecture notes, available from http://www-math.mit.edu/˜ goemans/, October 1994.

[Kar91] H. Karloff. Linear Programming. Birkhäuser, Boston, MA, 1991.