Optimization Theory (DS2) Lecture #3
Three Kinds of Certificates, Standard Equality Form, and a Simplex Iteration

Abstract

Today we are going to prepare for the simplex algorithm. We will cover Secs. 2.1 to 2.3 of the textbook.

Text and examples adapted from A Gentle Introduction to Optimization.

1 Review

Recall, last time we worked on how to formulate problems, going from a word description of a problem to a formal description including Variables, Constraints, and the Objective Function. Once you have those, you can produce a formal description that says, “Maximize (some function) such that (s.t.) (some constraints hold).”

2 Certificates

As the book says, “By definition, an LP [linear program] has only one optimal value, but it may have many optimal solutions.” However, it may not have a solution at all! Or, it may have a solution that runs off to infinity. So, there are three possible cases we would like to distinguish:

1. our problem is infeasible (has no solution),

2. it has one or more optimal solutions, or

3. it is unbounded.

Also, in the case where we have a solution that we think is optimal, it would be nice to be able to prove that to others. We will see that, once the problem is properly formulated, there are one or two vectors we can present that will show which of these cases holds. These vectors are called certificates of infeasibility, optimality, or unboundedness, respectively.

2.1 Infeasibility

We only need to look at the constraints, not the objective function, and demonstrate that they are mutually exclusive: by the time you take them all into account, there is no space left for a working solution at all.

Consider the constraints of a problem $(P)$,

	$4x_1+10x_2-6x_3-2x_4$	$=6$		(1)
	$-2x_1+2x_2-4x_3+x_4$	$=5$		(2)
	$-7x_1-2x_2+4x_4$	$=3,$		(3)
	$x_1,x_2,x_3,x_4$	$\ge 0.$		(4)

To show there is no solution, we create the equation

$$x_1+4x_2+2x_3=-1$$

(5)

and from this, we are done: all of the coefficients of the $x_i$ are positive (or non-negative, including $x_4$), and all of the $x_i$ are constrained to be non-negative, so there is no way this can be solved!

So how do we create that equation? Via the usual algebraic manipulation of the equations above:

$$1\times \text{(Eq. 1)}-2\times \text{(Eq. 2)}+1\times \text{(Eq. 3)}.$$

(6)

If we write down those values as $y_1=1$, $y_2=-2$, $y_3=1$, then we can create the vector

$$\overrightarrow{y}={(y_1,y_2,y_3)}^{⊺}={(1,-2,1)}^{⊺}.$$

(7)

The vector $\overrightarrow{y}$ is said to be our certificate of infeasibility.

We can write this down as

$$A\overrightarrow{x}=\overrightarrow{b},$$

(8)

where

	$A$			(9)
	$\overrightarrow{b}$	$=\left(\begin{array}{c}\hfill 6\hfill \\ \hfill 5\hfill \\ \hfill 3\hfill \end{array}\right)$		(10)
	$\overrightarrow{x}$	$=\left(\begin{array}{c}\hfill x_1\hfill \\ \hfill x_2\hfill \\ \hfill x_3\hfill \\ \hfill x_4\hfill \end{array}\right).$		(11)

We multiply both sides by the vector ${\overrightarrow{y}}^{⊺}$ to get

$${\overrightarrow{y}}^{⊺}A\overrightarrow{x}={\overrightarrow{y}}^{⊺}\overrightarrow{b},$$

(12)

Insert numbers for $\overrightarrow{y}$ into that equation, and you will get an equation with all non-negative values on the left and a negative value on the right. Formally,

Let $A$ be a matrix and $\overrightarrow{b}$ be a vector. Then the system

$$A\overrightarrow{x}=\overrightarrow{b},\overrightarrow{x}\ge \overrightarrow{0}$$

(13)

has no solution if there exists a vector $\overrightarrow{y}$ s.t.

1. ${\overrightarrow{y}}^{⊺}A\ge {\overrightarrow{0}}^{⊺}$, and

2. .

Farkas’ lemma tells us that there exists a certificate of infeasibility for every infeasible system.

2.2 Unbounded Linear Programs

Consider the already-formulated LP

$$max\{z(\overrightarrow{x})={\overrightarrow{c}}^{⊺}\overrightarrow{x}:A\overrightarrow{x}=\overrightarrow{b},\overrightarrow{x}\ge \overrightarrow{0}\},$$

(14)

where

	$A$			(15)
	$\overrightarrow{b}$	$=\left(\begin{array}{c}\hfill 7\hfill \\ \hfill -2\hfill \\ \hfill -3\hfill \end{array}\right)$		(16)
	$\overrightarrow{c}$	$=\left(\begin{array}{c}\hfill -1\hfill \\ \hfill 0\hfill \\ \hfill 3\hfill \\ \hfill 7\hfill \\ \hfill -1\hfill \end{array}\right)$		(17)
	$\overrightarrow{x}$	$=\left(\begin{array}{c}\hfill x_1\hfill \\ \hfill x_2\hfill \\ \hfill x_3\hfill \\ \hfill x_4\hfill \\ \hfill x_5\hfill \end{array}\right).$		(18)

If we pick vectors $\overrightarrow{x_a}={(2,0,0,1,2)}^{⊺}$ and $\overrightarrow{d}={(1,2,1,0,0)}^{⊺}$ and define a function

$$\overrightarrow{x}(t)=\overrightarrow{x_a}+t\overrightarrow{d},$$

(19)

then we can show that

$${\overrightarrow{c}}^{⊺}\overrightarrow{x}(t)={\overrightarrow{c}}^{⊺}\overrightarrow{x_a}+2t,$$

(20)

and so as $t\to \mathrm{\infty }$, so does our objective function. Formally,

Suppose there exists a problem $(P)$ as above, with a feasible solution $\overrightarrow{x_a}$ and a vector $\overrightarrow{d}$ such that:

1. $A\overrightarrow{d}=\overrightarrow{0}$, and

2. $\overrightarrow{d}\ge \overrightarrow{0}$, and

3. ${\overrightarrow{c}}^{⊺}\overrightarrow{d}>0$,

then $(P)$ is unbounded.

The pair of vectors $\overrightarrow{d}$ and $\overrightarrow{x_a}$ together are our certificate of unboundedness.

2.3 Optimality

See Sec. 2.1.3.

3 Standard Equality Form (SEF)

An LP is in standard equality form if:

1. it is a maximization problem;

2. other than the non-negativity constraints, all constraints are equalities; and

3. every variable has a non-negativity constraint.

That is, it can be written in the form

$$max\{z(\overrightarrow{x})={\overrightarrow{c}}^{⊺}\overrightarrow{x}:A\overrightarrow{x}=\overrightarrow{b},\overrightarrow{x}\ge \overrightarrow{0}\}.$$

(21)

Consider the LP

$$max(1,-2,4){(x_1,x_2,x_3)}^{⊺}$$

(22)

subject to

			(23)
	$x_1,x_2\ge 0.$		(24)

$x_3$ is called a free variable since it doesn’t have a non-negativity constraint. But a lot of our techniques and especially proofs depend on that. So, we introduce two new variables, $x_3^{+}$ and $x_3^{-}$, set $x_3=x_3^{+}-x_3^{-}$, and add constraints $x_3^{+},x_3^{-}\ge 0$. After some algebra, you get

$$max(1,-2,4,-4){(x_1,x_2,x_3^{+},x_3^{-})}^{⊺}$$

(25)

subject to

			(26)
	$x_1,x_2,x_3^{+},x_3^{-}\ge 0.$		(27)

Getting closer, but we’re still not quite there; we don’t have equalities everywhere yet. So we introduce two slack variables $x_4$ and $x_5$. Now we get

$$max(1,-2,4,-4,0,0){(x_1,x_2,x_3^{+},x_3^{-},x_4,x_5)}^{⊺}$$

(28)

subject to

			(29)
	$x_1,x_2,x_3^{+},x_3^{-},x_4,x_5\ge 0$		(30)

and we’re finally in SEF! Note that when we replaced a $\le$ constraint (second line of Eq. 26), we added a $1$ to the array, and when we replaced a $\ge$ constraint (first line of Eq. 26), we added $-1$ to the array. This helps us keep our non-negativity constraint for all variables.

4 A Simplex Iteration

The idea of the simplex algorithm is pretty simple: repeat a simplex operation until no more simplex operations improve the result, and you’re done. But there’s a catch: the second simplex operation might partly undo the work of the first! So there is something you have to do in order to make that second one work, and that’s what we’ll see next time.

…Oh, wait, we haven’t talked about a single simplex operation. Okay, let’s talk about that. (n.b.: It’s not really a simplex operation, in the mathematical sense, but it’s related, and the name has stuck.)

Consider the following LP in SEF:

$$maxz(\overrightarrow{x})=(2,3,0,0,0){(x_1,x_2,x_3,x_4,x_5)}^{⊺}$$

(31)

subject to

			(32)
	$x_1,x_2,x_3,x_4,x_5\ge 0.$		(33)

Given the solution $\overrightarrow{x_a}={(0,0,6,10,4)}^{⊺}$ (easy to see that’s a solution – look at the right side of the array), $z(\overrightarrow{x_a})=0$.

Now try increasing $x_1$, choosing $x_1=t$. A little algebra gives

$$\left(\begin{array}{c}\hfill x_3\hfill \\ \hfill x_4\hfill \\ \hfill x_5\hfill \end{array}\right)=\left(\begin{array}{c}\hfill 6\hfill \\ \hfill 10\hfill \\ \hfill 4\hfill \end{array}\right)-t\left(\begin{array}{c}\hfill 1\hfill \\ \hfill 2\hfill \\ \hfill -1\hfill \end{array}\right)$$

(34)

from which we get

$$t\left(\begin{array}{c}\hfill 1\hfill \\ \hfill 2\hfill \\ \hfill -1\hfill \end{array}\right)\le \left(\begin{array}{c}\hfill 6\hfill \\ \hfill 10\hfill \\ \hfill 4\hfill \end{array}\right)$$

(35)

which gives us the limiting inequality $t\le 10/2=5$. Setting $t=5$, $\overrightarrow{x^{\prime }}={(5,0,1,0,9)}^{⊺}$ and $z(\overrightarrow{x^{\prime }})=10$.

Unfortunately, we can’t yet apply the same trick to $x_2$, so that’s the topic for next time!

5 Homework

See the separate file uploaded to SFS.

1. Exercise 1 (parts a, b) in Sec. 2.1.

2. Exercise 1 (part a) in Sec. 2.2.

3. Exercise 1 (all parts) in Sec. 2.3.