% $Header: /home/rdv/quantum/genetic-paper/RCS/eaqc-paper.tex,v 1.12 2005/05/29 21:34:00 rdv Exp rdv $
% now how do I set the paper size?
% \documentclass[twocolumn]{article}

\documentclass[onecolumn,doublespace]{article}
\usepackage{times}
\usepackage{graphics}
\usepackage{lscape}
\usepackage{subfigure}
\usepackage{graphicx}
\usepackage{color}
\usepackage{amsmath,amsfonts,amssymb}
\usepackage{kbordermatrix,blkarray}
\pagestyle{plain}

\bibliographystyle{unsrt}   %>>>> makes bibtex use spiebib.bst

% \input{psfig}
\input{epsf}
% \epsfverbosetrue

%\def\comment#1{$\Rightarrow${\small\em #1}}
\def\comment#1{}
\newenvironment{chapterquote}
  {\begin{quotation}
      \sf}
  {\end{quotation}}
\def\cq#1#2{\begin{chapterquote}#1\flushright\textbf{#2}\end{chapterquote}}


\newcommand{\icol}[1]{% inline column vector
  \left(\begin{smallmatrix}#1\end{smallmatrix}\right)%
}

\newcommand{\irow}[1]{% inline row vector
  \begin{smallmatrix}(#1)\end{smallmatrix}%
}

\newcommand{\tvec}[1]{%
  \vec{#1}^{\thinspace\intercal}%
}

\begin{document}
\title{Optimization Theory (DS2) Lecture \#3\\
Three Kinds of Certificates, Standard Equality Form, and a Simplex Iteration
}
\maketitle
\begin{abstract}
  Today we are going to prepare for the simplex algorithm.  We will
  cover Secs. 2.1 to 2.3 of the textbook.
\end{abstract}

\emph{Text and examples adapted from \emph{A Gentle Introduction to
    Optimization}.}

\section{Review}

Recall, last time we worked on how to formulate problems, going from a
word description of a problem to a formal description including {\bf
  Variables}, {\bf Constraints}, and the {\bf Objective Function}.
Once you have those, you can produce a formal description that says,
``Maximize (some function) such that (s.t.) (some constraints hold).''

\section{Geometric Interpretation of Constraints and Objective Function}

First, let's review the idea of constraints with a 2-D example.  If I
give you the equation
\begin{equation}
y \ge x + 2
\end{equation}
you know that this cuts the X-Y plane in half; points in one half
satisfy the constraint, points in the other half don't.

If we add a second constraint, ...

\subsection{A Geometric Simplex}

A \emph{simplex} is an $n$-dimensional generalization of a triangle:
in one dimension it's a line segment, in two it's a triangle, in three
it's a tetrahedron, in four or more it's more generally called a
simplex.  It has $n+1$ vertices.  In terms of optimization problems,
it has $n+1$ independent constraints.

\subsection{More Complex Polytopes}

In the next lecture, we will see the $n$-dimensional generalization of
a polyhedron known as a \emph{polytope}.

\section{Certificates}

As the book says, ``By definition, an LP [linear program] has only one
optimal value, but it may have many optimal solutions.''  However, it
may not have a solution at all!  Or, it may have a solution that runs
off to infinity.  So, there are three possible cases we would like to
distinguish:

\begin{enumerate}
\item our problem is \emph{infeasible} (has no solution),
\item it has one or more \emph{optimal} solutions, or
\item it is \emph{unbounded}.
\end{enumerate}

Also, in the case where we have a solution that we think is optimal,
it would be nice to be able to prove that to others.  We will see
that, once the problem is properly formulated, there are one or two
vectors we can present that will show which of these cases holds.
These vectors are called \emph{certificates of infeasibility},
\emph{optimality}, or \emph{unboundedness}, respectively.

\subsection{Infeasibility}

We only need to look at the constraints, not the objective function,
and demonstrate that they are mutually exclusive: by the time you take
them all into account, there is no space left for a working solution
at all.

\subsection{2-D Example}

\emph{To be added.}

\subsection{Larger Example}

Consider the constraints of a problem $(P)$,
\begin{align}
4x_1 + 10x_2 - 6x_3 -2x_4 & = 6 \\
-2x_1 + 2x_2 - 4x_3 + x_4 & = 5 \\
-7x_1 - 2x_2 + 4x_4 & = 3, \\
x_1, x_2, x_3, x_4 & \ge 0.
\end{align}

To show there is no solution, we create the equation
\begin{equation}
x_1 + 4x_2 + 2x_3 = -1
\end{equation}
and from this, we are done: all of the coefficients of the $x_i$ are
positive (or non-negative, including $x_4$), and all of the $x_i$ are
\emph{constrained} to be non-negative, so there is no way this can be
solved!

So how do we create that equation?  Via the usual algebraic
manipulation of the equations above:
\begin{equation}
1\times\textrm{(Eq. 1)} -2\times\textrm{(Eq. 2)} + 1\times\textrm{(Eq. 3)}.
\end{equation}
If we write down those values as $y_1 = 1$, $y_2 = -2$, $y_3 = 1$,
then we can create the vector
\begin{equation}
  \vec{y} = (y_1, y_2, y_3)^\intercal = (1, -2, 1)^\intercal.
\end{equation}
The vector $\vec{y}$ is said to be our \emph{certificate of
  infeasibility}.

We can write this down as
\begin{equation}
  A\vec{x} = \vec{b}, \label{eq:axb}
\end{equation}
where
\begin{align}
  A & = \left(
    \begin{array}{cccc}
      4 & 10 & -6 & -2 \\
      -2 & 2 & -4 & 1 \\
      -7 & -2 & 0 & 4
    \end{array}
  \right) \\
\vec{b} & = \left(
  \begin{array}{c}
    6 \\
    5 \\
    3
  \end{array}
  \right) \\
\vec{x} & = \left(
  \begin{array}{c}
    x_1 \\
    x_2 \\
    x_3 \\
    x_4
  \end{array}
  \right).
\end{align}
We multiply both sides by the vector $\tvec{y}$ to get
\begin{equation}
  \vec{y}^{\thinspace\intercal} A\vec{x} = \tvec{y}\vec{b}, \label{eq:axb}
\end{equation}

Insert numbers for $\vec{y}$ into that equation, and you will get an
equation with all non-negative values on the left and a negative value
on the right.  Formally,
\begin{quotation}
  Let $A$ be a matrix and $\vec{b}$ be a vector. Then the system
  \begin{equation}
    A\vec{x} = \vec{b}, \vec{x} \ge \vec{0}
  \end{equation}
has no solution if there exists a vector $\vec{y}$ s.t.
\begin{enumerate}
\item $\tvec{y}A \ge
  \tvec{0}$, and
\item $\tvec{y}\vec{b} < 0$.
\end{enumerate}
\end{quotation}

Farkas' lemma tells us that there exists a certificate of
infeasibility for every infeasible system.

\subsection{Unbounded Linear Programs}

\emph{To be added.}

\subsection{2-D Example}

\emph{To be added.}

\subsection{Larger Example}

Consider the already-formulated LP
\begin{equation}
  \operatorname{max}\{z(\vec{x}) = \tvec{c}\vec{x} : A\vec{x} =
  \vec{b}, \vec{x} \ge \vec{0}\},
\end{equation}
where
\begin{align}
  A & = \left(
    \begin{array}{ccccc}
      1 & 1 & -3 & 1 & 2 \\
      0 & 1 & -2 & 2 & -2 \\
      -2 & -1 & 4 & 1 & 0
    \end{array}
  \right) \\
\vec{b} & = \left(
  \begin{array}{c}
    7 \\
    -2 \\
    -3
  \end{array}
  \right) \\
\vec{c} & = \left(
  \begin{array}{c}
    -1 \\
    0 \\
    3 \\
    7 \\
    -1
  \end{array}
  \right) \\
\vec{x} & = \left(
  \begin{array}{c}
    x_1 \\
    x_2 \\
    x_3 \\
    x_4 \\
    x_5
  \end{array}
  \right).
\end{align}

If we pick vectors $\vec{x_a} = (2, 0, 0, 1, 2)^\intercal$ and
$\vec{d} = (1, 2, 1, 0, 0)^\intercal$ and define a function
\begin{equation}
  \vec{x}(t) = \vec{x_a} + t\vec{d},
\end{equation}
then we can show that
\begin{equation}
  \tvec{c}\vec{x}(t) = \tvec{c}\vec{x_a} + 2t,
\end{equation}
and so as $t \rightarrow \infty$, so does our objective function.
Formally,
\begin{quotation}
  Suppose there exists a problem $(P)$ as above, with a feasible
  solution $\vec{x_a}$ and a vector $\vec{d}$ such that:
\begin{enumerate}
\item $A\vec{d} = \vec{0}$, and
\item $\vec{d} \ge \vec{0}$, and
\item $\tvec{c}\vec{d} > 0$,
\end{enumerate}
then $(P)$ is unbounded.
\end{quotation}

The pair of vectors $\vec{d}$ and $\vec{x_a}$ together are our
\emph{certificate of unboundedness}.

\subsection{Optimality}

See Sec. 2.1.3.

\section{Standard Equality Form (SEF)}

An LP is in \emph{standard equality form} if:
\begin{enumerate}
\item it is a maximization problem;
\item other than the non-negativity constraints, all constraints are
  \emph{equalities}; and
\item every variable has a non-negativity constraint.
\end{enumerate}
That is, it can be written in the form
\begin{equation}
  \operatorname{max}\{z(\vec{x}) = \tvec{c}\vec{x} : A\vec{x} =
  \vec{b}, \vec{x} \ge \vec{0}\}.
\end{equation}

Consider the LP
\begin{equation}
  \operatorname{max}(1,-2,4)(x_1,x_2,x_3)^\intercal
\end{equation}
subject to
\begin{align}
  \left(
    \begin{array}{ccc}
      1 & 5 & 3 \\
      2 & -1 & 2 \\
      1 & 2 & -1 
    \end{array}
  \right)
\left(
  \begin{array}{c}
    x_1 \\
    x_2 \\
    x_3
  \end{array}
  \right)
  \begin{array}{c}
    \ge \\
    \le \\
    =
  \end{array}
\left(
  \begin{array}{c}
    5 \\
    4 \\
    2
  \end{array}
  \right)\\
x_1, x_2 \ge 0.
\end{align}

$x_3$ is called a \emph{free variable} since it doesn't have a
non-negativity constraint.  But a lot of our techniques and especially
proofs depend on that.  So, we introduce two new variables, $x_3^+$
and $x_3^-$, set $x_3 = x_3^+ - x_3^-$, and add constraints $x_3^+,
x_3^- \ge 0$.  After some algebra, you get
\begin{equation}
  \operatorname{max}(1,-2,4,-4)(x_1,x_2,x_3^+,x_3^-)^\intercal
\end{equation}
subject to
\begin{align}
  \left(
    \begin{array}{cccc}
      1 & 5 & 3 & -3 \\
      2 & -1 & 2 & -2 \\
      1 & 2 & -1 & 1 
    \end{array}
  \right)
\left(
  \begin{array}{c}
    x_1 \\
    x_2 \\
    x_3^+ \\
    x_3^-
  \end{array}
  \right)
  \begin{array}{c}
    \ge \\
    \le \\
    =
  \end{array}
\left(
  \begin{array}{c}
    5 \\
    4 \\
    2
  \end{array}
  \right) \label{eq:sef-pre}\\
x_1, x_2, x_3^+, x_3^- \ge 0.
\end{align}

Getting closer, but we're still not quite there; we don't have
equalities everywhere yet.  So we introduce two \emph{slack variables}
$x_4$ and $x_5$.  Now we get

\begin{equation}
  \operatorname{max}(1,-2,4,-4,0,0)(x_1,x_2,x_3^+,x_3^-,x_4,x_5)^\intercal
\end{equation}
subject to
\begin{align}
  \left(
    \begin{array}{cccccc}
      1 & 5 & 3 & -3 & 0 & -1\\
      2 & -1 & 2 & -2 & 1 & 0 \\
      1 & 2 & -1 & 1 & 0 & 0
    \end{array}
  \right)
\left(
  \begin{array}{c}
    x_1 \\
    x_2 \\
    x_3^+ \\
    x_3^- \\
    x_4 \\
    x_5
  \end{array}
  \right)
    =
\left(
  \begin{array}{c}
    5 \\
    4 \\
    2
  \end{array}
  \right)\\
x_1, x_2, x_3^+, x_3^-, x_4, x_5 \ge 0
\end{align}
and we're finally in SEF!  Note that when we replaced a $\le$
constraint (second line of Eq.~\ref{eq:sef-pre}), we added a $1$ to
the array, and when we replaced a $\ge$ constraint (first line of
Eq.~\ref{eq:sef-pre}), we added $-1$ to the array.  This helps us keep
our non-negativity constraint for all variables.

\section{A Simplex Iteration}

The idea of the \emph{simplex algorithm} is pretty simple: repeat a
\emph{simplex operation} until no more simplex operations improve the
result, and you're done.  But there's a catch: the second simplex
operation might partly undo the work of the first!  So there is
something you have to do in order to make that second one work, and
that's what we'll see next time.

...Oh, wait, we haven't talked about a single simplex operation.
Okay, let's talk about that.  (n.b.: It's not \emph{really} a simplex
operation, in the mathematical sense, but it's related, and the name
has stuck.)

Consider the following LP in SEF:

\begin{equation}
  \operatorname{max} z(\vec{x}) = (2,3,0,0,0)(x_1,x_2,x_3,x_4,x_5)^\intercal
\end{equation}
subject to
\begin{align}
  \left(
    \begin{array}{ccccc}
      1 & 1 & 1 & 0 & 0 \\
      2 & 1 & 0 & 1 & 0 \\
      -1 & 1 & 0 & 0 & 1 
    \end{array}
  \right)
  \vec{x} =
\left(
  \begin{array}{c}
    6 \\
    10 \\
    4
  \end{array}
  \right) \\
x_1, x_2, x_3, x_4, x_5 \ge 0.
\end{align}

Given the solution $\vec{x_a} = (0,0,6,10,4)^\intercal$ (easy to see that's
a solution -- look at the right side of the array), $z(\vec{x_a}) =
0$.

Now try increasing $x_1$, choosing $x_1 = t$.  A little algebra gives
\begin{equation}
\left(
  \begin{array}{c}
    x_3 \\
    x_4 \\
    x_5
  \end{array}
  \right) 
= \left(
  \begin{array}{c}
    6 \\
    10 \\
    4
  \end{array}
  \right) 
- t \left(
  \begin{array}{c}
    1 \\
    2 \\
    -1
  \end{array}
  \right) 
\end{equation}
from which we get
\begin{equation}
  t
\left(
  \begin{array}{c}
    1 \\
    2 \\
    -1
  \end{array}
  \right) 
\le \left(
  \begin{array}{c}
    6 \\
    10 \\
    4
  \end{array}
  \right) 
\end{equation}
which gives us the limiting inequality $t \le 10/2 = 5$.  Setting $t = 5$,
$\vec{x'} = (5, 0, 1, 0, 9)^\intercal$ and $z(\vec{x'}) = 10$.

Unfortunately, we can't yet apply the same trick to $x_2$, so that's
the topic for next time!

\section{Homework}

See the separate file uploaded to SFS.

\begin{enumerate}
\item Exercise 1 (parts a, b) in Sec. 2.1.
\item Exercise 1 (part a) in Sec. 2.2.
\item Exercise 1 (all parts) in Sec. 2.3.
\end{enumerate}

\end{document}
