\input{template}
\input{macros}
\begin{document}

 \lecture{11}{Simplex Algorithm: Determining a neighbour of greater cost}{Rajhans Samdani}
\section{Simplex Algorithm}
First have a relook at the simplex algorithm :
\begin{enumerate}
 \item Start at an extreme point.
\item Move to a ``neighbour'' of greater cost if one exists and repeat the same procedure with it. If no such neighbour exists then exit with this point as the optimum point.
\end{enumerate}

\section{The notion of directions}
Lets consider an example
\begin{Exa}
Consider two half-planes in 2-d space given by
\begin{align*}
3x + 4y &= 12 \\
7x + 3y &= 6
\end{align*}

We have to find out their point of intersection and the direction of the vectors $v_{1}$ and $v_{2}$ ``along'' the lines. It turns out in this case that if the coordinates of the point of intersection are $(x,y)$ then we can find out $x$ and $y$ by solving the equation  
\begin{align*}
\begin{bmatrix} -7 & 3 \\ 3 & 4 \end{bmatrix}
\begin{bmatrix} x_{1} \\ x_{2} \end{bmatrix} 
= \begin{bmatrix} 6 \\ 12 \end{bmatrix}
\end{align*}

Also the direction of potential neighbours can be determined to be the vectors $\begin{bmatrix} 4 \\ -3 \end{bmatrix}$ and $\begin{bmatrix} -3 \\ -7 \end{bmatrix}$, respectively. If we write $v_{1}$ and $v_{2}$ as columns of a matrix $B$, $\begin{bmatrix} -3 & -7 \\ 4 & -3 \end{bmatrix}$ then we observe that $AB$ is a diagonal matrix with negative entries on the diagonal.
\qed
\end{Exa}


We generalize this to n dimensions. Consider the usual linear optimization problem with the constraint $Ax \leq b$ and the cost function $c^{\intercal}x$. Consider any of its extreme points $x_{o}$. The point $x_{o}$ satisfies the equation 
\begin{equation}
 A^{\prime} x_{o} = b^{\prime}
\end{equation}
where $A^{\prime}$ is the matrix formed by certain $n$ linearly independent rows of the matrix $A$ and $b^{\prime}$ is a vector having corresponding coefficients from $b$. Now, from $x_{o}$ there are $n$ lines or directions coming out which ``connect'' it to its neighbours. We wish to find out the vectors corresponding to these directions. \\
Let the matrix $A^{\prime}$ be written as 
\begin{align*}
\begin{bmatrix}
 A^{\prime}_{1} \\
  A^{\prime}_{2} \\
\vdots \\
A^{\prime}_{n}
\end{bmatrix}
\end{align*}
where $A_{i}^{\prime}$ is the $i^{th}$ row.  Now consider a point $x_{i}$ on any line $l_{i}$ coming out of $x_{o}$. Clearly this line is determined by the intersection of certain $n-1$ rows out of the $n$ linearly independent rows of $A^{\prime}$. Let $x_{i}$ be such that
%\begin{center}
%\begin{equation}
\begin{eqnarray}
 A_{j}^{\prime} x_{j} = b_{j}^{\prime} & \forall j, 1 \leq j \leq n, j \neq i\\
A_{j}^{\prime} x_{j} < b_{j}^{\prime}  & j = i 
\end{eqnarray}
%\end{equation}
%\end{center}
The direction of the vector along the line $l_{i}$ from the point $x_{o}$ towards $x_{i}$ is $x_{i} - x_{o}$. Now we construct a matrix, having as columns all such vectors for $i=1,2,..n$,
\begin{align*}
Z=
\begin{bmatrix}
x_{1} - x_{o}, & x_{2} - x_{o}, & \hdots, & x_{n} - x_{o}
\end{bmatrix}
\end{align*}

Now consider
%\begin{center}
\begin{equation}
A^{\prime} Z = 
\begin{bmatrix} 
  A^{\prime}x_{1} - A^{\prime}x_{o},&
A^{\prime}x_{2} - A^{\prime}x_{o},&
\hdots,&
A^{\prime}x_{n} - A^{\prime}x_{o} \\
 
\end{bmatrix}
\end{equation}
%\end{center}
Given the properties of $x_{i}$, we know that
\begin{eqnarray}
 (A^{\prime}x_{i})^{j} = (A^{\prime}x_{o})^{j} \quad j \neq i   \\
 (A^{\prime}x_{1})^{j} < (A^{\prime}x_{o})^{j} \quad j=i
\end{eqnarray}

So it is clear that the $i^{th}$ row of the matrix $A^{\prime} Z$ has a negative entry at the $i^{th}$ position and zeroes at all other positions which means that matrix $A^{\prime}Z$ is a diagonal matrix with negative entries on the diagonal. Clearly enough we can multiply each column of the matrix $Z$ with a suitable \textbf{negative} constant such that 
\begin{equation}
A^{\prime}Z = I, 
\end{equation}
 where $I$ is an identity matrix. And now since we only care about the sign of a direction vector and not its magnitude(which basically means we can ignore the magnitude of the constants multiplied with each column of $Z$ to get $A^{\prime}Z = I$) so we can say the following about the directions coming out of the point $x_{o}$
\newtheorem{thm1}{Theorem}
\begin{thm1}
 The direction vectors are the columns of the negative of the inverse of matrix $A^{\prime}$.
\end{thm1}


\section{Finding neighbouring points with higher cost}
One of the steps in the simplex algorithm says ``move to the neighbour with larger cost''.
Once we know the direction towards each neighbour, it is very easy to find out which of the  neighbours have higher cost. This is because of the fact that if one of the neighbours $x_{i}$ has higher cost than the current extreme point $x_{o}$, then every point on the line segment joining $x_{i}$ and $x_{o}$ has cost higher than $x_{o}$. Lets prove this.\\ \\	
\begin{proof}
Let $x^{\prime}$ be any arbitrary point on the line segment joining $x_{o}$ and $x_{i}$ such that $x^{\prime} \neq x_{o}$. Now, we know that any point on the line segment joining $x_{o}$ and $x_{i}$ can be written as a convex combination of these two. So,
\begin{equation}
 x^{\prime}= \lambda x_{i} + (1 - \lambda) x_{o}
\end{equation}
, where $\lambda > 0$. So we get,
\begin{equation}
 c^{\intercal}x^{\prime}= \lambda c^{\intercal} x_{i} + (1 - \lambda) c^{\intercal} x_{o}
\end{equation}
or,
\begin{equation}
 c^{\intercal}x^{\prime}= \lambda (c^{\intercal} x_{i} - c^{\intercal} x_{o}) + c^{\intercal} x_{o}
\end{equation}
And since 
$ c^{\intercal} x_{i} > c^{\intercal} x_{o} $
so clearly $ c^{\intercal} x^{\prime} > c^{\intercal} x_{o}$

\end{proof}

So choose any point on the line segment joining $x_{i}$ and $x_{o}$ given by a column of $(-A^{\prime})^{-1}$ and see if it has higher cost than $x_{o}$. If it has, then the neighbour $x_{i}$ will also have higher cost than $x_{o}$ otherwise not. \\ 
After having found out whether a particular direction fetches higher cost or not, now we move on to actually finding that neighbouring point. Given the direction vector $v_{i}$ from $x_{o}$ to $x_{i}$, any point $x^{\prime}$ on the line segment joining these two can be written as 
\begin{equation}
 x^{\prime} = x_{o} + \epsilon v_{i} \quad \epsilon \geq 0 
\end{equation}
where $\epsilon$ is suitably chosen so that the point $x^{\prime}$ doesn't shoot past $x_{i}$. This can be expressed in the constraint
\begin{equation}
 A(x_{o} + \epsilon v_{i}) \leq b
\end{equation}
We know that the point $x^{\prime} = x_{o} + \epsilon v_{i}$ satisfies 
\begin{equation}
 A^{\prime}_{j} x^{\prime} = b^{\prime}_{j} \quad \forall j \neq i, \quad 1 \leq j \leq n
\end{equation}
So we keep increasing the value of $\epsilon$ until $x^{\prime}$ starts satisfying 
\begin{equation}
 A_{k} x^{\prime} = b_{k}
\end{equation}
where $A_{k}$ is linearly independent of all rows of $A^{\prime}$. This $x^{\prime}$ will be the required neighbour.\\
Hence, 
\begin{equation}
\epsilon = min_{t} \frac{b_{t} - A_{t}x_{o}}{A_{t}v_{i}},
\end{equation}
where $t$ ranges over all those rows of $A$ which do not belong to $A^{\prime}$.

The neighbouring point can also be found in another way.
\begin{enumerate}
\item  We take $n-1$ linearly independent rows of $A^{\prime}$ which decide one particular direction.
\item  Then we add to them any row from the matrix $A$ such that this row is linearly independent of all the $n$ rows of the matrix $A^{\prime}$ and then we find out the point determined by this new set of rows. 
\item If this point is feasible, that is it satisfies $Ax \leq b$ then this is the required neighbouring point, else we repeat this process with other rows of the matrix $A$ until we find the neighbouring point.

\end{enumerate}
\end{document}