
\documentclass{article}
\usepackage[margin=1in]{geometry}
\usepackage{graphics}                 
\usepackage{graphicx}
\usepackage{caption}
\usepackage{amsmath,amssymb}
\newcounter{lecnum}

\newcommand{\R}{\mathbb{R}}
\newcommand{\transpose}{\mathtt{T}}   
\newcommand{\lecture}[4]{
   \newpage
   \setcounter{lecnum}{#1}
   \noindent

   \begin{center}
   \framebox{
      \vbox{\vspace{2mm}
    \hbox to 16cm { {\bf CS602 Applied Algorithms
                        \hfill 2019-20 Sem II} }
       \vspace{4mm}
       \hbox to 16cm { {\Large \hfill Lecture #1 -  I: #2  \hfill} }
       \vspace{2mm}
       \hbox to 16cm { {\it Scribe: #4  \hfill  Lecturer: #3} }
      \vspace{2mm}}
   }
   \end{center}
   \vspace*{4mm}
}

\newtheorem{theorem}{Theorem}[lecnum]
\newtheorem{lemma}[theorem]{Lemma}
\newtheorem{proposition}[theorem]{Proposition}
\newtheorem{claim}[theorem]{Claim}
\newtheorem{corollary}[theorem]{Corollary}
\newtheorem{definition}[theorem]{Definition}


\begin{document}

%\lecture{**LECTURE-NUMBER**}{**DATE**}{**LECTURER**}{**SCRIBE**}
\lecture{18}{March 24}{Rohit Gurjar}{Bismillah Hossainy}


\section{Methods for Solving LP Problems}

Till now, we have seen some primal-dual based methods to solve combinatorial optimization problems. 
Some of  these problems were polynomial-time solvable like
 %and we developed some problem-specific polynomial-time algorithms to solve them. We did this for
  bipartite matching and some were  NP-hard problems which we solved approximately, e.g., Steiner tree.
Until now, we have not used any generic LP solvers, instead, we just used some ideas from LP duality and problem specific techniques.
There are other paradigms to find either an exact or an approximate solution for combinatorial problems that 
actually use LP solvers (to be discussed in coming lectures).  
There are various methods to solve LP programs. In this lecture, we briefly discuss three of these methods in chronological order. 

	\begin{itemize}
		\item Simplex Algorithm (1950s)
		\item Ellipsoid Method (1970s)
		\item Interior Point Method (1980s)
	\end{itemize}

\section{Simplex Algorithm}

The simplex algorithm, the most famous and widely used method to solve linear programs, was developed in the 1950s by Dantzig. 
The Simplex method is robust; it solves any linear program, also detects if there is no feasible solution or unbounded optimal. 
%with one or more optimal solutions \cite{robust}. 
Although the method is known to be really good at practice, it is not known to be polynomial time.  

\noindent The Simplex algorithm works as follows:
	\begin{itemize}
		\item The method starts with a feasible vertex. 
		\item Then, it moves from this vertex to a neighboring vertex (connected by an edge), at each step improving the value of the objective function.
		\item The algorithm terminates after a finite number of such transitions.
	\end{itemize}
Figure-1 depicts the steps of the simplex algorithm. As seen on Figure-1.a, the process starts with the initial vertex of $v_1$, and then, it moves to vertices $v_2$, $v_3$, and so on until the algorithm finds the optimal solution. 

	\begin{center}
		\begin{tabular}{cc}
			\includegraphics[width=6cm]{simplex.png} &
			\includegraphics[width=5cm]{simplex2.png}\\
			a) Simplex Alg. in 2D & b) Simplex Alg. in 3D\\
		\end{tabular}
		\captionof{figure}{The Simplex Algorithm moves vertex by vertex while improving the objective value.\cite{simplex}}
		\label{fig:example}
	\end{center}

\noindent Let $Ax \le b$ or $a_i^\top x \leq b$ for $1 \leq i \leq m$ be the given LP, where  $ x = (x_1, x_2, ... x_n)$. 
%describes the polytope, which gives an LP problem. 
We know that it describes a polyhedron and any vertex of the polyhedron can be described by $n$ tight constraints.
There can be more than $n$ tight constraints for a vertex\footnote{for example, consider a pyramid in 3 dimensions; its apex has four planes meeting at it}, but for now, let us assume that every vertex has exactly $n$ tight constraints.  
Let us first see how do we go to a neighboring vertex (two vertices are neighbors if they are connected by an edge).
%There are $n$ independent tight constraints that define the vertices. So for every vertex $v_i$, there is at least one tight constraint. 
Suppose the starting vertex described by:
	\begin{eqnarray}
		v_1 & \equiv & \qquad a_i^\top x = b_i \text{ for }    1 \le i \le n \nonumber 
	\end{eqnarray} 

\noindent Now we want to move along with an edge. To do this  we keep $n-1$ constraints  tight and  relax one of the constraints.
 Note that the solution of $n-1$ tight constraints is a line. Without loss of generality, suppose we relax the $n^{th}$ constraint:  
	\begin{eqnarray}
		a_n^\top x= b_n \qquad \longleftarrow relax   \nonumber
	\end{eqnarray}  

\noindent From the starting vertex, we should move in the direction which keeps the feasibility condition. 
That is, when we relax $a_n x = b_n$, we should maintain $a_n x \leq b_n$.
We want to walk along the line given by 
\begin{equation}
\label{eq:line}
a_i^\top x= b_i, \quad 1 \le i \le n-1.
\end{equation}
 till the last point that is feasible. 
Take a generic point on this line satisfying 
%So we should change $x$ such that 
 $a_n^\top x=b_n-\lambda$ for  $\lambda \ge 0$. 
  %, that is,  for each $a_i^\top x \leq b_i$ for $n+1 \le i \le m$. 
 Such a point $x$ can be obtained by solving these $n$ equations (note that they give a unique solution), 
 which will be a function of $\lambda$.
%Here $x$ can be written as a function of $\lambda$, so 
Now, take the maximum possible $\lambda$ such that 
\[		a_i^\top x \le b_i  \text{ for each }  n+1 \leq i \leq m.
\]
Taking the maximum possible $\lambda$,   
one of these constraints ($(n+1)$-th to $m$-th) will become tight.
 Now, $v_2$ can be defined by this new set of $n$ tight constraints.
 
For any vertex, there are $n$ choices for the constraint to be relaxed, that means there are $n$ possible edges  incident a vertex. 
One can choose and walk along any one edge that increases the optimal value. 

\paragraph{Claim:} If a vertex $v$ is not optimal, then there exists an edge on $v$ that walking along it improves the objective value.

 The above claim implies that the simplex algorithm takes finitely many steps to find the optimal vertex. 
Prove this claim as an exercise. 

\noindent Tip: The argument is simply that from the starting vertex, we choose the next vertex, which strictly improves the objective value. How do we know if a vertex is optimal? A vertex is optimal when all edges on that vertex give a worse optimal value. As there are finitely many vertices to be chosen, so the simplex algorithm finishes in finitely many steps.

\paragraph{}\textbf{Running Time:} While using the simplex algorithm to solve an LP problem, how many steps are required? Can we bound the total number of steps? One trivial number of bounds is the number of vertices, which can be exponential in $n$. 
Another bound we can get is from the objective function values. If we know that the objective function takes values between some lower limit $\ell$ and an upper limit  $L$
 over the polytope. And at every step, the increment in the objective value is at least $\delta$. Then we can get a bound of  $(L- \ell)/\delta$ on the number of steps. 
In a  more rigorous analysis, the number of steps would depend on the edge selection mechanism.
It is possible that there is an edge selection rule that guarantees only polynomially many steps for any linear program. 
 However no such rule is known till now. 

\paragraph{Open:}  finding a rule to guarantee a polynomial bound on the number of steps in simplex for every possible LP.

There are two questions that we have left unanswered till now. 
One is the assumption we made in the beginning that every vertex has exactly $n$ tight constraints.
And the other is how to find the starting point for the simplex algorithm, that is, how to get a vertex of the polyhedron. 
Let us discuss them in order. 


\paragraph{Degenerate Case of Simplex Algorithm:} There is no upper bound on the number of tight constraints a 
vertex of an $n$-dimensional polytope can have. That means there is no upper bound on the number of edges a vertex can have. 
%on a point in an $n$-dimensional polytope. 
In general, the number of edges can be more than $n$. 
%So, there can be exponentially many edges on a vertex. Then how to go over all these possible edges, and which one to choose? However, remember that only a set of constraints are tight on a vertex.
 Suppose that there are $k$ tight constraints on a vertex, then there are $k \choose {n-1}$ choices of the constraints to be kept tight.
Many of these choices can give an edge and  thus, there can be exponentially many edges on a vertex. 
  %will be up to $ n \choose k-1$ possible edges on a given vertex. 
  From these edges, we have to choose one edge which gives the best improvement value. 
  How to go over all these possible edges, and which one to choose?
  There exist several ways of resolving this problem; one is \emph{random perturbation}.


\paragraph{Random perturbation.} Take the original set of constraints and add some small random number to  $b_i$ for every $1\leq i \leq m$.
 So for the $i^{th}$ constraint we will have  
	 \begin{eqnarray}
		 a_i^\top x \le b_i+\epsilon_i  \nonumber
	 \end{eqnarray}  
   What this does is that it puts the bounding hyperplanes in a general position.
   When a set of hyperplanes in $n$-dimensional space are in a general position, 
it is not possible that there more than $n$ hyperplanes that intersect on a  point.  
% In this case, all the hyperplane will be in the general position which 
 This means that every vertex has $n$ tight constraints and thus, only $n$ edges will be incident on a vertex.
  For example, in a $3D$ polytope, if the bounding hyperplanes are in a general position,
  it is not possible to have four planes intersecting on a  point. 
  This process is illustrated in Figure-2.  In Figure-2.a, four planes in 3-D are intersecting on a vertex.
  When we  add some random $\epsilon_i $ to the $i$-th plane, this single vertex will be split into two vertices, giving us Figure-2.b. 
  As seen in Figure-2.b, each vertex has only three planes that intersect on that vertex. 


\begin{center}
	\begin{tabular}{cc}
		\includegraphics[width=6cm]{four.png} &
		\includegraphics[width=6cm]{three.png}\\
		a) Before adding  $\epsilon$  & b) After adding  $\epsilon$ on every plane\\
	\end{tabular}
	\captionof{figure}{Random Perturbation}
	\label{fig:example2}
\end{center}

 Because these added numbers are so small, the optimal value does not change very much. \textbf{Prove it as an exercise.} 
 Therefore, random perturbation ensures that each vertex will have exactly $n$ tight constraints, which imply that each vertex has exactly $n$ edges. Now each of these edges can be checked by running the simplex algorithm to find the best promising edge. 

  
\paragraph{Finding a Feasible Solution:}    
Now we move on to the question of finding a starting vertex. 
Remember that finding a feasible solution is as hard as optimizing over an LP,
so it is not immediately clear how to find one. 
The idea is to use the simplex algorithm itself to find a initial feasible solution. 
Consider a given linear program, say LP(1), in the form  
  \begin{eqnarray*}
 	Ax = b \\
 	x \ge 0.
  \end{eqnarray*}
  We can assume that $b\geq 0$ (if $b_i <0$, we can multiply $-1$ in that constraint).
  Let us convert it to a new system by adding  extra variables $z = \{ z_i : 1 \leq i \leq m\}$, one for each constraint as follows.
   \begin{eqnarray*}
	   A x + z = b   \\
	   x \ge 0 \\
	   z \ge 0 
   \end{eqnarray*}  
   First of all, it is easy to find an initial feasible solution for this system:  take $x=0$ and $z=b$.
  %In this new system, we can find a feasible solution easily by assigning $z_i = b_i$ and $x_i = 0$, which satisfies the feasibility condition.
   Now the feasible solution for the original system can be found using the following linear program LP(2)
 \begin{eqnarray*}
		\min \quad \sum_{i=1}^m z_i  \quad s.t.  \\
		A x + z = b   \\
		x \ge 0 \\
		z \ge 0 
	\end{eqnarray*}  
 \paragraph{Claim:}   If $x$ is feasible for LP(1), then ($x, 0$) is a minimizing point for LP(2).
If there is no feasible point for LP(1), then the minimum value for the LP(2) is strictly more than zero.

\noindent \textbf{Prove this claim as an exercise.} To summarize: 
	\begin{itemize}
		\item Finding the initial feasible solution for LP(2) is easy: $z=b,x=0$.
		\item Use the simplex algorithm to find the optimal value for LP(2).
		\item Any optimal solution for LP(2) will give a feasible solution to LP(1). 
	\end{itemize}

\begin{thebibliography}{}
	
	\bibitem{robust} http://web.mit.edu/15.053/www/AMP-Chapter-02.pdf
	\bibitem{simplex} https://en.wikipedia.org/wiki/Simplex\_algorithm\#Overview
	
\end{thebibliography}
\end{document}