
\documentclass{article}
\usepackage[margin=1in]{geometry}
\usepackage{amsmath,amsthm}
\usepackage{relsize}
\newcounter{lecnum}
\usepackage{graphicx}
\graphicspath{./}
\newcommand{\lecture}[4]{
   \newpage
   \setcounter{lecnum}{#1}
   \noindent

   \begin{center}
   \framebox{
      \vbox{\vspace{2mm}
    \hbox to 16cm { {\bf CS602 Applied Algorithms
                        \hfill 2019-20 Sem II} }
       \vspace{4mm}
       \hbox to 16cm { {\Large \hfill Lecture #1: #2  \hfill} }
       \vspace{2mm}
       \hbox to 16cm { {\it Scribe: #4  \hfill  Lecturer: #3} }
      \vspace{2mm}}
   }
   \end{center}
   \vspace*{4mm}
}

\newtheorem{theorem}{Theorem}[lecnum]
\newtheorem{lemma}[theorem]{Lemma}
\newtheorem{proposition}[theorem]{Proposition}
\newtheorem{claim}[theorem]{Claim}
\newtheorem{corollary}[theorem]{Corollary}
\newtheorem{definition}[theorem]{Definition}




\begin{document}

%\lecture{**LECTURE-NUMBER**}{**DATE**}{**LECTURER**}{**SCRIBE**}
\lecture{11}{February 11}{Rohit Gurjar}{Vaibhav Kumar Brahmanya}


\section{LP Duality}
We have seen LP duality applies to Maximum Matching for Bipartite graph.
Also, for Bipartite Graphs we have:
\[|Maximum\  Matching|=|Minimum\  Vertex\  Cover|\]
Many combinatorial problems have this max-min structure. For instance, Maximum Flow has Min-Cut as its LP Dual.\\\\
\textbf{Homework:} Prove the Maximum Flow and Min-Cut  LP Duality.
\\\\
A simpler version of the Maximum Flow Min-Cut LP duality is the Menger's Theorem.
\begin{theorem}[Menger's Theorem]
	Maximum number of edge disjoint paths from s $\rightarrow$ t = min(s,t)-cut
\end{theorem}
A similar theorem is Dilworth's Theorem.\\
We can do all of the above because the Linear Program captures the problem at hand exactly. In general $\textbf{LP} \neq \textbf{ILP}$.
\section{Primal Dual Schema}
\[LP\rightarrow \text{Simpler } LP\]
\textbf{Complementary Slackness Conditions:}
When we have a LP and it's Dual LP, Complementary slackness conditions give us a way of knowing whether the feasible solutions are optimal or not. These conditions give us a way to characterize both primal and dual LP. \textit{This is a special case of \textbf{Karush-Kuhn-Tucker(KKT) conditions} for convex programs}.
Consider the following primal LP and it's dual LP:
\begin{center}
	\begin{tabular}{c|c}
		\textbf{Primal LP} 		& 		\textbf{Dual LP}\\
		\hline
		&\\
		$\min \; \mathlarger{\sum}_{i=1}^{n} w_{i}x_{i}$ 	& $ \max \; \mathlarger{\sum}_{j=1}^{m} b_j y_j $\\
		for $  1\leq i \leq n ,  \; x_i \geq 0$ 	& for  $1\leq j \leq m$, $y_j \geq 0$\\
		for $ 1 \leq j \leq m , \; \mathlarger{\sum}_{i=1}^n a_{ji} x_i \geq b_j  $ 		& for $ 1 \leq i \leq n , \mathlarger{\sum}_{j=1}^{m} a_{ji} y_j \leq w_i$\\
		\hline
	\end{tabular}
\end{center}
\textbf{Weak Duality}:  given $\alpha$ is feasible for LP and $\beta$ is feasible for $LP^*$, we have
%\[ \mathlarger{\sum}_{i=1}^{n} a_{ij} \alpha_i \leq b_j\]
\[\mathlarger{\sum}_{i=1}^{n}\alpha_i w_i \geq \mathlarger{\sum}_{i=1}^{n}\alpha_i \mathlarger{\sum}_{j=1}^{m}\beta_j a_{ji} = \mathlarger{\sum}_{j=1}^{m}\beta_j \mathlarger{\sum}_{i=1}^{n} a_{ji}\alpha_i \geq \mathlarger{\sum}_{j=1}^{m} b_j \beta_j \] 
%
From \textbf{Strong Duality} we know that if $\alpha, \beta$ are optimal for 
$LP$ and $LP^*$ respectively then \textbf{equality should hold} for both the above inequalities. Then, 
\[\Rightarrow \mathlarger{\sum}_{i=1}^{n} \alpha_i w_i = \mathlarger{\sum}_{i=1}^{n} \alpha_i \mathlarger{\sum}_{j=1}^{m} a_{ji} \beta_j \]
\[\Rightarrow \mathlarger{\sum}_{i=1}^{n} \alpha_i \left( w_i - \mathlarger{\sum}_{j=1}^{m} a_{ji} \beta_j \right) = 0 \] 
%
Since the summands are all non-negative, we can say % $\alpha$ and $\beta$ are optimal, therefore,
\[\Rightarrow \forall 1 \leq i \leq n,\  \alpha_i \left( w_i - \mathlarger{\sum}_{j=1}^{m} a_{ji} \beta_j \right) = 0 \ , i.e.\]
\[\Rightarrow \forall i\ \textbf{Either } \alpha_i = 0 \textbf{ or } \mathlarger{\sum}_{j=1}^{m} a_{ji} \beta_j = w_i  \]
That is, either the Primal variable is 0 or the Dual constraint is tight or both.
Also, from the above observation we can conclude the following points, that are known as \textbf{Complementary Slackness Conditions}.
\begin{enumerate}
	\item $\forall \ i, \ \alpha_i\  \neq \ 0\  \implies $ $i^{th}$ Dual constraint is tight.
	\item $\forall \ i, \ i^{th} $ dual constraint is not tight $ \implies \ \alpha_i= 0$ 
	\item $\forall \ j, \ j^{th} $ primal constraint is not tight $\implies \ \beta_j= 0$
	\item $\forall \ j, \ \beta_j\  \neq \ 0\  \implies $ Primal constraint is tight.
\end{enumerate}
Say we had a primal LP with two variables and 5 constraints then the dual LP will have 5 variables and 2 constraints. So, if we are constructing an optimal solution for the primal LP as shown in Figure 1, the optimal solution is constructed using only two tight constraints for the primal LP and the other three constraints are non tight. Thus for the optimal solution of the dual LP three of the dual variables will be zero and two of the dual variables will be non-zero as shown in figure below
\begin{figure}[h]
	\centering
	\includegraphics[width=9cm,height=7.5cm]{image1.png}
	\caption{Relation between the primal tight constraints and the dual variables for a LP with 2 variables and 5 constraints.}
\end{figure}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%left for diagram%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{theorem}
	If $\alpha , \ \beta$ are optimal for $LP$ and $LP^*$, then they satisfy complementary slackness.
\end{theorem}
\begin{theorem}
\label{thm:compSlackOpt}
	If $\alpha , \ \beta$ are feasible and they satisfy complementary slackness, then both are optimal.
\end{theorem}
Theorem~\ref{thm:compSlackOpt} is a useful technique as one can construct a feasible solution for dual LP and show that the primal and dual feasible solutions satisfy the complementary slackness conditions and thus prove primal feasible solution is optimal.\\

%%%%%%%%%%%%%%%%%%%%%%%
\section{Approximate Complementary Slackness Conditions}
Complementary slackness is also used widely in design of approximation algorithms. 
%
One can show that if a pair of a primal and a dual feasible solution satisfy complementary slackness approximately
then they will be approximately optimal.
%These are the conditions that the primal and dual satisfy for Approximate Algorithms.

\textbf{Approximate complementary slackness:} for some $0 \leq \delta \leq 1$,
$$ \forall i, \;  \ \alpha_i \neq 0  \implies \delta w_i \leq \sum_{j=1}^{m}a_{ji} \beta_j \leq w_i ,$$
$$ \text{ and } \forall j, \; \beta_j \neq 0 \implies \frac{b_j}{\delta} \geq \sum_{i=1}^{n} a_{ji} \alpha_i \geq b_j $$
%
Let $f(x) := \sum_{i=1}^n w_i x_i$ and $g(y) := \sum_{j=1}^m b_j y_j$. 
%
Let $\alpha^*$ and $\beta^*$ be primal optimal and dual optimal solutions respectively.
\begin{theorem}
	If $\alpha$ and $\beta$ satisfy the the approximate complementary slackness conditions above for some $0 \leq \delta \leq 1$, 
	then they are  approximately optimal, i.e.,
	$$f(\alpha) \leq (1/\delta^2) f(\alpha^*) \text{ and } g(\beta) \geq \delta^2 g(\beta^*).$$
\end{theorem}
\begin{proof}
We have,
\[ \delta \mathlarger{\sum}_{i=1}^{n}\alpha_iw_i \leq \mathlarger{\sum}_{i=1}^{n} \alpha_i \mathlarger{\sum}_{j=1}^{m} \beta_j a_{ji} = \mathlarger{\sum}_{j=1}^{m} \beta_j \mathlarger{\sum}_{i=1}^{n} a_{ji} \alpha_i \leq \mathlarger{\sum}_{j=1}^{m} \frac{\beta_j b_j}{\delta} \]
From duality we had, $f(\alpha) \geq g(\beta) $. Further from the approximate complementary slackness, we have
\[\delta^2 f(\alpha) \leq g(\beta).\]
%Let $w^TX=f(x)$ for the primal LP and $b^TY=g(y)$ for the dual LP, then 
The relation among 
these solutions and the optimal solution can be seen on a Real line as follows:
\begin{figure}[h]
	\centering
	\includegraphics[height=2cm,width=10cm]{image2.png}
	\caption{Relation between the optimal and the primal and dual LP.}
\end{figure}\\
%Thus, $ \delta^2 * f(\alpha) \leq g(\beta) $
From the above inequality, we get
\[ \delta^2 f(\alpha) \leq g(\beta) \leq g(\beta^*) = f(\alpha^*) \] 
\[
\implies	f(\alpha)\leq \frac{f(\alpha^*)}{\delta^2}
\]
Similarly,
$$g(\beta) \geq \delta^2 g(\beta^*).$$
\end{proof}
Thus if we choose a good $\delta$ we get a better solution from the approximation algorithm
 and if we choose a bad $\delta$ we get a bad solution from our approximation algorithm.


\section{Primal Dual Schema}
Primal-dual scheme is a generic method that reduces the given LP to a simpler LP. 
%
This is widely used in designing exact and approximation algorithms.
%
%For building a solution for the LP we try to convert it to a simpler LP i.e.
\[\textbf{LP} \rightarrow \textbf{simple LP} \]
\begin{center}
	\begin{tabular}{c|c}
		\textbf{Primal LP} 		& 		\textbf{Dual LP}\\
		\hline\\
		Min $w^T x$ & Max $ b^T y$\\
		$Ax=b$ 	& $y^TA \leq w^T$\\
		$x \geq 0$ 	& \\
	\end{tabular}
\end{center}
Let the primal and dual LP be as above, then:\\ 
\textbf{Steps for building a solution for a given problem:}
\begin{enumerate}
	\item Start with a feasible dual solution $\beta$.
	\item Try to construct a primal feasible solution that satisfies complemetary slackness with $\beta$.
	\item If we succeed then we are done, otherwise, it will give a way to improve the dual solution.
\end{enumerate}
This can be also understood as follows:\\
Firstly, we start with a $\beta$, then we have $ \beta^T A\leq w^T $. 
From complementary slackness we can say either the primal variable is zero or the dual constraint is tight. 
In other words, only those primal variables can be nonzero for which the corresponding dual constraints are tight. 
%
%Also, if the dual constraint is non tight then primal variable is zero.\\

Let $A^\prime$ be a column subset of $A$, which give \textbf{tight constraints} for $\beta$ such that, $ \beta^TA^\prime=w^{\prime T} $.
%
Let $x^\prime$ be the corresponding subset of $x$. 
%
Complementary slackness dictates that only the $x'$ part of $x$ can be nonzero.
 So, we can simply drop the other coordinates of $x$ and corresponding columns from $A$.
%
Thus, we need to find
$$x' \geq 0 \text{ such that } A' x' = b.$$
This is a simpler LP, where we are looking for just a feasible solution. 
%
%Then in order to convert the given LP to a "simpler" LP we look for $X\prime \leq 0 $ such that $A\prime X\prime = b$.
% This can also be understood as forcing some constraints to be zero by dropping these variables.
%Now for this simpler LP we try to find a feasible solution. 
If feasible solution exists then we are done because we found a primal-dual feasible pair that satisfies complementary slackness and thus,
must be optimal. 
%
If we fail to find such a feasible solution, then  we apply Farkas' Lemma to get 
\[ \exists u \text{ such that }u^T A^\prime \leq 0\text{ and }u^T b > 0 \]
Now, move in the direction of $u$. That is, take a new dual solution $ \beta_1 = \beta + \epsilon u $ for some $\epsilon \geq 0$. 
We can say
\[ \beta_1^T  A^{\prime} = \beta^T A^\prime + u^T A^\prime \leq w^{\prime T} \text{, since } u^T A^\prime \leq 0 .\]
%
We need to be careful here as, $ \beta_1^T  A^{\prime} \leq w^{\prime T}$ represents only a subset of all constraints.
%
What this is saying is that those constraints which were tight for $\beta$ are still valid for $\beta_1$.
%
But, what about other constraints? 
%
Since the other constraints were non-tight for $\beta$, there was some room available to move while maintaining their validity.
%
Thus, we can always choose some nonzero $\epsilon$, so that the new point $\beta_1$ is feasible for dual LP.
%
 Basically, we need to choose largest $ \epsilon $ such that $ \beta_1^T A \leq w^T $ remains valid. 
 %
 Let us see what happens to the dual value.
\[ g(\beta_1) = \beta_1^{T} b = \beta^T b + u^T b > \beta^T b. \]
That is, the new dual value is strictly better. Thus, we got a better dual solution as promised. 

We keep repeating this procedure till we find  optimal solutions. 
In every round, we keep increasing the dual value and thus, we finish in a finite number of rounds. 


 
\end{document}