\documentclass{article} \usepackage[margin=1in]{geometry} \usepackage{graphics} \usepackage{graphicx} \usepackage{subfig} \usepackage{amsmath,amssymb} \newcounter{lecnum} \newcommand{\R}{\mathbb{R}} \newcommand{\transpose}{\mathtt{T}} \newcommand{\lecture}[4]{ \newpage \setcounter{lecnum}{#1} \noindent \begin{center} \framebox{ \vbox{\vspace{2mm} \hbox to 16cm { {\bf CS602 Applied Algorithms \hfill 2019-20 Sem II} } \vspace{4mm} \hbox to 16cm { {\Large \hfill Lecture #1: #2 \hfill} } \vspace{2mm} \hbox to 16cm { {\it Scribe: #4 \hfill Lecturer: #3} } \vspace{2mm}} } \end{center} \vspace*{4mm} } \newtheorem{theorem}{Theorem}[lecnum] \newtheorem{lemma}[theorem]{Lemma} \newtheorem{proposition}[theorem]{Proposition} \newtheorem{claim}[theorem]{Claim} \newtheorem{corollary}[theorem]{Corollary} \newtheorem{definition}[theorem]{Definition} \begin{document} %\lecture{**LECTURE-NUMBER**}{**DATE**}{**LECTURER**}{**SCRIBE**} \lecture{1}{January 13}{Rohit Gurjar}{Harmeen Kaur} \section{Contents of the Course} \begin{enumerate} \item Linear Programming Basics: \begin{itemize} \item Duality \item Primal Dual Approach \end{itemize} \item Applications in Combinatorial optimization: \begin{itemize} \item Matching \item Flow \item Cuts \item Matroids \end{itemize} \item Submodular optimization \item Approximation Algorithms: \begin{itemize} \item Vertex cover \item Disjoint paths \item Scheduling \end{itemize} \item Online Algorithms: \begin{itemize} \item Matching \item Load Balanicing \item Semi Definite Programming (SDP) \item Continuous methods \end{itemize} \end{enumerate} \section{Linear Programming} \textbf{Linear Programming} is a technique for the optimization of a linear objective function, subject to linear equality and linear inequality constraints.\\ \\ It can be described as follows: \begin{itemize} \item we are given $n$ variables: $x_1, x_2, x_3 ........x_n \in \R$. \item $m$ linear inequalities in these variables (equalities are OK too) \item And we are supposed to optimise some linear function $$f(x_1, x_2, x_3 ........x_n) = w_1 x_1 + w_2 x_2 + \cdots + w_n x_n$$ in these variables, subject to the given linear constraints. \end{itemize} \paragraph{Example 1:} Let there be a set of variables in 2 dimensional space $$(x_1, x_2) \in \R^2 $$ which are subjected to the following constraints: $$x_1 \ge 0$$ $$x_2 \ge 0 $$ $$x_1 + x_2 \le 1$$ The goal is to maximize a linear objective function $f(x_1,x_2) = 2x_1+x_2.$ One can verify that $(x_1=1, x_2=0)$ is the point that satisfies the above constraints and achieves the maximum function value, $f(1,0)=2$. \paragraph{Example 2 (Transportation Problem):} Let there be $k$ coal mines $M_1,M_2, \dots, M_k$ and their respective capacities of production be $c_1,c_2,c_3, \dots, c_k$. Let there be $\ell$ industries $P_1,P_2,P_3, \dots, P_\ell$ having demands of coal from mines $d_1,d_2,d_3, \dots, d_\ell$ respectively. $p_{i,j}$ : Transportation cost per unit from $M_i$ to $P_j$ (given as part of the problem). \textbf{The Objective }is to minimize overall cost of transportation fulfilling all the demands of industries. Transportation problem can be expressed as a problem of Linear Programming. The linear program will use $k \ell$ variables $\{x_{i,j} \mid 1 \leq i \leq k, \; 1 \leq j \leq \ell\}$, where $x_{i,j}$ : amount of coal transferred from $M_i$ to $P_j$ \begin{itemize} \item for each $1 \le i \le k $ $$ \sum_{j=1}^{l} x_{i,j} \le c_i$$ \item for each $1 \le j \le l $ $$\sum_{i=1}^{k} x_{i,j} = d_j$$ \item The goal is to minimize: $$\sum_{i,j} p_{i,j} x_{i,j}.$$ \end{itemize} The above problem deals with continuous variables as the amount of coal transferred can be an arbitrary real number. Unlike this situation, in most combinatorial optimization problems our domain is actually discrete. Still linear programming proves to be a useful tool in combinatorial optimization. Let us try to express such a discrete problem via a linear program. \paragraph{Example 3 (Largest Independent Set):} Given a graph, the problem asks to find the largest set of vertices that have no common edge between them. Let us try to express this problem as an optimization problem with linear constraints. Let $G = (V,E)$ be the given graph where, $V$ is the vertex set and $|V| = n$ and $E$ is the edge set. Consider a variable $x_i$ for each vertex $v_i$ such that $$x_1, x_2, \dots, x_n \in \{0,1\} .$$ Here, $x_i=1$ is supposed to mean that we select vertex $v_i$ in the set and $x_i=1$ means we do not select it. Now, let us put the ``no common edge between two vertices in the set" condition as a linear inequality. $$\forall e = (v_i,v_j) \in E, \; x_i + x_j \le 1 .$$ The above constraint is simply expressing that for any edge, we are allowed to take at most one of its end points in the set. The objective is to maximise: $$\sum_{i=1}^{n}x_i.$$ One can verify that the maximizing solution will give us the largest independent set. Since the variables here are bound to take values from a discrete set of $\{0,1\}$, this is not a linear program but an integer program.\\ \paragraph{Linear Programming(LP) Relaxation:} The relaxation of a (mixed) integer linear program is the problem that arises by removing the integrality constraint of each variable. For example, in a 0-1 integer program, all constraints are of the form $$x_i \in \{0,1\} $$ The relaxation of the original integer program instead uses a collection of linear constraints: $$0\leq x_{i}\leq 1$$ The resulting relaxation is a linear program, hence the name. This relaxation technique transforms an NP-hard optimization problem (integer programming) into a related problem that is solvable in polynomial time (linear programming).\cite{LP}\\ \\ \textbf{Example: }So in the above example of {largest independent set} if we apply relaxation technique to transform it to a linear programming problem, it will look like: Consider a variable $x_i$ for each vertex $v_i$ such that : \\ $$\forall i, \; 0 \le x_i \le 1 $$ $$\forall e = (v_i,v_j) \in E, x_i + x_j \le 1 $$ \begin{itemize} \item The objective is to maximise: $$\sum_{i=1}^{n}x_i$$ \end{itemize} As the definition states, NP-hard problems after relaxation transforms to a related problem. Since linear programming problems can be solved in polynomial time, that related problem is not exactly equivalent to the original NP-hard problem. For above example of the largest independent set problem, consider the complete graph on three nodes. The following figure shows that the optimal values of the integer program and the corresponding linear program are different. \begin{figure}[!hbt] \centering \subfloat[Integer Programming Problem ( $OPT = 1$) ]{{\includegraphics[width=6cm]{im3.png} }}% \qquad \subfloat[Linear Programming Problem ($OPT = 3/2$) ]{{\includegraphics[width=6cm]{im4.png} }}% \label{fig:example}% \end{figure}\\ Here the largest independent set is of size 1 but, the corresponding linear programming relaxation has maximum value 3/2. \section{Various forms of linear programs} \paragraph{General form of a linear program: } %For n number of variables and m number of constraints: \\ For given $\{a_{i,j} \in \R \mid 1\leq i \leq m+m', \; 1\leq j \leq n\}$, $\{b_i \in \R \mid 1 \leq i \leq m+m'\}$, and $\{c_j \in \R \mid 1 \leq j \leq n\}$, we want to find $x_1, x_2, \dots, x_n \in {\R} $ such that \[ \text{for } 1 \leq i \leq m, \quad \sum_{j=1}^{n} a_{i,j} x_j\ge b_i, \] \[ \text{and for } m+1 \leq i \leq m', \quad \sum_{j=1}^{n} a_{i,j} x_j = b_i, \] and which maximizes $$\sum_{j=1}^{n}c_j x_j.$$ Note that $`\leq'$ kind of inequality can be converted to $`\geq'$ kind of inequality (and vice versa) by multiplying with $-1$. Also, maximization can be converted to minimization (and vice versa) by multiplying with $-1$. \paragraph{Standard Forms:} There are various standard forms to represent linear programs, which are as expressive as the general form. We present two of these and argue that they can be converted to one another. \begin{enumerate} \item minimize/maximize$$ \sum_{j=1}^{n} c_j x_j$$ such that for $1 \le i \le m $ $$\sum_{j=1}^{n} a_{ij} x_j \ge b_i $$ This actually gives intersections of half spaces formed by inequalities. \item minimize/maximize$$ \sum_{j=1}^{n} c_j x_j$$ such that \begin{itemize} \item for $1 \le j \le n $, \hspace{2mm} $$x_j \ge 0$$ \item and for $1 \le i \le m $ $$\sum_{j=1}^{n} a_{ij} x_j = b_i $$ \end{itemize} \end{enumerate} \paragraph{Converting one form to another:} \begin{itemize} \item Form 1 to 2 (shown by examples): Form 1: $$x_1 + x_2 \ge 3 $$ Equivalent Form 2 : $$x_1 + x_2 = y_1 + 3, \; y_1 \ge 0 $$ (converted inequality to equivalent equality by introducing a new variable). Note that in form 2, all variables have non-negativity constraints, while this need not be true form 1. We replace any such variable $x_i$ with $z_i - u_i$, where $z_i, u_i$ are two new variables and put non-negativity constraints $$ z_i \ge 0, \; u_i \geq 0$$ \item Form 2 to 1: Taking any general form 2 set of liner program, keeping the min/max objective the same and all $x_j \ge 0$ conditions the same, replace all equality constraint by two inequalities as: $$\sum_{j=1}^{n} a_{ij} x_j = b_i $$ can be replaced by: $$\sum_{j=1}^{n} a_{ij} x_j \ge b_i $$, $$-(\sum_{j=1}^{n} a_{ij} x_j) \ge -b_i $$ \end{itemize} \section{Geometric Perspective} \begin{definition}[Polyhedron] A polyhedron is the feasible region of a given set of linear inequalities. That is, for given $A \in \R^{m \times n}$ and $b \in \R^m$, the following set is called a polyhedron $$ P = \{ x \in \R^{n} : Ax \le b\} $$ \end{definition} \paragraph{Example:} In space $\R^{2}$, say the constrains are as follows: $$2x_1 + x_2 \le 2$$ $$x_1 + x_2 \ge 0 $$ $$x_1 \ge 0 $$ The yellow region in Figure~\ref{fig:polyhedron} shows the corresponding polyhedron. \begin{figure}[!hbt] \centering \includegraphics[width=50mm,scale=0.8]{im1.png} \caption{A Polyhedron} \label{fig:polyhedron} \end{figure} We will often deal with polyhedron which are bounded. \begin{definition}[Polytope] A set $S \subseteq \R^n$ is called bounded if $\exists r > 0 $ such that whole set $S$ is inside the the hypersphere of radius $r$ centered at origin. In other words, every point $p$ in the set $S$ satisfies $\lVert p \rVert_2 \leq r$. A bounded polyhedron is called a polytope. \end{definition} \paragraph{Example:} A cube is a polytope. Figure~\ref{fig:cube} shows the cube defined by the following constraints. $$ 0 \le x_1 \le 1 $$ $$ 0 \le x_2 \le 1 $$ $$ 0 \le x_3 \le 1 $$ \begin{figure}[h] \centering \includegraphics{im2.jpg} \caption{Cube \cite{cube}} \label{fig:cube} \end{figure} \paragraph{Facets:} In an $n$ dimensional polyhedron, its facets are $n-1$ dimensional hyperplane (segments) bounding the polyhedron. For example, in case of a 3 dimensional cube, its 2-D faces are facets. %\begin{definition}\textbf{Vertex: }Corners of the polyhedron are called vertices. %\end{definition} \begin{definition}[Face] Any dimensional boundary of a polyhedron is called a face. e.g. vertex, edge , or the polyhedron itself. We can get a face by replacing any subset of inequalities with equivalent equality. \end{definition} \paragraph{Example: } \begin{align} 2x_1 + x_2 \le 2 \label{eq:1}\\ x_1 + x_2 \ge 0 \label{eq:2}\\ x_1 \ge 0 \label{eq:3} \end{align} Now replace inequality \eqref{eq:3} with its corresponding equality. The new set of constraints will look like: \begin{align} 2x_1 + x_2 \le 2 \label{eq:4}\\ x_1 + x_2 \ge 0 \label{eq:5}\\ x_1 = 0 \label{eq:6} \end{align} which forms line/edge of the polyhedron $x_1 = 0$, which is a face. \begin{thebibliography}{} \bibitem{LP} source :https://en.wikipedia.org/wiki/Linear\_programming\_relaxation \bibitem{cube} source : www.math.brown.edu/~banchoff/Beyond3d/chapter8/section01.html \end{thebibliography} \end{document}