\input{template} \input{macros} \begin{document} \lecture{2}{Standard LP formulation, Linear algebra : solution to simultaneous linear equations $Ax = b$, Gaussian elimination}{Dinesh Gadge} \section{Formulation} A linear optimization problem can be formulated as \begin{gather} \text{max} \enskip c^{T}x \notag \\ Ax \leq b \end{gather} where, $A$ is an $m \times n$ matrix, $c$ a $n \times 1$ vector, $b$ a $m \times 1$ vector and $x$ a $n \times 1$ vector. We are given as {\bf Input:} $c$, $A$, $b$ and desire as {\bf Output} : $x$. Among all $x$ that satisfies $Ax \leq b$ find one which maximises $ c^{T}x $. The set $Ax \leq b$ is a set of points which have typical properties. Our first goal is to understand them and be able to describe some of these. \section{Understanding the set of all solutions to $Ax = b$} Before considering the set of inequalities, $Ax \leq b$, we consider the set of equalities $Ax = b$. This is short hand notation for the following set of equalities given below. \begin{align} a_{11}x_{1} + a_{12}x_{2} + a_{13}x_{3} + \cdots + a_{1n}x_{n} & = b_{1} \notag \\ a_{21}x_{1} + a_{22}x_{2} + a_{23}x_{3} + \cdots + a_{2n}x_{n} & = b_{2} \notag \\ & \vdots \notag \\ a_{m1}x_{1} + a_{m2}x_{2} + a_{m3}x_{3} + \cdots + a_{mn}x_{n} & = b_{m} \notag \end{align} We can solve such a system of equations using Gaussian Elimination. Here is an example. \begin{Exa}\label{lec2:exa1} \begin{align} 2x + 7y & = 13 \tag{I} \label{lec2:exa1:eq1}\\ x + 3y & = 4 \tag{II} \label{lec2:exa1:eq2} \end{align} Replacing ~\ref{lec2:exa1:eq2} by -$\tfrac{1}{2} \cdot$\ref{lec2:exa1:eq1} + ~\ref{lec2:exa1:eq2} gives \begin{align} -\dfrac{7}{2} y + 3y &= -\dfrac{13}{2} + 4 \notag \\ y &= 5 \label{lec2:exa1:eq3} \end{align} %Substituting eq~\ref{lec2:exa1:eq3} in \label{lec2:exa1:eq2}, we get %\begin{equation} % %\end{\equation} \end{Exa} \section{Gaussian Elimination} \begin{align} a_{11}x_{1} + a_{12}x_{2} + a_{13}x_{3} + \cdots + a_{1n}x_{n} & = b_{1} \notag \\ a_{21}x_{1} + a_{22}x_{2} + a_{23}x_{3} + \cdots + a_{2n}x_{n} & = b_{2} \notag \\ & \vdots \notag \\ a_{m1}x_{1} + a_{m2}x_{2} + a_{m3}x_{3} + \cdots + a_{mn}x_{n} & = b_{m} \notag \end{align} Assume that $a_{11} \neq 0$. If this is not so we exchange the first row with some other row which has a non-zero first co-ordinate. Then for each of the equations except the first, multiply the first equation by a suitable constant and subtract from the respective equations to get rid of $x_{1}$ in the other equations. We then do this with the other equations ignoring the first variable. There are 2 operations used in Gaussian Elimination. \begin{enumerate} \item Exchange two rows. \item Replace $row_{j}$ with $\alpha\cdot row_{i} + row_{j}$, where $\alpha$ is some constant. \end{enumerate} It is possible that using these two operations the co-efficients of some $x_{i}$ [for example $x_{2}$] in all rows except one are zeroed out. \begin{align} a_{11}x_{1} + a_{12}x_{2} + a_{13}x_{3} + \cdots + a_{1n}x_{n} & = b_{1} \notag \\ 0 + 0 + a_{23}x_{3} + \cdots + a_{2n}x_{n} & = b'_{2} \notag \\ 0 + 0 + a_{33}x_{3} + \cdots + a_{3n}x_{n} & = b'_{3} \notag \\ & \vdots \notag \\ 0 + 0 + a_{m3}x_{3} + \cdots + a_{mn}x_{n} & = b'_{m} \notag \end{align} In such a case, assume that, some $a_{ij} \neq 0$ [for example $a_{23} \neq 0$], then we repeat the process. Finally, we get something like, \begin{align} a_{11}x_1 + a_{12}x_2 + \cdots + a_{1n}x_n & = b_{1} \notag \\ 0 + 0 + \cdots + a_{2i_1}x_{i_1} + \cdots + a_{2n}x_n & = b''_{2} \notag \\ & \vdots \notag \\ 0 + 0 + \cdots + 0 + a_{ki_{k-1}}x_{i_{k-1}} + \cdots + a_{kn}x_n & = b''_{k} \notag \\ & \vdots \notag \\ 0 + 0 + \cdots + 0 + 0 + 0 + \cdots + 0 & = b''_{l} \notag\\ & \vdots \notag \\ 0 + 0 + \cdots + 0 + 0 + 0 + \cdots + 0 & = b''_{m} \notag \end{align} {\bf Observations:} \begin{enumerate} \item All zero rows occur after the non-zero rows. \item If, from the top, the first $t$ rows are nonzero, and the first non-zero entry of the $i$th row $i = 1, \dots, t$, is at the $k_i$th column, then $k_i > k_{i-1}$, $i = 2, \dots, t$. That is, the first non-zero entries in the rows appear later and later from top to bottom. \end{enumerate} \subsection{Existence of a solution for $Ax = b$} $Ax = b$ does not have a solution if in a particular row $i$, all coefficients $a_{ij} = 0$, but $b_{i} \neq 0$. This is both necessary and sufficient condition for non-existence of a solution for $Ax = b$ as we shall see later. Once we have the matrix in this form, it is easy to get solutions to the set of equations, if one exists. Except for $x_1, x_{i_1}, \ldots, x_{i_{k-1}}$ set any values to the other variables. Now solve for the variables $x_1, x_{i_1}, \ldots, x_{i_{k-1}}$ in the reverse order. If it has $n$ non-zero rows then it has only one solution. Otherwise $\{x : Ax = b\}$, in general, has many solutions. \subsection{Why is this procedure correct?} The procedure being correct means that the values of the variables obtained by the procedure indeed satisfy the original set of equations. So to prove the correctness of the procedure, we have to prove that the solution set does not change on applying the operations of Gaussian Elimination. Clearly exchanging two rows does not change the solution set. \begin{Thm} Given a set of equations, suppose $eq_j$ is replaced by \begin{equation} \alpha\cdot eq_i + eq_j\notag \end{equation} then the solutions set does not change. \end{Thm} \begin{proof} %Original set : %\begin{align} %&eq_1 \notag\\ %&eq_2 \notag\\ %&\vdots \notag\\ %&eq_i \notag\\ %&\vdots \notag\\ %&eq_j \notag\\ %&\vdots \notag\\ %&eq_n \notag %\end{align} %New set : %\begin{align} %&eq_1 \notag\\ %&eq_2 \notag\\ %&\vdots \notag\\ %&eq_i \notag\\ %&\vdots \notag\\ %&\alpha\cdot eq_i + eq_j \notag\\ %&\vdots \notag\\ %&eq_n \notag %\end{align} Let $eq_i$ be $a_{i1}x_1 + a_{i2}x_2 + \cdots + a_{in}x_n = b_i$ and $eq_j$ be $a_{j1}x_1 + a_{j2}x_2 + \cdots + a_{jn}x_n = b_j$. Consider the old and the new set of equations. Note that they differ only in the $j$th equation. This proof consists of 2 parts. {\bf Part 1:} {\it If the set of solutions satisfies original set of equations then it satisfies the new set.} Let $$ be the solution to the original set. Then $$ specifically satisfies $eq_i$ and $eq_j$. So, \begin{align} a_{i1}x'_1 + a_{i2}x'_2 + \cdots + a_{in}x'_n & = b_i \notag\\ a_{j1}x'_1 + a_{j2}x'_2 + \cdots + a_{jn}x'_n & = b_j \notag \end{align} that is, \begin{align} a_{i1}x'_1 + a_{i2}x'_2 + \cdots + a_{in}x'_n - b_i & = 0 \notag\\ a_{j1}x'_1 + a_{j2}x'_2 + \cdots + a_{jn}x'_n - b_j & = 0 \notag \end{align} So, \begin{align} & \alpha (a_{i1}x'_1 + a_{i2}x'_2 + \cdots + a_{in}x'_n - b_i) + (a_{j1}x'_1 + a_{j2}x'_2 + \cdots + a_{jn}x'_n - b_j) \notag\\ = & ~\alpha\cdot 0 + 0 \notag\\ = & ~0 \notag \end{align} Therefore, \begin{equation} \alpha (a_{i1}x'_1 + a_{i2}x'_2 + \cdots + a_{in}x'_n) + (a_{j1}x'_1 + a_{j2}x'_2 + \cdots + a_{jn}x'_n) = \alpha\cdot b_i + b_j \notag \end{equation} Hence $$ satisfies $\alpha\cdot eq_i + eq_j$. Since rest of the equations in both the sets are identical, $$ satisfies the new set. {\bf Part 2:} {\it If the set of solutions satisfies new set of equations then it satisfies the original set.} Let $$ be the solution to the new set. Then $$ specifically satisfies $\alpha\cdot eq_i + eq_j$ and $eq_i$. So, \begin{align} \alpha (a_{i1}x'_1 + a_{i2}x'_2 + \cdots + a_{in}x'_n) + (a_{j1}x'_1 + a_{j2}x'_2 + \cdots + a_{jn}x'_n) & = \alpha\cdot b_i + b_j \notag\\ a_{i1}x'_1 + a_{i2}x'_2 + \cdots + a_{in}x'_n & = b_i \notag \end{align} that is, \begin{align} \alpha (a_{i1}x'_1 + a_{i2}x'_2 + \cdots + a_{in}x'_n - b_i) + (a_{j1}x'_1 + a_{j2}x'_2 + \cdots + a_{jn}x'_n - b_j) & = 0\notag\\ a_{i1}x'_1 + a_{i2}x'_2 + \cdots + a_{in}x'_n - b_i & = 0\notag \end{align} So, \begin{align} \alpha\cdot 0 + (a_{j1}x'_1 + a_{j2}x'_2 + \cdots + a_{jn}x'_n - b_j) & = 0 \notag\\ \Rightarrow a_{j1}x'_1 + a_{j2}x'_2 + \cdots + a_{jn}x'_n & = b_j \notag \end{align} Hence $$ satisfies $eq_j$. Since rest of the equations in both the sets are identical, $$ satisfies the old set. \end{proof} \section{Understanding $Ax = b$ geometrically} Another way of looking at $Ax = b$ is through geometry. The operation of adding a constant times another equation to an equation rotates one of the hyperplanes in $\mathbb{R}^n$. Fig.~\ref{lec2:fig1} illustrates this as the case of rotation of lines (1-dimensional hyperplane) in a $x$-$y$ plane (i.e. $\mathbb{R}^2$). Here the solid lines are from example~\ref{lec2:exa1} and one of the dotted lines is obtained by rotating the line corresponding to eq.~\ref{lec2:exa1:eq2} to the one corresponding to eq.~\ref{lec2:exa1:eq3}. The other dotted line can be obtained by similarly manipulating eq.~\ref{lec2:exa1:eq1} \ffigureh{lecture2}{4in}{Geometrical way of looking at $Ax = b$}{lec2:fig1} \end{document}