\input{template}
\input{macros}

\usepackage{color, graphicx}
\usepackage{amssymb, amsmath}
\usepackage{epsfig}


\begin{document}
\lecture{7}{Linear Algebra}{Shaunak Godbole}


%% -----------------------------------------------------------------------------
%%                          Beginning of Section 1.
%% -----------------------------------------------------------------------------
\section{Convex Sets}
Let us start by defining a \emph{convex combination}.

\begin{Def}
Given a set of vectors $v_i$, any vector of the form
\begin{equation}
v = \sum_{i=1}^n\lambda_iv_i, \quad 0 \leq \lambda_i \leq 1,\quad \sum_{i=1}^n\lambda_i = 1
\notag
\end{equation}
is called a convex combination of the $v_i$'s.
\end{Def}

Now, using the above definition, a \emph{Convex Set} can be defined as: 
\begin{Def}
A set $S$ of vectors(points) is called convex if all convex combinations of vectors in $S$
are in $S$.
\end{Def}

Another way of looking at the definition is:
\begin{Def}
If for any two points in $S$, the line segment joining them is in $S$, then the set $S$ is
said to be convex.
\end{Def}

\begin{figure}[htp]
\centering
\includegraphics[scale=0.4]{cP}
\hspace{1.0in}
\includegraphics[scale=0.4]{nCP}
\caption{Example of a convex and a non-convex polygon}
\label{figure:convexPolygon}
\end{figure}


We can see in the figure that any line segment joining 2 points in the convex
region will lie in the region itself. Thus the region is convex.  This is not
true with the other non-convex figure.  We can see in the figure a line segment
which joins 2 points in the polygon is not completely situated in the polygon
and hence the polygon is not convex.

\newpage

\begin{Thm}
If $S_1$ and $S_2$ are two convex sets, then $S_1 \cap  S_2$ is a convex set.
\end{Thm}
\begin{proof}
Let $x_1, x_2 \in S_1 \cap S_2$.
Now since $x_1$ and $x_2$ belong to $S_1$ (which is convex), any convex combination of them lies in $S_1$.
Similarly we can say that this convex combination of $x_1$ and $x_2$ lies in
$S_2$. Thus the convex combination lies in $S_1 \cap  S_2$.
Thus $S_1 \cap  S_2$ is convex.
\end{proof}


\begin{Thm}
$\{x: Ax \leq b\}$ is a convex set.
\end{Thm}
\begin{proof}
Let $x_1$ and $x_2$ be in the set.
Then $$Ax_1 \leq b  \textrm{ and,}$$ $$Ax_2 \leq b$$
Consider $\lambda_1$ and $\lambda_2$ ($>0$) such that $\lambda_1 + \lambda_2 = 1$
\newline
Then, $$\lambda_1(Ax_1) + \lambda_2(Ax_2) \leq b(\lambda_1 + \lambda_2)$$
$$\Rightarrow \lambda_1(Ax_1) + \lambda_2(Ax_2) \leq b$$
$$\Rightarrow A(\lambda_1x_1 + \lambda_1x_2) \leq b$$

Thus, $\{x: Ax \leq b\}$ is convex.
\end{proof}

An alternate proof could be as follows. Let us look at $A_1x \leq b_1$. 
All the points satisfying this ineqality lie on
one side of the hyper-plane $A_1 x = b$. Thus the set formed by these points is convex (it is easy to check).
Similarly the sets of solutions to the inequalities $A_2x \leq b_2$, $A_3x \leq b_3$ \dots
($A = [A_1, A_2, \dots]$) are also convex.
\newline
\newline
Thus, $A$ can be seen as an intersection of Convex regions. And from Theorm 1
we can argue that since set $A$ is an intersection of convex regions, the set $A$ is
convex.

%% -----------------------------------------------------------------------------
%%                          Beginning of Section 2.
%% -----------------------------------------------------------------------------
\section{Maximize $c^Tx$}
How do we maximize $c^Tx$ over the set of all $x$ satisfying $Ax \leq b$?
\newline
\newline
First, consider maximizing over the region $x^Tx \leq 1$. This is a set such that all points are at a distance less than or equal to 1 from the origin. We can see that this is a convex set.
\newline
\newline
We know that $c^Tx$ increases in the direction of $c$. Thus we start moving in the
direction of $c$. The last point where $c^Tx$ touches the sphere is the point of
maxima. It can be easily seen that at this point, $c^Tx$ is a tangent to the
sphere $x^Tx$.
\newline
\newline
Now consider any convex polygon on a 2-d plane. To maximize $c^Tx$ in 2-d keep
moving along $c$. The last point where $c^Tx$ touches the polygon will the point
of maxima. It can be observed that this point will be a boundary point.
\newline
\newline
Extending this argument to a n-dimensional plane, it seems that $c^Tx$ will
attain its maximum value at the boundary points of the region $Ax \leq b$.
\newline
\newline
\newline
A local maximum of a function $f$ can be defined as follows.
\begin{Def}
If there exists a small neighbourhood $N$ of $x_0$ where $f(x_0) \geq f(x)  \forall
x \in N$, then $x_0$ is said to be a point of local maxima.
\end{Def}
$~$\newline
\begin{Thm}
If $f$ is linear and $f$ convex, then a local maximum is a global maximum.
\end{Thm}
\begin{proof}
Let $x_0$ be a local maximum and $y$ be a global maximum.
\newline
Consider a point $P = (1 - \epsilon)x_0 + \epsilon y$. If $\epsilon \rightarrow
0$,
then $P$ lies in any small neighbourhood of $x_0$ (in particular, in the neighbourhood where $x_0$ is the
point of local maximum).
\newline
Now, consider $f((1 - \epsilon)x_0 + \epsilon y)$. Since f is linear, this
can be written as $$(1 - \epsilon)f(x_0) + \epsilon f(y)$$
$$= f(x_0) + \epsilon (f(y) - f(x_0))$$
We can now observe that at point P, which is in the neighbourhood of $x_0$, the
value of the function $f((1 - \epsilon)x_0 + \epsilon y) \geq f(x_0)$. Thus
$f(x_0)$ can be maximum only if $f(x_0) = f(y)$.
\newline
Thus, we can see that a local maximum is the same as a global maximum.
\end{proof}

\end{document}