% % This is the LaTeX template file for lecture notes for CS294-8, % Computational Biology for Computer Scientists. When preparing % LaTeX notes for this class, please use this template. % % To familiarize yourself with this template, the body contains % some examples of its use. Look them over. Then you can % run LaTeX on this file. After you have LaTeXed this file then % you can look over the result either by printing it out with % dvips or using xdvi. % % This template is based on the template for Prof. Sinclair's CS 270. %\documentclass{article} \documentclass[]{article} %\usepackage[a4paper, total={7in, 8in}]{geometry} \usepackage{layout} \usepackage{tcolorbox} %\usepackage[usenames,dvipsnames,svgnames,table]{xcolor} \definecolor{darkgreen}{rgb}{0.0,0,0.9} \usepackage[colorlinks=true,pdfpagemode=UseNone,citecolor=OliveGreen,linkcolor=red,urlcolor=red, %pagebackref, pdfstartview=FitW]{hyperref} \usepackage{graphics,amsmath,amsthm,thmtools,amssymb,tikz,mathtools, algorithm, algpseudocode} \usepackage[capitalize]{cleveref} %\usepackage[backref=true, backend=biber, isbn=false, url=true, firstinits=true, maxnames=20, style=alphabetic,]{biblatex} \usepackage{paralist} %\usepackage{pgfplots} %\usepackage{xspace} \usepackage{natbib} \usetikzlibrary{calc} \setlength{\oddsidemargin}{0 in} \setlength{\evensidemargin}{0 in} \setlength{\topmargin}{-0.6 in} \setlength{\textwidth}{6.5 in} \setlength{\textheight}{8.5 in} \setlength{\headsep}{0.75 in} \setlength{\parindent}{0 in} \setlength{\parskip}{0.1 in} %\advance\hoffset by -3mm % A4 is narrower. %\advance\voffset by 8mm % A4 is taller. %\setlength{\footskip}{1cm} \def\X{b} \def\tX{\tilde{b}} \def\mX{B} \def\tmX{\tilde{B}} %\def\bone{{\bf 1}} %\def\N{\mathbb{N}} %\def\R{\mathbb{R}} \def\mP{\mathbb{P}} \def\mE{\mathbb{E}} \def\mI{\mathbb{I}} %\def\cT{{\cal T}} %\def\cB{{\cal B}} \def\reff{\textup{Reff}} % % The following commands set up the lecnum (lecture number) % counter and make various numbering schemes work relative % to the lecture number. % \newcounter{lecnum} \renewcommand{\thepage}{\thelecnum-\arabic{page}} %\renewcommand{\thesection}{\thelecnum.\arabic{section}} \renewcommand{\thesection}{\arabic{section}} %\renewcommand{\theequation}{\thelecnum.\arabic{equation}} \renewcommand{\theequation}{\thesection.\arabic{equation}} \renewcommand{\thefigure}{\thesection.\arabic{figure}} \renewcommand{\thetable}{\thesection.\arabic{table}} \newcommand{\PP}[2]{\mP_{#1}\left[#2\right]} \newcommand{\weight}[1]{w(#1)} \renewcommand{\P}[1]{\mP\left[#1\right]} \newcommand{\I}[1]{\mI\left[#1\right]} \newcommand{\E}[1]{\mE\left[#1\right]} \newcommand{\EE}[2]{\mE_{#1}\left[#2\right]} \newcommand{\norm}[1]{\|#1\|} \newcommand{\therank}{{\mathsf{rank}_{1-\eps}}} \newcommand{\cutcone}{{\textsf{CutCone}}} \newcommand{\mincut}{{\textsf{Min-Cut}}} \newcommand{\sparcut}{{\textsf{SparsestCut}}} \newcommand{\scut}{{\textsf{SC}}} \newcommand{\LR}{{\textsf{LR}}} %\DeclareMathOperator{\trace}{Tr} \DeclareMathOperator{\vol}{vol} \DeclareMathOperator{\sspan}{span} \newcommand{\algorithmautorefname}{Algorithm} \newcommand{\xmod}[1]{\ (\mathrm{mod}\ #1)} \newcommand{\np}{\textsf{NP}} %\newcommand{\nph}{\ensuremath{\mathsf{NP}}-\ensuremath{\mathsf{Hard}}\xspace} %\newcommand{\npc}{\ensuremath{\mathsf{NP-Complete}}\xspace} \newcommand{\ds}{\displaystyle} \newcommand{\den}{\texttt{den}} \newcommand{\dash}[1]{{#1}'} \renewcommand{\qedsymbol}{\ensuremath{\blacksquare}} % % The following macro is used to generate the header. % \newcommand{\handout}[5]{ \renewcommand{\thepage}{#1-\arabic{page}} \noindent \begin{center} \framebox{ \vbox{ \hbox to 6.35in { {\bf CS-786 Randomized Algorithms} \hfill #2 } \vspace{4mm} \hbox to 6.35in { {\Large #5 \hfill} } \vspace{2mm} \hbox to 6.35in { {\it #3 \hfill #4} } } } \end{center} \vspace*{4mm} } \newcommand{\lecture}[5]{\handout{#1}{#2}{Lecturer: #3}{Scribe: #4}{Lecture #1: #5}} \makeatletter \def\fnum@figure{{\bf Figure \thefigure}} \def\fnum@table{{\bf Table \thetable}} \long\def\@mycaption#1[#2]#3{\addcontentsline{\csname ext@#1\endcsname}{#1}{\protect\numberline{\csname the#1\endcsname}{\ignorespaces #2}}\par \begingroup \@parboxrestore \small \@makecaption{\csname fnum@#1\endcsname}{\ignorespaces #3}\par \endgroup} \def\mycaption{\refstepcounter\@captype \@dblarg{\@mycaption\@captype}} \makeatother %Use this command for a figure; it puts a figure in wherever you want it. %usage: \fig{NUMBER}{SPACE-IN-INCHES}{CAPTION} \newcommand{\fig}[3]{ \vspace{#2} \begin{center} Figure \thelecnum.#1:~#3 \end{center} } % Use these for theorems, lemmas, proofs, etc. %\newtheorem{theorem}{Theorem}[lecnum] \newtheorem{theorem}{Theorem}[section] \newtheorem{lemma}[theorem]{Lemma} \newtheorem{fact}[theorem]{Fact} \newtheorem{problem}[theorem]{Problem} \newtheorem{notation}[theorem]{Notation} \newtheorem{proposition}[theorem]{Proposition} \newtheorem{claim}[theorem]{Claim} \newtheorem{corollary}[theorem]{Corollary} \newtheorem{example}[theorem]{Example} \newtheorem{definition}[theorem]{Definition} \newtheorem{remark}[theorem]{Remark} %\newtheorem{item}[theorem]{Item} %\newtheorem{equation}[theorem]{Equation} \newenvironment{proofof}[1]{\textcolor{red}{\em Proof of #1.}}{\hfill%\rule{2mm}{2mm} \qed} \newenvironment{proofsk}[1]{\textcolor{red}{\em Proof Sketch for #1.}}{\hfill%\rule{2mm}{2mm} %\newenvironment{proofof}[1]{{\em Proof of #1.}}{\hfill%\rule{2mm}{2mm} \qed} \input{preamble} % **** IF YOU WANT TO DEFINE ADDITIONAL MACROS FOR YOURSELF, PUT THEM HERE: \begin{document} %FILL IN THE RIGHT INFO. %\lecture{**LECTURE-NUMBER**}{**DATE**}{**LECTURER**}{**SCRIBE**}{**TOPIC**} %TODO Add Lecture Number \lecture{01}{2 August 2024}{Rohit Gurjar}{Prerak, Ayush}{Limitations of Computation} %\footnotetext{These notes are partially based on those of Nigel Mansell.} % **** YOUR NOTES GO HERE: % Some general latex examples and examples making use of the % macros follow. %**** IN GENERAL, BE BRIEF. LONG SCRIBE NOTES, NO MATTER HOW WELL WRITTEN, %**** ARE NEVER READ BY ANYBODY. \section{Lower Bound on Sorting} \begin{theorem}\label{thm:comparison_lower_bound} Given an array of $n$ distinct numbers, any comparison based algorithm (an algorithm that can only make comparison between two elements to obtain an ordering amongst them, and does not use the value of the elements in any way) needs to perform at least $\theta(\log_2(n!)) = \theta (n \log_2(n))$ comparisons. \end{theorem} \begin{proofof}{\Thm{comparison_lower_bound}} Before we begin with the proof, let us first establish an equivalence between sorting and obtaining the permutation of the input array. \begin{claim}\label{clm:permutation_and_sorting_eq} Note: When we say permutation of an array, we mean the permutation which transforms the sorted version array to the given array. Given a comparison based algorithm that determines the permutation of an input array that performs at most $f(n)$ comparisons, we can construct a sorting algorithm which performs $\theta(f(n))$. In the reverse direction, given a comparison based algorithm that sorts an input array using at most $f(n)$ comparisons, we can construct an algorithm that determines the permutation of an input array. \end{claim} \begin{proofof}{\Clm{permutation_and_sorting_eq}} In the forward direction, given a sub-routine that determines the permutation of an input array after performing $f(n)$ comparisons, one can first call this subroutine which returns the permutation, and using this permutation, one can return the sorted array without any more queries to the array. Hence, we can sort the array in $\theta(f(n))$ comparisons. In the backward direction, consider augmenting the elements with their positions as well, that is we construct a new array where the $i^{th}$ element is $(a_i, i)$. Sorting this new array using the sorting algorithm, using the comparison of two elements in this new array as comparing the first number in the two tuples, we can obtain the permutation from the sorted array by then removing the first number in each tuple, and computing the inverse of the resulting permutation (which does not require any further queries to the array). Hence, we can compute the permutation using $\theta(f(n))$ queries if the sorting algorithm performed $f(n)$ queries. \end{proofof} With the above claim now, we have established that up to a constant, the two problems are equivalent in terms of minimum comparisons required, and hence it suffices to provide a lower bound for obtaining the permutation of the input array. To that extent, consider any algorithm that finds the permutation of the input array. Since this algorithm has to work for all inputs, it suffices to provide one adversarial input on which the algorithm requires $\Omega(n \log_2(n))$ comparisons. Let $S_0$ be the set of all possible inputs (permutations) of $n$ sized array, with $\vert S \vert = n!$. At each comparison step, the algorithm queries by providing indices $i, j$. Let us define set $S_k$ to be the set of all inputs that are consistent with the results of the queries in steps $1, 2, \ldots, k$. The algorithm can output an answer at step $k_f$ only when $\vert S_{k_f} \vert = 1$. For a set $S_k$ and query $q$, define a partition of $S_k = S_k[q_t] \cup S_k[q_f]$ where $S_k[q_t]$ represents the inputs from $S_k$ where the query holds true, and $S_k[q_f]$ as the inputs where query holds false. It is easy to see that: \begin{equation}\label{eq:comparison_update} S_{k+1} = \begin{cases} S_k[q_t] \qq{if the result of query was true} \\ S_k[q_f] \qq{if the result of query was false} \end{cases} \end{equation} Consider an adversary that, on query $q$ at $(k+1)^{th}$ step, returns true if $\vert S_k[q_t] \vert \geq \vert S_k[q_f] \vert$ and false otherwise. It follows from \ref{eq:comparison_update} that $\vert S_{k+1} \vert \geq 1/2 \vert S_{k} \vert \implies \vert S_{k} \vert \geq n!/(2^k) $. Hence, if $\vert S_{k_f} \vert = 1$, we have $k_f \geq \log_2(n!)$. This shows that for any behavior of the algorithm, we can generate an adversarial input on which the algorithm needs atleast $\log_2(n!)$ comparisons to obtain the permutation. \end{proofof} The above bound is an example of an information theoretic lower bound: We showed that we need at least $f(n)$ queries to obtain enough information about the input. And no matter how much computation power one can use, without $\log_2(n!)$ comparisons, one will not get enough information to sort the array (assuming that the algorithm gets no information about elements other than the ordering among the queried pair). \begin{problem}[Homework] Consider the following problem: You are given $n$ numbers, and exactly two of them are equal. You are allowed to compare two numbers $A_i \square A_j$, with the result being: $<, >$ or $=$. The task is to find the the pair that is equal. Show that any algorithm requires $\Omega(n \log_2(n))$ comparisons, and give an algorithm that does it in $O(n \log_2(n))$ comparisons. \end{problem} However, the kind of bounds that we shall be more interested in are complexity lower bounds: Given all the information that we need about the input, how much computation is required to obtain the answer. But before we answer this, a more important question to think about is: Can we even ``compute'' an answer for any given problem? As an example, consider the problem of weather forecasting. Let us look at some of the limitations we might face when trying to solve the problem: \begin{itemize} \item Lack of understanding: Maybe we are not experts in meteorology, and we do not understand the exact science behind how weather forecasting works. \item Lack of information: Maybe we do understand how weather works, but we do not have access to all the variables required to accurately predict the weather. \item Lack of computation power: Even with the above two, we might just not have enough computational power to solve the problem. \end{itemize} In this course, we shall assume that the first two are not limitations, that is we completely understand the objects and have complete information about the input. However, even if we overcome the first two limitations, and assume infinite computational power, is it still possible to solve all the problems? To answer this question, let us first formalize what a ``problem'' is: \section{Computational Task} \begin{definition} A computational task consists of an input $x \in \qty{0, 1}^*$, and a relation $R \subseteq \qty{0, 1}^* \times \qty{0, 1}^*$ (or a function $f : \qty{0, 1}^* \to \mathcal{P}({\qty{0, 1}^*})/\phi$ if one wants all the inputs to have some output). The task is to return an output $y$ such that $(x, y) \in R$. \end{definition} We are interested in two broad categories of computational tasks: \begin{enumerate} \item \textbf{Search Problem}: A task where the output $y \in \qty{0, 1}^*$ \item \textbf{Decision Problem}: A task where the output $y \in \qty{0, 1}$ \end{enumerate} It turns out that for all the search problems that we would be looking at, we can define a ``natural'' or ``equivalent'' decision problem (in loose terms) such using one as a sub-routine, one can solve the other with an at-most polynomial overhead (that is, the problem can be solved by performing polynomial steps, where each step is a computational step or a call to the sub-routine). \begin{example}{SAT} Input: A propositional formula $\phi$. Output: A satisfying assignment $\sigma$ for $\phi$, and some default value if $\phi$ is not satisfiable. If we try to define our decision problem as: Given a formula $\phi$ and an assignment $\sigma$ does $\sigma$ satisfy $\phi$, we do not know of a way to solve the search problem from the decision problem with at most a polynomial overhead, that is, this is not an example of a natural decision problem associated to the search problem. % A self reduction is a reduction from a search problem to a decision problem, such that with the decision problem as a subroutine, we can solve the search problem with an atmost polynomial overhead. Here is an example of a natural decision problem for SAT: Given a formula $\phi$, is it satisfiable? With this sub-routine, one algorithm to solve SAT is as follows: \begin{itemize} \item If $\phi$ is not satisfiable, return the default value. \item Else, we shall maintain a partial assignment for the propositional variables, and keep adding a new assignment to it after each step. We iterate over variables, and at each step $i$, we assign $x_i$ an arbitrary value (say $0$). Consider the formula obtained after the substitution for assigned variables, and ask the sub-routine if the new formula is satisfiable. If so, we do not modify the assignment for $x_i$, and otherwise we assign $x_i$ as the negation of its initial assignment ($1$ in this case). \end{itemize} It is easy to see that the above algorithm correctly returns a satisfiable assignment if $\phi$ is satisfiable, and runs in time polynomial to the size of the formula. However, to show that the above defined decision problem is in fact ``equivalent'' to the search problem, we also need to show the reverse direction: Using the search problem, we can solve the decision problem with an atmost polynomial overhead. But this is easy, since this can be done by checking if the returned assignment is the default value which corresponds to an unsatisfiable formula. \end{example} \begin{example}{PRIME FACTORS} Input: A non-negative integer $n$ (Note that the input size is $\log(n)$) Output: Prime factors of $n$ If we try to define the decision problem as: Given a prime $p$ and an integer $n$, does $p$ divide $n$, there again does not appear to be an efficient algorithm which can solve the search problem using the decision problem. If we try to iterate over all the primes from $1$ to $n-1$, we would need $\theta (n / \log(n))$ (approximately the number of primes from 1 to $n$) queries to the sub routine, which is not polynomial in input size. Instead consider the following decision problem: Given a number $n$ and a number $k$, is there a prime factor of $n$ less than $k$ (and greater than 1)? Using this subroutine, we can perform a binary search to obtain the smallest prime factor of $n$, keep dividing $n$ by $p$ until we obtain a number $n'$ not divisible by $p$, and recursively find the prime factors of $n'$. Since the number of prime factors of $n$ are at most $\log_2(n)$ ($n = \prod_i p_i^{\alpha_i} \geq \prod_i 2 = 2^{\text{\# primes}}$ ), and in each recursive call, we perform poly($\log(n)$) steps, the overall algorithm runs in time polynomial to the size of the input. The other way around, given an algorithm for the search problem, to solve the decision problem, we just iterate over all the prime factors of $n$, and if we find a factor less than $k$, we return YES, else we return NO. \end{example} We still need to define what it means to perform computation, without a proper definition, every problem can trivially be computed, by assuming our model of computation as a ``black-box'' which magically outputs the correct answer. However, we defer the formal definition for this to a later lecture. Here we shall assume that the computations are performed by \verb!C++! programs. With all the definitions, let us now try to answer the question: Given a problem, can we always write a program to solve it? \section{Undecidable Problems} \begin{claim}\label{clm:undecidability} You cannot write a program for solving every problem \end{claim}We will give two different proofs for the same. One will be based on the counting argument while other will be based on technique called diagonalization. \begin{proofof}{\Clm{undecidability} using counting} Firstly, we will give the proof based on counting. As two different decision problems can't have the same program, it's sufficient to show that the cardinality of the set of decision problems is greater than that of the set of programs. Each C++ program can be represented as a binary string in some suitable encoding (like ASCII). Since each binary string can be interpreted as a natural number (in binary representation), this establishes a injection between the set of all C++ programs and the set of natural numbers (\(\mathbb{N}\)). Injection of an infinite set with \(\mathbb{N}\) implies bijection, therefore we have established bijection between the set of all C++ programs and the set of natural numbers Now, consider problems as decision problems (yes/no questions) over the set of all possible inputs (encoded as binary strings). The set of all possible problems can be viewed as the set of all functions mapping binary strings to \(\{0, 1\}\). Each such function can be represented by an infinite binary sequence, where the \(i\)-th bit of the sequence corresponds to the output of the function on the \(i\)-th binary string. The set of infinite binary sequences has the same cardinality as the set of real numbers (\(\mathbb{R}\)), because each real number in the interval \((0, 1)\) can be uniquely represented by an infinite binary sequence (its binary expansion). To extend this bijection to all real numbers, we can use the function \[ f(x) = 2x - 1 \] which maps \((0, 1)\) bijectively onto \((-1, 1)\), and then extend it to \(\mathbb{R}\) by applying another transformation, e.g., \[ g(x) = \frac{x}{1-|x|} \] which maps \((-1, 1)\) onto \(\mathbb{R}\). This shows that there is a bijection between the set of infinite binary sequences and the set of real numbers \(\mathbb{R}\). Therefore, there is a bijection between the set of problems and the set of real numbers. Since the set of natural numbers (\(\mathbb{N}\)) is countable and the set of real numbers (\(\mathbb{R}\)) is uncountable, the cardinality of the set of real numbers is strictly greater than that of the set of natural numbers. Therefore, the set of problems is larger than the set of programs, proving that there exist problems for which no program can be written. This completes the proof using the counting argument. \end{proofof} The above proof only shows that there exists problems for which we cannot write a program. % But are there such explicit problems? The below proof shows how one can find such explicit problems. % Recall that above we have shown a bijection between set of natural numbers and set of all C++ programs. Thus, we can talk about an enumeration of the programs as in: 1st program, 2nd program, $\dots$, $i$th program, $\dots$. \begin{proofof}{\Clm{undecidability} using Diagonalization} Consider a function $g : \mathbb{N} \to \{0, 1\}$ defined as follows: \[ g(i) = \begin{cases} 1 & \text{if the \(i\)-th program halts on input \(i\),} \\ 0 & \text{otherwise.} \end{cases} \] We claim that there is no program that computes the function $g$. To show this we will take an arbitrary program $G$ and argue that the output of program $G$ must differ from function $g$ on some input. %Assume, for the sake of contradiction, that there exists a program \( G \) for function $g$. %which can decide whether any given program halts on a specific input. In other words, \( G(i) \) is Let $G$ be an arbitrary program. Now, consider the following program \( P \): \begin{quote} \textbf{Program \( P \):} \begin{itemize} \item Input: \( i \in \mathbb{N} \) \item Run program \( G \) on input \( i \). \item If \( G(i) = 1 \), then enter an infinite loop. \item Otherwise, return \( 0 \). \end{itemize} \end{quote} Let \( j \) be the index of program \( P \) in the enumeration of all programs. % Consider the behavior of \( P \) when given \( j \) as input: Now, consider the value of function $g(j)$ and output of program $G$ on input $j$. \begin{itemize} \item If \( g(j) = 1 \): According to the definition of \( g \), this means that program \( P \) halts on input \( j \). But program $P$ can halt only when the output of $G$ on input $j$ is different from $1$. Hence, $G(j)\neq 1$. % However, by the construction of \( P \), if \( G(j) = 1 \), \( P \) enters an infinite loop, which means \( P \) does not halt on input \( j \). \item If \( g(j) = 0 \): This implies that program \( P \) does not halt on input \( j \). That can happen in two ways: (i) program $G$ does not halt on input $j$ or (ii) program $G$ outputs $1$ on input $j$. In either case, we have $G(j)\neq 0$. % But according to the construction of \( P \), if \( G(j) = 0 \), \( P \) will return \( 0 \) and halt. \end{itemize} Here, we have shown that no program agrees with function $g$ on all inputs. % In other word, there is no program to compute function $g$. % Therefore, the assumption that \( G \) exists must be false. This implies that no program \( G \) can decide the halting problem for all programs and inputs. This completes the proof using diagonalization. \end{proofof} Using the above claim, now we will prove that there is no program for the halting problem. In other words, the halting problem is undecidable. %Now, we will talk about a common undecidable problem, which is the halting problem, and prove its undecidability using diagonalization. \textbf{Halting Problem:} Given a description of a program \(P\) and an input \(x\), determine whether \(P\) halts (i.e., finishes execution) when run with input \(x\), or \(P\) runs forever. Formally, define a function \( h \) such that: \[ h(P, x) = \begin{cases} 1 & \text{if program \(P\) halts when run with input \(x\)}, \\ 0 & \text{if program \(P\) does not halt when run with input \(x\)}. \end{cases} \] \textbf{Proof of Undecidability:} If there is a program for computing function $h$, then we can easily get a program for computing function $g$ defined above. For input $i$ for $g$, we simply need to find a description $P$ of the $i$th program and give input $(P,i)$ to the program computing function $h$. As we have shown that there is no program to compute function $g$, the same is true for function $h$. \begin{tcolorbox}[colframe=black,colback=white, title=\textbf{HW}] \textbf{Problem:} Prove that the following problem is undecidable: Given two C++ programs, determine whether they have the same behavior (i.e., whether they produce the same output for all possible inputs). \end{tcolorbox} %\bibliographystyle{alpha} %\bibliography{references} \end{document} Assume, for the sake of contradiction, that there exists a program \( H \) that can solve the halting problem for any program-input pair. That is, \( H(P, x) \) returns 1 if \( P \) halts on input \( x \), and 0 otherwise. Now, consider a new program \( D \) defined as follows: \begin{quote} \textbf{Program \( D \):} \begin{itemize} \item Input: a program \( P \) \item Run \( H(P, P) \): \begin{itemize} \item If \( H(P, P) = 1 \), then enter an infinite loop. \item If \( H(P, P) = 0 \), then halt. \end{itemize} \end{itemize} \end{quote} Now, consider the behavior of program \( D \) when given its own code as input, i.e., \( D(D) \): \begin{itemize} \item If \( H(D, D) = 1 \), this implies that \( D \) halts when given \( D \) as input. However, according to the definition of \( D \), if \( H(D, D) = 1 \), then \( D \) will enter an infinite loop, which is a contradiction. \item If \( H(D, D) = 0 \), this implies that \( D \) does not halt when given \( D \) as input. However, by the definition of \( D \), if \( H(D, D) = 0 \), then \( D \) will halt, which is again a contradiction. \end{itemize} In both cases, we arrive at a contradiction. Therefore, the assumption that such a program \( H \) exists must be false. This proves that the halting problem is undecidable.