how difficult is it to solve a problem?. computability and feasibility can a specific problem be...
TRANSCRIPT
Computability and Feasibility
How difficult is it to solve a problem?
Computability and Feasibility
Can a specific problem be solved by computation?
If so, can a suitably efficient solution be designed so that the ‘solution’ is feasible?
Computability is concerned with whether a problem can or cannot be solved.
Feasibility is concerned with how efficiently a problem can be solved.
We shall look at feasibility first then consider computability.
Key Questions
Computability and Feasibility
Feasibility
There are two standard measures of the efficiency of an algorithm:oTime: How long the algorithm takes to execute.oSpace: How much memory the algorithm uses for
the storage of data during execution.
When more than one algorithm exists which can solve a problem there is often a trade-off between time and space efficiency. i.e. one solution may be quicker but requires more memory than another solution.
Computability and Feasibility
How to measure efficiency? For a particular algorithm, the amount of
space and time required usually varies depending upon the number of inputs to the algorithm – e.g. adding up a thousand numbers takes a lot longer than adding up ten numbers.
Therefore, efficiency is normally expressed as a function of the number of inputs to an algorithm (commonly referred to as ‘n’).
Computability and Feasibility
How to measure efficiency? Space complexity can be easily measured - we
will look briefly at this later but shall focus on time complexity for now.
Measuring time complexity is more difficult than space complexity as the same algorithm run on two different computers could take very different amounts of time to execute as a result of factors such as:oProcessor clock speed.oEfficiency of hardware implementation of
operations used e.g. add.oOther processes executing concurrently.
Computability and Feasibility
How to measure efficiency? We need a measure of time complexity
that focuses on the algorithm rather than the hardware that it is being executed on.
We use Big O notation to do this.
Computability and Feasibility
Order of Growth / Big O
The time efficiency of an algorithm is established in the following way: Identify the basic operation that is performed
by the algorithm. Establish how many times this operation is
carried out as a function of the number of inputs (n).
Consider how quickly this value grows and identify the dominant part in this (e.g. in 4n2+3 the dominant part is n2).
This is the order of complexity of the algorithm. It is written as O(g) where g is the growth rate.
Computability and Feasibility
Linear Time Complexity (1)
Algorithm for summing a list of numbers in a file:
What is the basic operation? Total Total + Number How many times is it carried out? It is in a loop, executed
once for each number in the file. So, if a file contained n items the basic operation would be
executed n times. The algorithm is said to have order of complexity O(n).
Total 0WHILE NOT at end of file READ Number from file Total Total + NumberEND WHILEPRINT Total
Computability and Feasibility
Linear Time Complexity (2)Algorithm to sum a list of numbers entered at the keyboard, user terminating list by entering 0:
What is the basic operation? Total Total + Number How many times is it carried out? It is in a loop, executed
once for each number entered plus once more for the 0 terminator.
So, if n items were entered the operation would be executed n+1 times. The rate of growth is n as 1 is a constant.
The algorithm is said to have order of complexity O(n) since this is the dominant term in n+1.
Total 0REPEAT INPUT Number Total Total + NumberUNTIL Number = 0PRINT Total
Computability and Feasibility
Quadratic Time ComplexityA very naive sorting algorithm (based on bubble sort) to sort n items in an array:
The basic operation is the IF command that compares two numbers to check if they are out of order.
This is inside two loops. The outer one executes n-1 times and the inner one executes n-1 times.
So the IF command is executed (n-1)*(n-1)= n2-2n+1 times. n2 is the dominant term in this expression so the algorithm
is of order of complexity O(n2).
FOR i 1 TO n-1 DO FOR j 1 TO n-1 DO IF arr[j] > arr[j + 1] THEN SWAP (arr[j], arr[j + 1]) ENDIF ENDFORENDFOR
Computability and Feasibility
Worst, Best and Average Case In addition to the number of inputs, the
efficiency of some algorithms depends upon the values of the inputs made to them.
For example:oThe naive implementation of the bubble sort
algorithm on the previous slide takes the same amount of time to execute for a list of length n regardless of what the input values are.
oA more sophisticated implementation will terminate as soon as this list is sorted, so will be more efficient if the input list is already partially sorted than if it is not.
Computability and Feasibility
Worst, Best and Average Case In this improved version of the bubble sort algorithm
on the earlier slide, the outer FOR loop that repeated n-1 times is replaced by a REPEAT loop that terminates when the inner loop passes through the entire list without having to make any swaps i.e. the list is sorted.
i 1REPEAT swapmade FALSE FOR j 1 TO n-1 DO IF arr[j] > arr[j + 1] THEN SWAP (arr[j], arr[j + 1]) swapmade TRUE ENDIF ENDFOR i i + 1UNTIL swapmade = FALSE
Computability and Feasibility
Worst, Best and Average Case
Initial
i=1
i=2
i=3
i=4
[1] 30 30 30 20 10
[2] 50 20 20 10 20
[3] 20 10 10 30 30
[4] 10 40 40 40 40
[5] 40 50 50 50 50
Initial
i=1
[1] 10 10
[2] 20 20
[3] 30 30
[4] 40 40
[5] 50 50
In the best case the initial list is already ordered.
The algorithm terminates after one iteration of the outer loop because no swaps are made.
n-1 (4) comparisons have been made, so the best case order of complexity is O(n).
In the worst case the initial list is far from being ordered.
The outer loop iterates the maximum n-1 (4) times.
The inner loop iterates n-1 (4) times.So, the basic operation executes
(n-1)*(n-1) = n2-2n+1 times.So the worst case order of complexity is
O (n2), the same as the more naive algorithm.
Computability and Feasibility
Worst, Best and Average Case For the improved sort algorithm (as for many
algorithms), there is a significant difference between best and worst case performance:
Computability and Feasibility
Worst, Best and Average Case Three standard measures of efficiency are
therefore made for an algorithm:oWorst case complexity is the amount of resources
that an algorithm uses to solve a problem with the least favourable set of inputs. It is the upper bound on the amount of resources consumed by the algorithm.
oBest case complexity is the amount of resources that an algorithm uses to solve a problem with the most favourable set of inputs. It is the lower bound on the amount of resources consumed by the algorithm.
oAverage case complexity is the amount of resources that an algorithm uses to solve a problem averaged over all sets of possible inputs.
Computability and Feasibility
Algorithm vs Problem Complexity
For many problems, more than one algorithm can provide a solution.
For example an ordered list could be searched using a linear search or a binary search algorithm.
Computer scientists therefore often consider the complexity of a problem such as searching rather than the complexity of a particular algorithm such as binary search.
The complexity of a problem is the worst case complexity of the most efficient algorithm which solves the problem.
Computability and Feasibility
Classes of Complexity Problems are grouped into classes based upon their
complexity. This table shows how the execution time of an
algorithm increases as the number of inputs does for logarithmic, linear, polynomial and exponential classes:
Easy to solve Almost impossible to solve
Logarithmic
O(log2 n) O(n) O(2n) O(n2) O(n3) O(2n) O(3n)
1 0 1 2 1 1 2 32 1 2 4 4 8 4 95 2.3 5 10 25 125 32 243
10 3.3 10 20 100 1000 1024 5904920 4.3 20 40 400 8000 1048576 3.49E+0950 5.6 50 100 2500 125000 1.13E+15 7.18E+23
100 6.6 100 200 10000 1000000 1.27E+30 5.15E+471000 10 1000 2000 1000000 1000000000 1.1E+301 Too Big!
10000 13.3 10000 20000 100000000 1E+12 Too Big! Too Big!100000 16.6 100000 200000 10000000000 1E+15 Too Big! Too Big!
Linear (kn) Polynomial (nk) Exponential (kn)Number of inputs (n)
Computability and Feasibility
Tractability The time taken to solve exponential time complexity
problems grows rapidly as the number of inputs grows. e.g. for just 50 inputs an algorithm of time complexity
O(2n) would execute the basic operation more than 1130000000000000 times.
Problems that have a polynomial (or better) time complexity are known as tractable.
Problems that can be solved but which have a worse than polynomial time complexity are known as intractable.
Intractable problems are effectively unsolvable in the general case (except for very small numbers of inputs) as a solution would take such an unreasonably long time to produce that it is unlikely to be useful.
Intractable problems may sometimes be tackled using heuristics to produce a close to optimal solution.
Computability and Feasibility
Example Intractable Problems Travelling salesman: Given a set of n cities and
the distances between them, find the shortest tour that connects them all and allows the salesman to visit every city, before returning home.
Bin packing: There are n objects with sizes in the range 0 to 1 which must be fitted into bins, each of which has capacity 1. What is the minimum number of bins required?
Knapsack problem: A knapsack has a fixed capacity. There are a set of objects, each of which has a size and value. What is the most valuable set of objects that can be put into the knapsack?
Other types of problem which are often intractable include scheduling and optimisation problems.
Computability and Feasibility
Example Intractable Problems Why are these types of problem
intractable? Because the only way to produce an
optimal solution is to try out every possible solution – a brute force approach.
And, as the number of inputs to the problem increases, the number of possible solutions increase very rapidly.
Computability and Feasibility
Travelling Salesman e.g. For the travelling salesman problem:
A B20
A B20
C1016
2 cities = 1 route (ABA)Length 40
3 cities = 2 routes (ABCA), (ACBA) Both length 46 (as one is the reverse of the other)
A B20
C16
D
108 22 15
4 cities = 6 routes:ABCDA length 54 ADCBA length 54ABDCA length 73 ACDBA length 73ACBDA length 55 ADBCA length 55Optimum route is 54 long.
For n cities, (n-1)*(n-2)*....* 1 routes = (n-1)! routes.For just 20 cities, 121645100408832000 routes Conclusion: Intractable!!!
Computability and Feasibility
Space Efficiency
We have focussed on time efficiency as speed is the aspect of complexity that is usually of most concern.
However space efficiency can also be an important factor when choosing between different algorithms that have the same purpose.
Sometimes there can be a trade-off between space and time efficiency.
A good example of this is when sorting data.
Computability and Feasibility
Space Efficiency Example Simple sorting algorithms such as bubble sort and
insertion sort have very poor worst case time complexity of O(n2). The data is sorted ‘in situ’ within the existing list so to sort n items only n memory locations are required. No extra memory locations are required during the sort process.
Merge sort is a much more time efficient sort algorithm with worst case time complexity O(n log2n).
However, merge sort is less space efficient. This is because it is a recursive algorithm and copies of subsections of the list are usually generated each time the algorithm recurses.
The next two slides show the merge sort algorithm and why this extra storage space is required.
Computability and Feasibility
Space Efficiency Example
The merge sort procedure:
If list length <= 1 then exit as list is sorted If list length > 1 thenoSplit the unsorted list into two sublists of about
half the size.oRecursively call the merge sort procedure to sort
the first of the two sublists.oRecursively call the merge sort procedure to sort
the second of the two sublists.oMerge the two sublists back into one sorted list.
Computability and Feasibility
Space Efficiency Example
29 49 11 18 8037 22
29 49 1137 18 80 22
29 49 1137 18 80 22
2937 49 11 18 80
37 11 4929 18 80
29 37 4911 18 22 80
18 22 29 37 4911 80
Rec
urs
ing
Un
win
din
g
Rec
urs
ion
An example execution of merge sort on a seven item list:
Additional storage space for temporary storageof lists.
Computability and Feasibility
Computability
We have seen that some problems admit no effective solution because they take too long to compute (intractable problems).
But are there any problems that simply cannot be solved, however much time and space we could give them?
Yes – several problems have been proven to be non-computable.
The most famous non-computable problem is the halting problem.
Computability and Feasibility
The Halting Problem The halting problem: Is it possible to write a program
that can tell, given a program and its inputs, whether the program will halt? (without executing the program being tested)
The answer is no, it has been proven that in the general case this problem cannot be solved.
We shall look briefly at the proof of this although A Level candidates are only required to know what the problem is and that it is not computable.
It uses a standard technique called proof by contradiction in which we:o Make a proposition.o Show that if the proposition were true, a contradiction
must occur.o Therefore we prove that the proposition must be untrue.
Computability and Feasibility
Proof of Non-computability Assume that the halting function does exist.
It is defined as a function h with two inputs p and i.op is the program that the halting function is
testing.o i is the input that will be made to the program p.
The function h should output 0 if p halts on the input n and 1 if p does not halt.
h ( p , i ) =0, if p halts on input i1, if p does not halt{
Computability and Feasibility
Proof of Non-computability
Now construct a program r(n) which calls the function h using n as both parameters i and p:
r ( n ) Halts if h ( n , n ) = 1Loops forever otherwise{
Computability and Feasibility
Proof of Non-computability Now use the halting function h to test if the program r
terminates with input n. So:
Recall the definition of r:
And substitute the defintion of r into h gives:
Now, if r = n we would have:
A contradiction!
h ( r , n ) =0, if r halts on input n1, if r does not halt{
r ( n ) Halts if h ( n , n ) = 1Loops forever otherwise{
h ( r , n ) = 0, if h ( n , n ) = 11, if h ( n , n ) = 0{
h ( n , n ) = 0, if h ( n , n ) = 11, if h ( n , n ) = 0{
Computability and Feasibility
Proof of Non-computability We have arrived at a contradiction, i.e. that
h(n,n)=0 if h(n,n)=1 and vice-versa. Since the steps we have taken in arriving at this
contradiction are sound, it follows that the original proposition, i.e. that the halting function h exists must be untrue.
Therefore the halting problem is not computable.
Other problems which have proven to be non-computable are the Entscheidungsproblem, Hilbert’s tenth problem and the Post correspondence problem.
Computability and Feasibility
End of Presentati
on
Computability and Feasibility
Dominant Term Solution
Expression Dominant Term
Order of Complexity
n + 5 n O(n)6n + n2 + 12 n2 O(n2)2n + n3 + 12 2n O(2n)
Q1) Dominant terms and order of complexity:
Computability and Feasibility
Most to Least Efficient SolutionQ2) Orders of complexity in order, most
efficient first:
O(1) O(log2 n) O(n) O(n2) O(n4) O(3n)
Computability and Feasibility
Linear Search Solution
Basic Operation: Comparison of item in list with search value.
Order of Time Complexity: O(n) Explanation: The loop is repeated once for
each item in the array, so n times for an array containing n items.
INPUT SearchValueFound FalseFOR Index 1 TO ListLength DO IF ArrayValue[Index] = SearchValue THEN Found TrueENDFOROUTPUT Found
Q3)
Computability and Feasibility
Linear Search Solution
Why different complexities: This is because the loop terminates as soon as the item is found, if it is in the list. So the number of times the basic operation is executed depends upon where the item is in the list.
INPUT SearchValueFound FalseIndex 0REPEAT Index Index + 1 IF ArrayValue[Index] = SearchValue THEN Found TrueUNTIL Index = ListLength OR Found = TrueENDFOROUTPUT Found
Q4)
Computability and Feasibility
Linear Search Solution
Best Case: Item is at start of list, in which case only one item needs to be examined so O(1).
Worst Case: Item is at end of list so all n items must be checked, so O(n).
INPUT SearchValueFound FalseIndex 0REPEAT Index Index + 1 IF ArrayValue[Index] = SearchValue THEN Found TrueUNTIL Index = ListLength OR Found = TrueENDFOROUTPUT Found
Q4)
Computability and Feasibility
Tractability Solution
Order of Complexity
Tractable?
O(3n) NO(n10) Y
O(log2 n) YO(2n) N
Q5) Which algorithms are tractable? (though not necessarily quick to solve!)
Computability and Feasibility
Halting Problem Solution
Q6) The significance of the Halting problem is:
o That there are some problems that we can prove cannot be solved by a computer.
o That we know that it is not possible, in the general case, to write a program that will determine if another program will halt on a given set of inputs.