Algorithms and their Applications

CS2004 (2011-2012)

Dr Stasha Lauria

11.1 Applications, Introduction and Parameter Optimisation

Previously On CS2004…So far we have looked at:

Concepts of computation and algorithmsComparing algorithmsSome mathematical foundationThe Big-Oh notationComputational complexityData structuresSorting algorithmsHill Climbing and Simulated AnnealingTabu Search and Iterated Local Search

Applications, Introduction and Parameter Optimisation Slide 2

This TermWe are going to look into applications and advanced Heuristic Search methodsWe are going to cover:

Applications, introduction and parameter optimisationThis lecture!

Genetic AlgorithmsEvolutionary ComputationAnt Colony OptimisationParticle Swarm Optimisation Bin PackingData ClusteringThe Travelling Salesman ProblemGPU Computing

Applications, Introduction and Parameter Optimisation Slide 3

Applications – Part 1One of the aims of this module to provide a set of skills and techniques that can be applied to solving real world problems and applicationsHowever many applications require a high degree of previous knowledge and/or data

E.g. Gene expression data analysisVery large volumes of dataRequires in depth molecular and biological knowledge

Applications, Introduction and Parameter Optimisation Slide 4

Applications – Part 2However many problems can be solved if a similar problem can be solvedThe ScalesScales problem is easy to understand and requires only a little dataIf we have a method that can solve the Scales problem then we can solve the much more complex Subset SelectionSubset Selection problem

We will look at examples of this laterApplications, Introduction and Parameter Optimisation Slide 5

Applications – Part 3Nearly all of the problems we will look at are NP-CompleteNP-CompleteThere is no algebraic or direct solutionThe computation complexity is exponential in terms of problem sizeE.g. The ScalesScales problem is O(2n)I.e. The applications we are going to look at are very hard to solve….Applications, Introduction and Parameter Optimisation Slide 6

Optimisation – Definition

““The process of making something optimal”The process of making something optimal”

For the purposes of this module, the “thing” that we are making optimal is a solution to a problemWe will often look for the highest or lowest value of a fitness functionThe elements that we change to seek an optimal solution are often called parametersparameters, variablesvariables and/or featuresfeatures

We change the allocation of weights to optimise how well balanced a set of scales is using Hill Climbing to solve the Scales problem

Applications, Introduction and Parameter Optimisation Slide 7

Optimisation ProblemsWe will be looking at the following different types of Optimisation problems:

Parameter OptimisationCombinatorial Optimisation

Subset selectionPermutation problemsGrouping (Clustering) problemsBin Packing

Multi Objective ProblemsConstraint Satisfaction Problems

Applications, Introduction and Parameter Optimisation Slide 8

Parameter Optimisation – Part 1This is where we are searching for the values of a set of numbers that solve a problem, typically expressed as an equation

I.e. The maximisation or minimisation of a functionMore formally:

Find the X that maximises/minimises F(X) given:

X = (x1, x2, x3, ..., xn)

xi is a number

Note that the exact definition of F(X) is problem dependant

Applications, Introduction and Parameter Optimisation Slide 9

Parameter Optimisation – Part 2An example parameter optimisation problem is that of a very long range cannon

We have two parameters that need to be chosen so that we accurately hit a targetWe cannot use the laws of motion as air density and drag need to be accounted for

We can search for the optimal set of parametersHill ClimbingSimulated AnnealingMany, many more....We will cover this topic in this weeks laboratory...

Applications, Introduction and Parameter Optimisation Slide 10

Combinatorial Optimisation

““The process of finding anThe process of finding an

optimal ordering or set fromoptimal ordering or set from

a number of objects or a number of objects or items”items”

Applications, Introduction and Parameter Optimisation Slide 11

Subset Selection – Part 1

Applications, Introduction and Parameter Optimisation Slide 12

Given a set of items, select a subset of the set that optimises some requirementMore formally:

Given X, find a Y X such that F(Y) is a maximum/minimum where X is a set of objects/items

If you can solve the ScalesScales problem you can solve the subset selection problem

Subset Selection – Part 2

Applications, Introduction and Parameter Optimisation Slide 13

A good example of Subset Selection is the Feature Subset Feature Subset Selection Selection problemThis is where we are trying to model a set of data, but have too many features/variables/descriptionWe wish to select the best subset that models the problemThose of you who choose the AIAI option in level 3 will look into this problem

Machine learningClassification

Permutation Problems – Part 1

Applications, Introduction and Parameter Optimisation Slide 14

A permutationpermutation is defined as a shuffling of a set of objectsA permutation of A, B, C, D is B, C, A, DA permutation of 1, 2, 3, 4 is 4, 1, 2, 3If we have N objects then there are N! possible permutations4! = 24, 10! = 3,628,800, 100! 9.333 10157

Permutation Problems – Part 2

Applications, Introduction and Parameter Optimisation Slide 15

Without loss of generality we will assume that our solution is a permutation of the numbers

X = [1, 2, ..., N]We will use the notation Y = (X) to denote that Y is a permutation of XWe formally define the permutation problem as find a Y = (X) such that F(Y) is a maximum/minimumNote that the exact definition of F(Y) is problem dependant

Permutation Problems – Part 3The Travelling Salesman Problem (TSP) Travelling Salesman Problem (TSP) is a classic example of a permutation problem

A sales person has to visit N cities as part of their jobThe aim is to start off at one of the cities, visit each city exactly once and then to arrive back at the starting cityThis is called a tourtourThe objective is to find a tour where the sum of the total distance travelled is a minimumI.e. This is analogous to the sales person trying to minimise their petrol costs

Examples include parcel delivery, general vehicle routing and road grittingWe will cover this topic (TSP) in much more detail laterWe will look into the Eight QueensEight Queens problem when we look at Genetic AlgorithmsGenetic AlgorithmsApplications, Introduction and Parameter Optimisation Slide 16

Data Clustering – Part 1Data Clustering is the process of arranging objects (as points) into a number of sets (k) according to “distance” or similarityEach set will be referred to as a cluster/groupFor the purposes of this module, each set is mutually exclusive, i.e. an item cannot be in more than one cluster

Applications, Introduction and Parameter Optimisation Slide 17

Cluster 1Cluster 1

Cluster 2Cluster 2

Data Clustering – Part 2Why Cluster?

Knowing which objects are highly related to other objects is very useful within data analysisLess complex to modelMay give insight into the unknown properties of some of the objectsA useful pre-processing tool

The number of applications of data clustering is extensive

Gene Expression DataSoftware ModularisationClustering Customer BehaviourEtc...

We will look into data clustering in much more detail in a later lecture

Applications, Introduction and Parameter Optimisation Slide 18

Bin Packing and Data Clustering Slide 19

Bin Packing – Part 1

The bin packing bin packing problem is where a number of n items of size x1,…,xn, need putting into the smallest number of bins (or boxes) of size/capacity c

12

34

5

Capacity=6x1,...,x5

5

1

4

2

3

Bin Packing – Part 2

There are a large number of bin packing applications:

1-dimensional e.g. CD compilations2-dimensional e.g. stock cutting3-dimensional e.g. van packing

Again, we will look into this in more detail later…

Applications, Introduction and Parameter Optimisation Slide 20

Multi Objective OptimisationSome optimisation problems have more than one quality or fitness functionThus with these types of problems we might have more than one way of rating a solutionFor example we might be trying to optimise X = (x1, x2, x3, ..., xn)

We might have a number of fitness function F1(X), F2(X), ..., Fm(X)The problem is how to compare two potential solutions when only some of the fitness function evaluations are betterFor example if we had two potential solutions X1 and X2 and three fitness functions such that:

F1(X1) = 10, F2(X1) = 20 and F3(X1) = 30

F1(X2) = 20, F2(X2) = 10 and F3(X2) = 31

Which solution is better?

Applications, Introduction and Parameter Optimisation Slide 21

Weighted FitnessOne solution to this problem is to assign a numerical weight or importance to the fitness functionsFor example:

This assumes that we can assign a weightWe must have some prior knowledge of how important each function isThis also requires us to be aware of what the ranges of the fitness functions areFor example if one function is always less than 1 and another is always greater than 1,000,000 then equal weights would be pointlessApplications, Introduction and Parameter Optimisation Slide 22

m

iii XFwXF

1

)()(

Pareto Optimality – Part 1Pareto optimality is a very straight forward but highly effective techniqueIf we have two solutions X1 and X2 then we say that X1 Pareto DominatesPareto Dominates X2 is Fi(X1) is better than Fi (X2) for all iWe use the notation:

If neither solution dominates each other then they are Pareto Equivalent Pareto EquivalentThus we could search for a solution to a problem which is not dominated by any other solutionApplications, Introduction and Parameter Optimisation Slide 23

21 XX

Pareto Optimality – Part 2The Pareto Front Pareto Front is the set of Pareto Pareto Equivalent Equivalent solutions that are not dominateddominated by any other solutionMulti Objective Optimisation has a number of applications:

ManufacturingTelecommunications and computer networksLogistics and transportationTimetabling and scheduling

Applications, Introduction and Parameter Optimisation Slide 24

Constraint Satisfaction ProblemsConstraint Satisfaction Problems (CSP) are optimisation problems, where the valid solutions are constrained by a set of conditionsFor example, if we are seeking to optimise:

X = (x1, x2, x3, ..., xn)We might have a set of constraints such as

Applications, Introduction and Parameter Optimisation Slide 25...

54

321

Etc

xx

xxx

x

x

ii

ii

CSP ApplicationsAdvanced Planning and Scheduling

Train schedulingProduction scheduling

Assignment problemsGate allocation for airportsBalancing work among different people

Network management and configurationPlanning of cabling of telecommunication networksNetwork reconfiguration without service interruptions

Database systemsEnsure and/or restore data consistency

Molecular biologyDNA sequencingChemical hypothesis reasoningProtein docking

Electrical engineeringFault locationCircuit layout computation

Applications, Introduction and Parameter Optimisation Slide 26

CSP Example

Map ColouringColour all of the regions in a map so that no adjacent regions are the same colourE.g. xi xj

Sometimes we wish to minimise the number of colours (4!)

Applications, Introduction and Parameter Optimisation Slide 27

CSP StrategiesOften Heuristic Search methods are used to solve CSPGeneral Strategy

Instantiate all variables randomlyUse a repair or update strategy to move towards a more and more complete solutionStop if a complete solution is found

Hill ClimbingModify the value of one variable such that more constraints are satisfiedRestart with a random assignment if no more constraints can be satisfied (local minimum)

Minimising conflicts Randomly choose a variable that is involved with an unsatisfied constraintPick a value that reduces the number of unsatisfied constraintsIf no such value exists pick a random value which does not increase the number of unsatisfied constraints

Applications, Introduction and Parameter Optimisation Slide 28

The LaboratoryWe will be looking at solving the cannon problem using a Hill Climbing AlgorithmWe can re-use code from previous laboratories

Slide 29Applications, Introduction and Parameter Optimisation

Next LectureWe will be looking at Genetic Algorithms in the next lecture

Slide 30Applications, Introduction and Parameter Optimisation