A Simulated-annealing-based Approach for Simultaneous Parameter Optimization and Feature Selection of Back-Propagation Networks (BPN)

Shih-Wei Lin, Tsung-Yuan Tseng, Shuo-Yan Chou, Shih-Chieh Chen

National Taiwan University of Science and Technology

Expert Systems with Applications 2008

Introduction

The back-propagation network (BPN) can be used in various fields. -evaluating consumer loans

-diagnosing heart disease Different problems may require

different parameter settings for network architectures.

Rule of thumb or ‘‘trial and error’’ methods are usually used to determine them.

Introduction

Not all features are beneficial for classification in BPN.

Select the beneficial subset of features which result in a better classification.

Simulated-annealing (SA) -based approach, to obtain the optimal parameter settings for network architectures of BPN.

BPN

Before applying BPN to solve problems as follow:

(1) the parameter settings for network architectures (2) hidden layer number (3) learning rate (4) momentum term (5) number of hidden neurons (6) learning cycle

Feature Selection

The main benefits of feature selection are as follows:

(1) Reducing computational cost and storage requirements (2) Dealing with the degradation of classification efficiency due to the finiteness of training sample sets (3) Reducing training and prediction time (4) Facilitating data understanding and visualization

Problems

While using BPN, we confront two problems:

How to set the best parameters for BPN !

How to choose the input attributes for BPN !

SA-based approach that not only provided the best parameter settings for network architecture of BPN, but also found out the beneficial subset of features according to different problems.

BPN

BPN is a common neural network model whose architecture is the multilayer perceptorns (MLP).

Learning rate of BPN

Learning rate:

1.Too high a learning rate will cause the network architecture to oscillate and be hard to converge.

2.Too low a learning rate will cause slow convergence and may fall into local optimization.

Momentum term of BPN

Momentum term:

1.Too small a momentum term does not have an obvious effect and cannot increase the classification accuracy rate

2.Too big a momentum term can excessively affect the learning effect and cause extreme modification.

Number of hidden neurons of BPN

(3)Number of hidden neurons:

1.When there are too few hidden neurons, it is apt to cause a bigger error

2.Increasing the number of hidden neurons can affect the speeds of convergence and computing with almost no help in reducing errors

Learning cycle of BPN

(4) Learning cycle:

1.Too high a learning cycle will result in over-fitting 2.Too low a learning cycle can lead to too little training and result in a worse classification accuracy rate of testing data.

Some Solution

Search for the optimal weights after training

Search for the optimal parameter settings of BPN

Neural network pruning

Simulated Annealing Proposed by Kirkpatrick

(1985)1. Pick a random assignme

nt2. Make a small change3. Accept change if

cost is decreased; or Other criteria

First used by Kakuno et. al.

Simulated-annealing

Initial

random assignment

Make a small change

Accept?

Update current solution

temperature dropping

temperature dropped

Termination?

Optimized

No

Yes

Yes

Yes

No

No

Simulated-annealing

Solution representation

First variable is the learning rate Second is the momentum term Third is the number of hidden neurons Other is represented as feature selection

Parameters range

SA was set to 300 to find the optimal BPN parameter settings

The learning rate ranged from 0 to 0.45 The momentum term ranged from 0.4 t

o 0.9 The learning cycle of BPN was set as 50

0

Platform

Using the C language Windows XP operating system Pentium IV 3.0 GHz CPU 512 MB of RAM.

Cross-Validation

To guarantee that the present results are valid and can be generalized for making predictions regarding new data

Using k-fold-cross-validation This study used k = 10, meaning that

all of the data will be divided into ten parts, each of which will take turns at being the testing data set.

Datasets

System architecture

10-fold classification result of Breast Cancer dataset

The comparison results of approaches without feature selection

SA + BPN approach with feature selection and other approaches

Experimental results summary of with/without feature selection on datasets

Concusion

We proposed a SA-based strategy to select features subset and to set the parameters for BPN classification.

Compared to the previous studies, the classification accuracy rates of the proposed SA + BPN approach are better than those of other approaches.

Thank YouQ & A