the truth is in there: directions and challenges in...
TRANSCRIPT
The truth is in there: directions and challenges in extracting rulesfrom trained ar tificial neural networks
Alan B. Tickle� [email protected]
Robert Andrews� [email protected]
Mostefa Golea�
[email protected] Diederich
� joachim @fit.qut.edu.au�Neurocomputing Research Centre
Queensland University of TechnologyBox 2434 GPO Brisbane
Queensland 4001, Australiaand�
Department of Systems EngineeringResearch School of Information Sciences and Engineering
Australian National UniversityCanberra
Australian Capital Territory, Australia
AbstractA recognized impediment to the more widespread utilisation of Artificial NeuralNetworks (ANNs) is the absence of a capability to explain, in a human-comprehensible form, either the process by which a trained ANN arrives at a specificdecision/result or, in general, the totality of knowledge embedded therein. Recentlythere has been a proliferation of techniques aimed at redressing this situation byextracting the knowledge embedded in the trained ANN as a set of symbolic rules.This paper shows that the original schema for categorizing ANN rule extractiontechniques proposed by Andrews, Diederich, and Tickle (1995) is able toaccommodate these recent developments in the field. In addition attention is drawn towhat are seen to be some of the key challenges in the area including the identificationof factors which appear to limit what is actually achievable through the ruleextraction process.
1. IntroductionSince their renaissance in the mid-1980s, Artificial Neural Network (ANN) techniqueshave been successfully applied across a broad spectrum of problem domains such aspattern recognition and function approximation (Thrun et. al. 1991, Thrun 1994).However despite the proven capabilities of trained Artificial Neural Networks, to anend user an ANN is a numerical enigma comprising an arcane web of interconnectedinput, hidden, and output units. Moreover an ANN solution manifests itself entirely assets of numbers in the form of activation function parameters and weight vectors. Assuch a trained ANN offers little or no insight into the process by which it has arrived ata given result nor, in general, the totality of “ knowledge” actually embedded therein.This lack of a capacity to provide a “ human comprehensible” explanation is seen as aclear impediment to a more widespread acceptance of ANNs. In particular it is almostthe complete antithesis of what is required if the ANN is to be used for example in“ safety-critical” applications and other similar security related applications.
Recently, considerable effort has been directed towards redressing this situation byproviding ANNs with some form of an explanation capability. In particular a number ofmechanisms, procedures, and techniques have been proposed and developed to extractthe knowledge embedded in a trained ANN as a set of symbolic rules which in effectmimic the behaviour of the ANN. Techniques have also been developed which allow anANN to be used for the refinement of existing rules within symbolic knowledge bases.Recent surveys conducted by Andrews, Diederich, and Tickle (1995), Tickle,Hayward, and Diederich (1996), and Tickle, Andrews, Golea, and Diederich (1996)offered a detailed insight into the modus operandi of a broad cross-section of suchtechniques. In addition to these survey papers, a report produced by theNeuroComputing Research Centre at QUT (Andrews et. al. 1996) provided limitedconfirmation of the efficacy of the overall rule extraction concept by comparing theperformance of a subset of state-of-the-art rule extraction techniques on a (limited) setof benchmark problems.
The proliferation and diversity of techniques for extracting rules from trained ANNshighlight the fact that this is still an evolving discipline. Hence the purpose of theensuing discussion is:
a) to provide a brief critique of the taxonomy introduced in previous surveys forclassifying ANN rule extraction techniques;
b) to assess the current efficacy of these criteria by using them to classify arepresentative sample of the increasingly large number of contributions to thefield which have appeared since the original survey was completed;
c) to review some of the current issues on the theoretical aspects of the ruleextraction task.
The principal finding of this analysis is a confirmation of the overall utility of theexisting taxonomy for categorizing techniques for extracting rules from trained ANNs.Furthermore a set of factors are identified which at this stage appear to limit what isactually achievable through the rule extraction process. In particular one such factor isthe inherent (NP-hard) algorithmic complexity of the basic rule extraction tasks.Specifically this complexity problem means that there are situations in which any ruleextraction technique will fail (Golea 1996a, 1996b). In addition it is also noted that oneof the challenges in the field of rule extraction from trained ANNs is to identifysubclasses of ANNs and subclasses of rules where the rule extraction problem can besolved efficiently (ibid.).
The discussion also examines the merits of extending the scope of the rule extractionprocess to include the separate identification of what are termed causal factors andfunctional dependencies (and in particular, “ left reduced functional dependencies” )between the attributes from a problem domain and the corresponding decision/ result.It is argued that this strategy offers the potential to improve the tractability of some ofthe existing ANN rule extraction techniques by focusing the search for solutions onthose regions of the problem domain of most significance. Finally it is concluded that,at this stage, no compelling evidence has emerged which mandates the use of aparticular type of ANN architecture and/ or a particular type of rule extractiontechnique in a given class of problem domains.
2. A review of the salient points from previous surveys of ANN ruleextraction techniquesIn the original survey paper of Andrews et. al. (1995), a taxonomy was introduced forcategorizing the wide variety of techniques which have been developed for extractingrules from trained Artificial Neural Networks (ANNs). This taxonomy incorporated atotal of five primary classification criteria viz.
a) the expressive power of the extracted rules;b) the translucency of the view taken within the rule extraction technique of the
underlying Artificial Neural Network units;c) the extent to which the underlying ANN incorporates specialized training
regimes (i.e. a measure of the portability of the rule extraction technique acrossvarious ANN architectures);
d) the quality of the extracted rules; ande) the algorithmic complexity of the rule extraction/rule refinement technique.
These criteria for categorizing rule extraction techniques will now be reviewed brieflyin turn.
2.1 A criterion for categorizing rule extraction techniques based on theexpressive power of the extracted rulesThe primary dimension proposed by Andrews et. al. (1995) for classifying techniquesfor extracting rules from trained ANNs was termed the “ expressive power of theextracted rules” . In essence the purpose in using this dimension is to focus theclassification process on the output of the rule extraction process viz. the rule setsthemselves. The net effect of classifying rule extraction techniques according to thiscriterion is to coalesce into one grouping those techniques which generate sets ofif...then...else rules based on conventional (i.e. Boolean) symbolic logic. Similarly aseparate and distinct group is formed from those techniques which express the rulesextracted from the trained ANN using the concepts drawn from the field of fuzzy logic.A characteristic feature of rules of this latter type is that they use set membershipfunctions to deal with what are called “ partial truths” . (For example a fuzzy rulecould be expressed in the form if x is low and y is high then z is medium where low,medium, and high are fuzzy sets with corresponding membership functions.)
In formulating this categorization criterion, the authors also pre-empted the need for athird category of rule extraction techniques to accommodate rules represented in first-order logic form i.e. rules with quantifiers and variables. At the stage the survey wasundertaken, no ANN rule extraction techniques were found which came with the ambitof this category. However its inclusion was deemed important because of thecorrespondence with related work in the overall field of Artificial Intelligence.
As evidenced by the fact that a significant number of the publications in this fieldexplicitly nominate the type of rules being extracted (and, in particular, frequently inthe title of the work) this criterion has proven to be a simple but highly effective basisfor classifying rule extraction techniques.
2.2 A criterion for categorizing rule extraction techniques based ontranslucencyA second method of categorizing techniques for extracting rules from trained ANNs isbased on the perception (or “ view” ) of the granularity of the underlying ANN whichis either explicitly or implicitly assumed within the rule extraction technique. Andrewset. al. 1995 assigned the label “ translucency” (Figure 1) to describe the spectrum ofsuch perceived degrees of granularity.
Figure 1: Using the criterion of " translucency” to categorize ANN rule extractiontechniques, the two end points of the continuum are the “ decompositional”
techniques and the “ pedagogical” techniques respectively. Essentiallypedagogical techniques treat the ANN as a “ black box” and extract global
relationships directly. Alternatively, decompositional techniques first extractrules at the individual unit level and then aggregate these rules to form global
relationships.
Under this scheme of categorization of rule extraction techniques, at one end of thespectrum are positioned those rule extraction techniques which view the underlyingANN at the maximum level of granularity i.e. as a set of discrete hidden and outputunits. Craven and Shavlik (1994) categorized such techniques as “ decompositional”and Andrews et. al. adopted this nomenclature in their survey paper. The basic motif ofsuch decompositional rule extraction techniques is to extract rules at the level of eachindividual hidden and output unit within the trained ANN(e.g. at the level of individualhidden an output units within a classic/ conventional backpropagation ANN asdescribed in Rumelhart and McClelland 1986). In general, decompositional ruleextraction techniques incorporate some form of analysis of the weight vector andassociated bias of each unit in the trained ANN. Then, by treating each unit in theANN as an isolated entity, decompositional techniques initially yield rules in which theantecedents and consequents are expressed in terms which are local to the unit fromwhich they are derived. A process of aggregation is then required to transform theselocal rules into a composite rule base for the ANN as a whole.
Figure 2: An example of a decompositional technique operating at the individualunit level. The extracted rules show subsets of inputs which exceed the bias.
(Based on Towell and Shavlik 1993)
Two examples of decompositional rule extraction techniques cited both by Craven andShavlik (1994) and Andrews, Diederich, and Tickle (1995a,b) are the KT ruleextraction algorithm (Fu 1991, 1994) and the Subset algorithm (Towell and Shavlik1993). In particular the initial phase of the Subset algorithm proceeds in a mannerdescribed previously (Figure 2) with sets of rules being extracted at the level ofindividual (hidden and output) units within the trained ANN. In this case the process ofanalysing the weight vectors for a given unit takes the form of a search for subsets ofpositively weighted incoming links for which the summed weights exceed the bias onthe unit (Figure 2). In the second phase of the Subset algorithm these separate sets ofrules are then aggregated to form a composite rule base for the ANN as a whole.
In contrast to the decompositional approaches, the motif in the pedagogicalapproaches is to view the trained ANN at the minimum possible level of granularity i.e.as a single entity or alternatively as a “ black box” . The focus is then on finding rulesthat map the (ANN) inputs (i.e. the attribute/value pairs from the problem domain)directly into outputs (e.g. membership-of or exclusion-from some target class). In theirsurvey paper, Andrews, Diederich, and Tickle cite the Validity Interval Analysis (VIA)algorithm of Thrun (1994) and the rule-extraction-as-learning technique of Cravenand Shavlik (1994) as two examples of what might be considered as the epitome ofpedagogical approaches to extracting rules from a trained ANN.
In addition to these two main categories of rule extraction techniques, Andrews et. al.also propose a third category which they labelled as “ eclectic” to accommodate thoserule extraction techniques which incorporate elements of both the decompositional andpedagogical approaches. In reality this is a somewhat diffuse group. This can beillustrated by the fact that an example of an eclectic technique is the variant of rule-extraction-as-learning technique of Craven and Shavlik (1994) which utilies the(decompositional) KT algorithm of Fu (1991,1994) in the role of determining if a givenrule is consistent with the underlying ANN network. Andrews, Diederich, and Tickle(1995) also nominate the DEDEC approach (Tickle, Orlowski, Diederich, and Finn1995; and Tickle, Orlowski, and Diederich 1996) as another example of an ANN ruleextraction technique which falls within the ambit of the eclectic category.
In conclusion, notwithstanding the “boundary” problem in defining the eclectic group,at an overall level the basic decompositional/ pedagogical taxonomy proposed by theauthors for categorizing ANN rule extraction techniques appears to have beenembraced by people working in the field.
2.3 A criterion for categorizing rule extraction techniques based ontheir portability across different ANN architecturesA characteristic feature of techniques for extracting rules from trained ANNs is thepreponderance of what might be termed as “ specific purpose” techniques i.e. thoserule extraction techniques where the ANN has been specifically constructed tofacilitate the rule extraction process or where the rule extraction technique is specificto a particular ANN architecture. Examples of such techniques include RULEX(Andrews and Geva 1994, 1995), KBANN/ M-of-N (Towell and Shavlik 1993), andBRAINNE (Sestito and Dillon 1991, 1992, and 1994). Importantly, previous surveyshave noted the comparatively high level of success which was reported for suchtechniques in terms of the relative “quality” (see below) of the solutions.
Notwithstanding this situation, in their survey Andrews et. al (1995) commented onthe clear need for a portfolio of rule extraction techniques which could be applied ipsofacto to trained Artificial Neural Network (ANN) solutions already in situ.
2.4 A criterion for categorizing rule extraction techniques based on thequality of the extracted rulesApart from categorizing ANN rule extraction techniques on the basis of their modusoperandi and the expressive power of the rules they produce (e.g. Boolean or“ fuzzy” ), a third criterion for categorizing ANN rule extraction techniques is on thebasis of the “ quality” of the rules produced. In particular the set of criteria forevaluating rule quality adopted by Andrews et .al. is an extension of an initial setproposed by Towell and Shavlik (1993) and includes:
a) accuracy; andb) fidelity; andc) consistency; andd) comprehensibility
of the extracted rules.
Using the proposed terminology a rule set is considered to be accurate if it cancorrectly classify a set of previously unseen examples from the problem domain.Similarly a rule set is considered to display a high level of fidelity if it can mimic thebehaviour of the Artificial Neural Network from which it was extracted by capturingall of the information embodied in the ANN. Also under these designated criteria, anextracted rule set is deemed to be consistent if, under differing training sessions, theArtificial Neural Network generates rule sets which produce the same classifications ofunseen examples. Finally the comprehensibility of a rule set is determined by measuring
the size of the rule set (in terms of the number of rules) and the number of antecedentsper rule.
2.5 A criterion for categorizing rule extraction techniques based on thecomplexity of the core algorithm(s)The inclusion of a separate criteria on the algorithmic complexity of the various ANNrule extraction techniques is a reflection of algorithmic complexity as an issue in allcomputational processes. For example within decompositional approaches to ruleextraction such as the KT algorithm (Fu 1991, 1994), the problem of algorithmiccomplexity arises because the basic process of searching for subsets of rules at the levelof each (hidden and output) unit in the trained ANN is exponential in the number ofinputs to the node. Consequently heuristics are invoked to limit the depth to which thespace is explored (Craven and Shavlik 1994). In a similar manner, various heuristicsare employed in pedagogical approaches such as the rule-extraction-as-learningalgorithm and the VIA algorithm to obviate the need to enumerate all possibleexamples in the problem space in search of solutions.
While algorithmic complexity provides a useful measure of the efficiency of the ruleextraction process, few authors report or comment on this issue. The notableexceptions reported are in the discussion of the KT algorithm of Fu (1991,1994), theVIA algorithm of Thrun (1994) and the rule-extraction-as-learning algorithm ofCraven and Shavlik (1994). In each of these situations the authors all took fullcognizance of algorithmic complexity into consideration in the formulation andoperation of their respective rule extraction techniques.
3. Validation of the taxonomy in the light of new techniques and ideasfor ANN rule extractionBecause the field of rule extraction from trained ANNs is currently one which isattracting considerable interest, in the intervening period since the initial surveys ofAndrews et. al. (1995a,b) were undertaken, there has been a significant number of newtechniques proposed both for extracting symbolic rules from trained ANNs and theallied field of utilizing an ANN to refine an existing knowledge base. Rather thanattempt to review all of these developments in isolation, it is useful to assess theefficacy of the criteria discussed in the previous section in categorizing a cross sectionof such developments. For example:
3.1 Fuzzy ARTMAP (Carpenter and Tan 1995)In terms of the taxonomy discussed previously, the rule expressive power of the FuzzyARTMAP algorithm (Carpenter and Tan 1995) is “fuzzy” because the focus is onextracting rules of this type. In addition using the classification criterion of“translucency” the algorithm belongs to the “decompositional” category because acharacteristic feature of a Fuzzy ARTMAP system (and one which the authors wereable to exploit to their advantage), is that in Fuzzy ARTMAP, each (category) noderoughly corresponds to a rule. Furthermore the weight vector associated with eachnode can be directly translated into a verbal or algorithmic description of the ruleantecedents. This is in contrast to a classic/ conventional backpropagation network
where the role of each hidden unit in the total classification process is not usually asexplicit.
In terms of the criterion based on the “quality” of the rules produced by the FuzzyARTMAP system, Carpenter and Tan focus on the accuracy of the rules extractedfrom two benchmark problem datasets i.e. the Pima Indian Diabetes dataset and theEdible Mushroom dataset, both of which were drawn from the repository of machinelearning databases at the University of California at Irvine (UCI) (Murphy 1995). Inaddition they extracted a solution from a more recent variant of the DNA promoterdataset. (An earlier version of this dataset is also available from the UCI datasetrepository.)
While this technique is readily classified using criteria such as rule expressive power, atan overall level, the Fuzzy ARTMAP rule extraction technique is another example of asituation in which a highly effective rule extraction algorithm has been designed towork in conjunction with a specific supervised learning ANN (i.e. the Fuzzy ARTMAPSystem). As such the Fuzzy ARTMAP technique is not designed to be portable acrossdifferent ANN architectures (i.e. it does not have general utility).
3.2 Sensitivity analysis (Viktor et. al. 1995)In a similar vein, the expressive power of the Sensitivity Analysis algorithm (Viktor et.al. 1995) is directed towards representing the knowledge embedded in the trainedANN in terms of conventional (i.e. Boolean) symbolic logic. As in the previousexample, the Sensitivity Analysis technique is also an example of a decompositionaltechnique. This is because the basic rule extraction algorithm draws heavily on the ideafirst proposed by Towell and Shavlik (1993,1994) of using equivalence classes tocluster similarly weighted links (or inputs) to each individual unit in the trained ANN.As such it relies on two key assumptions about the underlying trained ANN viz.
1) “non-input” units are either maximally active (i.e. activation is near to 1) orinactive (i.e. activation is near to 0). [This requires that the steepness parameter s
in the standard logistic activation function is typically ≥ 5];
and2) each hidden and output unit in the trained ANN implements a symbolic rule.
While both of these assumptions tend to limit the general utility of the overalltechnique, the primary advantage of the clustering technique is that it obviates anumber of problems of tractability typically associated with search-baseddecompositional techniques.
As with the Fuzzy ARTMAP rule extraction technique discussed previously, Victor et.al. only assess the quality of the rules extracted by their technique in terms of theiraccuracy. Unfortunately even on this basis, comparisons with other rule extractiontechniques are limited by their decision to use a problem domain (i.e. the trainclassification problem) for which results are not reported by other workers in the field.
3.3 Extension to knowledge representation techniques (Dillon et al.1996)The primary overall aim in the work of Dillon et al. 1996 is take a more expansiveview of the process of eliciting the knowledge embedded in a trained ANN and relate itto the more general issues of knowledge representation and knowledge refinement(Diederich 1992). In particular, as part of their approach Dillon et. al. utilize aknowledge representation technique (SHRUTI) (Shastri and Ajjanagadde 1993) whichincorporates the notions of reflexive reasoning and the dynamic binding of variables.(For example in the backward reasoning process within SHRUTI, variables appearingin the antecedent must also appear in the consequent). The role of the ANN in thisapproach is to use data from the problem domain to refine existing knowledge whichhas been used to initialize an ANN. Similarly the rules extracted initially from a trainedANN can be transformed into the representation of more general concepts within aSHRUTI system.
In its current stage of developmemnt, this system is based on a restricted form of Hornlogic which, in turn, can be considered as being a subset of First Order Inductive Logic(FOIL). Hence in terms of the classification criteria proposed by Andrews, Diederich,and Tickle, this work is an example of the type of approaches envisaged in the originalsurvey paper in which the expressive power of the extracted rules is in a form whichtrancends Boolean or fuzzy rule representations and which begins to approximate“first-order logic” form.
4. Some current issues and recent developments in rule extraction fromANNsOne of the main challenges in the field of eliciting knowledge from trained ANNs is theformulation of a consistent theoretical basis for what has been, until recently, adisparate collection of empirical results. Other challenges include addressing andresolving issues such as:
4.1 Algorithmic complexity may limit the generality of the ruleextraction processWhile the algorithmic complexity of the rule extraction process is one of the primarycriteria proposed for categorizing rule extraction techniques, a number of authorsincluding Fu (1991,1994), Towell and Shavlik (1993), Craven and Shavlik (1994),Thrun (1994), and Viktor et. al. (1995), have commented on the potential problemsarising from the algorithmic complexity of various implementations of the ruleextraction process. Further such authors have typically shown how a variety ofheuristics may be employed to achieve tractability in “real world” problem domains.
Golea (1996a) extended this initial empirical work in an endeavour to identify issuesrelating to the intrinsic complexity of the rule-extraction problem. The two key resultswere that, in the worst case, extracting:
a) the minimum Disjunctive Normal Form (DNF) expression from a trained (feed-forward) ANN; and
b) both the best monomial rule and the best M-of -N rule (Towell and Shavlik1993) from a single perceptron within a trained ANN
are all NP-hard problems. In addition Golea (1996a) showed that the complexity ofone of the key algorithms in the Rule-extraction-as-learning technique of Craven andShavlik (1994) was also NP-hard as distinct from the O(n) originally suggested by theauthors.
Overall the important implication of these results is that it raises the clear prospect thatnotwithstanding successful solutions being found in specific situations, there may bylimits to the generality of rule extraction techniques.
4.2 Implicit assumptions about the “ goodness-of-fit” of the ANNtraining process and solution may be violatedIn most ANN architectures, the initial values for the weight vectors which ultimatelycharacterize an ANN solution are randomly assigned within the ANN trainingalgorithm. Hence the result of each separate instance of the ANN training process isnormally a unique ANN solution. Moreover during the ANN training phase thegoodness-of-fit (or alternatively the “quality” - although this term has already beenused in the context of evaluating the rules extracted from the ANN) of the ANNsolution is assessed using a number of different measures. The first is the size of theresidual error in applying the ANN solution to the set of training data i.e. the trainingset. The second is the size of the generalization error in applying the ANN to a set ofpreviously unseen examples from the problem domain i.e. a so-called test or validationset. A third measure of the goodness-of-fit of the ANN solution is the number ofhidden units required in the resultant ANN. Under most (if not all) ANN trainingregimes, some trade-off between these three measures (i.e. residual error,generalization error, and number of hidden units) is often required particularly insituations involving noise in the training set. However in most cases the implicit or insome cases explicit goal of the training process is to find an ANN solution with theminimum number of hidden units consistent with satisfying certain threshold criteria forthe residual/generalization errors.
An issue which has not been expanded upon in previous survey papers is the extent ofthe dependence of the efficacy of rule extraction techniques on the goodness-of-fit ofthe ANN solution (and, by extension, the algorithm used in the ANN training process).For example because of their algorithmic complexity, the tractability of mostdecompositional approaches to rule extraction (and specifically those which involvesome form of search process to find the possible rule sets) is heavily dependent on theANN having as close as practical to the minimum number of hidden units in the finalconfiguration. More importantly such decompositional techniques are criticallydependent on each (hidden and output) unit possessing a separate and distinct“meaning” or alternatively representing a single concept or feature about the problemdomain (Viktor et. al. 1995, Golea 1996a, 1996b).
However a recent result by Bartlett (1996) raises certain problems in this regard. Inparticular Bartlett (1996) has shown that the generalization error (ε) of a trained ANNcan be expressed in the form:
ε
wheren is the number of cases in the training set; and
i.e. the sum of the absolute values of the weights w for each (hiddenand output) unit in the trained ANN is bounded by some positive constant A.
This result shows that an ANN solution may be found which exhibits goodgeneralization behaviour (i.e. have an acceptably low generalization error ε) whilstbeing sub-optimal with respect to having as close as possible to the minimum numberof hidden units (Golea 1996b). An important corollary of this finding is that an ANNsolution could be found with good generalization capability in problem domainsinvolving binary classifications but in which some or all hidden units do not possesseither a separate and distinct “ meaning” . Moreover a given hidden or output unit maynot necessarily represent a single concept or feature about the problem domain. Assuch the result of Bartlett (1996) has important implications for all rule extractiontechniques and in particular decompositional rule extraction techniques which requireas a prerequisite the existence of a separate and distinct meaning being associated witheach unit in the ANN.
Earlier discussion on a set of criteria for evaluating the quality of a set of rulesextracted from a trained ANN defined “consistency” to reflect the degree of similaritybetween rule sets resulting from different training sessions of the ANN. From theviewpoint of rule extraction the principal ramification of this result is to highlight theneed for attaching a greater degree of importance to the consistency of an extractedrule set as defined previously. Moreover it also reinforces the point (e.g. in Andrewset. al. 1995) that one of the potential roles of rule extraction techniques is in being ableto identify ANN solutions which are sub-optimal.
4.3 In certain situations it may not be possible to achievesimultaneously high levels of accuracy, fidelity, and comprehensibilityin the rule extraction processRecently, rule extraction techniques have been applied in an increasingly diverse rangeof problem domains. This increased exposure has also brought to light a potentialconflict in attempting to maximize simultaneously the fidelity, accuracy, andcomprehensibility criteria for evaluating the quality of the rules extracted from atrained ANN. It is possible to illustrate the general nature of this problem in thefollowing way (Golea 1996b) (Figure 3) (Recall that rule accuracy is a measure of thecapability of the extracted rules to classify correctly a set of previously unseenexamples from the problem domain. Rule fidelity is a measure of the extent to whichthe extracted rules mimic the behaviour of the ANN from which they were drawnwhereas rule comprehensibility is assessed in terms of the size of the extracted rule setand the number of antecedents per rule):
Figure 3: For problem domains with small/ restr icted (or unrepresentative)training sets, substantial differences may exist between the target function andthe generalization of an ANN solution (i.e. the hypothesis). In this case, the goalof extracting rules with high fidelity, accuracy, and comprehensibility may proveto be elusive.
Let I be the complete input space for a given problem domain and let D be adistribution defined on I where D(x) represents the probability of seeing a given x in I.Let f(x) be the function which actually maps or classifies a set of training cases drawnfrom the problem domain (i.e. the function which is the “target” for the ANN training).In addition let h(x) be the functional representation of the ANN solution and let R bethe set of rules extracted from the ANN.
On the basis of the definitions presented above, the set of extracted rules R exhibits ahigh level of fidelity with respect to the ANN solution h if R can act as a surrogate forh (i.e. R and h can be used interchangeably). Hence a set of extracted rules R with ahigh level of fidelity will be as accurate as the ANN itself in classifying previouslyunseen examples from the problem domain. (In passing it should also be noted that anumber of authors e.g. Towell and Shavlik 1993 have reported situations in which theextracted rule set exhibits a better generalization performance than the trained ANNfrom which the rule set was extracted.)
A standard measure of the accuracy of the ANN solution (h) (which results from theANN training process involving the given training set) is given by the generalizationerror (ε) in using h (i.e. the ANN solution) as a surrogate for the target function f viz.
.
Hence, if the generalization error ε is small, a set of extracted rules R with a high levelof fidelity will simultaneously exhibit a high level of rule accuracy (as defined above).Moreover, R can therefore act as a surrogate for the original target function f.
However if the distribution D is localized in some region of the input space I then it ispossible to synthesize an ANN solution h for which the generalization error ε is smallbut for which h is neither equal to nor perhaps even close to f. Importantly, if thefunction h is significantly more complex than f, then the extracted rule set R will exhibithigh levels of accuracy and fidelity but a correspondingly lower level ofcomprehensibility than the set of rules extracted, for example, by applying a symbolicinduction algorithm directly to the data in the original training set. Hence one of thepotential benefits in extracting rules from a trained ANN is the ability to identifysituations in which certain regions of an input space are not sufficiently represented inthe set of data used in the ANN training process (Andrews et. al. 1995).
4.4 In certain situations, some merit may exist in extracting causalfactors and functional dependencies as a precursor to rule extractionfrom trained ANNsTo date almost the entire focus of the discussion regarding task of eliciting knowledgefrom trained ANNs has centred on the task of extracting as expeditiously as possible, aset of (symbolic) rules which capture and explain the quiddity of the trained ANN.Moreover, once extracted, such rules can in certain circumstances be used as asurrogate for the trained ANN.
While these are laudable goals, an argument exists that they are unnecessarilyrestrictive and fail to exploit the full potential of the capabilities of ANNs to discoverpreviously unknown dependencies and relationships in datasets. Moreover, in a rangeof applications such as those found in the field of information security, a useful interimresult from an end-user perspective might simply be one which identifies whichattributes, and in particular which combination of attributes from the problem domain,are the most significant (or, alternatively, the least significant) determinants of thedecision/ classification. Further, Holte (1993) showed that for broad range of datasetsinvolving in some situations a large number of attributes, it is frequently possible toclassify cases from the problem domain to an acceptable level of accuracy based ontwo or fewer dominant attributes (ibid.).
The task of identifying significant variables in relationships is not unique to the ANNrule extraction problem. In particular a numerous and diverse range of techniques hasbeen developed which are designed to enable the dominant attributes in a problemdomain to be isolated. This range includes multiple regression analysis and discriminantanalysis developed within the field of statistics as well as the determination of“reducts” in rough set theory (Pawlak 1991).
In the specific context of eliciting knowledge from trained ANNs, Garson (1991)focused on the task of determining the relative importance of input factors used by theANN to arrive at its conclusions. Garson (1991) termed these as “ causal factors”between ANN inputs and outputs. In a similar vein Tickle, Orlowski, Diederich, andFinn (1995) and Tickle, Orlowski, and Diederich (1996) identified certain parallelsbetween the process of identifying causal factors in a trained ANN as proposed byGarson (1991) and the processes for identifying keys, superkeys, and in particularfunctional dependencies (see, for example, Korth and Silberschatz 1991) within therealm of relational database design. (Specifically, for a given problem domain and a set
of cases drawn from the problem domain where each case comprises a set ofattribute/value pairs, then in essence a functional dependency (FD) X → Y { read Xdetermines Y} exists if the value of the attribute Y can be uniquely determined fromthe values of attributes belonging to the set X. More precisely, the functionaldependency X → Y is satisfied if for each pair of cases in a given problem
domain then, if (i.e. the set of attributes X for cases are
equivalent) it follows that ).
Both the notion of a causal factor as introduced by Garson (1991) and the concept of afunctional dependency as has been previously discussed can be viewed as one in whicha conceptual relationship is established between the domain attributes and the decisionattribute(s). However a functional dependency relies on having discrete values for theattributes whereas a causal factor as described by Garson can embrace both discretevalued and continuously valued attributes. Moreover the notion of a rule can be viewedas an extension of the functional dependency concept to the point of expressing arelationship between specific values of the domain attributes and the decisionattribute(s) (Geva and Orlowski 1996).
In any given problem domain there could exist numerous valid functionaldependencies. Hence in a relational database context, attention is primarily focused ondetermining only what are termed left-reduced functional dependencies i.e. thosewhich possess the property that while the functional dependency X → Y is satisfied,any proper subset R ⊂ X is not sufficient to determine Y. Applying these concepts inthe realm of ANNs, it is expected that a trained ANN would not necessarily reflect allof the possible functional dependencies in a given dataset. This is because the intrinsicnature of the ANN training process is to give prominence to those attributes/values orcombinations of attributes/values which lead to a global error minimization.
In both the relational database context and also in the context of applying thefunctional dependency concept to the task of eliciting knowledge from trained ANNs,the identification of the set of left reduced functional dependencies is importantbecause the goal is to identify and eliminate superfluous/ insignificant attributes. Inaddition eliminating such attributes augurs well for ultimately determining a set of ruleswith the minimum number of antecedents (Geva and Orlowski 1996).
As indicated previously in certain applications, the identification of causal factors and/or functional dependencies, may of themselves provide considerable insight into theproblem domain for the end user. However this may not be the only benefit. In thecontext of rule extraction from trained ANNs, one of the issues upon which commenthas already been made is the complexity of the various rule extraction algorithms. Inparticular as has already been indicated, the algorithmic complexity of rule extractionalgorithms such as KT and Subset is exponential in the number of ANN inputs (i.e. thenumber of problem domain attributes). Hence reducing the number of attributes byeliminating those which are irrelevant in determining the decision, has the potential ofmaking a direct impact on the tractability of such algorithms by significantly reducingthe size of the search space (Pawlak 1991, Geva and Orlowski 1996). In addition aswill be shown in the ensuing chapter, this approach has the potential to obviate one ofthe key problems in both the VIA (Thrun 1994) and the rule-extraction-as-learning
(Craven and Shavlik 1994) techniques discussed earlier viz. finding an initial set ofspecific cases which can then be use to synthesize more general rules.
4.5 Rules for classifying the initial training data vs. the totality ofknowledge in the trained ANNAnother issue in relation to the extraction of rules from a trained ANN is theobservation that only a subset of the total set of rules extracted may actually expressthe knowledge within the ANN required to classify the set of data used to train theANN initially. The remaining rules essentially reflect the capability of the ANN togeneralize beyond the attribute values corresponding to the initial training data. Thedilemma is that in certain situations the rules representing the totality of knowledgeembedded in the trained ANN may be more specific and in other cases more generalthan those required to classify just the original training data. Moreover as wasdiscussed in a previous section, a potential conflict exists in endeavouring to achievesimultaneously high levels of accuracy, fidelity, and comprehensibility of a set ofextracted rules in problem domains where the distribution of data points/cases islocalized with respect to the total input space. Hence a useful interim result could be toisolate and identify that subset of the extracted rules which are required to classify thedata used in the ANN training process.
An additional consideration in this context is that there exists a rich and diverse rangeof techniques called symbolic induction algorithms have been designed and developedspecifically to extract knowledge in the form of decision trees (Quinlan 1993, Holte1993), and symbolic rules (Quinlan 1993, Orlowski 1993, Geva 1995) directly fromdata. Hence a potentially useful result is to try to establish an equivalence between therules extracted by taking the ANN-rule extraction route and those obtained by applyingsymbolic induction algorithms such a C4.5 directly to the data. By isolating the (sub)set of rules which reflect the knowledge used by the ANN to classify the training data,this would provide a useful basis of comparison on the relative efficacy of the twoapproaches.
4.6 At this stage there appears to be no best way of matching ruleextraction techniques to a given problem domainNotwithstanding the overall utility in having a set of criteria such as those discussedpreviously for categorizing techniques for extracting rules from trained ANNs, oneissue that remains unresolved is whether either this (complete) set of criteria or perhapssome subset is sufficient to enable a user to make a decision a priori (rather than ipsofacto) as to which technique is optimum for a given situation. An initial attempt(Andrews et. al . 1996) at resolving this issue involved subjecting a cross-section oftechniques to a limited series of benchmark tests and comparing the results using threeof the four criteria for rule quality viz. accuracy, fidelity, and comprehensibility.
The results obtained from this initial exercise were regarded as being inconclusive. Inparticular they did not reveal any inherent superiority of those rule extractiontechniques which were designed to be used in conjunction with specific ANNarchitectures as distinct from general techniques intended for use with any ANN
architecture. In addition the results of the tests appeared to offer no clear basis fordifferentiating the relative merits of decompositional vs. pedagogical approaches.
4.7 Rule-extraction from ANNs and connectionist knowledgerepresentationAs explained earlier the “translucency” dimension for classifying ANN rule extractiontechniques is designed to reveal the relationship between the extracted rules and theinternal architecture of the trained artificial neural network (ANN). This dimensioncomprises two basic categories of rule-extraction techniques viz “decompositional” and“pedagogical” and a third - labelled as “eclectic” - which combines elements of the twobasic categories.
In particular the distinguishing characteristic of the “decompositional” rule-extractionapproach is that the focus is on rules at the level of individual (hidden and output) unitswithin the trained ANN. Hence the “view” of the underlying trained ANN is one of“transparency” . In addition the rules, the motif of decompositional rule extraction isthat the sets of rules extracted at the individual unit level are then aggregated to formthe composite rule base for the ANN as a whole.
The translucency dimension - “pedagogical” is given to those rule-extractiontechniques which treat the trained ANN as a “black box” i.e. the view of the underlyingtrained ANN is “opaque”. The core idea in the “pedagogical” approach is to “viewrule-extraction as a learning task where the target concept is the function computed bythe network and the input features are simply the network's input features (Craven &Shavlik, 1994). Hence the “pedagogical” techniques aim to extract rules that mapinputs directly into outputs. Where such techniques are used in conjunction with asymbolic learning algorithm, the basic motif is to use the trained ANN to generateexamples for the learning algorithm.
“Pedagogical” or learning-based rule-extraction techniques are attractive because theymake no assumption about the underlying neural network architecture and are offeredas an efficient alternative to decompositional algorithms (Craven & Shavlik, 1994).However, more recently, Golea (1996a) pointed out that the time-complexity of theCraven & Shavlik (1994) algorithm is not polynomial in the worst-case and even moreimportantly, that rule-extraction from single-layer perceptrons should generally beeasy. However, Golea (1996b) has shown that extracting the best M-of-N rule from asingle-layer perceptrons is hard. These results make decompositional algorithmsgenerally attractive. A decompositional algorithm could decompose a multi-layernetwork is into single-layer networks followed by the application of decompositionalrule extraction methods.
Unfortunately, the situation is more complicated as the following discussion indicates.In particular a pedagogical rule-extraction algorithm accepts as input a data set D withtwo subsets: D1 which is the original training data and D2 which is the data generatedby the neural network via generalisation. Typically D2 is larger than D1 and crucial forthe evaluation of the pedagogical rule extraction. Moreover in order to achieve good“fidelity” , the pedagogical rule extraction has to learn D2 particularly well in order toclosely approximate the neural network.
A decompositional rule extraction-type algorithm accepts as input the neural networkN and possibly the data set D1. The network is a set of hidden unit and output unitrepresentations (i.e. weight vectors including thresholds). One or more hidden andoutput units are decompiled into rules which may later be combined and simplified toarrive at a more comprehensible rule set. Note that hidden units and output units areassumed to be compact representations which can be problematic as discussed earlierin the context of Bartlett (1996).
Lets compare a decompositional rule extraction method with a symbolic learningalgorithm. The symbolic algorithm can be “knowledge-intensive” i.e. does notrepresent learning from scratch but might include pre-defined representations. For afair comparison, we have to compare the decompositional rule extraction method witha knowledge-intensive symbolic learning method such as EBG (Mitchell et. al. 1986)or ILP (Quinlan 1996) because the decompositional rule extraction algorithm accepts a“representation” as input: the hidden units and output units of the trained artificialneural network (ANN).
To make this point clearer assume the task is recognise the object “chair” . Thesymbolic algorithm might use the predicates “edge(pos_1, pos_2)” and “bar(pos_1,pos_2, pos_3, pos_4)” where the “pos” variables can take any value. The crucial pointis that the decompositional rule extraction is disadvantaged because the hidden unitand output unit representations it takes as input (the weight vectors) are primitivecompared to the prior knowledge of the symbolic algorithm (the predicates “edge” and“bar”).
A system that can take advantage of the generic predicates “edge” and “bar” cancombine these to form “leg” and “seat” representations (which are part of a “chair” ).Hence, these predicates help to recognise a chair at various positions, rotations etc.Learning is easier if these predicates can be used because they represent part of thesolution of the learning problem. But it is important to note that the generic predicatesare not only an advantage for learning: The symbolic system can do questionanswering, i.e. can answer questions like “How many legs has a chair?” and “What arethe parts of a chair?” .
From this perspective, a decompositional rule extraction is inherently disadvantagedcompared to a symbolic induction algorithm. However, there are at least two solutionsto the problem:
1. make use of a connectionist knowledge representation system which providesthe "expressiveness" of a symbolic algorithm; or
2. utilise modular networks, with one network pre-processing for another network.(For instance, one network recognises “edges” and “bars” and another combinesthese.)
4.8 Developments in rule refinementOf increasing interest and importance in the field of connectionist inductive learning isthe ability to utilize prior knowledge to bias the learning of a neural network. There
are four places in which it is possible to program prior knowledge into the network,viz,
a) the network architecture;b) the choice of learning algorithm;c) the activation function/s of hidden and output units; andd) the numeric connection weight parameters.
Clearly every time a neural network is constructed certain a priori knowledge isutilized in the form of input feature selection, and choice of learning algorithm,activation functions and network architecture. These choices are generally guided byknowledge of the problem domain. For instance if the problem to be learned is thegrammar of a regular language then a recurrent network may be a more suitablearchitecture than a 2 layer feedforward network.
A problem that has proven more difficult is the proper utilization of existing domainspecific symbolic knowledge. Giles and Omlin (1993) state that the utilization of suchknowledge can
a) simplify the learning task (by allowing other than a tabula rasa starting point forlearning);
b) yield better generalisation performance; andc) allow validation or correction of the prior knowledge.
Two further points need to be made about the utilization of symbolic knowledge of theproblem domain. Firstly, the network training phase provides for not only thevalidation and correction of the prior knowledge, but may also add new knowledge tothe partial domain theory used to initialise the network. Secondly, as well assimplifying the learning task it is possible that the initialisation of the network with apartial domain theory will allow the network to discover a different solution to thatwhich it would have discovered had a tabula rasa configuration been adopted.
Rule refinement is a three step process that allows for the utilization of prior symbolicknowledge, (given in the form of a rule base), by a neural network.. Rule refinementinvolves encoding a given rule base into the network before training begins, thentraining the network with examples drawn from the problem domain, and finally theextraction of the refined knowledge after training has been completed.
Fu (1989) describes a method for utilizing prior symbolic knowledge by transformingthe rule base and inference engine of a knowledge based system to a conceptualisationnetwork. Conceptualisation networks utilize a combination of the backpropagationlearning rule and the hill climbing search strategy to modify a conceptualisationnetwork through training.. Mahoney and Mooney (1993) describe the RAPTUREsystem for revising probabilistic knowledge bases using a combination of connectionistand symbolic learning methods. RAPTURE maps a certainty factor rule base into anequivalent network in which the certainty factors become the weights on connectionsbetween nodes. The network is trained using a modified version of backprop similar tothat described by Fu (1989). If all the training examples can be classified usingbackprop alone the network is considered trained. Otherwise ID3’s, (Quinlan 1986),
information gain criteria is applied to alter the network architecture by adding featuresto discriminate between incorrectly classified patterns. This cycle of training andfeature addition continues until all the patterns are correctly classified. The network’sstructure allows a set of revised rules to be read directly from the network aftertraining has been completed.
Towell, Shavlik and Noordewier (1990) describe the KBANN algorithm which allowsa Knowledge Based Neural Network, (KBNN), to be constructed from a set ofpropositional rules in such a way that the network topology reflects the dependencystructure of the rules. The resulting KBNN exhibits the same classification behaviouras that of the set of rules from which the network was constructed. The originalformulation of KBANN meant that the system was purely a knowledge refinementsystem as there was no way in which the network could learn new knowledge. Towell(1992) showed that KBNNs produced by KBANN are good at removing irrelevantinformation in approximately correct domain theories, (in fact KBNNs were shown tobetter refine domain theory than purely symbolic methods), but do not deal with“impoverished” domains theories well.
Radial Basis Function, (RBF) type networks lend themselves naturally to the processof rule refinement and have two major strengths in this area. Firstly, it is conceptuallyeasy to see how a local response unit can be converted to a symbolic rule and viceversa. The conversion from local response unit to rule is achieved by describing thearea of response of the individual units in terms of a reference vector that representsthe centre of the unit and a set of radii that determine the effective range of the unit ineach input dimension, (and hence the boundaries of the unit). The rule associated withthe unit is formed by the conjunct of these effective ranges in each dimension. Rulesextracted from each local response unit are thus propositional and of the form:
IF ∀ 1 ≤ i ≤ n : xi ∈ [ xi lower , xi upper ]THEN pattern belongs to class c
where [ xi lower , xi upper ] represents the effective range in the ith input dimension.
The conversion from rule to local function is achieved by stating the symbolic rule tobe inserted into the network in the above format and then deriving the n (centre,radius) pairs necessary to specify the local response unit in n dimensional space. Ingeneral each rule in the knowledge based is converted to a local response unit in thenetwork.
The second strength of local function networks in rule refinement derives from theabove. Because each local response unit can be described by the conjunct of somerange of values in each input dimension it makes it easy to add units to the networkduring training such that the added unit has a meaning that is directly related to theproblem domain The unit is chosen such that its reference vector, ie, the centre of theunit, is one of the as yet unclassified points in the training set. Thus the premise of therule that describes the new unit is the conjunction of the attribute values of the datapoint with the rule consequent being the class to which the point belongs. Networksconstructed this way can be used to not only refine an existing domain theory but toadd new knowledge/rules to the existing rule base.
Tresp, Hollatz and Ahmad, (1993) describe a method for performing rule refinementusing RBF networks. Andrews and Geva (1995) describe the RULEX algorithm forperforming rule refinement with the Constrained Error Backpropagation, (CEBP)network which uses sigmoid based local function units.
Frasconi et.al. (1995), describe a technique for injecting finite state automata into firstorder Recurrent Radial Basis Function (R2BF) networks for problems of inductiveinference of regular grammars. R2BF networks consist of an input layer, two hiddenlayers, (a locally tuned processing unit layer and a sigmoidal unit layer termed the statelayer), and an output layer. Feedback connections exist from the state layer to theradial basis function layer.
The role of prior knowledge in R2BF networks is twofold. Firstly knowledge of thenetwork architecture is utilized to define a penalty function that is applied to the statelayer neurons. This forces the output of the state neurons to approximate booleanvalues and hence biases the network towards finite state behaviour resulting in thelearned automata states being represented by very small ’clusters’ in the network statespace. Secondly, prior knowledge about the problem domain is injected into thenetwork by programming the radial basis function centres and widths to represent theminterms of the canonical form of the next-state function, (where the minterm isdefined as the (current state, new input) pair. There is one radial basis function unitfor each possible minterm in the input.) Frasconi et.al also describe an algorithm forextracting the learned automata from the trained network that is based on theassociation of symbolic states to the clusters in the network state space.
Omlin and Giles, (1996), also attack the problem of inferring the grammar of a regularlanguage generated by a non-trivial, random, 10 state DFA. They describe a rulerefinement method that involves firstly injecting known state transitions into secondorder recurrent networks by programming the weights representing the {input, currentstate}⇒next state transition, then training the network, and finally extracting rules inthe form of a deterministic finites state automata. They show that the network iscapable of preserving correct rules as well as correcting through training inserted rulesthat were initially incorrect, ie, rules that were not the ones in the randomly generatedgrammar.
Rule refinement provides a link between the symbolic and connectionist learningparadigms. To date the symbolic domain knowledge incorporated into neural networkshas been in the form of propositional, fuzzy, or probabilistic rules. As Shavlik (1994)pointed out, a useful area of research is to investigate how other forms of symbolicknowledge, (first order theories for instance), can be refined by neural networks. TheSHRUTI system, (Shastri and Ajjanagadde 1993), is a connectionist knowledgerepresentation network that is capable of dynamically representing relationshipsbetween values that are dynamically bound to variables. In its original formulation thenetwork was unable to learn from examples. Hayward and Diederich (1996) haveextended the network towards a realisation of connectionist Explanation BasedGeneralisation, (EBG), (Mitchell, Keller, and Kedar-Cabelli 1986), however themethod is currently applicable to only complete and correct domain theories. Moreresearch needs to be done to extend the method to weaker domain theories.
5. Summary and conclusionThe preceding discussion highlights the fact that, for purposes of comparison, there issome continuing benefit in being able to categorize the various contributions to thefield of rule extraction from trained ANNs, particularly as the field continues to evolve.In addition based on recent work on the theoretical aspects of rule extraction fromtrained ANNs, the discussion has also identified a set of factors relating to algorithmiccomplexity which at this stage appear to limit what is actually achievable through therule extraction process.
The preceding discussion also raised the issue of whether the current focus of ANNrule extraction techniques may be unnecessarily restrictive. In particular it was notedthat some merit exists in identifying what were termed as causal factors oralternatively as left-reduced functional dependencies (i.e. the attributes and inparticular the combination of attributes which are most significant in determining thedecision attribute);
Finally it was also noted that the majority of current techniques and procedures foreliciting knowledge from trained ANNs are specific to a particular ANNimplementation or architecture. On this basis it was argued that notwithstanding theinitial success of such decompositional techniques as the KT algorithm of Fu (1991,1994) and pedagogical techniques such as the VIA algorithm of Thrun (1994),considerable scope still exists for synthesizing methodologies and techniques which areapplicable across a broad spectrum of ANN implementations and architectures. Inparticular considerable potential appears to exist in exploiting the pedagogicalapproach to rule extraction from trained ANNs to develop a set of techniques withgeneral utility.
6. AcknowledgementsSupport for this work has been provided by the Australian Research Council via anARC Large Grant and an ARC Small Grant to the last of the four authors. Initialsupport was provided under the QUT Meritorious Project Grant Scheme to the firstand fourth authors.
7. Bibliography1. Andrews, R and Geva, S “Rule extraction from a constrained error back
propagation MLP” Proc 5th Australian Conference on Neural Networks BrisbaneQueensland Australia (1994) pp9-12.
2. Andrews, R and Geva, S “Inserting and extracting knowledge from constrainederror back propagation networks” Proc. 6th Australian Conference on NeuralNetworks Sydney NSW Australia (1995).
3. Andrews, R and Geva, S “RULEX and CEBP networks as a basis for a rulerefinement system” in Hallam, J (Ed) “Hybrid problems, hybrid solutions”Amsterdam IOS Press (1995) pp1-12
4. Andrews, R, Diederich, J, and Tickle A B “A survey and critique of techniques forextracting rules from trained artificial neural networks” Knowledge-Based SystemsVolume 8 Number 6 (December 1995) pp 373-389
5. Andrews, R, Cable, R, Diederich, J, Geva, S, Golea, M, Hayward, R, Ho-Stuart,C, and Tickle A B “An evaluation and comparison of techniques for extracting andrefining rules from artificial neural networks” QUT NRC (February 1996)
6. Bartlett, P L “For valid generalisation, the size of the weights is more importantthan the size of the network” Working Paper From the Research School ofInformation Sciences and Engineering Australian National University 1996
7. Carpenter, A and Tan, A H, “Rule extraction: from neural architecture to symbolicrepresentation” Connection Science Vol 7 No 1 (1995) pp 3-27
8. Craven, M W and Shavlik, J W “Using sampling and queries to extract rules fromtrained neural networks” Machine Learning: Proceedings of the EleventhInternational Conference, San Francisco, CA, USA (1994)
9. Diederich, J “Explanation and artificial neural networks” International Journal ofMan-Machine Studies Vol 37 (1992) pp 335-357
10. Dillon, L, Hayward, R, Hogan, J, and Diederich, J “Constructive learning for anoscillatory connectionist reasoning system” QUT NRC Technical Report (ToAppear) (1996)
11. ������ ������������������������������� �!��� ��#"%$&�'"&(�)��*,+-�!� ./� 0'"213��450��6��45� ���7��.8:9�;=< � ��>4�?@�!��A@<>0'"��0B��C"ED�0'������>���-FHG-8H9��IJ;=< 0)� �7KH0� ML!���M0��!4�N-0�45A)���MOP�RQS�138:8H8UT����C�V���7?)����A)< 0'"W�60B��#"YX(�4�@8:�!��� ��0�0����>���-Z\[�]']�^�_a`H��<Pb@N-��c!�;&;ed�f!g'h3d�f!i
12. ������ ������������������������������� �!��� ��#"%$&�'"&(�)��*,K:0�;=��0'��0���4j�45� ���7��.!��� �!� 450$&4��450)k:L�45��Il�4�@�>�7K:0� �L�����0���4�KY�"W�5�<�FY����5�V��L!�� �45� ���7N-0�45A)���MOP�RQS���� �m�� �!0@DP0'������ �!�-`H��<�c!d@;&;e^�h/d�c
13. ��LC��Dn����*M13��450��6��45� ���7��.!N-0�L!���<�op0�L!���5��45� �V� �!4q�)?@�!��A@<>0'"��0)F)��R0'"13��./0���0��� �0'��r�������0� �45� ���s$& �� 0��! M0�QM��Z\[�]�t�]�_�`:��<P[@N-�Bd@;&;ed�c!^'h3d�f!g
14. Fu, L M “Rule learning by searching on adapted nets” Proceedings of the NinthNational Conference on Artificial Intelligence, Anaheim CA USA (1991) pp 590-595
15. Fu, L M "Rule generation from neural networks" IEEE Transactions on Systems,Man, and Cybernetics (1994) Vol 28 No 8 pp 1114-1124
16. Garson, D G "Interpreting Neural-Network Connection Weights" AI Expert (April1991) pp 47-51
17. Geva, S (1995) “Direct production rule extraction” To Appear.18. Geva S, and Orlowski M, “On identification of decision rules with functional
dependencies preprocessing” Proceedings 4th European Congress on IntelligentTechniques and Soft Computing Aachen, Germany Sept.1996b, pp 1600-1604
19. ���>< 0'�VD���C"2u�Iv< �>�sr:��*,8H9�4q�� �45� ���C�'13�C��0���45� ���s��#"YK:0�./� �!0�IJ0��!4���.$&G6Ivw���<>� :KHL�<>0'�P�>�7X@G��#�IJ�> �< <>G-X)�M�>x�0��7K:0� �L�����0���4�Ny0�L���<�Ny0�4qA)����OP��QS�Z\[�]�]�d�_!re���!��0� �4q�>���s$& M�>0��� �0)`:��<�f(N-�Bd@;&;ed�g�b�h/d�d'b
20. Golea, M “On the complexity of rule extraction from neural networks and networkquerying” Proceedings of the Rule Extraction From Trained Artificial NeuralNetworks Workshop, Society For the Study of Artificial Intelligence andSimulation of Behavior Workshop Series (AISB’96) University of Sussex,Brighton, UK (April 1996a) pp 51-59.
21. Golea, M “On the complexity of rule extraction from trained neural nets andrelated issues” To appear (June 1996b).
22. oz�G�A(��"YK{��C"YX)� 0'"W0���� �m7|!��*S$&450�;}�VT���AB��"=��r������!0� M45� ���!�5��48:9�;=<5��#�4q�>���7F)��R0'"%��0��!0���< ���~�45� ����QM�������� :��.!4qm!0@N�13�H$}��]�i@K:L�<>08:9�45��� M45� ���7�P�M��I�T���� �!0'"Yk:��45� ./� ��5�<�N-0�L���<�Ny0�4qA)����OP�V������OP��m!��;}�
k:�#"��0�AB�PK{��C"YX)� 0'"W0���� �m7|(Z38Y"&��_3�'��L!0�0��C��<���C"Y+y���>x60�����>4qGy��.T�0� �m��!��< ����G���Z\[�]�]�i�_�;&;=c!]�h/d�b
23. Holte, R C “Very simple classification rules perform well on most commonly useddatasets” Machine Learning Volume 11 (1993) pp 63-91
24. Korth, H F and Silberschatz, A “ Database systems concepts” McGraw-Hill (1991)25.
���m����!0�G-|z��C"E�n�����!0�G-|!��*Sre��Ivw�� �!� �!�vre���!��0� �45� ��������4H��#"%$&G�IJw���<>� D�0'������>���y4q�)K:0�./� �!0(r�0���4���>��45G-�H� M45���:KHL�<>0@FY���0'��QS�'re�����!0� �4q�>���s$' ��>0��� �0Z\[�]�]�d�_�`:��<�^�Ny�'d���f(;=;}d'd�]�h/d�i�f
26. Mitchell, T, Keller, R and Kedar-Cabelli, S “Explanation Based Generalization: aunifying view,” Machine Learning Volume 1 (1986) pp47-80
27. Murphy, P “The UC Irvine Repository of machine learning databases - a machine-readable data repository” (1995) [available by anonymous ftp from ics.uci.edu inthe directory pub/machine-learning-databases]
28. u�Iv< �>�srU��C"2���>< 0'�VDH��*,KHL�<>0@KH0�x6����� ���7A)� 45m7K:0� �L!�M��0��!4�N-0�L!���<N-0�45A@����OP��QM��138:8H8UT����C�V���7?)����A)< 0'"W�60B��#"YX(�4�@8:�!��� ��0�0����>���-Z\[�]']�i�_`:��<Pt@N-�B[@;}['t�d
29. Orlowski, M “On The Integration Of Knowledge Bases” in Lasker, G E (ed)Advances in Computers and Information Engineering, The International Institutefor Advanced Studies in Systems Research and Cybernetics, Canada (1993)
30. Pawlak, Z “Rough sets - theoretical aspects of reasoning about data” KulwerAcademic Publishers 1991
31. ��L!� �!<5��7|(KU*S13�#"L! M45� ���7��.!X@0� ������ ���7T���0�0'��Q���� �m�� �!0@DP0'������ �!�-`H��<P[
;&;et�[�h/[�g'i32. Quinlan, J R “C4.5: programs for machine learning” Morgan-Kaufmann (1993)33. Quinlan, J R “Learning first-order definitions of functions” Journal of Artificial
Inteligence Reserach Volume 5 (1996) 139-16134. Rumelhart, D E, and McClelland, J L “Parallel distributed processing: explorations
in the microstructure of cognition (Volume 1)” The MIT Press 198635. Sestito, S and Dillon, T “The use of sub-symbolic methods for the automation of
knowledge acquisition for expert systems” Proc. 11th International Conference onExpert Systems and their Application, Avignon, France (July 1991) pp 317-328
36. Sestito, S and Dillon, T “Automated knowledge acquisition of rules withcontinuously valued attributes” Proc. 12th International Conference on ExpertSystems and their Application, Avignon, France (May 1992) pp 645-656
37. Sestito, S and Dillon, T “Automated knowledge acquisition” Prentice Hall(Australia) (1994)
38. Shastri, L and Ajjanagadde, V “From simple associations to systematic reasoning: aconnectionist representation of rules, variables, and dynamic bindings usingtemporal synchrony” Behavioral and Brain Sciences Volume 16 (1993) pp 417-494
39. $&mC�x�< �>OP��|(*Sre��Ivw��>��� �!�v$&G�IJw'��< � Y��C"EN-0�L���<�DP0'������ �!��Q���� �m!� ��0D�0'������>���yZR[']�]�f!_�`:��<P[�f�;&;ed�c
42. Tickle, A B, Hayward, R, and Diederich, J “Recent developments in techniques forextracting rules from trained artificial neural networks” HerbstschuleKonnektionismus (HeKonn ‘96) Munster, (October 1996)
43. Tickle, A B, Andrews, R, Golea, M, and Diederich J “Rule extraction from trainedartificial neural networks” in Browne, A (Ed) “Current perspectives in neuralcomputing” Institute of Physics Publishing (Bristol UK) (To appear)
44. Tickle, A B, Orlowski, M, Diederich J, and Finn G “DEDEC: decision detection byrule extraction from neural networks” 3rd Conference of the Australian CognitiveScience Society, The University of Queensland (April 1995) pp 117
45. Tickle, A B, Orlowski, M and Diederich J “DEDEC: a methodology for extractingrule from trained artificial neural networks” Proceedings of the Rule ExtractionFrom Trained Artificial Neural Networks Workshop, Society For the Study ofArtificial Intelligence and Simulation of Behavior Workshop Series (AISB’96)University of Sussex, Brighton, UK (April 1996) pp 90-102
46. T���A)0�< <5�'��*S$&G�IJw���<>� H?@�!��A@<>0'"��0B��C"YNy0�L���<�Ny0�4qA)����OP�~��13�C�R0��M45� ���#�K:0�./� �!0�IJ0��!4H��#"Y8:9�45��� M45� ���!QS��X)�� �4q����<�X)�5�~�R0��M4j�45� ���#��+-�!� x�0����� 45G-��.������ ����C���>�C��[']�]�c
47. Towell, G and Shavlik, J “The extraction of refined rules from knowledge basedneural networks” Machine Learning Vol 13 No 1 (1993) pp 71-101
48. Towell, G and Shavlik, J “Refining symbolic knowledge based neural networks”Machine Learning Vol 14 (1994) pp 321-331
49. T���A)0�< <P����$&mC�x�< �>OP��|z��C"EN-�����"0�A)� 0������*,K:0�./�>��0�Iv0���4���.!k:;=;&����9�� Il�450�< Gre������0� �4�X@��Il��>�7T�m�0������ 0'�VFHG-?)����A)< 0'"W�60)F)'��0'"YN-0�L!��<�N-0�45A@����OP��Q}������ ��.�45m�0)8:�>�6m!4qm7N��45� ���C�<Pre����./0���0��� �0)���7k:��45� ./� ��5�<P13��450�< < �>��0��� �0'��k:kHk)1����0'�~�~��FH�'��45���C��Z\[�]�]'g�_a;=;}t'i�[�h/t�i�i
50. Tresp, V, Hollatz, J and Ahmad, S “Network structuring and training using rule-based knowledge” Advances in Neural Information Processing Systems Vol 5(1993) pp871-878
51. Viktor, H L, Engelbrecht A P, and Cloete I “Reduction of symbolic rules fromartificial neural networks using sensitivity analysis” IEEE Conference on NeuralNetworks Perth Western Australia (November 1995) pp 1788-1793