Cover Page 

Uploaded June 24, 2011 

 

Managing and 

Benefitting from Multi‐

Million Rule Systems  

Author: Jeffrey G. Long (jefflong@aol.com) 

Date: October 31, 2007 

Forum: Poster session presented at the 2007 Conference of the New England 

Complex Systems Institute.

Contents 

Page 1: Abstract 

Pages 2‐26: Slides (but no text) for presentation 

 

License 

This work is licensed under the Creative Commons Attribution‐NonCommercial 

3.0 Unported License. To view a copy of this license, visit 

http://creativecommons.org/licenses/by‐nc/3.0/ or send a letter to Creative 

Commons, 444 Castro Street, Suite 900, Mountain View, California, 94041, USA. 

Managing and Benefitting From Multi-Million Rule Systems

Abstract Jeffrey G. Long

October 31, 2007 This talk will discuss the idea that better representation and understanding of complex systems will require new abstractions and new uses of existing abstractions. One approach I have been exploring is taking system rules out of software and representing them as data. I will discuss several abstractions I have found useful in representing various kinds of complex business, linguistic, and biological systems as data. These include (1) the notion of tens of thousands of complex, contingent "Competency Rules" that define or describe the behavior of a system, (2) the implementation of those rules partly in software (like an inference engine) and primarily in data (like an expert system); (3) the notion of contingent rules having multiple "factors" or primary drivers and zero or more "considerations" that the system must review before deciding what to do next; and (4) the notion of the form of a rule, as contrasted with its content (like algebra). Reducing complexity cannot mean ignoring details, but must include seeing the larger picture presented by ruleforms. Several specific examples will be given from current and past projects.

Managing & Benefiting from Multi-Million

Rule SystemsRule Systems

International Conference on Complex Systems

ICCS2007 – Boston, MAJeffrey G. Long

October 31, 2007

jefflong@aol.com

Studying a Variety of Notational Systems

Wh t k th f l?speech & writingcartographyarithmetic & algebra

What makes them powerful?

What is their nature & structure?

Can their design be facilitated?arithmetic & algebrageometrychemical notationd / t t ti

How and why did they evolve?

Who created them?

What accelerated or impeded dance/movement notationmusic notationlogic notation

ptheir general usage and acceptance?

What effects did they have on society? on cognition?money society? on cognition?

How do we know if we’re at the limits of usefulness of a notational system?y

Key PointsModern society is critically dependant upon a number of different kinds of rule systems Yet weModern society is critically dependant upon a number of different kinds of rule systems. Yet we

have (increasingly) enormous problems creating and managing large rule systems.

This arises from how we currently represent rules and data. We cannot solve them by means of faster computers or other extensions of current representations. Reducing complexity cannot mean ignoring details, but must include seeing an even larger picture. g g , g g p

We can look to the past for guidance. Many times in the past, society has overcome “complexity barriers” by means of new notational systems. These events are what I call “notational revolutions”, and they affect how we see the world, how we think about the world, and how readily and what we can communicate with others.

My experience is that representing rules and data as an integrated whole, and using a place-value representation, does make large rule systems much more comprehensible, therefore more manageable, and therefore more able to safely grow and change as needed (i.e. evolve). My name for this approach is “Ultra-Structure”.

I hope other proposed Rule Calculi will consider the issues and approaches I’m suggesting here.

R l S t Ubi itRule Systems are Ubiquitous

Business Scientific LegalSubject

Small

# RulesBusiness

RulesScientific

RulesLegal Rules

jetc.

< 1,000Medium

< 100,000

Large

< 10 000 000< 10,000,000

Very Large

> 10 000 000> 10,000,000

Many Types of RulesOntological Rules (what exists how entities relate)Ontological Rules (what exists, how entities relate)Operating Rules (how a system nominally works)Strategy Rules (how to optimize a process; win; be artful)Ethical Rules (additional guidelines for a clear conscience)Ethical Rules (additional guidelines for a clear conscience)Evaluation Rules (how to tell if making progress/“winning”, or detecting that rules are not working well) Learning Rules (rules for changing rules)Learning Rules (rules for changing rules)Historical Rules (past events; custom)

Rules are multi notational: largely qualitative but may includeRules are multi-notational: largely qualitative but may include quantities or other kinds of abstractions (e.g. musical notes)Rules are probabilistic but can be treated as deterministic

Characteristics of Notational RevolutionsSome involve looking at the world from a different viewpoint, e.g. a birds-eye rather than a g p , g yground-truth viewpoint, or indirect rather than direct reference to the world.

Some involve moving from a relative-value representation to a place-value representation.

Some involve the introduction of new abstractions such as zero musical notes or mapSome involve the introduction of new abstractions, such as zero, musical notes, or map coordinates.

Physics has benefited from and might be said to have even co evolved with improvedPhysics has benefited from and might be said to have even co-evolved with improved notational systems such as calculus, Feynman Diagrams, Riemannian geometry, tensors

They all greatly expand the sphere of what can be readily said; the notation is the limitation.

Th l f “ t ti l i i ” i ith t th b fit f t tiThey are examples of “notational engineering” occurring without the benefit of systematic guidelines from the experience of others, or of a general theory of notation derived from a longitudinal and comparative study of humanity’s notational systems

1. Separation of Algorithms from Data

Traditional separation contributes to and is caused by object-centered view of the world.

In a process-centered worldview, everything is a process and every process is only describable in terms of r lesterms of rules.

Traditional Management Info System

Events WorkSoftware/ Algorithms

Data

Conventional Data are Rule Fragments

Bin Part QOH QOO

A X 5 4

etc.

A X 5 4Rule

FragmentsB B 15 7B B 15 7

Satisfies TNF requirements, but is still not flexible enough.

Data-Inclusive Rules Include Conventional Data as Part of Larger Rules

QtyUniversals

Provide

Loc’n PartQty Type Qty

A X QOH 5

ProvideContextetc.

A X QOH 5

Simple 4A X QOOSimple Sourcing

RulesB B 15

4

QOH

A X QOO

B B 15

7

QOH

B B QOOB B Q

2. Examples of Relative to Place ValueRoman to Hindu Arabic NumeralsRoman to Hindu-Arabic Numerals

500 BCE, 200 CE, 875 CE, 1200 CE, 1600 CE

Neumatic to Staff Notationeu at c to Sta otat o

500 CE, 800 CE, 1025 CE, 1300 CE, 1600 CE

Peripli to Coordinate-System maps

500 BCE, 100 CE, 1600 CE

Pl V l b Q titPlace-Value by QuantityHindu-Arabic

Roman Numerals Numerals

100101102103

IV 4IV 4

CXII

MCMIX

21

1 9 0 9

1

MCMIX 1 9 0 9

Without a placeholder, you can’t reliably have columnsp , y y

Pl V l b Pit hPlace Value by Pitch

Neume direction indicated voice interval

F

D

B

G

E

E

C

A

FE

A

F

D

B

G

E

C

AB

G

A

Pl V l b C di tPlace Value by Coordinates

RuleML Adds More Complexity<imp>< head><_head><atom><_opr><rel>isAvailable</rel></_opr><var>Car</var></atom></_head><_body><and><atom>< opr><rel>isPresent</rel></ opr>

“A car is available for rental if it is h i ll t i t i d t_opr rel isPresent /rel /_opr

<var>Car</var></atom><not><atom><_opr><rel>isAssignedToRentalOrder</rel></_opr><var>Car</var></atom></not><not>

physically present, is not assigned to any rental order, is not scheduled for service, and does not require service.”

<atom><_opr><rel>isScheduledForService</rel></_opr><var>Car</var></atom></not><not><atom><_opr><rel>requiresService</rel></_opr><var>Car</var></atom></not></and></_body></imp>

H. Boley, S Tabet, G. Wagner, “Design Rationale for RuleML: A Markup Language for Semantic Web Rules”

Place-Value of RulesConventional rules are semantically informal and multiplex (many parts)Conventional rules are semantically informal and multiplex (many parts)

Exceptions to rules are themselves rules

Any conventional rule can be converted into >1 “simple” rulesy co e t o a u e ca be co e ted to s p e u es

Each “simple” rule has the form:

“If a and b and c…Then Consider x and y and z”, where>= 1 Ifs>= 0 Then Considers

Rules converted into simple form are grouped based on their format (# Ifs, # Then-Considers) and meaning (= function) e.g. agencies versus locations

d tversus products

Result is a small (< 100) set of tables each having different structure and/or function (syntax and semantics)

Each simple rule is represented as one record in one table (out ofEach simple rule is represented as one record in one table (out of n tables)

Each column of each table has a general meaning that is used to g gassign context to that part of each rule in that table

Initial rule selection for inspection (the If component) constitutes the primary key column(s)the primary key column(s)

Subsequent rule evaluation and possible execution (the Then Consider component) constitutes most other columnsp )

There are usually several columns of rule metadata at the end

Software implements a Competency Rule Engine that (ideally) doesn’t know anything about world, only about how to read the rules for a broad application area (e.g. business, games, law)

Rule systems have several kinds of Existential Ruleforms as aRule systems have several kinds of Existential Ruleforms as a foundation

agenciesproducts/serviceslocationstime periods

Existential rules are referenced by foreign key constraints to form Compound Ruleforms

network ruleforms define relations among same kind of entitiesattribute ruleforms define characteristics of entitiesauthorization ruleforms define relations among different kinds of entities protocol ruleforms define processesp p

Most columns are foreign keys to a particular existential table (this can cause problems with some RDBMS)

3 Competency Rule Engine (CoRE)3. Competency Rule Engine (CoRE)

Very small t famount of

code in engine (~100K LOC)

Stimulus ResponseControl Logic

Conventional Competency

Rulesdata is ab-sorbed into rules; every-thing is a rule!

BenefitsRepresenting rules as “data” rather than software decreasesRepresenting rules as data rather than software decreases required amount of software by 1-2 orders of magnitude:

reduced amount of software may reduce initial development costreduced amount of software definitely reduces chances for bugs, thus

d i d l t d i t treducing development and maintenance costsRules as “data” can be directly accessed and managed by subject experts, without reliance on programmers:

changes in rules normally do not require changes in software reducingchanges in rules normally do not require changes in software, reducing maintenance costsreduces/eliminates communication requirements from subject expert to programmer

As rules are externalized corporate knowledge can be seenAs rules are externalized, corporate knowledge can be seen, studied, and improved by many

with added metadata regarding each rule, and hyperlinks, this can become a true knowledgebase

Exploratory CoREsCoRE650 Business (wholesaler with 10 000 orders/day)CoRE650 – Business (wholesaler with 10,000 orders/day)

CoRE415 – Language (search documents for concepts)

CoRE576 – Biology (various toy models in proteomics lab)

I’m always eager to try this theory on new kinds of rule systemsI m always eager to try this theory on new kinds of rule systems.

R l C tRule Counts# Existential Rules

Over Time

400,000

450,000

250,000

300,000

350,000

,

AgencyLocationMaster Protocol

50,000

100,000

150,000

200,000 Master ProtocolProduct or Service

0

9/22

/200

5

10/2

2/20

05

11/2

2/20

05

12/2

2/20

05

1/22

/200

6

2/22

/200

6

3/22

/200

6

4/22

/200

6

5/22

/200

6

6/22

/200

6

7/22

/200

6

8/22

/200

6

9/22

/200

6

10/2

2/20

06

11/2

2/20

06

12/2

2/20

06

1/22

/200

7

2/22

/200

7

3/22

/200

7

4/22

/200

7

5/22

/200

7

6/22

/200

7

7/22

/200

7

8/22

/200

7

9/22

/200

7

SummaryRules are a type of abstraction, and should be studied as such. There areRules are a type of abstraction, and should be studied as such. There are higher-level abstractions than individual rules, and many rule types; we need a discipline whose object of study is rules/laws.

Putting more rules into software is not the solution, nor is building new layers on top of existing layers Software is the problem It substitutes for a formalizedtop of existing layers. Software is the problem. It substitutes for a formalized, place-value representation of rules, enforces a divide between algorithms and data, and obscures the rules with significant ancillary syntax.

Rule systems must be conceived at a higher level of abstraction to be y gmanageable while still maintaining all necessary detail

< 100 ruleforms and their interactions are comprehensible1+ million individual rules are not comprehensible

Th lti t t b bl t f b th d d ti i f dThe resulting system must be able to perform both deductive inference and computations, and be managed directly by subject experts (not programmers)

QuestionsMight the problems of large rule systems arise from the way weMight the problems of large rule systems arise from the way we represent them?

Wh t i th ti l t ti f l b ( illi ) fWhat is the optimal representation of large numbers (millions) of complex, contingent rules?

What might a place-value system for representing rules look like?

What is the relationship of algorithms and data? Is there benefit in conceiving and representing both as rules?

Ult St t R fUltra-Structure ReferencesLong, J., and Denning, D., “Ultra-Structure: A design theory for complex systems and processes.” In Communications of the ACM (January 1995)

Long, J., “A new notation for representing business and other rules.” In Long, J. (guest editor), SemioticaSpecial Issue: Notational Engineering, Volume 125-1/3 (1999)

Shostko, A., “Design of an automatic course-scheduling system using Ultra-Structure.” In Long, J. (guestShostko, A., Design of an automatic course scheduling system using Ultra Structure. In Long, J. (guest editor), Semiotica Special Issue: Notational Engineering, Volume 125-1/3 (1999)

Long, J., “Automated Identification of Sensitive Information in Documents Using Ultra-Structure.” Proceedings of the 20th Annual ASEM Conference, American Society for Engineering Management (1999)

Oh Y and Scotti R “Analysis and Design of a Database using Ultra Structure Theory (UST)Oh, Y., and Scotti, R., Analysis and Design of a Database using Ultra-Structure Theory (UST) –Conversion of a Traditional Software System to One Based on UST,” Proceeding of the 20th Annual Conference, American Society for Engineering Management (1999)

Parmelee, M., “Design For Change: Ontology-Driven Knowledgebase Applications For Dynamic Biological Domains.” Master’s Paper for the M.S. in I.S. degree, University of North Carolina, Chapel Hill (November 2002)2002)

Maier, C., CoRE576 : An Exploration of the Ultra-Structure Notational System for Systems Biology Research. Master’s Paper for the M.S. in I.S. degree, University of North Carolina, Chapel Hill (April 2006)