exceptional exceptions - 12th contecsi 34th wcars
Post on 18-Aug-2015
23 Views
Preview:
TRANSCRIPT
EXCEPTIONAL EXCEPTIONS
Hussein Issa
(Spring 2015)
BACKGROUND
Background
• Audit by Exception (Vasarhelyi and Halper, 1991)
• Continuous Auditing literature is rich with studies that propose
statistical and machine learning techniques to identify
exceptions (Dull et al., 2006; Groomer & Murthy, 1989a; Alexander Kogan et al.,
1999; Vasarhelyi & Halper, 1991)
• Problem?
• Result?
3 5/31/2015 34th WCARS
Large numbers of Exceptions!
-Information overload!
-Ambiguity
-Information relevance
Motivation & Contribution
Motivation:
• Prior continuous auditing research focused on detecting
exceptions efficiently
• We need to develop a classification hierarchy or ranking of
detected exceptions to limit the need for human intervention
Contribution:
• Fill the gap in the continuous auditing literature with regards to
providing the human users with assistance in dealing with
large number of exceptions
• Propose exceptions prioritization methodologies
4 5/31/2015 34th WCARS
Relevant Literature-Exceptional Exceptions
• Analysis usually yields large amounts of exceptions, causing
information overloads due to sub-optimal business processes
or over-conservative CA/CM system (Alles et al 2006, 2008 ;
Debreceny et al. 2003)
• Human users perform complex aggregation and processing
tasks poorly (Iselin, 1988; Kleinmunitz, 1990)
• Murthy 2012 emphasized the lack of and need for studies
addressing the problem of large numbers of exceptions
5 5/31/2015 34th WCARS
IDENTIFYING AND PRIORITIZING
IRREGULARITIES USING A RULE-
BASED MODEL WITH A WEIGHTING
SYSTEM DERIVED FROM EXPERTS’
KNOWLEDGE
Motivation, Research Questions, & Findings
Motivation: • Plenty of studies for exception identification, few address processing
• Majority of expert systems assign the same weight to rules
• HOWEVER: business rules, and accordingly their violations, do not have
the same importance
Research Questions: 1. How can we integrate the judgment of the domain experts (in this case the
auditors) in a rule-based expert system?
2. How can we develop a weighting system for the various rules in that expert
system?
3. How can we use this weighting system to prioritize exceptions?
7 5/31/2015 34th WCARS
Suspicion Score
8 5/31/2015 34th WCARS
Rank
4
2
3
1
6
5
Framework
9 5/31/2015 34th WCARS
Framework
10 5/31/2015 34th WCARS
6
Rule-Based System
• Set of IF-THEN rules
• Popularity stems from simplicity, interpretability, flexibility
• Data: Simulated Order to Cash data
• Originally 33 analytics, narrowed down to 12 by experienced auditors
• 12 analytics can be categorized as tests for: – Segregation of duties
– Unauthorized transactions
– Missing documents
– Non-matching documents.
11 5/31/2015 34th WCARS
Rules Weights Inference
• Business rules and accordingly their violations, do not have
the same significance
• 28 participants with 3 and more years of experience:
– Conduct 17 pairwise comparisons
– Select the transaction they believe to present higher control risk
– Provide justification if their assessment
12 5/31/2015 34th WCARS
Customer
Transaction
ID
Custom
er ID
Invoice
Number
Invoice
Date
Created
by
Inventory
Item ID
Invoice
Selling
Price
Invoiced
Amount
Shipment
ID
Shipment
document
Created By
Shipment
document
Approved By
10034 1146 10001203 10/15/1997 1546 63
800 9600 10000877 1002 1546
4110 1000 10000262 3/1/1998 1117 628
600 1800
SOD Missing Values
(Orphaned invoices)
Rules Weight Inference-LP
• Special Case Linear Program
– 16 pairs
– Each transaction can violate one and only one
• General Case Linear Program
– 17 pairs
– One transaction violated two rules
13 5/31/2015 34th WCARS
Exceptions Identification & Prioritization
Identification:
• Apply expert system to the whole population to find all the records
that violate one or more rules
• Remaining records are assumed to be normal, thus presenting
negligible risk
Prioritization:
• Calculate the Suspicion Score for each exception such that:
14 5/31/2015 34th WCARS
Exceptions Prioritization-Example
15 5/31/2015 34th WCARS
Rank
4
2
3
1
6
5
Investigation & Feedback
Investigation:
• Auditors are provided with Prioritized exceptions
• Scope of investigation depends on their time/budget
constraints
Feedback:
• Helps adjusting the rules that make up the expert system.
• Enables us to modify the weights of the rules according to the
audit teams’ findings
– incorporated as a new set of constraints in the general case Model
• Effect of the original experiment will decrease over time with
more feedback from auditors
16 5/31/2015 34th WCARS
Weights –Special vs. General Models
17 5/31/2015 34th WCARS
• Excessive Write offs ranked highest
• SOD in general ranked low
• Rules with direct impact on financial numbers ranked high
• Operational controls ranked low Analytic Special Case Model General Case Model
Weight Rank Weight Rank
Analytic_1_SOD_Customers 1.00 11 1.00 11 Analytic_2_Unauthorized_Sales_Order 2.86 2 2.71 3 Analytic_3_Unathorized_Price 1.38 9 1.35 9 Analytic_4_SOD_Credit_Adjustment 1.21 10 1.20 10 Analytic_5_Match_Shipping_to_SO 1.79 8 1.72 8 Analytic_6_Match_Invoice_to_Ship 2.33 4 2.22 5 Analytic_7_Missing_Sales_Orders 2.31 5 3.18 1 Analytic_8_Unauthorized_Shipments 2.27 6 2.17 6 Analytic_9_SOD_Ship_Invoice 1.88 7 1.81 7 Analytic_10_Orphaned_Invoices 2.85 3 2.70 4 Analytic_11_SOD_Invoice_Receipt 1.00 12 1.00 12 Analytic_12_Excessive_Write_Offs 3.11 1 2.94 2
Analytic Special Case Model General Case Model
Weight Rank Weight Rank
Analytic_12_Excessive_Write_Offs 3.11 1 2.94 2
Analytic_2_Unauthorized_Sales_Order 2.86 2 2.71 3
Analytic_10_Orphaned_Invoices 2.85 3 2.7 4
Analytic_6_Match_Invoice_to_Ship 2.33 4 2.22 5
Analytic_7_Missing_Sales_Orders 2.31 5 3.18 1
Analytic_8_Unauthorized_Shipments 2.27 6 2.17 6
Analytic_9_SOD_Ship_Invoice 1.88 7 1.81 7
Analytic_5_Match_Shipping_to_SO 1.79 8 1.72 8
Analytic_3_Unathorized_Price 1.38 9 1.35 9
Analytic_4_SOD_Credit_Adjustment 1.21 10 1.2 10
Analytic_1_SOD_Customers 1 11 1 11
Analytic_11_SOD_Invoice_Receipt 1 12 1 12
Weights – Internal vs. External Auditors
18 5/31/2015 34th WCARS
All Responses Internal Auditors External Auditors
Analytic Weight Rank Weight Rank Weight Rank
Analytic_1_SOD_Customers 1.00 11 1.00 10 1.00 11
Analytic_2_Unauthorized_Sales_Order 2.71 3 2.65 4 2.84 3
Analytic_3_Unathorized_Price 1.35 9 1.51 8 1.24 9
Analytic_4_SOD_Credit_Adjustment 1.20 10 1.45 9 1.06 10
Analytic_5_Match_Shipping_to_SO 1.72 8 1.98 7 1.59 8
Analytic_6_Match_Invoice_to_Ship 2.22 5 2.29 6 2.22 5
Analytic_7_Missing_Sales_Orders 3.18 1 3.10 1 3.37 1
Analytic_8_Unauthorized_Shipments 2.17 6 2.36 5 2.08 6
Analytic_9_SOD_Ship_Invoice 1.81 7 1.00 11 1.79 7
Analytic_10_Orphaned_Invoices 2.70 4 2.75 3 2.76 4
Analytic_11_SOD_Invoice_Receipt 1.00 12 1.00 12 1.00 12
Analytic_12_Excessive_Write_Offs 2.94 2 2.93 2 3.05 2
Takeaway from today
I think the main take away from this presentation is:
A. Continuous auditing sucks, dude!
B. It is better to stick to traditional sample based auditing
C. Number of exceptions is always very low
D. Hussein is awesome!!! (Hint: correct answer)
19 5/31/2015 34th WCARS
Too many exceptions!
Information Overload
Research Opp. in EE
top related