TECHNOLOGY-ASSISTED REVIEW 2013
Bob Gillespie, Vice President of ESI Consulting, eTERA Richard Perrin, E-Discovery Counsel, Dickstein Shapiro LLP
June 6, 2013
#3169610
TEC
HN
OLO
GY-A
SSISTED REV
IEW 2
01
3
Copyright © 2013 eTERA Consulting
• Introductions
• What is Technology Assisted Review (“TAR”)?
• Court’s and Regulators’ Views on TAR
• Factors Relating to the Use of TAR
• Options for using TAR
• Use of TAR in Recent Cases
• Q&A session
Agenda
TEC
HN
OLO
GY-A
SSISTED REV
IEW 2
01
3
Copyright © 2013 eTERA Consulting
Richard Perrin Richard Perrin is E-Discovery Counsel at Dickstein Shapiro LLP. Rick brings broad experience to this position, having been a member of the firm’s litigation practice for more than 20 years. As litigation counsel, Rick developed expertise in case management and designed client-specific programs to improve the discovery process for both litigated matters and government investigations. Rick focuses on developing strategies to limit the costs associated with document preservation, review and production through the effective use of technology and process planning. He co-chairs his firm’s Litigation Steering Committee and is active in developing internal best practice guidelines relating to e-discovery and other litigation matters. Rick is a member of The Sedona Conference Working Group on Electronic Document Retention and Production (WG1) and the District of Columbia Bar E-Discovery Committee. Bob Gillespie As Vice President of ESI Consulting, Bob Gillespie specializes in consulting with clients on the efficient and defensible management of ESI collections. Mr. Gillespie’s experience in managing complex litigation matters as well as his expertise in all aspects of electronic discovery throughout the EDRM make him a trusted advisor to eTERA’s law firm and corporate clients in key industries such as energy and pharmaceuticals. With fifteen years of law firm experience in the litigation support area and twelve years of legal solutions provider experience, he is uniquely qualified to provide consulting on all aspects of data management. In particular, Mr. Gillespie is well positioned to help clients address the myriad of challenges posed by ESI collections including information readiness, litigation preparedness and budget predictability.
Today’s Speakers
TEC
HN
OLO
GY-A
SSISTED REV
IEW 2
01
3
Copyright © 2013 eTERA Consulting
What is Technology-Assisted Review (TAR)?
ABA Journal (April 2012)
Method of review that utilizes computer “learning” to accurately “predict” and apply attorney decisions to large document sets, thereby supplementing human review
TEC
HN
OLO
GY-A
SSISTED REV
IEW 2
01
3
Copyright © 2013 eTERA Consulting
Create Random Sample
Document Set
Subjectively Review Sample
Set
Categorize Document Universe
Validate Results
TAR Workflow – The Art of Refinement
TEC
HN
OLO
GY-A
SSISTED REV
IEW 2
01
3
Copyright © 2013 eTERA Consulting
Five Misconceptions About TAR • TAR is New Technology
Artificial Intelligence algorithms used in government, law enforcement and Wall Street for years
• TAR is Just Technology
Computer assisted review is a process
• TAR Excludes Keyword Searching
The goal is to train the computer using TAR, keyword search, concept search, similarity search, and other methods
• TAR Eliminates Manual Review
Manual review is a key component of the TAR process
• TAR Must Be Perfect to Be Useful
TAR only has to be better than human review while being faster and cheaper.
What is Technology-Assisted Review?
TEC
HN
OLO
GY-A
SSISTED REV
IEW 2
01
3
Copyright © 2013 eTERA Consulting
• Prioritize document review
batches based on categorization score
• Potentially eliminate review batches based on sampling methodologies
• Quality control of reviewed documents prior to production
• Review opposing party’s document production
TAR Workflow – Options
Technology Assisted Review combines the expertise of human reviewers while leveraging the power of data analytics
TAR Process: Review/Production Option 1
1,000,000 Total Collection
20% “Responsive” 200,000 documents
10% “Uncategorized” 100,000 documents
70% “Not Responsive” 700,000 documents
Produce
Review
Ignore
2.5% error rate; 95% confidence level * Sample size: 1,537 * 5 iterations: 8,000 docs reviewed * 9 iterations: 14,000 docs reviewed
1,000,000 Total Collection
20% “Responsive” High Priority
200,000 documents 10% “Uncategorized” Medium Priority
100,000 documents
70% “Not Responsive” Low Priority
700,000 documents
Higher cost reviewers
Lower cost reviewers
Lowest cost reviewers
2.5% error rate; 95% confidence level * Sample size: 1,537 * 5 iterations: 8,000 docs reviewed * 9 iterations: 14,000 docs reviewed
TAR Process: Review/Production Option 2
TEC
HN
OLO
GY-A
SSISTED REV
IEW 2
01
3
Copyright © 2013 eTERA Consulting
• Are you really confident that your search terms will hit the key evidence?
• Do you test your key word search results?
• Judges Andrew Peck, John Facciola and Paul Grimm challenge the myth of key words “gold standard.”
“The way lawyers choose keywords is the equivalent of the child’s game of ‘Go Fish’.”
-Judge Andrew Peck
Keyword Search - Old Ways are Not the Best
TEC
HN
OLO
GY-A
SSISTED REV
IEW 2
01
3
Copyright © 2013 eTERA Consulting
• Cases “Approving” Use of TAR
Da Silva Moore v. Publicis Groupe, (S.D.N.Y., Judge Andrew Peck)
First case to approve use of “predictive coding”
Global Aerospace v. Landow Aviation, (Circuit Court, Loudoun County, Virginia)
Published results of TAR review
In re: Actos Antitrust Litigation (D.C., Louisiana)
Cumbersome review protocol adopted by the court
EORHB, Inc. v. HOA Holdings LLC, (Delaware Chancery, October 15, 2012 and May 6, 2013)
Sua Sponte order directed use of computer-assisted review
Recently, the Court relieved parties of requirement to use TAR
What have the Courts said about TAR?
TEC
HN
OLO
GY-A
SSISTED REV
IEW 2
01
3
Copyright © 2013 eTERA Consulting
• Gabriel Technologies v. Qualcomm, (U.S.D.C., S.D. California, February 1, 2013)
Court approved Qualcomm’s application for fees, including $2.8 million for e-discovery vendor TAR services
What have the Courts said about TAR?
“...[counsel’s] decision to undertake a more efficient and less time-consuming method of document review to be reasonable under the circumstances ... and … reduced the overall fees and attorney hours required by performing electronic document review at the outset.”
more efficient less time-consuming
reduced the overall fees and attorney hours
TEC
HN
OLO
GY-A
SSISTED REV
IEW 2
01
3
Copyright © 2013 eTERA Consulting
• In re Biomet Hip Implant Products Liability Litigation (N.D. Ind. April 13, 2013)
• Biomet combined keyword searching with Technology Assisted Review Keyword searches were the first step in culling Biomet’s documents
• Plaintiffs objected to Biomet’s process Application of keyword search prior to employing TAR tainted the entire
process
• “The issue…isn’t whether predictive coding is a better way of doing things than keyword searching prior to predictive coding. I must decide whether Biomet’s procedure satisfies it discovery obligations and, if so, whether it must also [apply predictive coding to the entire document collection]. Judge Robert Miller, Jr., framing the Biomet discovery issues
In re Biomet Hip Implants - Keyword Culling Used with TAR
TEC
HN
OLO
GY-A
SSISTED REV
IEW 2
01
3
Copyright © 2013 eTERA Consulting
• Biomet’s procedure satisfied its FRCP discovery obligations “I can’t find that the likely benefits of the discovery proposed by
[plaintiffs] equals or outweighs its additional burden on, and additional expense to, Biomet.”
• Basis for the Decision Proportionality and Cost Analysis
Statistical Sampling
Acceptance of TAR discovery tools
• Final Considerations Court anticipated continued meet & confers on discovery issues
Additional reasonably-targeted search terms
Production of non-privileged documents in the statistical sample
Plaintiffs may pay to use TAR on the entire document collection
In re Biomet – Holding and Other Considerations
TEC
HN
OLO
GY-A
SSISTED REV
IEW 2
01
3
Copyright © 2013 eTERA Consulting
Contact Information
Contact information:
Dickstein Shapiro LLP
Richard Perrin E-Discovery Counsel
[email protected] 202.420.3198
www.dicksteinshapiro.com
eTERA Consulting, LLC
Bob Gillespie Vice President, ESI Consulting
[email protected] 202.905.4473
www.eteraconsulting.com