technology assisted review (tar) - dickstein shapiro llp

TECHNOLOGY-ASSISTED REVIEW 2013

Bob Gillespie, Vice President of ESI Consulting, eTERA Richard Perrin, E-Discovery Counsel, Dickstein Shapiro LLP

June 6, 2013

#3169610

TEC

HN

OLO

GY-A

SSISTED REV

IEW 2

01

3

Copyright © 2013 eTERA Consulting

• Introductions

• What is Technology Assisted Review (“TAR”)?

• Court’s and Regulators’ Views on TAR

• Factors Relating to the Use of TAR

• Options for using TAR

• Use of TAR in Recent Cases

• Q&A session

Agenda

TEC

HN

OLO

GY-A

SSISTED REV

IEW 2

01

3


Richard Perrin Richard Perrin is E-Discovery Counsel at Dickstein Shapiro LLP. Rick brings broad experience to this position, having been a member of the firm’s litigation practice for more than 20 years. As litigation counsel, Rick developed expertise in case management and designed client-specific programs to improve the discovery process for both litigated matters and government investigations. Rick focuses on developing strategies to limit the costs associated with document preservation, review and production through the effective use of technology and process planning. He co-chairs his firm’s Litigation Steering Committee and is active in developing internal best practice guidelines relating to e-discovery and other litigation matters. Rick is a member of The Sedona Conference Working Group on Electronic Document Retention and Production (WG1) and the District of Columbia Bar E-Discovery Committee. Bob Gillespie As Vice President of ESI Consulting, Bob Gillespie specializes in consulting with clients on the efficient and defensible management of ESI collections. Mr. Gillespie’s experience in managing complex litigation matters as well as his expertise in all aspects of electronic discovery throughout the EDRM make him a trusted advisor to eTERA’s law firm and corporate clients in key industries such as energy and pharmaceuticals. With fifteen years of law firm experience in the litigation support area and twelve years of legal solutions provider experience, he is uniquely qualified to provide consulting on all aspects of data management. In particular, Mr. Gillespie is well positioned to help clients address the myriad of challenges posed by ESI collections including information readiness, litigation preparedness and budget predictability.

Today’s Speakers

TEC

HN

OLO

GY-A

SSISTED REV

IEW 2

01

3


What is Technology-Assisted Review (TAR)?

ABA Journal (April 2012)

Method of review that utilizes computer “learning” to accurately “predict” and apply attorney decisions to large document sets, thereby supplementing human review

TEC

HN

OLO

GY-A

SSISTED REV

IEW 2

01

3


Create Random Sample

Document Set

Subjectively Review Sample

Set

Categorize Document Universe

Validate Results

TAR Workflow – The Art of Refinement

TEC

HN

OLO

GY-A

SSISTED REV

IEW 2

01

3


Five Misconceptions About TAR • TAR is New Technology

Artificial Intelligence algorithms used in government, law enforcement and Wall Street for years

• TAR is Just Technology

Computer assisted review is a process

• TAR Excludes Keyword Searching

The goal is to train the computer using TAR, keyword search, concept search, similarity search, and other methods

• TAR Eliminates Manual Review

Manual review is a key component of the TAR process

• TAR Must Be Perfect to Be Useful

TAR only has to be better than human review while being faster and cheaper.

What is Technology-Assisted Review?

TEC

HN

OLO

GY-A

SSISTED REV

IEW 2

01

3


• Prioritize document review

batches based on categorization score

• Potentially eliminate review batches based on sampling methodologies

• Quality control of reviewed documents prior to production

• Review opposing party’s document production

TAR Workflow – Options

Technology Assisted Review combines the expertise of human reviewers while leveraging the power of data analytics

TAR Process: Review/Production Option 1

1,000,000 Total Collection

20% “Responsive” 200,000 documents

10% “Uncategorized” 100,000 documents

70% “Not Responsive” 700,000 documents

Produce

Review

Ignore

2.5% error rate; 95% confidence level * Sample size: 1,537 * 5 iterations: 8,000 docs reviewed * 9 iterations: 14,000 docs reviewed

1,000,000 Total Collection

20% “Responsive” High Priority

200,000 documents 10% “Uncategorized” Medium Priority

100,000 documents

70% “Not Responsive” Low Priority

700,000 documents

Higher cost reviewers

Lower cost reviewers

Lowest cost reviewers

2.5% error rate; 95% confidence level * Sample size: 1,537 * 5 iterations: 8,000 docs reviewed * 9 iterations: 14,000 docs reviewed

TAR Process: Review/Production Option 2

TEC

HN

OLO

GY-A

SSISTED REV

IEW 2

01

3


• Are you really confident that your search terms will hit the key evidence?

• Do you test your key word search results?

• Judges Andrew Peck, John Facciola and Paul Grimm challenge the myth of key words “gold standard.”

“The way lawyers choose keywords is the equivalent of the child’s game of ‘Go Fish’.”

-Judge Andrew Peck

Keyword Search - Old Ways are Not the Best

TEC

HN

OLO

GY-A

SSISTED REV

IEW 2

01

3


• Cases “Approving” Use of TAR

Da Silva Moore v. Publicis Groupe, (S.D.N.Y., Judge Andrew Peck)

First case to approve use of “predictive coding”

Global Aerospace v. Landow Aviation, (Circuit Court, Loudoun County, Virginia)

Published results of TAR review

In re: Actos Antitrust Litigation (D.C., Louisiana)

Cumbersome review protocol adopted by the court

EORHB, Inc. v. HOA Holdings LLC, (Delaware Chancery, October 15, 2012 and May 6, 2013)

Sua Sponte order directed use of computer-assisted review

Recently, the Court relieved parties of requirement to use TAR

What have the Courts said about TAR?

TEC

HN

OLO

GY-A

SSISTED REV

IEW 2

01

3


• Gabriel Technologies v. Qualcomm, (U.S.D.C., S.D. California, February 1, 2013)

Court approved Qualcomm’s application for fees, including $2.8 million for e-discovery vendor TAR services

What have the Courts said about TAR?

“...[counsel’s] decision to undertake a more efficient and less time-consuming method of document review to be reasonable under the circumstances ... and … reduced the overall fees and attorney hours required by performing electronic document review at the outset.”

more efficient less time-consuming

reduced the overall fees and attorney hours

TEC

HN

OLO

GY-A

SSISTED REV

IEW 2

01

3


• In re Biomet Hip Implant Products Liability Litigation (N.D. Ind. April 13, 2013)

• Biomet combined keyword searching with Technology Assisted Review Keyword searches were the first step in culling Biomet’s documents

• Plaintiffs objected to Biomet’s process Application of keyword search prior to employing TAR tainted the entire

process

• “The issue…isn’t whether predictive coding is a better way of doing things than keyword searching prior to predictive coding. I must decide whether Biomet’s procedure satisfies it discovery obligations and, if so, whether it must also [apply predictive coding to the entire document collection]. Judge Robert Miller, Jr., framing the Biomet discovery issues

In re Biomet Hip Implants - Keyword Culling Used with TAR

TEC

HN

OLO

GY-A

SSISTED REV

IEW 2

01

3


• Biomet’s procedure satisfied its FRCP discovery obligations “I can’t find that the likely benefits of the discovery proposed by

[plaintiffs] equals or outweighs its additional burden on, and additional expense to, Biomet.”

• Basis for the Decision Proportionality and Cost Analysis

Statistical Sampling

Acceptance of TAR discovery tools

• Final Considerations Court anticipated continued meet & confers on discovery issues

Additional reasonably-targeted search terms

Production of non-privileged documents in the statistical sample

Plaintiffs may pay to use TAR on the entire document collection

In re Biomet – Holding and Other Considerations

TEC

HN

OLO

GY-A

SSISTED REV

IEW 2

01

3


Contact Information

Contact information:

Dickstein Shapiro LLP

Richard Perrin E-Discovery Counsel

[email protected] 202.420.3198

www.dicksteinshapiro.com

eTERA Consulting, LLC

Bob Gillespie Vice President, ESI Consulting

[email protected] 202.905.4473

www.eteraconsulting.com

http://www.eteraconsulting.com/

technology assisted review (tar) - dickstein shapiro llp

Documents