computer assisted decision analysis in drug naming · predicts visual perception errors and some...

Computer Assisted Decision Analysis in Drug Naming

Bruce L. Lambert, Ph.D.Department of Pharmacy Administration

Department of Pharmacy PracticeUniversity of Illinois at Chicago

[email protected]

OverviewHow can computers be used to objectively measure similarities/differences between name pairs?What evidence is there that these measures are valid?How do we evaluate a name-retrieval system?What are the next steps, challenges, limitations

Preface: Need to Change Focus!Names are not enough: Must focus on drug products not drug namesSimilarity is not enough: Must focus on similarity and frequency (of prescribing)Error reduction is not enough: Must focus on harm reductionPairs are not enough: Must focus on overall confusability/intelligibility of individual namesHow to balance public risk against private/corporate benefit?

Why Do These Errors HappenSimilarity- and frequency-based errors in cognitive processingMemory (recall and recognition)Perception (visual and auditory)Motor control (picking wrong drug from drop-down menu)Poorly designed systems (e.g., handwritten orders, oral orders, no CPOE, etc.)

Naming Decisions Must Be Grounded in Best Available Scientific Evidence

Public policy decisions must be based on validated scientific evidence, not marketing claims

Peer reviewed publicationsRelevant evidenceValidated measuresTransparency, full disclosure of methods, reproducibility, objectivity

Different Types of SimilarityName similarity

Orthographic (spelling)Predicts visual perception errors and some motor control errors

Phonetic or phonological (pronunciation)Predicts auditory perception errors and short-term memory errorsNeed to account for dialect and foreign language accent

Product similarityComposite of name and non-name (e.g., strength, dosage form, route, schedule, color, shape, etc.)Predicts variety of practical confusions and moderates effects of other types of similarity

Phonological Similarity Not Enough

Avandia 4mg p.o. ?

http://www.ismp.org/msaarticles/a072600safety.html

Coumadin 4mg p.o. ?

Tegretol 4mg p.o. ?Tequin 400mg p.o. ?

http://www.medmal-law.com/illegibl.htm

Plendil or Isordil?

Isordil® prescribedPlendil® dispensedCardiologist found negligent$450,000 damage awardFirst ever award for bad penmanship!

Different Types of SimilarityDifferent types of similarity are associated with different types of errorsWe want to prevent all types of errorsTherefore, screening process must use validated measure of all three types of similarity

Orthographic, phonetic, productHandwritten (cursive, print), spoken (dialect, foreign language accent)

Objective Measures of Name Similarity

N-gram: based on proportion of n-letter subsequences that two names have in common

Bigram: two-letter subsequences (e.g., Premarin=Pr, re, em, ma, ar, ri, in)Trigram: three-letter subsequences (Pre, rem, etc.)Dice coefficient: Proportion of n-grams in common

Edit distance: the number of insertions deletions or substitutions needed to transform one name into anotherAlignment distance methods

Kondrak’s ALINEFisher’s AL-DIST

Objective Measures of Name Similarity

N-gram and edit distance measures can be used on any formal representation of the name (spelling or phonological)

Use phonetic alphabet (e.g., International Phonetic Alphabet or ARPAbet)ARPAbet: Zyprexa: z ay p r eh k s ax Phoneme bigrams: z ay, ay p, p r, r eh, eh k, k s, s ax

Objective Measures of Similarity

There are many variations on these basic measures

Add spaces before or after to emphasize beginning or ending of nameUse different weights depending on position of lettersUse different equations to compute numerical similarity (Dice, Hamming, etc.)Allow approximate matches between letters (e.g., m=n, a=e=i=o=u)

Objective Measures of Similarity

Power of simple descriptive analysesTen most common three-letter prefixes in US brand names:

Pro-, Bio-, Car-, Tri-, Vit-, Pre-, Nut-, Ult-, Con-, Per-

Lambert BL, Chang KY, Lin SJ. Descriptive analysis of the drug name lexicon. Drug InfJ. 2001;35:163-172.

Distribution of Distance Scores

0%

10%

20%

30%

40%

50%

Normalized Edit Distance

Perc

ent

USPTO 0.00% 0.00% 0.00% 0.00% 0.05% 0.36% 1.11% 5.51% 20.68% 41.25% 31.04%

USAN 0.04% 0.00% 0.01% 0.04% 0.18% 1.28% 3.95% 13.81% 28.32% 31.17% 21.20%

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 More

Lambert BL, Chang KY, Lin SJ. Descriptive analysis of the drug name lexicon. Drug Inf J. 2001;35:163-172.

Distribution of Similarity Scores

0 %

1 0 %

2 0 %

3 0 %

4 0 %

5 0 %

6 0 %

7 0 %

8 0 %

9 0 %

1 0 0 %

T rig ra m -2 b S im i la r i ty

Perc

ent

U S P T O 9 2 .4 9 % 0 .4 4 % 5 .9 0 % 0 .7 3 % 0 .3 1 % 0 .0 9 % 0 .0 3 % 0 .0 1 % 0 .0 0 % 0 .0 0 % 0 .0 0 %U S A N 8 0 .0 7 % 1 2 .1 3 % 5 .9 1 % 1 .4 2 % 0 .2 8 % 0 .1 0 % 0 .0 4 % 0 .0 1 % 0 .0 0 % 0 .0 0 % 0 .0 4 %

0 0 .1 0 .2 0 .3 0 .4 0 .5 0 .6 0 .7 0 .8 0 .9 M o re

Lambert BL, Chang KY, Lin SJ. Descriptive analysis of the drug name lexicon. Drug Inf J. 2001;35:163-172.

Objective Measures Do Predict Probability of Human Error

Similarity accurately distinguishes between known error pairs and non-error pairsGreater objective similarity correlated with higher rates of recognition memory errors by laypeople and pharmacistsGreater similarity correlated with lower rates of free recall errorsObjective similarity correlated with subjective similarity (for experts and laypeople)Similarity neighborhoods predict visual perception errors

Similarity accurately distinguishes between known error pairs and non-error pairs

Histogram of trigram string similarities for 969 error pairs and 969 control pairs. Vertical axis is on a logarithmic scale. Light bars represent error pairs. Dark bars represent control pairs. Values at ends of vertical bars are frequencies. Values on the horizontal axis represent the bins for the histogram. For example, (0, 0.1) means "greater than 0 and less than 0.1," and [0.1, 0.2) means "greater than or equal to 0.1 and less than 0.2." From: Lambert: Am J Health Syst Pharm, Volume 54(10).May 15, 1997.1161-1171

0

50

100

150

200

250

1 2 3 4 5

Similarity Level

Freq.

0

0.2

0.4

0.6

0.8

1 Error Rate

CorrectErrorObservedPredicted

Figure 1. Effect of spelling similarity on pharmacists recognition memory errorsLambert BL, Chang KY, Lin SJ. Effect of orthographic and phonological similarity on false recognition of drug names. Soc Sci Med. 2001;52:1843-1857.

0

10

20

30

40

50

1 2 3Similarity Level

Freq

uenc

y

0%

20%

40%

60%

80%

100%

Rat

e

0 1 2 3 Observed Predicted

Figure 2. Effect of spelling similarity on pharmacists free recall errors.Lambert BL, Chang K-Y, Lin S-J. Immediate free recall of drug names: effects of similarity and availability. Am J Health-Syst Pharm. 2003;60:156-168.

0

10

20

30

40

50

60

70

1 2 3

Similarity Level

Freq

uenc

y

0%

20%

40%

60%

80%

100%

Rat

e

0 1 2 3 Observed Predicted

Figure 3. Effect of phonological similarity on pharmacists’ free recall errors.Lambert BL, Chang K-Y, Lin S-J. Immediate free recall of drug names: effects of similarity and availability. Am J Health-Syst Pharm. 2003;60:156-168.

0

0.5

1

1.5

2

2.5

3

0 0.2 0.4 0.6 0.8 1

Trigram Similarity (Dice Coefficient)

Pros

cale

D

issi

mila

rity

Figure 4. Relationship between objective similarity and lay people’s subjective dissimilarity.Lambert BL, Donderi D, Senders J. Similarity of drug names: Objective and subjective measures. Psychology and Marketing. 2002;19(7-8):641-661.

Neighborhoods Matter Concept of similarity “neighborhood” is key part of modern theories of visual and auditory perceptionNeighborhood characteristics

Frequency (of prescribing)DensityRadius (i.e., how close does a name have to be to be in another name’s neighborhood?)

Neighborhood Illustration

X

High frequency neighbor

Low frequency neighbor

Target name

Neighborhood radius

Dense Neighborhoods: High and Low Frequency

X

X

X

X

1) High SF, High NF, High ND 2) High SF, Low NF, High ND

4) High SF, Low NF, Low ND

X

3) High SF, High NF, Low ND

XX

X

X

7) Low SF, High NF, Low ND 8) Low SF, Low NF, Low ND

X

5) Low SF, High NF, High ND 6) Low SF, Low NF, High ND

X X

1) High SF, High NF, High ND 2) High SF, Low NF, High ND

4) High SF, Low NF, Low ND

XX

3) High SF, High NF, Low ND

XXXX

X

XX

7) Low SF, High NF, Low ND 8) Low SF, Low NF, Low ND

XX

5) Low SF, High NF, High ND 6) Low SF, Low NF, High ND

XX XX

ExamplesHigh log SF names (log SF > 7): Ventolin®, Dyazide®, Provera®

Low log SF names (log SF < 3): Vistazine®, Antispas®, Protaphane®

Name from a sparse neighborhood: Flexeril®(no neighbors in NAMCS/NHAMCS)Name from a dense neighborhood: Dynabac®, Synalar®, Rynatan®, Dynapen®, Dynacirc®, Dynacin®, Cynobac®

0

0.2

0.4

0.6

0.8

1

1.2

0 2 4 6 8 10

Log SF

Erro

r Rat

eHigh NDLow ND

Figure 5. Effect of similarity neighborhood on RPh visual perception of drug names.Lambert BL, Chang K-Y, Gupta P. Effects of frequency and similarity neighborhoods on pharmacists' visual perception of drug names. Soc Sci Med. in press.

Objective Measures: Conclusions

They work.They are not perfect.Better on population basis than on individual basisBetter for public health than for legal wranglingWe should be using them.

Software DemonstrationName searching

SpellingN-gram, edit distance

PronunciationN-gram, edit distance

Product SearchingName, dosage form, strength, routeEach weighted for importance

Lambert BL, Yu C, Thirumalai M. A system for multi-attribute drug product comparison. Journal of Medical Systems. in press.Lambert BL, inventor. Apparatus, method, and product for multi-attribute drug comparison. US patent 6,529,892. March 4, 2003.

How can computer resources be used to calculate weights for various elements in name similarity?

One way is to calculate a composite similarity score using multiple distinct similarity measuresUse multiple measures to predict error probability or some other outcomeExpert = 0.69 - 0.01*Editex - 0.30*NED + 0.22*Trigram2b - 0.02*EditSoundex

R2 = 0.406

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0.00 0.10 0.20 0.30 0.40 0.50 0.60 0.70 0.80 0.90 1.00

Observed

Pred

icte

d

Figure 6. Using multi-measure regression model to predict expert similarity judgments.Lambert BL, Yu C, Thirumalai M. A system for multi-attribute drug product comparison. Journal of Medical Systems. in press.

Actual Name Retrieval Results for Query Name: Curosurf®

CurecalCurasilkCurafilCurasaltInfasurfCurasolCurasaltCuragardCurisoneAtrosulfCurasolUrocurExosurfVirosureCurasilkExosurfCurasoreCurasoreCurasorbCurasorb

Expert RatingsCombined Model

See: Lambert, B. L., Yu, C., Thirumulai, M. (2004). A system for multiattribute drug product comparison. Journal of Medical Systems, 28(1), 29-54.

Evaluating a Drug Product Search/Retrieval System

From the FDA’s RFQ 223-02-5618: Empirical comparisons of the system's ability to identify at least 75% of the potentiallyconfusing names that have been identified by all the safety evaluators and included in the review of the proposed proprietary name. This will be done with the most recently completed 100 proposed proprietary name reviews.

This is not the appropriate test. Easy to identify 75% of confusing names, just return a list of all names, and it is guaranteed to include 100% of confusing names. Trick is for search system to retrieve confusing names without lots of false positives.


From the FDA’s RFQ 223-02-5618:System's capability to identify 95% of the known and published known drug name pairs. (Presumably referring to error pairs.)Pairs can be useful unit of analysis in assessing association between similarity and risk, but not correct unit of analysis in assessing retrieval performance.Published lists do not distinguish between actual errors and near misses or pairs that simply raised concerns. These lists containvoluntary reports that are known to miss > 99% of actual errors.Regulatory agencies must approve individual names, not pairs of names.N pairs of names yields [N(N-1)]/2 pairs. Positive predictive value of tests based on rare events from large populations is very poor.


Appropriate measures are recall and precisionRecall: the number of relevant names retrieved divided by the total number of relevant names in the databasePrecision: the number of relevant names retrieved divided by the total number of names retrievedRelevant names identified by method of pooled relevance judgments (see http://trec.nist.gov) See: Lambert, B. L., Yu, C., Thirumulai, M. (2004). A system for multiattribute drug product comparison. Journal of Medical Systems, 28(1), 29-54.

Example of Recall/Precision Curve

http://www.itl.nist.gov/iaui/894.02/works/presentations/bcs-irsg/sld012.htm

What level of performance can be expected? Variety of tasks.

http://www.itl.nist.gov/iaui/894.02/works/presentations/bcs-irsg/sld014.htm

What level of performance can be expected?

0.00

0.10

0.20

0.30

0.40

0.50

0.60

0.70

0.80

0.90

1.00

0.00 0.10 0.20 0.30 0.40 0.50 0.60 0.70 0.80 0.90 1.00Recall

Prec

isio

n

Figure 9. Precision of editex retrieval method at 11 levels of recall (mean precision = 17.4%).


How Well Do Human Experts Do?

0.00

0.10

0.20

0.30

0.40

0.50

0.60

0.70

0.80

0.90

1.00

0.00 0.10 0.20 0.30 0.40 0.50 0.60 0.70 0.80 0.90 1.00

Recall

Prec

isio

n

Figure 13. Precision of expert rating retrieval method at 11 levels of recall (mean precision = 26.7%).


How can computer resources be used to calculate weights for various elements in name similarity?

Compute distinct similarity score for each product attribute

Name, Dosage Form, Strength, Route of Administration, ScheduleIndication, Shape, Color, etc.Use equivalence classes for approximate matching of attribute values (e.g., tablet and capsule)

Use regression or other modeling techniques to assign weights to various attributesUse ISMP/MERP error data to estimate the importance of various attributes

Can computer assisted pattern recognition support the decision process to determine name/name similarities?

Yes, but…Problems

False positivesFalse negativesReliability of data for modeling (too often based on voluntary reports)Determination of a threshold beyond which a name is “too confusing to approve”Validation of predictive modelsWhich database to search?

SummaryComputers can be used to objectively measure differences between name pairs, but need to assess all dimensions of similarity.Computers can be used to calculate weights for various elements in name similarity, but good evidence about the relative importance of non-name attributes is lackingPlease contact me to discuss further. [email protected].

computer assisted decision analysis in drug naming · predicts visual perception errors and some...

Documents