latanya sweeney, ph.d.washstat.org/materials.html/20180125/20180125_sweeney.pdf · traditional...
TRANSCRIPT
![Page 1: Latanya Sweeney, Ph.D.washstat.org/materials.html/20180125/20180125_Sweeney.pdf · Traditional Belief System. Sweet spot. Who can be re-identified? Everyone! No one! You are wrong](https://reader033.vdocuments.site/reader033/viewer/2022050305/5f6e0df68e84ba3b6d617840/html5/thumbnails/1.jpg)
The elusive sweet spotsof privacy and utility
Latanya Sweeney, Ph.D.
![Page 2: Latanya Sweeney, Ph.D.washstat.org/materials.html/20180125/20180125_Sweeney.pdf · Traditional Belief System. Sweet spot. Who can be re-identified? Everyone! No one! You are wrong](https://reader033.vdocuments.site/reader033/viewer/2022050305/5f6e0df68e84ba3b6d617840/html5/thumbnails/2.jpg)
Privacy
Utility
![Page 3: Latanya Sweeney, Ph.D.washstat.org/materials.html/20180125/20180125_Sweeney.pdf · Traditional Belief System. Sweet spot. Who can be re-identified? Everyone! No one! You are wrong](https://reader033.vdocuments.site/reader033/viewer/2022050305/5f6e0df68e84ba3b6d617840/html5/thumbnails/3.jpg)
Privacy
Utility
“make up values”
“share raw values”
![Page 4: Latanya Sweeney, Ph.D.washstat.org/materials.html/20180125/20180125_Sweeney.pdf · Traditional Belief System. Sweet spot. Who can be re-identified? Everyone! No one! You are wrong](https://reader033.vdocuments.site/reader033/viewer/2022050305/5f6e0df68e84ba3b6d617840/html5/thumbnails/4.jpg)
Privacy
UtilityTraditional Belief System
![Page 5: Latanya Sweeney, Ph.D.washstat.org/materials.html/20180125/20180125_Sweeney.pdf · Traditional Belief System. Sweet spot. Who can be re-identified? Everyone! No one! You are wrong](https://reader033.vdocuments.site/reader033/viewer/2022050305/5f6e0df68e84ba3b6d617840/html5/thumbnails/5.jpg)
Privacy
UtilityTraditional Belief System Sweet spots
![Page 6: Latanya Sweeney, Ph.D.washstat.org/materials.html/20180125/20180125_Sweeney.pdf · Traditional Belief System. Sweet spot. Who can be re-identified? Everyone! No one! You are wrong](https://reader033.vdocuments.site/reader033/viewer/2022050305/5f6e0df68e84ba3b6d617840/html5/thumbnails/6.jpg)
Privacy
UtilityTraditional Belief System Sweet spot
![Page 7: Latanya Sweeney, Ph.D.washstat.org/materials.html/20180125/20180125_Sweeney.pdf · Traditional Belief System. Sweet spot. Who can be re-identified? Everyone! No one! You are wrong](https://reader033.vdocuments.site/reader033/viewer/2022050305/5f6e0df68e84ba3b6d617840/html5/thumbnails/7.jpg)
Who can be re-identified?
Everyone!
No one!
You are wrong.
We are right.
Ohm
Yakowitz
El Emam
![Page 8: Latanya Sweeney, Ph.D.washstat.org/materials.html/20180125/20180125_Sweeney.pdf · Traditional Belief System. Sweet spot. Who can be re-identified? Everyone! No one! You are wrong](https://reader033.vdocuments.site/reader033/viewer/2022050305/5f6e0df68e84ba3b6d617840/html5/thumbnails/8.jpg)
The Year is 1997. Building my thinking machine… when…
![Page 9: Latanya Sweeney, Ph.D.washstat.org/materials.html/20180125/20180125_Sweeney.pdf · Traditional Belief System. Sweet spot. Who can be re-identified? Everyone! No one! You are wrong](https://reader033.vdocuments.site/reader033/viewer/2022050305/5f6e0df68e84ba3b6d617840/html5/thumbnails/9.jpg)
Visit date
Diagnoses
Procedures
ZIP
Birth date
Sex
Name
Address
Date registered
Party affiliation
Date last voted
Dataset Voter List
Sweeney Simple Demographics Often Identify People Uniquely. 2000. dataprivacylab.org/projects/identifiability/index.html
![Page 10: Latanya Sweeney, Ph.D.washstat.org/materials.html/20180125/20180125_Sweeney.pdf · Traditional Belief System. Sweet spot. Who can be re-identified? Everyone! No one! You are wrong](https://reader033.vdocuments.site/reader033/viewer/2022050305/5f6e0df68e84ba3b6d617840/html5/thumbnails/10.jpg)
[02138, M,7/31/1945]
1997 Cambridge Voter List foreverdata.org
![Page 11: Latanya Sweeney, Ph.D.washstat.org/materials.html/20180125/20180125_Sweeney.pdf · Traditional Belief System. Sweet spot. Who can be re-identified? Everyone! No one! You are wrong](https://reader033.vdocuments.site/reader033/viewer/2022050305/5f6e0df68e84ba3b6d617840/html5/thumbnails/11.jpg)
[02138, M,7/31/1945]
1997 Cambridge Voter List foreverdata.org
![Page 12: Latanya Sweeney, Ph.D.washstat.org/materials.html/20180125/20180125_Sweeney.pdf · Traditional Belief System. Sweet spot. Who can be re-identified? Everyone! No one! You are wrong](https://reader033.vdocuments.site/reader033/viewer/2022050305/5f6e0df68e84ba3b6d617840/html5/thumbnails/12.jpg)
Date of Birth Mon/Yr Birth Year of Birth
ZIP5-digit
Town/Place
County
Gender
87%
58.4%
18.1%
3.7%
3.6%
0.04%
0.04%
0.04%
0.00004%
![Page 13: Latanya Sweeney, Ph.D.washstat.org/materials.html/20180125/20180125_Sweeney.pdf · Traditional Belief System. Sweet spot. Who can be re-identified? Everyone! No one! You are wrong](https://reader033.vdocuments.site/reader033/viewer/2022050305/5f6e0df68e84ba3b6d617840/html5/thumbnails/13.jpg)
My work was cited in the preamble of the
HIPAA Privacy Rule
(and in other policies worldwide).
![Page 14: Latanya Sweeney, Ph.D.washstat.org/materials.html/20180125/20180125_Sweeney.pdf · Traditional Belief System. Sweet spot. Who can be re-identified? Everyone! No one! You are wrong](https://reader033.vdocuments.site/reader033/viewer/2022050305/5f6e0df68e84ba3b6d617840/html5/thumbnails/14.jpg)
![Page 15: Latanya Sweeney, Ph.D.washstat.org/materials.html/20180125/20180125_Sweeney.pdf · Traditional Belief System. Sweet spot. Who can be re-identified? Everyone! No one! You are wrong](https://reader033.vdocuments.site/reader033/viewer/2022050305/5f6e0df68e84ba3b6d617840/html5/thumbnails/15.jpg)
.
Re-identification stimulates improvement.
![Page 16: Latanya Sweeney, Ph.D.washstat.org/materials.html/20180125/20180125_Sweeney.pdf · Traditional Belief System. Sweet spot. Who can be re-identified? Everyone! No one! You are wrong](https://reader033.vdocuments.site/reader033/viewer/2022050305/5f6e0df68e84ba3b6d617840/html5/thumbnails/16.jpg)
33 States Sell or Share Personal Health Data
Hooley S and Sweeney L. Survey of Publicly-Available State Health Databases. Paper 1075. 2013. thedatamap.org/states.html
![Page 17: Latanya Sweeney, Ph.D.washstat.org/materials.html/20180125/20180125_Sweeney.pdf · Traditional Belief System. Sweet spot. Who can be re-identified? Everyone! No one! You are wrong](https://reader033.vdocuments.site/reader033/viewer/2022050305/5f6e0df68e84ba3b6d617840/html5/thumbnails/17.jpg)
Only 3 States Used HIPAA Standards
Hooley S and Sweeney L. Survey of Publicly-Available State Health Databases. Paper 1075. 2013. thedatamap.org/states.html
![Page 18: Latanya Sweeney, Ph.D.washstat.org/materials.html/20180125/20180125_Sweeney.pdf · Traditional Belief System. Sweet spot. Who can be re-identified? Everyone! No one! You are wrong](https://reader033.vdocuments.site/reader033/viewer/2022050305/5f6e0df68e84ba3b6d617840/html5/thumbnails/18.jpg)
TechScience.org/a/2015092903/
![Page 19: Latanya Sweeney, Ph.D.washstat.org/materials.html/20180125/20180125_Sweeney.pdf · Traditional Belief System. Sweet spot. Who can be re-identified? Everyone! No one! You are wrong](https://reader033.vdocuments.site/reader033/viewer/2022050305/5f6e0df68e84ba3b6d617840/html5/thumbnails/19.jpg)
TechScience.org/a/2015092903/
![Page 20: Latanya Sweeney, Ph.D.washstat.org/materials.html/20180125/20180125_Sweeney.pdf · Traditional Belief System. Sweet spot. Who can be re-identified? Everyone! No one! You are wrong](https://reader033.vdocuments.site/reader033/viewer/2022050305/5f6e0df68e84ba3b6d617840/html5/thumbnails/20.jpg)
TechScience.org/a/2015092903/
![Page 21: Latanya Sweeney, Ph.D.washstat.org/materials.html/20180125/20180125_Sweeney.pdf · Traditional Belief System. Sweet spot. Who can be re-identified? Everyone! No one! You are wrong](https://reader033.vdocuments.site/reader033/viewer/2022050305/5f6e0df68e84ba3b6d617840/html5/thumbnails/21.jpg)
TechScience.org/a/2015092903/
![Page 22: Latanya Sweeney, Ph.D.washstat.org/materials.html/20180125/20180125_Sweeney.pdf · Traditional Belief System. Sweet spot. Who can be re-identified? Everyone! No one! You are wrong](https://reader033.vdocuments.site/reader033/viewer/2022050305/5f6e0df68e84ba3b6d617840/html5/thumbnails/22.jpg)
TechScience.org/a/2015092903/
![Page 23: Latanya Sweeney, Ph.D.washstat.org/materials.html/20180125/20180125_Sweeney.pdf · Traditional Belief System. Sweet spot. Who can be re-identified? Everyone! No one! You are wrong](https://reader033.vdocuments.site/reader033/viewer/2022050305/5f6e0df68e84ba3b6d617840/html5/thumbnails/23.jpg)
TechScience.org/a/2015092903/
![Page 24: Latanya Sweeney, Ph.D.washstat.org/materials.html/20180125/20180125_Sweeney.pdf · Traditional Belief System. Sweet spot. Who can be re-identified? Everyone! No one! You are wrong](https://reader033.vdocuments.site/reader033/viewer/2022050305/5f6e0df68e84ba3b6d617840/html5/thumbnails/24.jpg)
TechScience.org/a/2015092903/
Matched correct names to 43 percent of 81 samples
of shared “anonymous” data.
![Page 25: Latanya Sweeney, Ph.D.washstat.org/materials.html/20180125/20180125_Sweeney.pdf · Traditional Belief System. Sweet spot. Who can be re-identified? Everyone! No one! You are wrong](https://reader033.vdocuments.site/reader033/viewer/2022050305/5f6e0df68e84ba3b6d617840/html5/thumbnails/25.jpg)
Only 2 of the 30 states respondedto the WA re-identification
![Page 26: Latanya Sweeney, Ph.D.washstat.org/materials.html/20180125/20180125_Sweeney.pdf · Traditional Belief System. Sweet spot. Who can be re-identified? Everyone! No one! You are wrong](https://reader033.vdocuments.site/reader033/viewer/2022050305/5f6e0df68e84ba3b6d617840/html5/thumbnails/26.jpg)
So 20 years later, we have to build evidence state-by-state…
![Page 27: Latanya Sweeney, Ph.D.washstat.org/materials.html/20180125/20180125_Sweeney.pdf · Traditional Belief System. Sweet spot. Who can be re-identified? Everyone! No one! You are wrong](https://reader033.vdocuments.site/reader033/viewer/2022050305/5f6e0df68e84ba3b6d617840/html5/thumbnails/27.jpg)
Yoo J, Thaler A, Sweeney L, and Zang J. Risks to Patient Privacy: a re-identification of patients in Maine and Vermont statewide hospital data. Technology Science. 2017110701. November 7, 2017. http://techscience.org/a/2017110701 (White paper: dataprivacylab.org/projects/me/)
![Page 28: Latanya Sweeney, Ph.D.washstat.org/materials.html/20180125/20180125_Sweeney.pdf · Traditional Belief System. Sweet spot. Who can be re-identified? Everyone! No one! You are wrong](https://reader033.vdocuments.site/reader033/viewer/2022050305/5f6e0df68e84ba3b6d617840/html5/thumbnails/28.jpg)
.
Re-identification stimulates improvement.20 years later: privacy vulnerabilities exist.
![Page 29: Latanya Sweeney, Ph.D.washstat.org/materials.html/20180125/20180125_Sweeney.pdf · Traditional Belief System. Sweet spot. Who can be re-identified? Everyone! No one! You are wrong](https://reader033.vdocuments.site/reader033/viewer/2022050305/5f6e0df68e84ba3b6d617840/html5/thumbnails/29.jpg)
HIPAA Safe Harbor, Remove following:
(A) Names; (B) All geographic subdivisions, except first 2 digits
of ZIP code (no digits if ZIP population < 20K)(C) All elements of dates (except year) for dates (D) Telephone numbers; (E) Fax numbers; (F) Electronic mail addresses; (G) Social security numbers; (H) Medical record numbers; and other numbers(N) Web Universal Resource Locators (URLs); (O) Internet Protocol (IP) address numbers; (P) Biometric identifiers, etc
![Page 30: Latanya Sweeney, Ph.D.washstat.org/materials.html/20180125/20180125_Sweeney.pdf · Traditional Belief System. Sweet spot. Who can be re-identified? Everyone! No one! You are wrong](https://reader033.vdocuments.site/reader033/viewer/2022050305/5f6e0df68e84ba3b6d617840/html5/thumbnails/30.jpg)
Date of Birth Mon/Yr Birth Year of Birth
ZIP5-digit
Town/Place
County
Gender
87%
58.4%
18.1%
3.7%
3.6%
0.04%
0.04%
0.04%
0.00004%
![Page 31: Latanya Sweeney, Ph.D.washstat.org/materials.html/20180125/20180125_Sweeney.pdf · Traditional Belief System. Sweet spot. Who can be re-identified? Everyone! No one! You are wrong](https://reader033.vdocuments.site/reader033/viewer/2022050305/5f6e0df68e84ba3b6d617840/html5/thumbnails/31.jpg)
How many re-identifications under HIPAA Safe Harbor?
Everyone!
Less than 1%
You are wrong.
We are right.
Ohm
Lafky
El Emam
![Page 32: Latanya Sweeney, Ph.D.washstat.org/materials.html/20180125/20180125_Sweeney.pdf · Traditional Belief System. Sweet spot. Who can be re-identified? Everyone! No one! You are wrong](https://reader033.vdocuments.site/reader033/viewer/2022050305/5f6e0df68e84ba3b6d617840/html5/thumbnails/32.jpg)
Sweeney L, Yoo J, Perovich L, Boronow K, Brown P, Brody J. Re-identification Risks in HIPAA Safe Harbor Data: A study of data from one environmental health study. Technology Science. 2017082801. August 28, 2017. https://techscience.org/a/2017082801
![Page 33: Latanya Sweeney, Ph.D.washstat.org/materials.html/20180125/20180125_Sweeney.pdf · Traditional Belief System. Sweet spot. Who can be re-identified? Everyone! No one! You are wrong](https://reader033.vdocuments.site/reader033/viewer/2022050305/5f6e0df68e84ba3b6d617840/html5/thumbnails/33.jpg)
Sweeney L, Yoo J, Perovich L, Boronow K, Brown P, Brody J. Re-identification Risks in HIPAA Safe Harbor Data: A study of data from one environmental health study. Technology Science. 2017082801. August 28, 2017. https://techscience.org/a/2017082801
![Page 34: Latanya Sweeney, Ph.D.washstat.org/materials.html/20180125/20180125_Sweeney.pdf · Traditional Belief System. Sweet spot. Who can be re-identified? Everyone! No one! You are wrong](https://reader033.vdocuments.site/reader033/viewer/2022050305/5f6e0df68e84ba3b6d617840/html5/thumbnails/34.jpg)
When the redacted data contained the exact birth year, as allowed by the HIPAA Safe Harbor, we correctly identified 8 of 32 (25 percent) participants by name.Sweeney L, Yoo J, Perovich L, Boronow K, Brown P, Brody J. Re-identification Risks in HIPAA Safe Harbor Data: A study of data from one environmental health study. Technology Science. 2017082801. August 28, 2017. https://techscience.org/a/2017082801
![Page 35: Latanya Sweeney, Ph.D.washstat.org/materials.html/20180125/20180125_Sweeney.pdf · Traditional Belief System. Sweet spot. Who can be re-identified? Everyone! No one! You are wrong](https://reader033.vdocuments.site/reader033/viewer/2022050305/5f6e0df68e84ba3b6d617840/html5/thumbnails/35.jpg)
.
Re-identification stimulates improvement.20 years later: privacy vulnerabilities exist. 25% re-identified in HIPAA Safe Harbor study
![Page 36: Latanya Sweeney, Ph.D.washstat.org/materials.html/20180125/20180125_Sweeney.pdf · Traditional Belief System. Sweet spot. Who can be re-identified? Everyone! No one! You are wrong](https://reader033.vdocuments.site/reader033/viewer/2022050305/5f6e0df68e84ba3b6d617840/html5/thumbnails/36.jpg)
What about expert determinations
and best practices?
![Page 37: Latanya Sweeney, Ph.D.washstat.org/materials.html/20180125/20180125_Sweeney.pdf · Traditional Belief System. Sweet spot. Who can be re-identified? Everyone! No one! You are wrong](https://reader033.vdocuments.site/reader033/viewer/2022050305/5f6e0df68e84ba3b6d617840/html5/thumbnails/37.jpg)
A Public Records Request (FOIA)on the Original Bar Dataset.
Sweeney L, Von Lowenfeldt M and Perry M. Saying it's Anonymous Doesn't Make It So: re-identifications of "anonymized" law school data. Harvard University. Data Privacy Lab. White Paper. Oct 25, 2017. (White paper: dataprivacylab.org/projects/lawdata)
![Page 38: Latanya Sweeney, Ph.D.washstat.org/materials.html/20180125/20180125_Sweeney.pdf · Traditional Belief System. Sweet spot. Who can be re-identified? Everyone! No one! You are wrong](https://reader033.vdocuments.site/reader033/viewer/2022050305/5f6e0df68e84ba3b6d617840/html5/thumbnails/38.jpg)
Experts submitted protocols and datasets that they asserted were privacy protected and useful for the study.
We assumed the experts were correct and then tested whether we found examples of re-identifications that should not occur (“null hypothesis test”).
![Page 39: Latanya Sweeney, Ph.D.washstat.org/materials.html/20180125/20180125_Sweeney.pdf · Traditional Belief System. Sweet spot. Who can be re-identified? Everyone! No one! You are wrong](https://reader033.vdocuments.site/reader033/viewer/2022050305/5f6e0df68e84ba3b6d617840/html5/thumbnails/39.jpg)
Sweeney L, Von Lowenfeldt M and Perry M. Saying it's Anonymous Doesn't Make It So: re-identifications of "anonymized" law school data. Harvard University. Data Privacy Lab. White Paper. Oct 25, 2017. (White paper: dataprivacylab.org/projects/lawdata)
Null Hypothesis
Testing
![Page 40: Latanya Sweeney, Ph.D.washstat.org/materials.html/20180125/20180125_Sweeney.pdf · Traditional Belief System. Sweet spot. Who can be re-identified? Everyone! No one! You are wrong](https://reader033.vdocuments.site/reader033/viewer/2022050305/5f6e0df68e84ba3b6d617840/html5/thumbnails/40.jpg)
Protocol Description Test Result11-Anonymity Recodings and
aggregations, some subsets had 11 copies
Failed! Re-identifications.
11-Anonymity Plus
Same as above with less precise GPA and 25% drop of records
Failed! Re-identifications.
Enclave 5-anonymity version of first protocol in a secured setting
Failed! Re-identifications.
Standardized All values replaced with z-scores
Failed! Re-identifications. Effort, Know-how.
![Page 41: Latanya Sweeney, Ph.D.washstat.org/materials.html/20180125/20180125_Sweeney.pdf · Traditional Belief System. Sweet spot. Who can be re-identified? Everyone! No one! You are wrong](https://reader033.vdocuments.site/reader033/viewer/2022050305/5f6e0df68e84ba3b6d617840/html5/thumbnails/41.jpg)
11-Anonymized Protocol, Steps 1-2/141. Preliminary Steps
a. Recode School Name Abbreviationsb. Compute number of times bar taken for each personc. Recode Bar’s assignment of race differently
2. “Cleaning Steps”a. Drop students attending one than one law schoolb. Eliminate students who graduated prior to 1982c. Drop students that have a graduation date of 0d. Rename graduation year variablee. Eliminate schools that are not accredited or correspondence law
schools from AMSCHMP filef. Drop unaccredited law schoolsg. Drop correspondence law schools
![Page 42: Latanya Sweeney, Ph.D.washstat.org/materials.html/20180125/20180125_Sweeney.pdf · Traditional Belief System. Sweet spot. Who can be re-identified? Everyone! No one! You are wrong](https://reader033.vdocuments.site/reader033/viewer/2022050305/5f6e0df68e84ba3b6d617840/html5/thumbnails/42.jpg)
11-Anonymized Protocol, Step 3/143. Create graduation year ranges and recode variables by law school class:
Class One: 1982-1987, 1988-1990, 1991-1993, 1994-1996, 1997-1999, 2000-2002, 2003-2005, 2006-2008
Class Two: 1982-1990, 1991-1999, 2000-2008Class Three: 1982-1990, 1991-1999, 2000-2008
![Page 43: Latanya Sweeney, Ph.D.washstat.org/materials.html/20180125/20180125_Sweeney.pdf · Traditional Belief System. Sweet spot. Who can be re-identified? Everyone! No one! You are wrong](https://reader033.vdocuments.site/reader033/viewer/2022050305/5f6e0df68e84ba3b6d617840/html5/thumbnails/43.jpg)
11-Anonymized Protocol, Step 4/144. LSAT Protocol Two Cleaning
a. Drop students from Class Three Schools that are not in a 9 year band of at least 20 students
b. Create median LSAT value for law schools that remainc. Divide law schools into two categories: those on the 10-48 scale and
those on the 120-180 d. Categorize 1982-1990 students, Class Three schools into quintilese. Ascribe median school value to all students in school/period group,
even if no LSAT scoref. Categorize 1991-1999 students, Class Three schools into decilesg. Categorize 2000-2008 students, Class Three schools into decilesh. Set law school name to missing for students for any of the above
that have a quintile and decile values i. Drop remaining Class Three students. These are students that did
not fall into the quantile or decile distributions.
![Page 44: Latanya Sweeney, Ph.D.washstat.org/materials.html/20180125/20180125_Sweeney.pdf · Traditional Belief System. Sweet spot. Who can be re-identified? Everyone! No one! You are wrong](https://reader033.vdocuments.site/reader033/viewer/2022050305/5f6e0df68e84ba3b6d617840/html5/thumbnails/44.jpg)
11-Anonymized Protocol, Steps 5-6/145. Compute cell counts.6. Racial redaction for Class One school students, black and Hispanic:
a. Create a variable to count school/Class One graduation year band/race groupings
b. Create variable to indicate the number of students in the groupingc. Create URM variable that collapses race if school/graduation year band
group has less than 11 blacks or less than 11 hispanics, or less than 11 blacks and 11 hispanics. The URM variable sums black and hispanic students within the schoo/graduation year band group, based on one of these three conditions, and assigns the sum value to all black or hispanic students in that group. If the URM sum for the group is less than 11, all black and hispanic students within group have race set to missing.
![Page 45: Latanya Sweeney, Ph.D.washstat.org/materials.html/20180125/20180125_Sweeney.pdf · Traditional Belief System. Sweet spot. Who can be re-identified? Everyone! No one! You are wrong](https://reader033.vdocuments.site/reader033/viewer/2022050305/5f6e0df68e84ba3b6d617840/html5/thumbnails/45.jpg)
11-Anonymized Protocol, Steps 7-9/147. Racial redaction for Class One school students, white and other:
a. Create White/Other variable that collapses race if school/graduation year band group has less than 11 white or less than 11 other, or less than 11 whites and 11 others. The White/Other variable sums white and other students within the schoo/graduation year band group, based on one of these three conditions, and assigns the sum value to all white or other students in that group. If the White/Other sum for the group is less than 11, all white and other students within group have race set to missing.
8. If a school/graduation year group has a value for URM or White/Other (or both variables) that is less than 11, set race equal to missing for all students in that school/graduation year group
9. If there is a URM or White/Other variable value for a student (meaning race had to be collapsed) code race as URM or White/Other and set race for those students to missing
![Page 46: Latanya Sweeney, Ph.D.washstat.org/materials.html/20180125/20180125_Sweeney.pdf · Traditional Belief System. Sweet spot. Who can be re-identified? Everyone! No one! You are wrong](https://reader033.vdocuments.site/reader033/viewer/2022050305/5f6e0df68e84ba3b6d617840/html5/thumbnails/46.jpg)
11-Anonymized Protocol, Step 10/1410. Racial redaction for Class Two school students, white and other
a. Create a variable to count band/race groupingsb. Create variable to indicate whether the groupings created have more or
less than 11 students in them c. Create White/Other variable that collapses race if school/graduation
year band group has less than 11 white or less than 11 other, or less than 11 whites and 11 others. The White/Other variable sums white and other students within the schoo/graduation year band group, based on one of these three conditions, and assigns the sum value to all white or other students in that group. If the White/Other sum for the group is less than 11, all white and other students within group have race set to missing.
d. If the White/Other sum for the group is less than 11, all white and other students within group have race set to missing.
![Page 47: Latanya Sweeney, Ph.D.washstat.org/materials.html/20180125/20180125_Sweeney.pdf · Traditional Belief System. Sweet spot. Who can be re-identified? Everyone! No one! You are wrong](https://reader033.vdocuments.site/reader033/viewer/2022050305/5f6e0df68e84ba3b6d617840/html5/thumbnails/47.jpg)
11-Anonymized Protocol, Step 11-12/14
11. If a school/graduation year group has a value for URM or White/Other (or both variables) that is less than 11, set race equal to missing for all students in that school/graduation year group
12. If there is a URM or White/Other variable value for a student (meaning race had to be collapsed) code race as URM or White/Other and set race for those students to missing
![Page 48: Latanya Sweeney, Ph.D.washstat.org/materials.html/20180125/20180125_Sweeney.pdf · Traditional Belief System. Sweet spot. Who can be re-identified? Everyone! No one! You are wrong](https://reader033.vdocuments.site/reader033/viewer/2022050305/5f6e0df68e84ba3b6d617840/html5/thumbnails/48.jpg)
11-Anonymized Protocol, Steps 13-14/14
13. Racial redaction for Class Three school studentsRace is not Redacted for Class Three students as all Class Three students have already had law school name set to missing.
14. Create graduation year category variable to replace student's actual graduation year
![Page 49: Latanya Sweeney, Ph.D.washstat.org/materials.html/20180125/20180125_Sweeney.pdf · Traditional Belief System. Sweet spot. Who can be re-identified? Everyone! No one! You are wrong](https://reader033.vdocuments.site/reader033/viewer/2022050305/5f6e0df68e84ba3b6d617840/html5/thumbnails/49.jpg)
Sample of 11-Anonymity Dataset.
![Page 50: Latanya Sweeney, Ph.D.washstat.org/materials.html/20180125/20180125_Sweeney.pdf · Traditional Belief System. Sweet spot. Who can be re-identified? Everyone! No one! You are wrong](https://reader033.vdocuments.site/reader033/viewer/2022050305/5f6e0df68e84ba3b6d617840/html5/thumbnails/50.jpg)
Identifiability of 11-Anonymity Dataset
![Page 51: Latanya Sweeney, Ph.D.washstat.org/materials.html/20180125/20180125_Sweeney.pdf · Traditional Belief System. Sweet spot. Who can be re-identified? Everyone! No one! You are wrong](https://reader033.vdocuments.site/reader033/viewer/2022050305/5f6e0df68e84ba3b6d617840/html5/thumbnails/51.jpg)
Law School Graduation Programs
![Page 52: Latanya Sweeney, Ph.D.washstat.org/materials.html/20180125/20180125_Sweeney.pdf · Traditional Belief System. Sweet spot. Who can be re-identified? Everyone! No one! You are wrong](https://reader033.vdocuments.site/reader033/viewer/2022050305/5f6e0df68e84ba3b6d617840/html5/thumbnails/52.jpg)
Official List of Attorneys
![Page 53: Latanya Sweeney, Ph.D.washstat.org/materials.html/20180125/20180125_Sweeney.pdf · Traditional Belief System. Sweet spot. Who can be re-identified? Everyone! No one! You are wrong](https://reader033.vdocuments.site/reader033/viewer/2022050305/5f6e0df68e84ba3b6d617840/html5/thumbnails/53.jpg)
Self-Published Resumes w GPAs
![Page 54: Latanya Sweeney, Ph.D.washstat.org/materials.html/20180125/20180125_Sweeney.pdf · Traditional Belief System. Sweet spot. Who can be re-identified? Everyone! No one! You are wrong](https://reader033.vdocuments.site/reader033/viewer/2022050305/5f6e0df68e84ba3b6d617840/html5/thumbnails/54.jpg)
Self-Published Resumes w LSATs
![Page 55: Latanya Sweeney, Ph.D.washstat.org/materials.html/20180125/20180125_Sweeney.pdf · Traditional Belief System. Sweet spot. Who can be re-identified? Everyone! No one! You are wrong](https://reader033.vdocuments.site/reader033/viewer/2022050305/5f6e0df68e84ba3b6d617840/html5/thumbnails/55.jpg)
Photos of Graduation
Classes
![Page 56: Latanya Sweeney, Ph.D.washstat.org/materials.html/20180125/20180125_Sweeney.pdf · Traditional Belief System. Sweet spot. Who can be re-identified? Everyone! No one! You are wrong](https://reader033.vdocuments.site/reader033/viewer/2022050305/5f6e0df68e84ba3b6d617840/html5/thumbnails/56.jpg)
Club Memberships
![Page 57: Latanya Sweeney, Ph.D.washstat.org/materials.html/20180125/20180125_Sweeney.pdf · Traditional Belief System. Sweet spot. Who can be re-identified? Everyone! No one! You are wrong](https://reader033.vdocuments.site/reader033/viewer/2022050305/5f6e0df68e84ba3b6d617840/html5/thumbnails/57.jpg)
![Page 58: Latanya Sweeney, Ph.D.washstat.org/materials.html/20180125/20180125_Sweeney.pdf · Traditional Belief System. Sweet spot. Who can be re-identified? Everyone! No one! You are wrong](https://reader033.vdocuments.site/reader033/viewer/2022050305/5f6e0df68e84ba3b6d617840/html5/thumbnails/58.jpg)
.
Re-identification stimulates improvement.20 years later: privacy vulnerabilities exist. 25% re-identified in HIPAA Safe Harbor studyBest practices have privacy vulnerabilities.
![Page 59: Latanya Sweeney, Ph.D.washstat.org/materials.html/20180125/20180125_Sweeney.pdf · Traditional Belief System. Sweet spot. Who can be re-identified? Everyone! No one! You are wrong](https://reader033.vdocuments.site/reader033/viewer/2022050305/5f6e0df68e84ba3b6d617840/html5/thumbnails/59.jpg)
Formal Protection Models Provide Privacy, but utility?
• K-anonymity
• Differential privacy
• Synthetic Data
![Page 60: Latanya Sweeney, Ph.D.washstat.org/materials.html/20180125/20180125_Sweeney.pdf · Traditional Belief System. Sweet spot. Who can be re-identified? Everyone! No one! You are wrong](https://reader033.vdocuments.site/reader033/viewer/2022050305/5f6e0df68e84ba3b6d617840/html5/thumbnails/60.jpg)
.
Re-identification stimulates improvement.20 years later: privacy vulnerabilities exist. 25% re-identified in HIPAA Safe Harbor studyBest practices have privacy vulnerabilities.
Formal protection models work but not appropriate for all uses.
![Page 61: Latanya Sweeney, Ph.D.washstat.org/materials.html/20180125/20180125_Sweeney.pdf · Traditional Belief System. Sweet spot. Who can be re-identified? Everyone! No one! You are wrong](https://reader033.vdocuments.site/reader033/viewer/2022050305/5f6e0df68e84ba3b6d617840/html5/thumbnails/61.jpg)
What can be done?
1. Risk assessment instead of prescription
2. Sliding scales of access3. Transparent processes
U.S. Commission on Evidence-Based Policy Making. CEP Final Report: The Promise of Evidence-Based Policymaking. https://www.cep.gov/cep-final-report.html
![Page 62: Latanya Sweeney, Ph.D.washstat.org/materials.html/20180125/20180125_Sweeney.pdf · Traditional Belief System. Sweet spot. Who can be re-identified? Everyone! No one! You are wrong](https://reader033.vdocuments.site/reader033/viewer/2022050305/5f6e0df68e84ba3b6d617840/html5/thumbnails/62.jpg)
Published 2015-10-16
Sweeney L, Crosas M, Bar-Sinai M. Sharing Sensitive Data with Confidence: The Datatags System. Technology Science. 2015101601. October 16, 2015. http://techscience.org/a/2015101601
techscience.org
![Page 63: Latanya Sweeney, Ph.D.washstat.org/materials.html/20180125/20180125_Sweeney.pdf · Traditional Belief System. Sweet spot. Who can be re-identified? Everyone! No one! You are wrong](https://reader033.vdocuments.site/reader033/viewer/2022050305/5f6e0df68e84ba3b6d617840/html5/thumbnails/63.jpg)
Multi-level Repository DataTagsTag Type Description Security Features Access Requirements
Blue Public Clear storageClear transmission Open
Green Controlled public Clear storageClear transmission
Email, OAuth verified registration
Yellow Accountable Clear storageEncrypted transmit
Password, Registered , Approval, Click DUA
Orange More accountable Encrypted storageEncrypted transmit
Password, Registered, Approval, Signed DUA
Red Fully accountable Encrypted storageEncrypted transmit
Two-factor authentication, Approval, Signed DUA
Crimson Maximally restricted
MultiEncrypt storeEncrypted transmit
Two-factor authentication, Approval, Signed DUA
![Page 64: Latanya Sweeney, Ph.D.washstat.org/materials.html/20180125/20180125_Sweeney.pdf · Traditional Belief System. Sweet spot. Who can be re-identified? Everyone! No one! You are wrong](https://reader033.vdocuments.site/reader033/viewer/2022050305/5f6e0df68e84ba3b6d617840/html5/thumbnails/64.jpg)
Multi-level Repository DataTagsTag Type Description Security Features Access Requirements
Blue Public Clear storageClear transmission Open
Green Controlled public Clear storageClear transmission
Email, OAuth verified registration
Yellow Accountable Clear storageEncrypted transmit
Password, Registered , Approval, Click DUA
Orange More accountable Encrypted storageEncrypted transmit
Password, Registered, Approval, Signed DUA
Red Fully accountable Encrypted storageEncrypted transmit
Two-factor authentication, Approval, Signed DUA
Crimson Maximally restricted
MultiEncrypt storeEncrypted transmit
Two-factor authentication, Approval, Signed DUA
![Page 65: Latanya Sweeney, Ph.D.washstat.org/materials.html/20180125/20180125_Sweeney.pdf · Traditional Belief System. Sweet spot. Who can be re-identified? Everyone! No one! You are wrong](https://reader033.vdocuments.site/reader033/viewer/2022050305/5f6e0df68e84ba3b6d617840/html5/thumbnails/65.jpg)
Multi-level Repository DataTagsTag Type Description Security Features Access Requirements
Blue Public Clear storageClear transmission Open
Green Controlled public Clear storageClear transmission
Email, OAuth verified registration
Yellow Accountable Clear storageEncrypted transmit
Password, Registered , Approval, Click DUA
Orange More accountable Encrypted storageEncrypted transmit
Password, Registered, Approval, Signed DUA
Red Fully accountable Encrypted storageEncrypted transmit
Two-factor authentication, Approval, Signed DUA
Crimson Maximally restricted
MultiEncrypt storeEncrypted transmit
Two-factor authentication, Approval, Signed DUA
![Page 66: Latanya Sweeney, Ph.D.washstat.org/materials.html/20180125/20180125_Sweeney.pdf · Traditional Belief System. Sweet spot. Who can be re-identified? Everyone! No one! You are wrong](https://reader033.vdocuments.site/reader033/viewer/2022050305/5f6e0df68e84ba3b6d617840/html5/thumbnails/66.jpg)
Latanya [email protected]