DIGITAL PUBLIC RECORDS

Ryan ThornburgUniversity of North Carolina

DIGITAL PUBLIC RECORDS

• What are they?• How do I get them?• What do I do with them?

WHAT ARE DIGITAL PUBLIC RECORDS?

“all documents, papers, letters, maps, books, photographs, films, sound recordings, magnetic or other tapes, electronic data processing records‑ , artifacts, or other documentary material, regardless of physical form or characteristics”

WHAT ARE DIGITAL PUBLIC RECORDS?

“Persons requesting copies of public records may elect to obtain them in any and all media in which the public agency is capable of providing them. No request for copies of public records in a particular medium shall be denied on the grounds that the custodian has made or prefers to make the public records available in another medium.”

WHAT ARE DIGITAL PUBLIC RECORDS?

“No request to inspect, examine, or obtain copies of public records shall be denied on the grounds that confidential information is commingled with the requested nonconfidential information. If it is necessary to separate confidential from nonconfidential information in order to permit the inspection, examination, or copying of the public records, the public agency shall bear the cost of such separation.”

WHAT ARE DIGITAL PUBLIC RECORDS?

“Nothing in this section shall … require a public agency to respond to a request for a copy of a public record by creating or compiling a record that does not exist. … Nothing in this section shall be construed to require a public agency to put into electronic medium a record that is not kept in electronic medium.”

WHAT ARE USABLE PUBLIC RECORDS?

DATA. NOT DOCUMENTS.

OPEN GOVERNMENTDATA PRINCIPALS

1. CompleteAll public data is made available. Public data is data that is not subject to valid privacy, security or privilege limitations.

“ONGOING INVESTIGATION”?

• Time • Date• Location• Nature of a violation• Name of a person arrested,

charged, or indicted.• Sex of a person arrested,

charged, or indicted.

“ONGOING INVESTIGATION”?

• Age of a person arrested, charged, or indicted

• Address of a person arrested, charged, or indicted

• Employment of a person arrested, charged, or indicted

• Whether the arrest involved resistance

• Whether the arrest involved possession or use of weapons

“ONGOING INVESTIGATION”?

• Whether the arrest involved pursuit

• Description of any items seized in connection with the arrest

• Name of complaining witness • Sex of complaining witness• Age of complaining witness• Address of a complaining witness

• EXCEPT …

“ONGOING INVESTIGATION”?

• if release of the information is reasonably likely to pose a threat to the mental health, physical health, or personal safety of the complaining witness or materially compromise a continuing or future criminal investigation or criminal intelligence operation. (You can appeal)

“ONGOING INVESTIGATION”?

• if it will jeopardize the right of the State to prosecute a defendant or the right of a defendant to receive a fair trial or will undermine an ongoing or future investigation (police must first get a court order)

“ONGOING INVESTIGATION”?

“The use of a public record in connection with a criminal investigation or the gathering of criminal intelligence shall not affect its status as a public record.”

OPEN GOVERNMENTDATA PRINCIPALS

2. PrimaryData is as collected at the source, with the highest possible level of granularity, not in aggregate or modified forms.

OPEN GOVERNMENTDATA PRINCIPALS

3. TimelyData is made available as quickly as necessary to preserve the value of the data.

4. AccessibleData is available to the widest range of users for the widest range of purposes.

OPEN GOVERNMENTDATA PRINCIPALS

5. Machine processableData is reasonably structured to allow automated processing.

Not PDF.

OPEN GOVERNMENTDATA PRINCIPALS

6. Non-discriminatoryData is available to anyone, with no requirement of registration.

“No person requesting to inspect and examine public records, or to obtain copies thereof, shall be required to disclose the purpose or motive for the request.”

OPEN GOVERNMENTDATA PRINCIPALS

7. Non-proprietaryData is available in a format over which no entity has exclusive control.

OPEN GOVERNMENTDATA PRINCIPALS

8. License-freeData is not subject to any copyright, patent, trademark or trade secret regulation. Reasonable privacy, security and privilege restrictions may be allowed.

HOW DO I GET THEM?

• Request• Download• Scrape• API

REQUESTING DATA

• Know the ‘Custodian’• Know the Name of the Paper• Know the Fields You Want

REQUESTING DATAEvery public agency shall create an index of computer databases compiled or created (after 1998)• list of the data fields• description of the format or record layout• frequency with which the database is updated• list of any data fields to which public access is restricted• description of each form in which the database can be

copied using the agency's computer facilities• a schedule of fees for the production of copies in each

available form

(Database = “data or documents residing in a database management program or spreadsheet software”)

REQUESTING DATA“Pursuant to the state open records act, N.C. General Statute 132-1 to 132-10, I am requesting access to or copies of information contained in the [name of database], specifically the following fields: …“I request that the data be made available in the [filetype] format and copied onto either a CD-ROM or portable USB disk that I will provide.“If our request is denied in whole or part, I ask that you justify all deletions by reference to the specific exemptions of the act. We will also expect you to release all segregable portions of otherwise exempt material.“This material will be used in the preparation of news articles. As such, I ask that you waive all fees in the public interest. However, I am willing to pay up to $500 in your agency’s direct costs for search and copy. Please contact me if this amount is insufficient.”

REQUESTING DATA

• Reply: “as prompt as possible”• Copies: “as soon as reasonably

possible”• Denials: Should be in writing,

citing specific statutory reasons.

REQUESTING DATA

Costs:• “extensive use of information technology

resources or extensive clerical or supervisory assistance”

• “producing the record in the medium requested results in a greater use of information technology resources than that established by the agency for reproduction of the volume of information requested "

REQUESTING DATA

Costs:• “a special service charge, which shall be

reasonable and shall be based on the actual cost incurred for such extensive use of information technology resources or the labor costs of the personnel providing the services, or for a greater use of information technology resources that is actually incurred by the agency or attributable to the agency”

SCRAPING DATA

• OutWit– Firefox Add-On– www.outwit.com

SCRAPING DATA

• ScreenScraper– Downloadable Program for Windows & Mac– www.screen-scraper.com

SCRAPING DATA

API

WHAT DO I DO WITH THEM?

• Access & Excel• Google Fusion Tables & Maps• Tableau Public• Socrata• OpenBlock• PANDA

ACCESS & EXCEL

Sum Sort Filter Functions

GOOGLE FUSION TABLES

TABLEAU PUBLIC

TABLEAU PUBLIC

SOCRATA

OPENBLOCK

OPENBLOCK

OPENBLOCK

• http://github.com/openrural

• http://openblock.readthedocs.org

• http://openblockproject.org

PANDA

• pandaproject.net• demo.pandaproject.net

DIGITAL PUBLIC RECORDS

Ryan Thornburgryan.thornburg@unc.edu

@rtburg