the hathitrust research center (htrc): an overview and demo
DESCRIPTION
The session will provide an overview of the HathiTrust Research Center including its mission and current status. It will also include a demonstration of current HTRC phase one technology and services. Additionally, the speakers will address the HTRC's role in supporting humanities research at scale.TRANSCRIPT
The HathiTrust Research Center (HTRC): An Overview and Demo
IU Librarians’ Day | IUPUI Libraries | 06.07.13
Robert H. McDonald -‐ @mcdonald -‐ IU Libraries Yiming Sun – IU Data to Insight Center Miao Chen – IU Data to Insight Center
Tweet US -‐ @HathiTrust #HTRC
Speaker Deck Slides
06.07.13 IU Librarians’ Day #HTRC @HathiTrust
hHp://bit.ly/13gWD7C
HTRC Mission
• Public research arm of the HathiTrust • Help researchers world-‐wide to accomplish tera-‐scale text data-‐mining and analysis – Develop cuOng-‐edge soPware tools for processing, analyzing text
– Develop cyberinfrastructure to enable HPC access to the HathiTrust Digital Library
• Established: July, 2011 • CollaboraWve center: Indiana University & University of Illinois
06.07.13 IU Librarians’ Day #HTRC @HathiTrust
HTRC Governance • Reports to the HathiTrust Board of Governors • HTRC ExecuWve Commi[ee
– J. Stephen Downie (Co-‐director), Professor and Associate Dean for Research, University of Illinois GSLIS
– Beth Plale (Co-‐director and Chair), Director Data To Insight Center and professor in the School of InformaWcs and CompuWng at Indiana University
– Robert H. McDonald, Associate Dean of Libraries/Deputy Director Data to Insight Center at Indiana University
– Beth Sandore Namachchivaya, Associate University Librarian for InformaWon Technology Planning & Policy at the University of Illinois
– John Unsworth, Vice Provost for Library & Technology Services and Chief InformaWon Officer at Brandeis University
• HTRC Advisory Board (See members next slide) • Google Public Domain agreement – in place for IU and
UIUC
HTRC Advisory Board • Cathy Blake, University of Illinois, Urbana-‐Champaign • Beth Cate, Indiana University • Greg Crane, TuPs University • Laine Farley, California Digital Library • Brian Geiger, University of California at Riverside • David Greenbaum, University of California at Berkeley • FoWs Jannidis, University of Wurzberg, Germany • Ma[hew Jockers, Stanford University • Jim Neal, Columbia University • Bill Newman, Indiana University • Bethany Nowviskie, University of Virginia • Andrey Rzhetsky, University of Chicago • Pat Steele, University of Maryland • Craig Stewart, Indiana University • David Theo Goldberg, University of California at Irvine • John Towns, NaWonal Center for SupercompuWng ApplicaWons • Madelyn Wessel, University of Virginia
HTRC Timeline
• Phase I: 18-‐month development cycle – Began 01 July 2011 – Demo of capability September 2012 (14 mo mark) at HTRC UnCamp I
• Phase II: broad availability of resource, begins 31 March 2013 – New HTRC Asst. Director for EducaWon and Outreach (Miao Chen)
– New listserv to drive user input: htrc-‐usergroup-‐l @ list.indiana.edu
06.07.13 IU Librarians’ Day #HTRC @HathiTrust
HTRC Next Steps
• Phase 2 availability of resource 31 March 2013 • Thanks to:
Photos from HTRC UnCamp 9.10.12 at Indiana University
HTRC Phase 2: Current Thrusts
• Grow HTRC User-‐base – Outreach and Engagement
• Input from HTRC Advisory Board • Input from HT BOG
– Town Hall Groups at DH, JCDL, JADH – Online Town Hall Groups
• Develop New SpecificaWons from User-‐Based Agile Development Methodology
• Develop and Integrate Sloan Cloud Components into the HTRC Infrastructure
06.07.13 IU Librarians’ Day #HTRC @HathiTrust
06.07.13 IU Librarians’ Day #HTRC @HathiTrust
HTRC Architecture Overview
Data API access interface
Portal
Security (OAuth2 WSO2 IS)
Algorithms and Worksets
Registry (WSO2 GR)
ApplicaWon submission
Audit
Cassandra cluster volume store
Solr index EnWty
ExtracWon Topic
Modeling Sentence Tokenizer
Word posiWon
Latent semanWc analysis
High level apps
Compute resources Storage resources
Blacklight
06.07.13 IU Librarians’ Day #HTRC @HathiTrust
VM Request
VM Image
Manager
VM Image Store
VM Image Builder
VM Manager
VM instance
Sloan Cloud
SSH Non-‐consumpWve Output Storage
user
HTRC Non-‐ConsumpUve Research Access (Sloan Cloud)
HTRC DemonstraWon
• Yiming Sun – Lead Technical Architect HTRC
06.07.13 IU Librarians’ Day #HTRC @HathiTrust
Metadata Enhancement
• Current metadata fields are MARC-‐based – E.g. publicaWon date, authors, Wtle, subject
• MARC fields are fundamental • Needed more fields of users’ interest for granular analyWcs (Metadata Enhancement)
• Solicit user requirements and prioriWze for implementaWon – Mainly digital humaniWes uses now
06.07.13 IU Librarians’ Day #HTRC @HathiTrust
Top Metadata Enhancement Items
• 1st round user requirement collecWon, top 3 items were metadata related: – Word frequency count and document length for a volume
– Metadata de-‐duplicaWon – Author Gender Analysis
• These top 3 items are in process for funcWonality within the current producWon system and will be available in the next quarter.
06.07.13 IU Librarians’ Day #HTRC @HathiTrust
Other Metadata Enhancement Items
• Stats analysis: m-‐idf • Readability score • Language • Topic modeling (e.g. LDA probability) • Genre • Era of compilaWon • Book length (e.g. short or long) • Concordance index (indexing with context) 06.07.13 IU Librarians’ Day #HTRC @HathiTrust
HTRC Upcoming Events
• DH 2013 – July 16-‐29, 2013 • JCDL 2013 – July 22-‐26, 2013 • HathiTrust Research Center UnCamp – Sept 8-‐9, 2013 – University of Illinois
• Catapult Symposium – IUB – Sept 2013 • JADH 2013 – September 19-‐21, 2013 • Ohio State University – Library Symposium – October 2013
• Educause 2013 – October 15-‐18, 2013
Thank You • This presentaWon was made possible with content provided by many HTRC colleagues John Unsworth, J. Stephen Downie, Robert H. McDonald, Beth Sandore, Yiming Sun, Guangchen Ruan, Lore[a Auvil, Kirk Hess, and many others…
• The HTRC Non-‐ConsumpWve Research Grant is graciously funded by the Alfred P. Sloan FoundaWon
• IU D2I-‐PTI is graciously funded by The Lilly Endowment, Inc.
• HTRC -‐ h[p://www.hathitrust.org/htrc • IU D2I Center -‐ h[p://d2i.indiana.edu/ • UIUC GSLIS -‐ h[p://www.lis.illinois.edu/
06.07.13 IU Librarians’ Day #HTRC @HathiTrust
Contact InformaWon
• General – Robert H. McDonald, HTRC ExecuWve Commi[ee
– [email protected] • Technical – Yiming Sun, Chief Architect, [email protected]
• Requests for capability, interest – Miao Chen, HTRC Asst. Director of EducaWon and Outreach, [email protected]
06.07.13 IU Librarians’ Day #HTRC @HathiTrust