APPLICATION INFORMATION Title: TRIPTYCH: A Multimodal Interface for Text Recognition, Image Parsing, and Translation, for InterdisciplinarY Collaboration in the Humanities Field of Project: Humanities Description of Project: The relationships between art, literature, and institution have long inspired Himalayan religious narrative expression, and currently inform the study of Buddhism across a range of disciplines including religious studies, philology, cultural history, and the history of art. We propose TRIPTYCH, a multimodal, HCI interface that will combine with Mirador Viewer to allow for simultaneous searching, annotation, and collaboration across three sources depicting the Life of the Buddha: image, machine-readable transcript, and text. Building on current technologies for annotation, shareable collections, robust filtering and display mechanisms, and collaboration, this web-based platform will provide a dynamic space for engaging and analyzing the relationships between texts and images that is both content and data driven, and that is relational and non-linear. It will be useful for any discipline where comparison of digitized material artifacts, machine-readable text, and associated literary documents is paramount, and in learning environments that emphasize undergraduate research with a focus on active learning and collaboration, visual and data analysis, and markup of documents and images.

TRIPTYCH: A Multimodal Interface for Text Recognition, Image Parsing, and Translation, for InterdisciplinarY

Collaboration in the Humanities A Level II Start-Up Project

1. TABLE OF CONTENTS 1. TABLE OF CONTENTS 2. LIST OF PARTICIPANTS

Directors and Staff Advisory Board Letters of Support

3. ABSTRACT Statement of Innovation Statement of Humanities Significance Enhancing the Humanities Through Innovation

(1) Examining relationships on a large scale (2) Accessibility and use for undergraduate research (3) Analyzing and Sharing Discoveries (4) Extensibility to Other Disciplines

Environmental Scan History and Duration of the Project Work plan Staff Final Product Dissemination

5. PROJECT BUDGET 6. BIOGRAPHIES 7. DATA MANAGEMENT PLAN

Roles and Responsibilities Data to Be Generated User Annotation and Tour Permission Matrix Period of Data Retention Data Formats and Dissemination

8. LETTERS 9. APPENDICES

A. Publications, Conferences, and Workshops on Life of the Buddha B. Completed Digitized Materials To-date (fully edited for ingestion into interface) C. Examples of Coursework and Undergraduate Research D. Wireframes E. Robust Tools Tested F. Technical Specifications G. Glossary of Components (identified as Phase I of Stage I start-up, and Phases II and III to be created through additional funding or post start-up) H. Work Plans (Grant Period: Stage I, and Overall Project: Stages I-III) I. Bibliography

3. ABSTRACT The challenge of studying visual art, literature, and their institutional contexts in a synthetic fashion is widespread across the humanities today. TRIPTYCH uses the rich example of narrative mural art in Tibet to enable scholars to find relational pathways between image, text, and context. The project will result in an unprecedented multimodal interface for interactive visualization including representation at various scales, markup of text-image relationships, tagging and annotation, and qualitative and quantitative data analysis. Focused on the most extensive mural of the life of the Buddha in Tibet, the project also provides the first digital platform for researching the foundational Buddhist narrative. Building on the Mirador Viewer framework, TRIPTYCH will create a collaborative environment for research across the disciplines, including religious studies, philology, cultural history, and art history, as well as an engaging learning environment for students. Statement of Innovation Augmenting technologies for annotation, shareable collections, robust filtering, and collaboration, TRIPTYCH will provide a dynamic space for analyzing relationships between texts and images that is both content and data driven, and that is relational and non-linear. It will be of value for disciplines where simultaneous exploration and analysis of digitized material artifacts, associated literary documents, and their relationships and context is paramount, both for research and teaching. Statement of Humanities Significance This project addresses a question of abiding interest in the humanities and of central importance to the study of religion: How can we best explore art, literature, and institution as a synthetic whole? If the Buddha narrative is paramount in Buddhism, scholars working in disciplinary isolation have overlooked its complex expression. This interface breaks down boundaries by creating a collaborative digital environment enabling research across distinctive modes of inquiry.

4. NARRATIVE Enhancing the Humanities Through Innovation From the Egyptian Coffin Texts and Book of the Dead carved on pyramid walls and sarcophagi, to Chinese and Japanese scrolls illustrating scenes from literary works, to Greek vase paintings, to medieval manuscript illuminations, to financial artifacts from antiquity to modern times, authors and artists have been inspired to transpose images into text and transcriptions into images, and to share those works through acts of publication and translation. Yet, never before the advent of digital scholarship were scholars able to examine the relationships between texts and images on a large scale, save and share their discoveries with others, or find meaningful ways to visualize and publish the results for a range of users and disciplines. As a multimodal interface building upon the Mirador Viewer developed by Stanford University with support from the Andrew W. Mellon Foundation, TRIPTYCH is a packaged solution to this pervasive set of challenges: (1) How do we design an interface that examines relationships on a large scale between material artifacts and images, foreign language texts and translations, and scholarly documentation? (2) How do we make digital research tools accessible and useful for undergraduate research? (3) As researchers in a specific discipline, how do we analyze and share our discoveries in meaningful ways? (4) How do we make it extensible to other disciplines? (1) Examining relationships on a large scale The Life of the Buddha (LOTB) project will provide material for TRIPTYCH to serve as the first interactive digital presentation of visual and literary narratives of the Buddhas life story as recorded in Tibet. The LOTB project is based on a detailed series of murals preserved at the famed Tibetan Buddhist monastery of Jonang, together with an extended literary narrative by the monasterys founder Taranatha (1575-1634). These murals date from the 1630s, and are among only a handful of fully preserved narrative paintings in Central Tibet. They are also some of the few murals in Tibet explicitly linked to an extant collection of narrative, poetic, ritual, and technical painting literature about the Buddha. Practically nothing has been written about the Jonang murals, and no complete visual documentation has ever been attempted (Appendix I). TRIPTYCH acts as an extension to Mirador Viewer; it is a discovery and analysis interface that will enable scholars to find relational pathways in the Life of the Buddha across three panels (image, text, and context, with accompanying metadata), and then curate and analyze four specific pattern types: identification of topics and themes; delineation of varied narrative structures; relationships between image and text, including spatial relations; and language usage within the images and texts (Appendix D). We believe these patterns serve as an exemplar for humanities studies that engage in analysis of visual and textual narratives, and connect those narratives to historical and cultural contexts. In the next three points, we further describe how this interface will facilitate authentic learning and research. (2) Accessibility and use for undergraduate research Scholarly research conducted by experts should, in our view, be directly related to the goals of undergraduate education (Appendix C). TRIPTYCH pedagogically echoes the call for best practices in undergraduate education: personal database construction, collaborative research, and interactive teaching (Cohen); text creation partnership (Hirsch, Bunde, Martin, Tomasek); identifying taxonomies to discover the message in the document (Marsh); and moving effortlessly from the macro to the granular (Lavin). Perhaps more importantly, we see TRIPTYCH as an interface for obtaining transferable skills to succeed in the globally networked world, to produce knowledge in and with the network (Davis). To this end, our project seeks to encourage and make possible a synergy between undergraduate learning and scholarly research about a particular corpus of visual and textual resources in a digital environment, in a manner that creates new public knowledge for all participants, novice and expert alike. We expect users to develop the following topical and technical skills through manipulation of the TRIPTYCH-Mirador Viewer interface:

Topical Skills Technical Skills

*Distinguish a central narrative by identifying key figures, events, places, and ideas *Understand the context of the narrative within architectural contexts *Understand the distinctions and interrelations of textual and visual narrative (Djeraba) *Analyze visual and textual narratives at the micro level in order to explain how distinct elements combine to create larger patterns (Benitez and Chang) *Contextualize visual and textual narratives in relation to larger cultural and historical practices *Gain insights into media collections through experiential and affective computing methods (Lew)

*Facility in collaborative, project-based, or inquiry-based learning *Annotate, perform parametric and faceted searches, use spatial markup *Curate and share tours *Perform data analysis and data visualization on gathered research *Conduct and present research in primary textual and visual sources in the following manner (Ferster): 1. Ask a question 2. Search for information 3. Structure the information 4. Envision an answer 5. Represent answer visually and textually 6. Tell a story that uses evidence to present the answer

(3) Analyzing and Sharing Discoveries As for the humanistic aspect of our project, the literary source for the narrative, the Hundred Acts of the Buddha, is one of the most extensive Tibetan compositions portraying the deeds of the Buddha, and also contains detailed information about the authors sources and literary aims. A related text, the Descriptive Guide, presents a frame-by-frame discussion of the iconography, composition, and symbolism in Jonangs narrative murals. A further work, the Guide to Jonang Monastery, together with other related works, maps out a plan of the murals architectural setting within the monastic complex including the assembly hall and upper gallery, and present a catalogue of religious objects, statues, and other materials present at the time of their construction. To date, no detailed studies or translations of this literature have been published (Appendix I). In order to foster collaborative research on this literary and visual material amongst scholars of various disciplines, TRIPTYCH will support research and results presentation on the narrative, topical, and thematic aspects of the visual/textual narrative corpus as a whole. The interface will allow, for the first time, comparative analysis across visual narrative (Jonang murals), textual narrative (Hundred Acts), and the guidelines for visual narrative construction (Descriptive Guide).

Researchers will be able to navigate chronologically or non-linearly through the visual narrative according to the divisions of a literary work (Benitez and Chang), and select foci to zoom and center a particular panel region (in the LOTB, foci represent the Hundred Acts upon which the mural is based). In the LOTB project murals, hundreds of topics and themes appear through the juxtaposition of active human figures, diverse landscape and architectural settings, and textual inscriptions that bind Taranatha's longer literary renderings to the visual narrative. Through the use of controlled vocabulary search functions (based on standard systems of visual analysis such as Getty Vocabularies), users can identify visual elements on both macro and micro scales (individual figures, locations, narrative vignettes) and create collections of those elements to understand their narrative and thematic relationships (Djerba). The

ability to share insights, questions, and even debates seamlessly across two distinctive media, visual and textual, will allow researchers in distinct fields such as philology, history, art history, architectural history, and others to address a common object of interest, the life of the Buddha, in synthetic fashion. Users will be able to share and disseminate their analysis of the images, texts, and their relationships through saved searches, curated tours, and robust annotations. (4) Extensibility to Other Disciplines While our humanities focus is the interface between visual and literary Buddha narratives in Tibet, we believe TRIPTYCH-Mirador Viewer will be the first truly extensible interface dedicated to a ubiquitous need in the digital humanities, namely, the synthetic analysis of digitized material artifacts, machine-readable text, and associated documents (Appendix B). Likewise, the project will also be productive in learning environments that emphasize undergraduate research with a focus on active learning and collaboration, visual and data analysis, and markup of documents and images (Appendix C). In our effort to document potential use cases for TRIPTYCH, we identified several courses and projects at X University where a robust interface of this nature has already been requested, including: Dante in Translation, Modern Greek: MetaGallery, Financial History, and Genjis World. During phases II and III of our project, we hope to make this interface applicable to other disciplines by adding extensibility features including: the ability to perform faceted searches and share historical search information, and third-party extensions for more robust sharing capabilities and data analysis (Appendix G). The need expressed to develop a multimodal discovery interface of this nature is sizeable; we address this trend in the environmental scan. Environmental Scan We envision TRIPTYCH as a multimodal interface that will work together with Mirador Viewer. The technical specifications were selected for their compatibility with Mirador Viewer (as currently being developed by Yales Digital Collections Center and Stanford University), cost benefit, and long term goals (Appendix F). TRIPTYCH works differently from sibling interfaces in that it is less focused on annotation and more focused on the discovery and analysis of materials and the generation of non-linear tours by users. No sibling interface emphasizes the ability to construct new paths for reading images and documents; annotation has been either linear or random, with no functionality for making and sharing tours.

The NCTE summary on Multimodal Literacies (2008) suggests that in multimodal environments, there are increased cognitive demands on the audience to interpret the intertextuality of communication events that include combinations of print, speech, images, sounds, movement, music, and animation. Some remarkable prototypes for a multimodal interface that include a panel navigator, robust search and markup tools, and collaborative editing have surfaced in the past decade, and we have tested several of these for learning and research (Appendix E). Most of these designs heavily emphasize annotation rather than creation of relational tours, and most belong to one of two camps, which we identify as linguistic: focused on word patterns, and visual: focused on material artifacts.

Multimodal interfaces with a linguistic approach focus heavily on language components; they allow users to search and visualize linguistic and historical data mined from the digital corpus (SCRIPTORIUM), to perform word searches, annotation to manuscripts, inscriptions, and music notes without knowledge of markup languages (ImAnTo), to analyze the graphical aspects specific to digitized printed texts (Visual Page Project), and to provide a virtual research environment (VRE) for humanities disciplines that allows users to select image sections and link them to text, use thumb views, and XML editor (TextGrid). None of these interfaces allows for cross-media or non-linear searching, and none permits the discovery and creation of tours. TextGrid is similar to TRIPTYCH in offering multifaceted search capabilities; but, it does not function as a multimodal, unified interface that is both visually accommodating and accessible for novice users, nor can users harvest and edit materials directly on the TextGrid site.

Multimodal interfaces focused on material artifacts generally include an interactive application for a similarity based retrieval, visually assisted tagging, relevance feedback, and zoom functionality to explore and annotate images. In addition to building in useful metadata, tools for visual-based projects allow users to identify correspondences between illustrations and text (Rerum Novarum, Dead Sea Scrolls), to avoid distortions of the temporal and spatial qualities caused by photographing or improperly exhibiting the work (University of Chicagos Chinese Handscrolls, Princetons Piero Project), and to develop a system of ontologies in order to permit the automatic connection of images, transcriptions and other resources (Virtual Manuscript Room-VRM). The VRM may be similar in scope to TRIPTYCH with regard to cross-functional searches and annotation, and with the goal of ingesting new resources from others repositories; however, VRMs first phase works more as a digital library, and in later phases there is no published plan to discover and share non-linear or new narratives, which we see as the most critical and beneficial feature of our interface.

TRIPTYCH complements the objectives for projects in both camps, but moves well beyond their functionality by focusing on the creation and sharing of tours, and by offering the first multimodal interface to be viable for research distributed across three types of materials. For the start-up phase of this project, TRIPTYCH will combine with Mirador Viewer to allow viewers to interact with the Life of Buddha panels by zooming and panning the images with mouse clicks or touch points (Appendices G, H). The panels will also contain interactive points of interest that can be clicked on to display further information about that particular panel, scene, or specific image or text within the panel. These points of interest, both in the original language and the translation, can be searched via a robust search box. A search will return a list of matches within any of the Life of Buddha panels and corresponding links to view those points. In addition to the general viewing interface, the combined TRIPTYCH-Mirador Viewer application will allow users (or contributors) to log in and modify or edit any of the existing text and points of interest within the panels. New points of interest can be added at any point, by any number of users.

As represented in the Technical Specifications (Appendix F), the X will commit project direction, development, and infrastructure to build TRIPTYCH as an extension to Mirador Viewer, which currently allows scholars to study high resolution of images and their accompanying metadata through various views (image, thumbnail, scroll), but does not currently include tiles for simultaneously searching, annotating, and creating tours of machine-readable texts or contextual documents. The simultaneous and relational analysis of images and machine-readable text represents an entirely new level of functionality on the Mirador Viewer platform, one that is not currently in scope at Harvard, Stanford, or Yale. A further advantage of the Mirador Viewer is its reliance on data models such as the Open Annotation Collaboration and the IIIF, which encourage interoperability and exchange of text and images across repositories and software tools. In 2012-2013, X incorporated the hosting components of the interoperable environment within the X Content Platform to support Medieval Studies research funded by the Andrew W. Mellon Foundation, thereby allowing information to flow between the relevant systems. The X framework utilizes the Shared Canvas Data Model designed to accommodate the complexities of digitally representing manuscripts, the Open Annotation Model, a W3C Specification to support annotation, and the International Image Interoperability Framework (IIIF) for standardized delivery of images. Because various user types will create tours, a permissions filter and viewer filter will enable faculty researchers, undergraduates, and public or casual users to create and view tours with different degrees of access; Xs annotation service currently enforces access control (see data management plan for user matrix).

We now describe a sample use case for TRIPTYCH, and how synthetic forms of inquiry could be achieved using it in conjunction with Mirador Viewer: Through a landing page, a user enters the image window and selects locator map as the view mode. The entire mural displays, and the user selects teaching as a topical element across the map, where the controlled vocabulary term teaching has already been used to identify panel locations in which the Buddha is depicted teaching to an audience of people, deities, and other figures. Once TRIPTYCH returns teaching vignettes from across the mural, the user can begin to analyze those scenes and relate them to the literary data, thereby identifying image-text relationships. Her analysis of the scenes focuses first on identifying the individual recipients by proper

name, social group, and ontological typology (animal, human, god, demon, etc.). She creates annotations with those identifications. She then turns to context, noting the location of the teaching, first as expressed in the visual narrative, then in painting manual, and lastly in the textual narrative. Finally, she analyzes the formal visual relationships between the Buddha, his disciples, and their context, noting relative size and position, distinctive and repeating gestures, and placement in architectural or landscape features, in order to understand the murals depiction of the social/hierarchical relationships between the Buddha and others. She then may want to focus on a particular audience type: Having discerned an abundance of occasions in which the Buddha is teaching serpent demons called nagas, she then searches the term naga in the Tibetan source text and accompanying scholarly documents in English. A cross-panel search reveals twenty instances of the Buddha teaching nagas. It also reveals nine further instances of naga teaching in the literary materials that were not represented in the murals. She then notices that in every case, nagas are depicted visually in close relationship to water even though water is not mentioned in the literary sources. Through deft analysis of the visual and textual, she comes to understand how the creators of mural and the text capitalize on their distinctive media to highlight, sublimate, complexify, or simplify elements of the narrative in a manner that would not have been possible through attention to either the visual or the textual in isolation. The entire analytical process also allows her to contribute to the structured terminology of Buddhist imagery in the Getty Vocabularies. The user creates a tour titled Buddha teaches the Nagas, which she both saves to the interface and exports for incorporation in a collaborative research project. No interface or toolkit exists to elegantly display, curate, and share the image-text discoveries that will be realized through TRIPTYCH-Mirador Viewer. This vehicle will allow for the first truly world-wide collaborative editing canvas on which researchers are able to document three types of source material. History and Duration of the Project This project emerged from a series of courses on Tibetan Buddhism and Buddhist Traditions offered by at X since 2011 (Appendix C). Students in these courses actively participated in identifying mural features, annotating details, and analyzing image-text relationships. Graduate and undergraduate students received basic training in color correcting and photo stitching the image panels; acquired basic skills and theory of translation, text mark-up and OCR; and contributed to the creation of metadata associated with digital assets (Appendix B). Since 2011, the X has designed and supported the course site, maintained the database, and continues to experiment with various zoom and annotation tools (Appendix E) to find workarounds for active learning and collaborative use in the live classroom. Our work with various tools reveals similar shortcomings, which we hope to resolve through TRIPTYCH (Appendix D).

TRIPTYCH is envisioned as a multi-year project planned in three stages (Appendix H). With the NEH Start-Up grant, we will implement Stage I, Phase I of the project. We divide the multiyear project as follows: (STAGE 1) Development of the interface (see Appendix F), (STAGE 2) ingestion of resources from other researchers and archives, (STAGE 3) use of the curated tours for computational research in the humanities and to create a high performance computing simulation. Both directors have participated conferences and panel discussions for the LOTB project (Appendix A). They are also co-authoring a book titled The Life of the Buddha at Jonang: Literature and Art in Place. The directors plan to apply for additional funding through the Ho Foundation Collaborative Research Grant in Buddhist Studies. No monies from NEH will be used for the digitization of additional materials. Any non-NEH funding secured during the Start-Up phase will be used to add secondary features described in the work plans (Appendix H). Work plan The work plan is divided into four phases during the start-up period (Appendix H).

Final Product Dissemination The TRIPTYCH interface, learning materials, and project updates will be disseminated on the TRIPTYCH website under open-source licenses, and hosted on servers maintained by Xs ITS Department for 60 months after start-up completion. In addition to conferences and a book publication (Appendix A), metrics and research data will be presented at conferences and submitted to digital humanities journals such as the International Journal of Humanities Arts and Computing and annual meetings hosted by DARIAH and ADHO. Discipline-specific findings will be submitted to a leading journal of Tibetan or Buddhist Studies, such as Journal of the International Association of Buddhist Studies, Numen, or History of Religions. The projects white paper will be published on the TRIPTYCH site.

7. DATA MANAGEMENT PLAN Roles and Responsibilities Project Directors will oversee the data management plan and development of the public web platform. X will direct the creation of digital tools, manage the server for the public repository, and manage the versioning software for the repository and pre-publication. Data to Be Generated Type of data When shared? Under what conditions? Source code At conclusion of the project. Apache 2 License. Mural Images Downloadable by users. No restrictions. Machine-Readable OCR of Tibetan source text Downloadable by users within the TRIPTYCH context. Files will be

downloadable in UTF-8 text format.

No restrictions. Digital textual data of the original Tibetan Manuscript is based on transcriptions of ancient texts, which are not subject to copyright.

English translations of Tibetan source text Downloadable by users within the TRIPTYCH context. Digital textual

data of the original Tibetan Manuscript is based on translations of ancient texts. Project Directors hold copyright.

Translation is not subject to copyright because it was work for hire under the direction of a Project Director.

User-generated annotations Displayed on screen for the contributing user using Open

Annotation data format. Exportable according to permissions matrix (below).

Only downloadable when the contributing user generates an export; users agree when they participate that their annotations will be visible to editors and administrators.

Assessment data generated during the testing phase Aggregated data will be shared via the white paper & final report to the NEH. No information will be shared that could identify individual users. White Paper At the projects conclusion. Freely available on website. A multimedia report At the projects conclusion. Freely available on website. Final report to NEH At the projects conclusion Dissemination will be the

responsibility of the NEH.

User Annotation and Tour Permission Matrix Users Annotation Users Tour Contributing User Read/Write Read/Write Other Users Read (if permission granted) Read (if permission granted) Editor Read Read Administrator Read/Write Read/Write

Period of Data Retention All data will be securely stored and backed up on Xs source control and file storage systems (GitHub Enterprise). Source code, application data curated by project directors (images, manuscripts and translations) and user-generated data (annotations, saved searches, etc.) will be retained for a minimum of five years beyond the completion of the start-up phase of TRIPTYCH. Formal reports will be publicly available within one year of project completion on the project site, and copies will be stored long term in Xs source control and file storage systems. Data Formats and Dissemination

Source code will be available under an Apache 2.0 license and will be stored in a publicly accessible GitHub code repository.

Primary Source Documents (machine-readable OCR text of Tibetan texts) will be available for download in UTF-8 text format.

English Translations of Tibetan Texts, commissioned for this project by the Project Directors, will be available for download in UTF-8 text format.

User annotations will be stored internally using the Open Annotation data format with extensions specified by the IIIF Presentation API.

Metadata associated with media documents in shared multimedia collections in TRIPTYCH will be freely available on the project website in the Metadata Encoding and Transmission Standard.

Data Management and Maintenance Source code will be stored in GitHub. All other data, including user generated texts and annotations that are exportable, assessment data, as well as any reports and publications, will be made available through the project website.

9. APPENDICES A. Publications, Conferences, and Workshops on Life of the Buddha

Books The Life of the Buddha at Jonang: Literature and Art in Place. (in-progress) Conferences 2014. Life of the Buddha in Tibet, presentation at Harvard University. 2013. kyamuni in the Service of Jo nang: Trantas Jo bo Phyogs las rnam royal,

International Association of Tibetan Studies, Ulaanbaatar, Mongolia. 2013. Life of the Buddha at Jonang Monastery: Literature, Art, and Institution, Annual Tibetan

Collection Lecture, Newark Museum of Art, Newark, NJ. 2012. The Life of kyamuni at Jonang: Preliminary Observations on Literature, Art, and

Institution, Institutional Landscapes and Intellectual Codifications in Tibets Long Seventeenth Century, Columbia University & Rubin Museum of Art.

2012. Refiguring the Buddha in Tibet, Roundtable discussion, Association for Asian Studies. 2011. The Life of the Buddha in Tibet: Art, Literature, Institution, SSEASR Conference,

Thimphu, Bhutan. B. Completed Digitized Materials To-date (fully edited for ingestion into interface)

Prepared digital photographs of entire mural Created Portfolio database of full Image set Ingested metadata: identifying narrative topics, figures, literary references Processed Images for Panels A, B, and O (featuring the birth, early life, and death of the Buddha) Stitched images of Panels A, B, and O Created narrative maps of Panels A and B Transcribed inscriptions (complete) Translated inscriptions (complete) Prepared English translations of first 25 chapters of the Hundred Acts Prepared electronic text of the Hundred Acts and the Painting Manual, unedited Prepared concordances for Acts in the Hundred Acts and Painting Manual

C. Examples of Coursework and Undergraduate Research Currently, Life of the Buddha materials are used for undergraduate research in two X courses, offered annually: Tibetan Buddhism (RLST 126) and Introduction to Buddhist Thought and Practice (RLST 125). Both courses include collaborative final assignments using the multimedia materials of the LOTB project. The assignments present an opportunity to consider the visual logic of painted narratives and its relationship to the textual logic of the story as written. Students are divided into groups of 3-4, and each are assigned a different section of the Buddha life story based on the Jonang murals and English translations of the Hundred Acts of the Buddha: Act 1, Act 2, Act 3, etc. Each act contains a number of associated scenes, which in turn can contain numerous human and divine figures, architectural structures, environmental features, inscriptions, and other objects. Each scene may also depict one or several activities. Projects have so far concentrated on the early portion of the life story (mural acts 1-10, Hundred Acts chapters 1-25), which depicts the time prior to the Buddhas birth up to his enlightenment and early teaching. Students are responsible for identifying, describing, and analyzing visual elements of the narrative, both discrete details (people, architectural structures, ritual objects, etc.) and thematic elements (e.g., the story of the buddha crowning a regent, the subjugation of demons). Students are asked to consider such questions as:

Where in the larger mural panel context does the act occur? How, if at all, is the act related to the images that surround it? How is the space used? Is there a linear progression? If so, in what direction? Is the movement

obvious or intuitive? How does the visual representation compare to the literary version of the story? Is there anything

added or left out? Student groups are given access to high resolution images of full mural panels and individual acts, as well as English translations of the Hundred Acts. They are also provided mural episode maps that identify where specific narrative episodes occur on the mural panel (Figure 1). They first determine the order of the painted scenes within each act, taking in to account the visual features that separate scenes from one another: architectural elements or environmental features such as mountains, rivers, or clouds. Using basic image editing tools (such as Preview), students identify and label all human figures, architectural elements, environmental features, clothing, ritual objects, etc. (Figures 2-3). They next create a reference map indicating the narrative flow within the act across the mural surface (Figure 4). Finally, groups create their own narrative description of their findings, indicating the actions taking place and elements involved. Some groups are assigned the task of mapping the narrative of all the acts within a larger mural panel (Act 1 Act 2 Act 3, etc.), thus creating a complete topographical survey of the visual narrative (Figure 5). Groups post all of their materials to a course blog to form an extended and detailed curated tour through the Buddhas life story. Collaborative work and final presentations are conducted in Xs TEAL (teaching enabled active learning) classroom. By working on these materials, students are able to develop some of the topical and technical skills outlined in part a(2) of the proposal narrative above. In their analysis of the images and texts of the Buddhas life story from Jonang Monastery, students were able to make genuine discoveries. For example, students identified numerous figures located throughout the mural that appear identical in shape, form, and posture, but in different forms of dress and ornamentation, and distinguished as different

individuals. This led them to conclude that the artists likely used pounces or patterns when executing the mural, a practice that has not been traditionally associated with Tibetan narrative mural painting. The public-facing site is housed at

Figure 1 Panel B location map

Figure 2 The Buddhas Regent Figure 3 The bodhisatttva prince encounters death

Figure 4 Act 10 narrative chronology (major scenes in purple)

Figure 5 Transitions between Acts 6-8

D. Wireframes For the more robust research site, we created (as an expansion to Mirador Viewer) a basic mockup of the three-panel interface with Activity Log, to be accessed from a landing page where users will have designated a point of entry, and user roles will be assigned:

Close-up view of the Image Panel building on Mirador Viewer, which illustrates core features for the Phase 1 Start-Up Project (View, Zoom, Annotate, Parametric Search, Metadata); and secondary features to be added in Phase 1 and 2 (Faceted Search, Spatial Markup, Rotate).

Basic mockup of the Activity Log as a workspace for advanced research, showing both core features to be developed in the Phase 1 Start-Up project (Saved Searches, Locally Saved Tours, Basic Export); and secondary or advanced features to be added in Phase 2 and later (Advanced Export, Analysis tools including Metrics and Automated Computer Visual Analysis). The mockup also includes notes about types of tours (walk throughs) the users might make, as an expansion of Mirador Viewer: E. Robust Tools Tested Tool Dates Tested Results MIT Annotation Studio summer 2014 focus is on PDF and documents linked to

images rather than seamless emphasis across all three

Omeka summer 2014- no relational connections possible: users

can see images but not make relations; limitation of connection between three types of media in Neatline

Mirador Viewer spring 2014- provides relational features, but the annotation tool cannot handle large images, and no connections are possible for text and scholarly documents

Media Thread 2013-2014 tool can handle scenes and panels, but not the entire mural; no more than ten simultaneous users can collaborate without system crashing; analyze this browser only works with Firefox

Zoomify 2012-2013 no relational connections: annotations do not adequately connect scenes or hotspots; center display area is inadequate; difficult for users to navigate

F. Technical Specifications Programming Language Ruby (server-side) JavaScript (client) Frontend HTML5, CSS, JavaScript Backend Ruby on Rails Platforms Safari, Firefox, Chrome, IE 9 + Encoding Unicode G. Glossary of Technical Components (identified as Phase I of Stage I start-up, and Phases II and III to be created through additional funding or post start-up) Panel and Historical Text Search Window (I): The robust search engine will also be one of a kind, as it will pull together the ability to search within the panels curated hotspot text, as well as the historic Buddha texts and their English translations. With the ability to pull in results from all three sources, the Panel and Historical Text Search Window will allow users and researches to find truly unique references between texts and the visual representations of The Buddhas life. This search will be primarily keyword based, but extended to all three databases. If created in an extensible manner, this interface could be applied to an array of historical texts, documents and maps, as well as gigapixel images and large, complex diagrams and plans.

Advanced, Relative, and Faceted Search (I-II): As an addition to the already robust keyword search, the advanced and relative search will add intuitive connections between the growing databases of imagery, hotspots, and texts. To extend the searching and linking capabilities even further, a faceted search tool will be constructed to give the ability to find elements of a certain ontology within the panels and historical texts. By creating each option within the faceted options explicitly, users will be able to find important and relevant information about things like specific locations, buildings, periods of Buddhas life and other particularly vital pieces that may be more often searched by users and researchers. Shareable History and Tours (I-II): As the traffic on the site and collaboration increases within the online tool, users will have the ability to share previous searches and a history of things that they found within the panels and texts. By way of the social tools listed below, users can send these tours to other users or save them for later viewing. Social Tools and Third- Party Extensibility (III): The ability to extend the web application to third party note- taking and social applications will be added to allow users to keep personal journals of their findings, images, and searches, as well as share those items through modern social tools including social media, email, and applications such as Evernote. Data Visualization Tools & Metrics (III): A data visualization panel will be created to allow for viewing of the searches, collaboration, and traffic within the panels in a given timeframe. Administrative users and researchers will see how the tool is being used, and glean associations and references from the activity. H. Work Plans (Grant Period: Stage I, and Overall Project: Stages I-III) START-UP: STAGE I-PHASE 1A (June 2015-December 2015) Plan and finalize the interface architecture, design elements, and wireframes for Phase I elements

Hold one-week, in-person working session in late May; hold weekly team meetings via teleconference for the duration of the project

STAGE I-PHASE 1B (January 2016-May 2016) Deliver and review Alpha software

Author and disseminate documentation STAGE 1-PHASE 1C (May 2016-November 2016) Deliver and review Beta software

Author the joint white paper and disseminate documentation

POST START-UP (January 2017) Publicize the interface through conferences and publications for Digital Humanities and Tibetan or Buddhist Studies Use the interface in undergraduate research courses on Tibetan Studies; gather metrics and

data on user experience Create an extensible framework and Phase II and III elements when additional funding is secured Ingest new materials from scholarly collections Identify and apply for additional funding for Phase II and Phase III components Make the source code public under an open source license

TECHNICAL RESOURCES REQUIRED Panel Navigator, Collaborative Hotspot Editing and Viewing Window, Panel and Historical Text Search Window, Robust Search Panel All software will be open source Database and servers are provided by X Universitys ITS Department as in-kind service

I. Bibliography

Technology-Related Materials Ali, Ahmed, Yahya Al-Hajj and Marc Kuster. The Text-Image-Link Editor: A Tool for Linking Facsimiles and Transcriptions and Image Annotations. Literary and Linguistic Computing 28.2 (2013): 190-98. Benitez, A.B. and S.F. Chang. Semantic Knowledge Construction from Annotated Image Collection. In Proceedings of the 12th Annual ACM International Conference on Multimedia. IEEE Computer Society Press, 2002. Borghesani, Daniele. Rerum Novarum: Interactive Exploration of Illuminated Manuscripts. Department of Computer Science, University of Modena and Reggio Emilia, 2010. White paper on Rerum Novarum interface. Bunde, Janet, and Deena Engle. Computing in the Humanities: An Interdisciplinary Partnership in Undergraduate Education. Journal of Archival Organization 8.2 (2010): 149-59. Cohen, Kathleen, et al. Digital Culture and the Practices of Art and Art History. The Art Bulletin 79.2 (1997): 187-216. Davis, Rebecca Frost. Crowdsourcing, Undergraduates, and Digital Humanities Projects. Published September 3, 2012 on http://rebeccafrostdavis.wordpress.com/.

Djeraba, C. Content-based Multimedia Indexing and Retrieval. IEEE Multimedia 9 (2002). Ferster, Bill. Interactive Visualization. Cambridge, MA: MIT Press, 2013. Hazan, Susan. The Dead Sea Scrolls Online: Taking On a Second Life of Their Own. White Paper from The New Media Unit, The Israel Museum, 2009. http://www.musesphere.com/images/DeadSeaScrollsonline.pdf Hirsch, Brett D., ed. Digital Humanities Pedagogy. Cambridge: Open Book, 2012 Hung, Wu, et al. Digital Scrolling Paintings Project. University of Chicago, 2011. http://scrolls.uchicago.edu/ Lavin, Marilyn. Making Computers Work for the History of Art. The Art Bulletin 79.2 (1997): 187-216. Lew, Michael. Content-Based Multimedia Information Retrieval: State of the Art and Challenges. ACM Transactions in Multimedia Computing, Communications and Applications 2.1 (February 2006). Marsh, Emily, and Marilyn Domas White. A Taxonomy of Relationships Between Images and Text. College Information Studies 59.6 (2003): 647-72. NCTE (National Council of Teachers of English). Multimodal Literacies and Technology. Published 2008 on http://www.ncte.org/governance/multimodalliteracies. Raaf, Manuel. Easily Annotating Manuscripts Online: A Web-Application for Linking Images to Texts. LOEWE-Center, 2012. White paper on ImAnTo (Image Text Annotation Tool). Reside, Doug, et al. TILE: Text-Image Linking Environment. NEH Final Performance Report. October 31, 2011. URI http://hdl.handle.net/1903/14737. Robison, Peter. The Virtual Manuscript Room. A Digital Humanities Project through the University of Birmingham Institute for Textual Scholarship and Electronic Editing (ITSEE), 2011-15. http://vmr.bham.ac.uk/ Schroeder, Caroline, and Amir Zeldes. Towards Digital Coptic: Searching and Visualizing Coptic Manuscript Data. Berlin Digital Classicist Seminar 14.1 (2014). White paper on SCRIPTORIUM. Snyder, James. The Digital Dead Sea Scrolls. The Israel Museum, 2013. http://dss.collections.imj.org.il/ Tomasek, Kathryn. Integrating Digital Humanities Projects into the Undergraduate Curriculum. Presented at the NITLE Conference, 2011. Tibetan Language Materials Trantha (1575-1634). Bcom ldan das thub pai mdzad pa mdo tsam brjod pa mthong bas don ldan rab tu dga ba dang bcas pas dad pai nyin byed phogs brgyar char ba. Various editions. [Biography entitled Hundred Acts of the Buddha]

_____. Ston pa Shkyai dbang poo mdzad pa brgya pai bris yig rje btsun kun snying gis mdzad pa. Various editions. [Descriptive guide to the murals at Jonang Monastery based on the Hundred Acts of the Buddha] _____. Dga ldan phun tshogs gling gi gnas bzhad. Various editions. [Descriptive guide to Jonang Monastery] Discipline-Specific Materials Cummings, Mary. The Lives of the Buddha in the Art and Literature of Asia. Ann Arbor, MI: Center for South and Southeast Asia, University of Michigan, 1982. Foucher, Alfred. The Life of the Buddha According to the Ancient Texts and Monuments of India. Translated by S. B. Boas. 1st ed. Middletown, CT: Wesleyan University Press, 1963. Original edition, 1949. Templeman, David. Becoming Indian: A Study of the 16-17th Century Tibetan Lama Trantha. PhD Dissertation, Monash University, 2008.