creating, visualizing, and exploring knowledge...
Post on 23-Feb-2018
224 Views
Preview:
TRANSCRIPT
Creating Visualizing and Exploring Knowledge Maps
Colorado ReedUC Berkeley
coloradoberkeleyedu
Figure 1 The knowledge map visualization and exploration system described in this paper reduces the number of visual edges for large knowledgemaps In addition it provides a number of interactive components to help users explore and understand these maps The left figure shows a knowledgemap (layered Sugiyama-style directed graph) of machine learning concepts The large number of crisscrossing edges makes the graph difficult tovisually parse The top right figure shows the same graph produced by our system Notice that the node locations are the same but a large number ofvisible edges have been replaced with ldquowisp edgesrdquo small dashed protruding lines from the source and target nodes that become full edges when theuser hovers on the wisp or the sourcetarget node The bottom right figure shows the focused mode that is displayed after clicking on a concept Thefocused mode shows the inlinks and outlinks of the targeted node and the nodes associated with these edges
ABSTRACTKnowledge maps are cognitive aids designed to help users ex-plore a body of knowledge in a nonlinear way Specificallya knowledge map is a graphical representation of knowledgechunks eg single-digit addition linked together by directededges that express prerequisite relationships In this paperwe describe a novel knowledge map creation visualizationand exploration system We employ edge reduction and in-teractive techniques to aid users in exploring large knowledgemaps Furthermore we describe a visual knowledge map ed-itor that allows users to create new knowledge maps asso-ciate related content with the map (eg textbook referencesof video lectures) and make local changes to large preexist-ing knowledge maps
INTRODUCTIONA knowledge map is a graphical representation of conceptsthat reveals the relationships and dependencies among the
concepts In this work a knowledge map is a directed graphin which the nodes are concepts and the edges specify prereq-uisite relationships between the concepts see Figure 1 Thisgraphical representation is a variant of the concept maps com-mon in educational psychology whereby the nodes are con-cepts and the edges are linking words that create a meaning-ful statement or proposition when combined with the nodesStated another way a knowledge map primarily serves as areference tool while a concept map serves as a learning toolThat is a knowledge map typically has associated contentwith each concept eg textbook references or video lecturethat a student would use to learn the material while a con-cept map summarizes a body of knowledge and is oftentimescreated by the student to help learn the material
In recent years several online education companies andschools have created knowledge maps to help students vi-sualize and navigate their content in a nonlinear way see
eg Khan Academy [3] Duo Lingo [2] and Metacademy[4]Khan Academy a nonprofit online educational company withover six million unique monthly visitors has a large (700+node) knowledge map that founder Salman Khan discussedon p 52 of [13]
[The knowledge map] became a core piece of the KhanAcademy software platform In stressing the connec-tions among subjects and giving learners a visual pictureof where theyrsquove been and where theyrsquore going we hopeto encourage students to follow their own pathmdashto moveactively up down and sideways wherever their imagi-nations lead
A problem with knowledge maps however is that the largenumber of relationships between the concepts produces alarge number of intertwined and crisscrossing edges in thegraph In turn it is difficult to interpret the conceptual re-lationships see Figure 2 for examples from Khan Academyand Metacademy These ldquohairballrdquo graphs are common innetwork visualizations and indeed there is a deep field ofresearch focused on visualizing networks eg see [7] Thehierarchical placements of the nodes and edges in knowledgemaps express the dependency structure within the body ofknowledge and for this reason many network visualizationtechniques will not work with knowledge maps as they ob-scure or break these relationships
In this work we create a novel interactive system for creat-ing visualizing and exploring knowledge maps On the cre-ation side we constructed a graphical user interface (GUI)for visually specifying concepts their relationships and as-sociated content On the visualization and exploration sideour system uses a layered (Sugiyama) directed graph place-ment algorithm [17] to initially place the nodes a topologi-cal sort to determine an initial set of visible edges (the topo-logical sort maintains the connectivity of the graph) an edgelength threshold to determine which remaining edges shouldbe shown and edge ldquowispsrdquo to indicate the absence of anedge These techniques greatly reduce the number of visibleedges in large knowledge graphs while maintaining the hier-archical and connectivity structure of the content
To further accommodate the goals of knowledge map userswe incorporated several interactive components that aid theuser in locating concepts of interest and understanding theirrelationships to other concepts In particular our system al-lows users to step through the dependency structure of a graphby sequentially visualizing contextual views of the concepts(the direct dependencies and outlinks) Furthermore we in-clude a number of hover and click operations that reveal thereason for the relationships and associated content present fora given concept
RELATED WORKIn this section we discuss three areas of previous research anddevelopment relevant to this paper online knowledge graphsystems visualizations of large directed graphs and tools forcreating knowledge maps
Online Knowledge GraphsThe knowledge map creation visualization and explorationsystem described in this paper was designed for Meta-cademymdashan open source platform for collaboratively cre-ating refining and sharing knowledge maps and associatedlearning material [4] Figure 2 (right) shows a Metacademyknowledge graph one of the two main components of Meta-cademy The other major component is a text-based view thatprovides the user with the following information for each con-cept a description of the concept goals for the learner theprerequisites of the concept learning resources for the con-cept and related (non-dependency) concepts Metacademy istypically used as a targeted learning resource whereby a usersearches for a particular concept and is then presented witha knowledge map with that concept as the sole leafmdashmostgraphs have 5-50 nodes and less than 100 edges
As mentioned in the introduction Khan Academy [3]mdasha freevideo lecture and exercise repositorymdashalso provides a knowl-edge map of their material Rather than showing target-basedcomponents of the graph Khan Academy shows users theirentire 700+ node knowledge map This knowledge map ismuch larger than any of Metacademyrsquos knowledge maps butthe techniques discussed in this work are applicable to theirgraph and we will format their content for our system in fu-ture work
Visualization and Exploration of Directed GraphsOur knowledge map visualization and exploration systemdraws on a number contributions from the graph visualizationcommunity In particular the authors of [18] developed an in-teractive visualization system for large networks eg threemillion nodes The authors focused on providing context-specific visualizations of large directed graphs and stated that
Although providing a structural overview of the graphis a laudable goal there are many cases where the useris simply not interested in a global view of the wholegraph but wants to solve a particular concrete task onthe graph instead
Rather than show all direct inlinks and outlinks as is donein several related systems [5 6] the authors devised a de-gree of interest (DOI) term for each node that they use todetermine which subset of contextual nodes to show the userSince individual nodes in knowledge maps have a relativelysmall degree it is typically possible to visualize the fullcontext of each node rather than a subset and we do notuse the DOI technique in this work The authors use wispedgesmdashtruncated inlinksoutlinks to a nodemdashto indicate thepresence of an edge without cluttering the visualization withlong crisscrossing edges We incorporate this technique intoour visualization (see the next section)
Figure 2 The left figure shows a section of the Khan Academy knowledge map the right figure shows a Metacademy knowledge map
There are a number of software packages and algorithms forvisualizing (large) directed graphs that inspired our system
bull the dot placement algorithm described in [12] is theunderlying placement algorithm for the popular graphvizC++ graph visualization software The dot algorithm isused in the dagre JavaScript graph placement library whichis used in this work (see the next item)
bull dagre [1] a JavaScript library for directed graph placementis used to generate node and edge placements in our knowl-edge map system
bull Dynasty [11] a system for browsing large directed graphswhere only a small portion of the graph is visible at anygiven time Dynasty inspired the ldquofocused moderdquo aspectof our system where a concept and its dependencies andoutlinks are visible at a given time
bull the transition animations in Open Graph Drawing frame-work [10] a large C++ library of algorithms for graphdrawing inspired the transition animations sequence andtiming used in this work
Furthermore Misue et alrsquos mental map preservation proper-ties for directed graphs [15] motivated our use of visualizationmodes rather than dynamic directed graphs
Finally many directed graph visualization systems transformthe data for visualization For instance hive plots [14] pro-duce a linear layout for grouping nodes by type and then ar-ranges the nodes along a radial axis Arranging knowledgemap concepts along a radial axis does not work well for a vi-sualization as the concepts commonly have relationships thatspan multiple levels and as a result the radial network be-comes polluted with edges that span most of the circumfer-ence Strictly hiding these types of dependencies masks theprerequisite complexity for various concepts and could de-ceive the user
Knowledge Map CreationAs of the time of this writing the author currently does notknow of any tools that are explicitly used for creating knowl-edge maps as defined in this paper Both Khan Academyand Metacademy use text-based systems eg html formsto specify the relationships in the graph However there is alarge number of tools for creating similar structures such as
diagrams mind maps and concept maps We briefly discussthese tools and their relevance to our system
Cmap tools [9] is a Java-based GUI for creating conceptmaps directed graphs where the nodes are concepts and theedges are linking words so that tracing through a path is akinto reading a sentence The Cmap tools GUI inspired our edi-tor as it has simple drag and click operations to construct thegraph The Cmap tool editor however does not have asso-ciated content with the graph and is not browser-based Fur-thermore the Cmap tools editor requires the user to manuallyplace the nodes and edges Our editor operates in a similarfashion but users have the ability to algorithmically arrangethe nodes and edges using the dagre javascript library Fi-nally our editor is browser-based while Cmap tools is a Javaapplet
There are countless diagramming tools such as gliffy1drawio2 and dia 3 Many of these of tools are polished in-tuitive and downright beautiful diagramming tools but theyare more general purpose than what is needed for knowledgemap creation Letting users choose the color and shape of thenodes the thickness of the edges etc detracts from the mainpurpose of specifying the concept relationships and providingassociated content
METHODOLOGYIn this section we describe the visualization and interactionmethods employed by our knowledge map system In par-ticular our system builds upon the Metacademy knowledgemap system and we have recently acquired the data for KhanAcademyrsquos knowledge map and plan to extend our system forthis data in the near future
Knowledge Map Creation and ExplorationFrom an informal usability study conducted with seven UCBerkeley students using the think-aloud protocol[16] and theMetacademy knowledge map system we identified the fol-lowing common objectives the students had when interactingwith the knowledge map4
1httpwwwgliffycomindex-gphp2httpwwwdrawio3httpsourceforgenetprojectsdia-installer4Specifically we observed the students interacting with relativelylarge (30-50 nodes) and small (circa 10 nodes) knowledge maps
1 visualize the overall complexity of the targeted conceptie the depth and breadth of the graph and the number ofedges and nodes
2 locate specific concepts within the knowledge map
3 identify the prerequisites and outlinks for a concept theprerequisites of its prerequisites outlinks of its outlinksetc
4 focus on a particular concept
5 explore related concepts not shown on the current graph(inclusion of new data)
Metacademyrsquos primary form of interaction is allowing usersto mark concepts they have learned and subsequently removethem from the graph When removing the concepts the Meta-cademy system naively recalculates the graph layout with-out taking into account the previous layout The system theninstantaneously flashes to the new knowledge graph withoutshowing the user the corresponding transitions see Figure 3This dynamic form of interaction does not directly addressany of the objectives expressed by the students in our usabil-ity test Furthermore according to Misue et alrsquos theory ofmental map preservation for dynamic directed graphs [15]this transition destroys the userrsquos mental map of the system asit violates the following three mental-map-preservation prop-erties
1 orthogonal ordering maintaining the relative directionsbetween the nodes (these updates can shift the relative di-rections see eg the ldquorandom variablesrdquo and ldquoexpectationand variancerdquo nodes in Figure 3)
2 proximity model maintaining the relative distance of thenodes (these updates can change the relative distance of thenodes see eg the ldquorandom variablesrdquo and ldquoexpectationand variancerdquo nodes in Figure 3)
3 topology model graphical objects within a relative regionshould stay within that region (these updates can changethe regional placement of nodes this is not present in thesupplied example but it tends to occur with larger graphs)
While letting users remove nodes and edges associated withlearned concepts helps reduce the number of objects in thegraph this form of interaction does not address the identifiedobjectives of the users and comes at the cost of nullifying theuserrsquos previous mental map From these observations we didnot include this functionality within our system
To address the visualization and exploration objectives abovewe created two modes within our system (1) an overviewmode in which the entire graph is shown and (2) a focusedmode in which a particular concept is shown along withits immediate dependencies and outlinks The goal of theoverview mode is to allow users to visualize the overall com-plexity of the targeted concept as well as quickly trace the
and associated content for 30-60 minutes while verbalizing theirthoughts These maps showed the full prerequisite structure for agiven concept in machine learning or probabilistic artificial intelli-gence
paths of various concepts (objectives 1 and 3) as well asquickly find specific concepts within the knowledge map andexplore related concepts by loading them into the currentgraph (objectives 2 and 5) The focused mode inspired bythe focused graph view from [11] address objective 4 Inthe following paragraphs we describe these modes in greaterdetail
Both Metacademy and Khan Academy knowledge maps havea large number of long crisscrossing edges that make it dif-ficult to trace the paths within the knowledge map We ad-dressed this problem in our system by employing the follow-ing algorithm to reduce the number of visible long crisscross-ing edges in our graph
1 remove all transitive edges from the graph
2 compute the graph layout using dagre
3 set visible-outlinkslarr empty
4 for each node n in the graphset visible-outlinkslarr visible-outlinkscup outlinks(n) with path length lt Lcup shortest-outlink(n)
5 for all edges e not in visible-outlinks display an edge wispprotruding from the source and target of e
This algorithm is used to help accurately convey the relation-ships between the concepts while reducing the edge satura-tion within the graph The motivating philosophy behind thistechnique is to show users edges when they are interested inseeing them while still providing enough edges to maintainthe proper structure of the graph
In step 1 we remove the transitive edges in the graph becausethey do not affect the relative prerequisite structure of thegraph That is if we have edges (AB) (BC) and (AC) thenremoving (AC) from the graph will not affect the ABC or-dering and from a pedagogical perspective it is a redundantedge In step 2 we compute the edge and node placements us-ing dagre though any graph placement algorithm can be usedduring this step
In step 4 we remove all edges exceeding a length thresholdL while ensuring that each node has at least one outlink (theshortest outlink) By including the shortest outlink for eachnode we maintain the visual connectivity of the graph afterhiding other long edges Naively truncating all edges couldinduce artificially separated clusters of concepts that would bedetermined by the graph placement algorithm rather than thecontent We note that with a small enough threshold L eachnode would have one outlink and the graph would become atree Therefore if we simply did not include the non-visibleedges in the graph placement algorithm it would be possibleto obtain a tree layout with zero edge crossings This tree lay-out however could visually invalidate the relative prerequi-site structure of the graph Using the previous example withnodes A B and C we could have a tree with visible edges(BC) and (AC) and invisible edge (BC) which would resultin visually placing nodes A and B on the same level
Figure 3 One of Metacademyrsquos primary forms of interaction is allowing users to remove concepts they know (left) the original graph (middle) thedimmed node is marked as known (right) the node is removed from the graph In removing the node the Metacademy system naively redeterminesthe location of the edges and nodes without taking into account their previous locations In the text we argue that this form of interaction nullifies theuserrsquos mental map and as a result we have not included this feature in our system
The edge wisps from step 4 give a simple visible indication ofthe number of prerequisites and outlinks for a given conceptand are used in the interactive components discussed belowFigure 4 shows a medium-sized Metacademy knowledge mapof 39 nodes rendered directly from dagre (74 visible edgeswith 61 visible edge crossings) and the same knowledge mapobtained using the above algorithm (38 visible edges with 0visible edge crossings)
We employ the following interactive tools and techniques tohelp the user explore and understand the knowledge map gen-erated using the visible-edge-reduction-algorithm
bull hovering over nodes highlights the inlinksoutlinks fromthat node as well as the source (target) of the inlink (out-link) and shows a set of clickable icons that the user canuse to eg show the content associated with the node
bull clicking on a node triggers a series of animations that tran-sitions to the focused mode for that node (see below)
bull clicking on an edge shows a pop-up justification for theedge
bull a contractible side menu gives users the ability to searchfor a concept in the graph
The ldquofocused moderdquo shows the user a selected concept aswell its dependencies and outlinks We use a series of ani-mations to contract and expand the graph when a user entersthe focused mode the node is highlighted then centered onthe screen then the dependencies and outlinks are translatedso that they are adjacent to the focused node while the othernodes and edges fade out This sequence is based on similartransitions in the Open Graph Drawing framework [10] Theanimations and interactive tools were implemented using thed3 JavaScript library [8]
In the focused mode a small information box appears atthe bottom of the screen and indicates the number of hid-den nodes and edges The entire animation sequence takesslightly longer than one second Clicking on an adjacent nodewithin focused mode moves that node into focus while click-ing on the focused node runs the sequence of animations in
reverse and returns the user to the full graph Figure 5 showsa visualization of focused mode and the associated transition
Knowledge Map CreationIn this subsection we describe our visual knowledge map cre-ator While this tool is independent of a particular platformits current implementation is aligned for the Metacademy sys-tem We designed this visual editor using the d3 JavaScriptlibrary [8] and only included features that benefitted the fol-lowing operations addremove concepts addremove rela-tionships between concepts (edges) addremove associatedcontent and preview the knowledge map using the techniquesdiscussed in the previous subsection
Figure 6 shows the knowledge map creator The main viewconsists of a large graph editing surface and a small toolboxThe user can perform the following editing operations
bull add node shift + click on the editing surface
bull remove node click on a node and press delete or backspace
bull move node click + drag node
bull change node title shift + click on the node
bull edit content associated with node click on the small circlethat appears when hovering the node
bull expandcontract a nodersquos dependencies (outlinks) click theplusminus icon at the top (bottom) of the node
bull add directed edge shift + click on the source node and dragto the target node
bull remove edge click on a edge and press delete or backspace
bull scale graph scroll
bull translate graph click + drag on the editing surface
In addition the toolbox shown in Figure 7 provides the userwith the ability to optimize the graph location using dagreclear all nodes and edges from the graph preview the graphusing the knowledge map exploration system described in this
Figure 4 The left figure shows the knowledge map obtained directly using the dagre output the right figure shows the knowledge map obtained usingour visible-edge-reduction algorithm
Figure 5 The left figure shows the highlighted edges and nodes when hovering over a specific node The right figure shows the ldquofocused moderdquo thatoccurs after clicking the node Clicking the focused node in the focused mode returns to the full-graph display while clicking the adjacent nodes placesthe focus on them
paper and downloadupload a graph tofrom a JSON5 repre-sentation
The search bar at the bottom of the toolbox gives users theability to add preexisting concepts to graph When a user addsa preexisting concept to the graph the inlinksoutlinks of thatnode are contracted by default so that eg 20-50 prerequi-site nodes are not immediately placed into the graph Whenexpandingcontracting nodes a small information box at thebottom of the screen displays the number of hidden nodes andedges
Figure 6 This figure displays the graph editor described in the text
Figure 7 This figure shows the annotated toolbox for the graph editor
When hovering over a node a small circle appears on theouter radius of the node If the user clicks the small circle thena content editor overlays the graphs with roughly a 08 alphavalue Figure 8 This content editor lets the user provide asummary for the concept as well as provide links to learningresources and justifications for the prerequisites Presentlythis form is hard-coded html but in the near future we wouldlike to create a simple interface for specifying the types dataassociated with the knowledge map In this way we hope tomake this system usable outside of Metacademy
DISCUSSIONThe Metacademy system was designed so that individuals invariegated disciplines could both contribute content and ex-plore preexisting content Until now this process involved
5httpwwwjsonorg
Figure 8 This figure shows the content-editing form that overlays theknowledge map creator This form allows users to associate content withthe graph
exploring large static graphs and editing text files The con-tributions from this paper will help users interactively explorethe knowledge maps and associated content Furthermoreusers can now contribute content via a visual knowledge mapcreator
From watching users interact with our system during aresearch-poster session we found that several users had hadcomments along the line of ldquovisualizing the edges isnrsquot nec-essary until yoursquove chosen a reference node and you want tosee itrsquos relationshipsrdquo While this certainly is not a formal us-ability study we found it encouraging the users organicallyagreed with one of the driving forces for this work We maypursue a formal usability study in the future
This work and its inevitable progressions and exten-sions can be found at httpwwwmetacademyorgThe exact system described in this paper can befound at httpgithubcommetacademymetacademy|applicationtreegraph|editor
FUTURE WORKThe Khan Academy knowledge map inspired this paper andin the coming weeks we plan to make our system compatiblewith their data As mentioned in the previous section a for-mal user study would help validate or nullify our conjecturethat our knowledge map exploration and creation tools helpusers efficiently explore and create educational material
Beyond the realm of visualization we would like to explorehow users interact with these graphs in an online educationalsetting Some interesting questions to answer include whatadvantages and disadvantages does a knowledge map systemhave over traditional content presentations such as a textbookor wiki page do users that create knowledge maps performbetter in the class (not a concept map as has been studiedcountless times in educational psychology) and to what ex-tent are large knowledge maps useful ie how big is too big
ACKNOWLEDGEMENTSColorado Reed would like to thank Kristin Stephens for herfantastic feedback throughout this project and Dan Allan andZach MacHardy for testing early versions of the knowledgemap creator
REFERENCES1 dagre javascript library
httpsgithubcomcpettittdagre Accessed2013-12-01
2 Duolingo httpwwwduolingocom Accessed2013-12-01
3 Khan academy httpwwwkhanacademyorgAccessed 2013-12-01
4 Metacademy httpwwwmetacademyorg Accessed2013-12-01
5 Palantir httpwwwpalantirtechcom Accessed2013-12-01
6 Touchgraph httpwwwtouchgraphcomnavigatorAccessed 2013-12-01
7 Batagelj V and Mrvar A Pajekanalysis andvisualization of large networks Springer 2004
8 Bostock M Ogievetsky V and Heer J D3Data-driven documents IEEE Trans Visualization ampComp Graphics (Proc InfoVis) (2011)
9 Canas A J Hill G Carff R Suri N Lott JEskridge T Gomez G Arroyo M and Carvajal RCmaptools A knowledge modeling and sharingenvironment In Concept maps Theory methodologytechnology Proceedings of the first internationalconference on concept mapping vol 1 (2004) 125ndash133
10 Chimani M and Gutwenger C The open graphdrawing framework (ogdf) httpwwwogdfnet
11 Eisner J Kornbluh M Woodhull G Buse RHuang S Michael C and Shafer G Visual navigation
through large directed graphs and hypergraphs InProceedings of the IEEE Symposium on InformationVisualization (InfoVisrsquo06) PosterDemo Session(Baltimore Oct 2006) 116ndash117
12 Gansner E R Koutsofios E North S C and VoK-P A technique for drawing directed graphs SoftwareEngineering IEEE Transactions on 19 3 (1993)214ndash230
13 Khan S The one world schoolhouse Educationreimagined Hachette Digital Inc 2012
14 Krzywinski M Birol I Jones S J and Marra M AHive plotsrational approach to visualizing networksBriefings in Bioinformatics 13 5 (2012) 627ndash644
15 Misue K Eades P Lai W and Sugiyama K Layoutadjustment and the mental map Journal of visuallanguages and computing 6 2 (1995) 183ndash210
16 Nielsen J Evaluating the thinking-aloud technique foruse by computer scientists In Advances inhuman-computer interaction (vol 3) Ablex PublishingCorp (1993) 69ndash82
17 Sugiyama K Tagawa S and Toda M Methods forvisual understanding of hierarchical system structuresSystems Man and Cybernetics IEEE Transactions on11 2 (1981) 109ndash125
18 Van Ham F and Perer A Search show contextexpand on demand Supporting large graph explorationwith degree-of-interest Visualization and ComputerGraphics IEEE Transactions on 15 6 (2009) 953ndash960
- Introduction
- Related Work
-
- Online Knowledge Graphs
- Visualization and Exploration of Directed Graphs
- Knowledge Map Creation
-
- Methodology
-
- Knowledge Map Creation and Exploration
- Knowledge Map Creation
-
- Discussion
- Future Work
- Acknowledgements
- REFERENCES
-
eg Khan Academy [3] Duo Lingo [2] and Metacademy[4]Khan Academy a nonprofit online educational company withover six million unique monthly visitors has a large (700+node) knowledge map that founder Salman Khan discussedon p 52 of [13]
[The knowledge map] became a core piece of the KhanAcademy software platform In stressing the connec-tions among subjects and giving learners a visual pictureof where theyrsquove been and where theyrsquore going we hopeto encourage students to follow their own pathmdashto moveactively up down and sideways wherever their imagi-nations lead
A problem with knowledge maps however is that the largenumber of relationships between the concepts produces alarge number of intertwined and crisscrossing edges in thegraph In turn it is difficult to interpret the conceptual re-lationships see Figure 2 for examples from Khan Academyand Metacademy These ldquohairballrdquo graphs are common innetwork visualizations and indeed there is a deep field ofresearch focused on visualizing networks eg see [7] Thehierarchical placements of the nodes and edges in knowledgemaps express the dependency structure within the body ofknowledge and for this reason many network visualizationtechniques will not work with knowledge maps as they ob-scure or break these relationships
In this work we create a novel interactive system for creat-ing visualizing and exploring knowledge maps On the cre-ation side we constructed a graphical user interface (GUI)for visually specifying concepts their relationships and as-sociated content On the visualization and exploration sideour system uses a layered (Sugiyama) directed graph place-ment algorithm [17] to initially place the nodes a topologi-cal sort to determine an initial set of visible edges (the topo-logical sort maintains the connectivity of the graph) an edgelength threshold to determine which remaining edges shouldbe shown and edge ldquowispsrdquo to indicate the absence of anedge These techniques greatly reduce the number of visibleedges in large knowledge graphs while maintaining the hier-archical and connectivity structure of the content
To further accommodate the goals of knowledge map userswe incorporated several interactive components that aid theuser in locating concepts of interest and understanding theirrelationships to other concepts In particular our system al-lows users to step through the dependency structure of a graphby sequentially visualizing contextual views of the concepts(the direct dependencies and outlinks) Furthermore we in-clude a number of hover and click operations that reveal thereason for the relationships and associated content present fora given concept
RELATED WORKIn this section we discuss three areas of previous research anddevelopment relevant to this paper online knowledge graphsystems visualizations of large directed graphs and tools forcreating knowledge maps
Online Knowledge GraphsThe knowledge map creation visualization and explorationsystem described in this paper was designed for Meta-cademymdashan open source platform for collaboratively cre-ating refining and sharing knowledge maps and associatedlearning material [4] Figure 2 (right) shows a Metacademyknowledge graph one of the two main components of Meta-cademy The other major component is a text-based view thatprovides the user with the following information for each con-cept a description of the concept goals for the learner theprerequisites of the concept learning resources for the con-cept and related (non-dependency) concepts Metacademy istypically used as a targeted learning resource whereby a usersearches for a particular concept and is then presented witha knowledge map with that concept as the sole leafmdashmostgraphs have 5-50 nodes and less than 100 edges
As mentioned in the introduction Khan Academy [3]mdasha freevideo lecture and exercise repositorymdashalso provides a knowl-edge map of their material Rather than showing target-basedcomponents of the graph Khan Academy shows users theirentire 700+ node knowledge map This knowledge map ismuch larger than any of Metacademyrsquos knowledge maps butthe techniques discussed in this work are applicable to theirgraph and we will format their content for our system in fu-ture work
Visualization and Exploration of Directed GraphsOur knowledge map visualization and exploration systemdraws on a number contributions from the graph visualizationcommunity In particular the authors of [18] developed an in-teractive visualization system for large networks eg threemillion nodes The authors focused on providing context-specific visualizations of large directed graphs and stated that
Although providing a structural overview of the graphis a laudable goal there are many cases where the useris simply not interested in a global view of the wholegraph but wants to solve a particular concrete task onthe graph instead
Rather than show all direct inlinks and outlinks as is donein several related systems [5 6] the authors devised a de-gree of interest (DOI) term for each node that they use todetermine which subset of contextual nodes to show the userSince individual nodes in knowledge maps have a relativelysmall degree it is typically possible to visualize the fullcontext of each node rather than a subset and we do notuse the DOI technique in this work The authors use wispedgesmdashtruncated inlinksoutlinks to a nodemdashto indicate thepresence of an edge without cluttering the visualization withlong crisscrossing edges We incorporate this technique intoour visualization (see the next section)
Figure 2 The left figure shows a section of the Khan Academy knowledge map the right figure shows a Metacademy knowledge map
There are a number of software packages and algorithms forvisualizing (large) directed graphs that inspired our system
bull the dot placement algorithm described in [12] is theunderlying placement algorithm for the popular graphvizC++ graph visualization software The dot algorithm isused in the dagre JavaScript graph placement library whichis used in this work (see the next item)
bull dagre [1] a JavaScript library for directed graph placementis used to generate node and edge placements in our knowl-edge map system
bull Dynasty [11] a system for browsing large directed graphswhere only a small portion of the graph is visible at anygiven time Dynasty inspired the ldquofocused moderdquo aspectof our system where a concept and its dependencies andoutlinks are visible at a given time
bull the transition animations in Open Graph Drawing frame-work [10] a large C++ library of algorithms for graphdrawing inspired the transition animations sequence andtiming used in this work
Furthermore Misue et alrsquos mental map preservation proper-ties for directed graphs [15] motivated our use of visualizationmodes rather than dynamic directed graphs
Finally many directed graph visualization systems transformthe data for visualization For instance hive plots [14] pro-duce a linear layout for grouping nodes by type and then ar-ranges the nodes along a radial axis Arranging knowledgemap concepts along a radial axis does not work well for a vi-sualization as the concepts commonly have relationships thatspan multiple levels and as a result the radial network be-comes polluted with edges that span most of the circumfer-ence Strictly hiding these types of dependencies masks theprerequisite complexity for various concepts and could de-ceive the user
Knowledge Map CreationAs of the time of this writing the author currently does notknow of any tools that are explicitly used for creating knowl-edge maps as defined in this paper Both Khan Academyand Metacademy use text-based systems eg html formsto specify the relationships in the graph However there is alarge number of tools for creating similar structures such as
diagrams mind maps and concept maps We briefly discussthese tools and their relevance to our system
Cmap tools [9] is a Java-based GUI for creating conceptmaps directed graphs where the nodes are concepts and theedges are linking words so that tracing through a path is akinto reading a sentence The Cmap tools GUI inspired our edi-tor as it has simple drag and click operations to construct thegraph The Cmap tool editor however does not have asso-ciated content with the graph and is not browser-based Fur-thermore the Cmap tools editor requires the user to manuallyplace the nodes and edges Our editor operates in a similarfashion but users have the ability to algorithmically arrangethe nodes and edges using the dagre javascript library Fi-nally our editor is browser-based while Cmap tools is a Javaapplet
There are countless diagramming tools such as gliffy1drawio2 and dia 3 Many of these of tools are polished in-tuitive and downright beautiful diagramming tools but theyare more general purpose than what is needed for knowledgemap creation Letting users choose the color and shape of thenodes the thickness of the edges etc detracts from the mainpurpose of specifying the concept relationships and providingassociated content
METHODOLOGYIn this section we describe the visualization and interactionmethods employed by our knowledge map system In par-ticular our system builds upon the Metacademy knowledgemap system and we have recently acquired the data for KhanAcademyrsquos knowledge map and plan to extend our system forthis data in the near future
Knowledge Map Creation and ExplorationFrom an informal usability study conducted with seven UCBerkeley students using the think-aloud protocol[16] and theMetacademy knowledge map system we identified the fol-lowing common objectives the students had when interactingwith the knowledge map4
1httpwwwgliffycomindex-gphp2httpwwwdrawio3httpsourceforgenetprojectsdia-installer4Specifically we observed the students interacting with relativelylarge (30-50 nodes) and small (circa 10 nodes) knowledge maps
1 visualize the overall complexity of the targeted conceptie the depth and breadth of the graph and the number ofedges and nodes
2 locate specific concepts within the knowledge map
3 identify the prerequisites and outlinks for a concept theprerequisites of its prerequisites outlinks of its outlinksetc
4 focus on a particular concept
5 explore related concepts not shown on the current graph(inclusion of new data)
Metacademyrsquos primary form of interaction is allowing usersto mark concepts they have learned and subsequently removethem from the graph When removing the concepts the Meta-cademy system naively recalculates the graph layout with-out taking into account the previous layout The system theninstantaneously flashes to the new knowledge graph withoutshowing the user the corresponding transitions see Figure 3This dynamic form of interaction does not directly addressany of the objectives expressed by the students in our usabil-ity test Furthermore according to Misue et alrsquos theory ofmental map preservation for dynamic directed graphs [15]this transition destroys the userrsquos mental map of the system asit violates the following three mental-map-preservation prop-erties
1 orthogonal ordering maintaining the relative directionsbetween the nodes (these updates can shift the relative di-rections see eg the ldquorandom variablesrdquo and ldquoexpectationand variancerdquo nodes in Figure 3)
2 proximity model maintaining the relative distance of thenodes (these updates can change the relative distance of thenodes see eg the ldquorandom variablesrdquo and ldquoexpectationand variancerdquo nodes in Figure 3)
3 topology model graphical objects within a relative regionshould stay within that region (these updates can changethe regional placement of nodes this is not present in thesupplied example but it tends to occur with larger graphs)
While letting users remove nodes and edges associated withlearned concepts helps reduce the number of objects in thegraph this form of interaction does not address the identifiedobjectives of the users and comes at the cost of nullifying theuserrsquos previous mental map From these observations we didnot include this functionality within our system
To address the visualization and exploration objectives abovewe created two modes within our system (1) an overviewmode in which the entire graph is shown and (2) a focusedmode in which a particular concept is shown along withits immediate dependencies and outlinks The goal of theoverview mode is to allow users to visualize the overall com-plexity of the targeted concept as well as quickly trace the
and associated content for 30-60 minutes while verbalizing theirthoughts These maps showed the full prerequisite structure for agiven concept in machine learning or probabilistic artificial intelli-gence
paths of various concepts (objectives 1 and 3) as well asquickly find specific concepts within the knowledge map andexplore related concepts by loading them into the currentgraph (objectives 2 and 5) The focused mode inspired bythe focused graph view from [11] address objective 4 Inthe following paragraphs we describe these modes in greaterdetail
Both Metacademy and Khan Academy knowledge maps havea large number of long crisscrossing edges that make it dif-ficult to trace the paths within the knowledge map We ad-dressed this problem in our system by employing the follow-ing algorithm to reduce the number of visible long crisscross-ing edges in our graph
1 remove all transitive edges from the graph
2 compute the graph layout using dagre
3 set visible-outlinkslarr empty
4 for each node n in the graphset visible-outlinkslarr visible-outlinkscup outlinks(n) with path length lt Lcup shortest-outlink(n)
5 for all edges e not in visible-outlinks display an edge wispprotruding from the source and target of e
This algorithm is used to help accurately convey the relation-ships between the concepts while reducing the edge satura-tion within the graph The motivating philosophy behind thistechnique is to show users edges when they are interested inseeing them while still providing enough edges to maintainthe proper structure of the graph
In step 1 we remove the transitive edges in the graph becausethey do not affect the relative prerequisite structure of thegraph That is if we have edges (AB) (BC) and (AC) thenremoving (AC) from the graph will not affect the ABC or-dering and from a pedagogical perspective it is a redundantedge In step 2 we compute the edge and node placements us-ing dagre though any graph placement algorithm can be usedduring this step
In step 4 we remove all edges exceeding a length thresholdL while ensuring that each node has at least one outlink (theshortest outlink) By including the shortest outlink for eachnode we maintain the visual connectivity of the graph afterhiding other long edges Naively truncating all edges couldinduce artificially separated clusters of concepts that would bedetermined by the graph placement algorithm rather than thecontent We note that with a small enough threshold L eachnode would have one outlink and the graph would become atree Therefore if we simply did not include the non-visibleedges in the graph placement algorithm it would be possibleto obtain a tree layout with zero edge crossings This tree lay-out however could visually invalidate the relative prerequi-site structure of the graph Using the previous example withnodes A B and C we could have a tree with visible edges(BC) and (AC) and invisible edge (BC) which would resultin visually placing nodes A and B on the same level
Figure 3 One of Metacademyrsquos primary forms of interaction is allowing users to remove concepts they know (left) the original graph (middle) thedimmed node is marked as known (right) the node is removed from the graph In removing the node the Metacademy system naively redeterminesthe location of the edges and nodes without taking into account their previous locations In the text we argue that this form of interaction nullifies theuserrsquos mental map and as a result we have not included this feature in our system
The edge wisps from step 4 give a simple visible indication ofthe number of prerequisites and outlinks for a given conceptand are used in the interactive components discussed belowFigure 4 shows a medium-sized Metacademy knowledge mapof 39 nodes rendered directly from dagre (74 visible edgeswith 61 visible edge crossings) and the same knowledge mapobtained using the above algorithm (38 visible edges with 0visible edge crossings)
We employ the following interactive tools and techniques tohelp the user explore and understand the knowledge map gen-erated using the visible-edge-reduction-algorithm
bull hovering over nodes highlights the inlinksoutlinks fromthat node as well as the source (target) of the inlink (out-link) and shows a set of clickable icons that the user canuse to eg show the content associated with the node
bull clicking on a node triggers a series of animations that tran-sitions to the focused mode for that node (see below)
bull clicking on an edge shows a pop-up justification for theedge
bull a contractible side menu gives users the ability to searchfor a concept in the graph
The ldquofocused moderdquo shows the user a selected concept aswell its dependencies and outlinks We use a series of ani-mations to contract and expand the graph when a user entersthe focused mode the node is highlighted then centered onthe screen then the dependencies and outlinks are translatedso that they are adjacent to the focused node while the othernodes and edges fade out This sequence is based on similartransitions in the Open Graph Drawing framework [10] Theanimations and interactive tools were implemented using thed3 JavaScript library [8]
In the focused mode a small information box appears atthe bottom of the screen and indicates the number of hid-den nodes and edges The entire animation sequence takesslightly longer than one second Clicking on an adjacent nodewithin focused mode moves that node into focus while click-ing on the focused node runs the sequence of animations in
reverse and returns the user to the full graph Figure 5 showsa visualization of focused mode and the associated transition
Knowledge Map CreationIn this subsection we describe our visual knowledge map cre-ator While this tool is independent of a particular platformits current implementation is aligned for the Metacademy sys-tem We designed this visual editor using the d3 JavaScriptlibrary [8] and only included features that benefitted the fol-lowing operations addremove concepts addremove rela-tionships between concepts (edges) addremove associatedcontent and preview the knowledge map using the techniquesdiscussed in the previous subsection
Figure 6 shows the knowledge map creator The main viewconsists of a large graph editing surface and a small toolboxThe user can perform the following editing operations
bull add node shift + click on the editing surface
bull remove node click on a node and press delete or backspace
bull move node click + drag node
bull change node title shift + click on the node
bull edit content associated with node click on the small circlethat appears when hovering the node
bull expandcontract a nodersquos dependencies (outlinks) click theplusminus icon at the top (bottom) of the node
bull add directed edge shift + click on the source node and dragto the target node
bull remove edge click on a edge and press delete or backspace
bull scale graph scroll
bull translate graph click + drag on the editing surface
In addition the toolbox shown in Figure 7 provides the userwith the ability to optimize the graph location using dagreclear all nodes and edges from the graph preview the graphusing the knowledge map exploration system described in this
Figure 4 The left figure shows the knowledge map obtained directly using the dagre output the right figure shows the knowledge map obtained usingour visible-edge-reduction algorithm
Figure 5 The left figure shows the highlighted edges and nodes when hovering over a specific node The right figure shows the ldquofocused moderdquo thatoccurs after clicking the node Clicking the focused node in the focused mode returns to the full-graph display while clicking the adjacent nodes placesthe focus on them
paper and downloadupload a graph tofrom a JSON5 repre-sentation
The search bar at the bottom of the toolbox gives users theability to add preexisting concepts to graph When a user addsa preexisting concept to the graph the inlinksoutlinks of thatnode are contracted by default so that eg 20-50 prerequi-site nodes are not immediately placed into the graph Whenexpandingcontracting nodes a small information box at thebottom of the screen displays the number of hidden nodes andedges
Figure 6 This figure displays the graph editor described in the text
Figure 7 This figure shows the annotated toolbox for the graph editor
When hovering over a node a small circle appears on theouter radius of the node If the user clicks the small circle thena content editor overlays the graphs with roughly a 08 alphavalue Figure 8 This content editor lets the user provide asummary for the concept as well as provide links to learningresources and justifications for the prerequisites Presentlythis form is hard-coded html but in the near future we wouldlike to create a simple interface for specifying the types dataassociated with the knowledge map In this way we hope tomake this system usable outside of Metacademy
DISCUSSIONThe Metacademy system was designed so that individuals invariegated disciplines could both contribute content and ex-plore preexisting content Until now this process involved
5httpwwwjsonorg
Figure 8 This figure shows the content-editing form that overlays theknowledge map creator This form allows users to associate content withthe graph
exploring large static graphs and editing text files The con-tributions from this paper will help users interactively explorethe knowledge maps and associated content Furthermoreusers can now contribute content via a visual knowledge mapcreator
From watching users interact with our system during aresearch-poster session we found that several users had hadcomments along the line of ldquovisualizing the edges isnrsquot nec-essary until yoursquove chosen a reference node and you want tosee itrsquos relationshipsrdquo While this certainly is not a formal us-ability study we found it encouraging the users organicallyagreed with one of the driving forces for this work We maypursue a formal usability study in the future
This work and its inevitable progressions and exten-sions can be found at httpwwwmetacademyorgThe exact system described in this paper can befound at httpgithubcommetacademymetacademy|applicationtreegraph|editor
FUTURE WORKThe Khan Academy knowledge map inspired this paper andin the coming weeks we plan to make our system compatiblewith their data As mentioned in the previous section a for-mal user study would help validate or nullify our conjecturethat our knowledge map exploration and creation tools helpusers efficiently explore and create educational material
Beyond the realm of visualization we would like to explorehow users interact with these graphs in an online educationalsetting Some interesting questions to answer include whatadvantages and disadvantages does a knowledge map systemhave over traditional content presentations such as a textbookor wiki page do users that create knowledge maps performbetter in the class (not a concept map as has been studiedcountless times in educational psychology) and to what ex-tent are large knowledge maps useful ie how big is too big
ACKNOWLEDGEMENTSColorado Reed would like to thank Kristin Stephens for herfantastic feedback throughout this project and Dan Allan andZach MacHardy for testing early versions of the knowledgemap creator
REFERENCES1 dagre javascript library
httpsgithubcomcpettittdagre Accessed2013-12-01
2 Duolingo httpwwwduolingocom Accessed2013-12-01
3 Khan academy httpwwwkhanacademyorgAccessed 2013-12-01
4 Metacademy httpwwwmetacademyorg Accessed2013-12-01
5 Palantir httpwwwpalantirtechcom Accessed2013-12-01
6 Touchgraph httpwwwtouchgraphcomnavigatorAccessed 2013-12-01
7 Batagelj V and Mrvar A Pajekanalysis andvisualization of large networks Springer 2004
8 Bostock M Ogievetsky V and Heer J D3Data-driven documents IEEE Trans Visualization ampComp Graphics (Proc InfoVis) (2011)
9 Canas A J Hill G Carff R Suri N Lott JEskridge T Gomez G Arroyo M and Carvajal RCmaptools A knowledge modeling and sharingenvironment In Concept maps Theory methodologytechnology Proceedings of the first internationalconference on concept mapping vol 1 (2004) 125ndash133
10 Chimani M and Gutwenger C The open graphdrawing framework (ogdf) httpwwwogdfnet
11 Eisner J Kornbluh M Woodhull G Buse RHuang S Michael C and Shafer G Visual navigation
through large directed graphs and hypergraphs InProceedings of the IEEE Symposium on InformationVisualization (InfoVisrsquo06) PosterDemo Session(Baltimore Oct 2006) 116ndash117
12 Gansner E R Koutsofios E North S C and VoK-P A technique for drawing directed graphs SoftwareEngineering IEEE Transactions on 19 3 (1993)214ndash230
13 Khan S The one world schoolhouse Educationreimagined Hachette Digital Inc 2012
14 Krzywinski M Birol I Jones S J and Marra M AHive plotsrational approach to visualizing networksBriefings in Bioinformatics 13 5 (2012) 627ndash644
15 Misue K Eades P Lai W and Sugiyama K Layoutadjustment and the mental map Journal of visuallanguages and computing 6 2 (1995) 183ndash210
16 Nielsen J Evaluating the thinking-aloud technique foruse by computer scientists In Advances inhuman-computer interaction (vol 3) Ablex PublishingCorp (1993) 69ndash82
17 Sugiyama K Tagawa S and Toda M Methods forvisual understanding of hierarchical system structuresSystems Man and Cybernetics IEEE Transactions on11 2 (1981) 109ndash125
18 Van Ham F and Perer A Search show contextexpand on demand Supporting large graph explorationwith degree-of-interest Visualization and ComputerGraphics IEEE Transactions on 15 6 (2009) 953ndash960
- Introduction
- Related Work
-
- Online Knowledge Graphs
- Visualization and Exploration of Directed Graphs
- Knowledge Map Creation
-
- Methodology
-
- Knowledge Map Creation and Exploration
- Knowledge Map Creation
-
- Discussion
- Future Work
- Acknowledgements
- REFERENCES
-
Figure 2 The left figure shows a section of the Khan Academy knowledge map the right figure shows a Metacademy knowledge map
There are a number of software packages and algorithms forvisualizing (large) directed graphs that inspired our system
bull the dot placement algorithm described in [12] is theunderlying placement algorithm for the popular graphvizC++ graph visualization software The dot algorithm isused in the dagre JavaScript graph placement library whichis used in this work (see the next item)
bull dagre [1] a JavaScript library for directed graph placementis used to generate node and edge placements in our knowl-edge map system
bull Dynasty [11] a system for browsing large directed graphswhere only a small portion of the graph is visible at anygiven time Dynasty inspired the ldquofocused moderdquo aspectof our system where a concept and its dependencies andoutlinks are visible at a given time
bull the transition animations in Open Graph Drawing frame-work [10] a large C++ library of algorithms for graphdrawing inspired the transition animations sequence andtiming used in this work
Furthermore Misue et alrsquos mental map preservation proper-ties for directed graphs [15] motivated our use of visualizationmodes rather than dynamic directed graphs
Finally many directed graph visualization systems transformthe data for visualization For instance hive plots [14] pro-duce a linear layout for grouping nodes by type and then ar-ranges the nodes along a radial axis Arranging knowledgemap concepts along a radial axis does not work well for a vi-sualization as the concepts commonly have relationships thatspan multiple levels and as a result the radial network be-comes polluted with edges that span most of the circumfer-ence Strictly hiding these types of dependencies masks theprerequisite complexity for various concepts and could de-ceive the user
Knowledge Map CreationAs of the time of this writing the author currently does notknow of any tools that are explicitly used for creating knowl-edge maps as defined in this paper Both Khan Academyand Metacademy use text-based systems eg html formsto specify the relationships in the graph However there is alarge number of tools for creating similar structures such as
diagrams mind maps and concept maps We briefly discussthese tools and their relevance to our system
Cmap tools [9] is a Java-based GUI for creating conceptmaps directed graphs where the nodes are concepts and theedges are linking words so that tracing through a path is akinto reading a sentence The Cmap tools GUI inspired our edi-tor as it has simple drag and click operations to construct thegraph The Cmap tool editor however does not have asso-ciated content with the graph and is not browser-based Fur-thermore the Cmap tools editor requires the user to manuallyplace the nodes and edges Our editor operates in a similarfashion but users have the ability to algorithmically arrangethe nodes and edges using the dagre javascript library Fi-nally our editor is browser-based while Cmap tools is a Javaapplet
There are countless diagramming tools such as gliffy1drawio2 and dia 3 Many of these of tools are polished in-tuitive and downright beautiful diagramming tools but theyare more general purpose than what is needed for knowledgemap creation Letting users choose the color and shape of thenodes the thickness of the edges etc detracts from the mainpurpose of specifying the concept relationships and providingassociated content
METHODOLOGYIn this section we describe the visualization and interactionmethods employed by our knowledge map system In par-ticular our system builds upon the Metacademy knowledgemap system and we have recently acquired the data for KhanAcademyrsquos knowledge map and plan to extend our system forthis data in the near future
Knowledge Map Creation and ExplorationFrom an informal usability study conducted with seven UCBerkeley students using the think-aloud protocol[16] and theMetacademy knowledge map system we identified the fol-lowing common objectives the students had when interactingwith the knowledge map4
1httpwwwgliffycomindex-gphp2httpwwwdrawio3httpsourceforgenetprojectsdia-installer4Specifically we observed the students interacting with relativelylarge (30-50 nodes) and small (circa 10 nodes) knowledge maps
1 visualize the overall complexity of the targeted conceptie the depth and breadth of the graph and the number ofedges and nodes
2 locate specific concepts within the knowledge map
3 identify the prerequisites and outlinks for a concept theprerequisites of its prerequisites outlinks of its outlinksetc
4 focus on a particular concept
5 explore related concepts not shown on the current graph(inclusion of new data)
Metacademyrsquos primary form of interaction is allowing usersto mark concepts they have learned and subsequently removethem from the graph When removing the concepts the Meta-cademy system naively recalculates the graph layout with-out taking into account the previous layout The system theninstantaneously flashes to the new knowledge graph withoutshowing the user the corresponding transitions see Figure 3This dynamic form of interaction does not directly addressany of the objectives expressed by the students in our usabil-ity test Furthermore according to Misue et alrsquos theory ofmental map preservation for dynamic directed graphs [15]this transition destroys the userrsquos mental map of the system asit violates the following three mental-map-preservation prop-erties
1 orthogonal ordering maintaining the relative directionsbetween the nodes (these updates can shift the relative di-rections see eg the ldquorandom variablesrdquo and ldquoexpectationand variancerdquo nodes in Figure 3)
2 proximity model maintaining the relative distance of thenodes (these updates can change the relative distance of thenodes see eg the ldquorandom variablesrdquo and ldquoexpectationand variancerdquo nodes in Figure 3)
3 topology model graphical objects within a relative regionshould stay within that region (these updates can changethe regional placement of nodes this is not present in thesupplied example but it tends to occur with larger graphs)
While letting users remove nodes and edges associated withlearned concepts helps reduce the number of objects in thegraph this form of interaction does not address the identifiedobjectives of the users and comes at the cost of nullifying theuserrsquos previous mental map From these observations we didnot include this functionality within our system
To address the visualization and exploration objectives abovewe created two modes within our system (1) an overviewmode in which the entire graph is shown and (2) a focusedmode in which a particular concept is shown along withits immediate dependencies and outlinks The goal of theoverview mode is to allow users to visualize the overall com-plexity of the targeted concept as well as quickly trace the
and associated content for 30-60 minutes while verbalizing theirthoughts These maps showed the full prerequisite structure for agiven concept in machine learning or probabilistic artificial intelli-gence
paths of various concepts (objectives 1 and 3) as well asquickly find specific concepts within the knowledge map andexplore related concepts by loading them into the currentgraph (objectives 2 and 5) The focused mode inspired bythe focused graph view from [11] address objective 4 Inthe following paragraphs we describe these modes in greaterdetail
Both Metacademy and Khan Academy knowledge maps havea large number of long crisscrossing edges that make it dif-ficult to trace the paths within the knowledge map We ad-dressed this problem in our system by employing the follow-ing algorithm to reduce the number of visible long crisscross-ing edges in our graph
1 remove all transitive edges from the graph
2 compute the graph layout using dagre
3 set visible-outlinkslarr empty
4 for each node n in the graphset visible-outlinkslarr visible-outlinkscup outlinks(n) with path length lt Lcup shortest-outlink(n)
5 for all edges e not in visible-outlinks display an edge wispprotruding from the source and target of e
This algorithm is used to help accurately convey the relation-ships between the concepts while reducing the edge satura-tion within the graph The motivating philosophy behind thistechnique is to show users edges when they are interested inseeing them while still providing enough edges to maintainthe proper structure of the graph
In step 1 we remove the transitive edges in the graph becausethey do not affect the relative prerequisite structure of thegraph That is if we have edges (AB) (BC) and (AC) thenremoving (AC) from the graph will not affect the ABC or-dering and from a pedagogical perspective it is a redundantedge In step 2 we compute the edge and node placements us-ing dagre though any graph placement algorithm can be usedduring this step
In step 4 we remove all edges exceeding a length thresholdL while ensuring that each node has at least one outlink (theshortest outlink) By including the shortest outlink for eachnode we maintain the visual connectivity of the graph afterhiding other long edges Naively truncating all edges couldinduce artificially separated clusters of concepts that would bedetermined by the graph placement algorithm rather than thecontent We note that with a small enough threshold L eachnode would have one outlink and the graph would become atree Therefore if we simply did not include the non-visibleedges in the graph placement algorithm it would be possibleto obtain a tree layout with zero edge crossings This tree lay-out however could visually invalidate the relative prerequi-site structure of the graph Using the previous example withnodes A B and C we could have a tree with visible edges(BC) and (AC) and invisible edge (BC) which would resultin visually placing nodes A and B on the same level
Figure 3 One of Metacademyrsquos primary forms of interaction is allowing users to remove concepts they know (left) the original graph (middle) thedimmed node is marked as known (right) the node is removed from the graph In removing the node the Metacademy system naively redeterminesthe location of the edges and nodes without taking into account their previous locations In the text we argue that this form of interaction nullifies theuserrsquos mental map and as a result we have not included this feature in our system
The edge wisps from step 4 give a simple visible indication ofthe number of prerequisites and outlinks for a given conceptand are used in the interactive components discussed belowFigure 4 shows a medium-sized Metacademy knowledge mapof 39 nodes rendered directly from dagre (74 visible edgeswith 61 visible edge crossings) and the same knowledge mapobtained using the above algorithm (38 visible edges with 0visible edge crossings)
We employ the following interactive tools and techniques tohelp the user explore and understand the knowledge map gen-erated using the visible-edge-reduction-algorithm
bull hovering over nodes highlights the inlinksoutlinks fromthat node as well as the source (target) of the inlink (out-link) and shows a set of clickable icons that the user canuse to eg show the content associated with the node
bull clicking on a node triggers a series of animations that tran-sitions to the focused mode for that node (see below)
bull clicking on an edge shows a pop-up justification for theedge
bull a contractible side menu gives users the ability to searchfor a concept in the graph
The ldquofocused moderdquo shows the user a selected concept aswell its dependencies and outlinks We use a series of ani-mations to contract and expand the graph when a user entersthe focused mode the node is highlighted then centered onthe screen then the dependencies and outlinks are translatedso that they are adjacent to the focused node while the othernodes and edges fade out This sequence is based on similartransitions in the Open Graph Drawing framework [10] Theanimations and interactive tools were implemented using thed3 JavaScript library [8]
In the focused mode a small information box appears atthe bottom of the screen and indicates the number of hid-den nodes and edges The entire animation sequence takesslightly longer than one second Clicking on an adjacent nodewithin focused mode moves that node into focus while click-ing on the focused node runs the sequence of animations in
reverse and returns the user to the full graph Figure 5 showsa visualization of focused mode and the associated transition
Knowledge Map CreationIn this subsection we describe our visual knowledge map cre-ator While this tool is independent of a particular platformits current implementation is aligned for the Metacademy sys-tem We designed this visual editor using the d3 JavaScriptlibrary [8] and only included features that benefitted the fol-lowing operations addremove concepts addremove rela-tionships between concepts (edges) addremove associatedcontent and preview the knowledge map using the techniquesdiscussed in the previous subsection
Figure 6 shows the knowledge map creator The main viewconsists of a large graph editing surface and a small toolboxThe user can perform the following editing operations
bull add node shift + click on the editing surface
bull remove node click on a node and press delete or backspace
bull move node click + drag node
bull change node title shift + click on the node
bull edit content associated with node click on the small circlethat appears when hovering the node
bull expandcontract a nodersquos dependencies (outlinks) click theplusminus icon at the top (bottom) of the node
bull add directed edge shift + click on the source node and dragto the target node
bull remove edge click on a edge and press delete or backspace
bull scale graph scroll
bull translate graph click + drag on the editing surface
In addition the toolbox shown in Figure 7 provides the userwith the ability to optimize the graph location using dagreclear all nodes and edges from the graph preview the graphusing the knowledge map exploration system described in this
Figure 4 The left figure shows the knowledge map obtained directly using the dagre output the right figure shows the knowledge map obtained usingour visible-edge-reduction algorithm
Figure 5 The left figure shows the highlighted edges and nodes when hovering over a specific node The right figure shows the ldquofocused moderdquo thatoccurs after clicking the node Clicking the focused node in the focused mode returns to the full-graph display while clicking the adjacent nodes placesthe focus on them
paper and downloadupload a graph tofrom a JSON5 repre-sentation
The search bar at the bottom of the toolbox gives users theability to add preexisting concepts to graph When a user addsa preexisting concept to the graph the inlinksoutlinks of thatnode are contracted by default so that eg 20-50 prerequi-site nodes are not immediately placed into the graph Whenexpandingcontracting nodes a small information box at thebottom of the screen displays the number of hidden nodes andedges
Figure 6 This figure displays the graph editor described in the text
Figure 7 This figure shows the annotated toolbox for the graph editor
When hovering over a node a small circle appears on theouter radius of the node If the user clicks the small circle thena content editor overlays the graphs with roughly a 08 alphavalue Figure 8 This content editor lets the user provide asummary for the concept as well as provide links to learningresources and justifications for the prerequisites Presentlythis form is hard-coded html but in the near future we wouldlike to create a simple interface for specifying the types dataassociated with the knowledge map In this way we hope tomake this system usable outside of Metacademy
DISCUSSIONThe Metacademy system was designed so that individuals invariegated disciplines could both contribute content and ex-plore preexisting content Until now this process involved
5httpwwwjsonorg
Figure 8 This figure shows the content-editing form that overlays theknowledge map creator This form allows users to associate content withthe graph
exploring large static graphs and editing text files The con-tributions from this paper will help users interactively explorethe knowledge maps and associated content Furthermoreusers can now contribute content via a visual knowledge mapcreator
From watching users interact with our system during aresearch-poster session we found that several users had hadcomments along the line of ldquovisualizing the edges isnrsquot nec-essary until yoursquove chosen a reference node and you want tosee itrsquos relationshipsrdquo While this certainly is not a formal us-ability study we found it encouraging the users organicallyagreed with one of the driving forces for this work We maypursue a formal usability study in the future
This work and its inevitable progressions and exten-sions can be found at httpwwwmetacademyorgThe exact system described in this paper can befound at httpgithubcommetacademymetacademy|applicationtreegraph|editor
FUTURE WORKThe Khan Academy knowledge map inspired this paper andin the coming weeks we plan to make our system compatiblewith their data As mentioned in the previous section a for-mal user study would help validate or nullify our conjecturethat our knowledge map exploration and creation tools helpusers efficiently explore and create educational material
Beyond the realm of visualization we would like to explorehow users interact with these graphs in an online educationalsetting Some interesting questions to answer include whatadvantages and disadvantages does a knowledge map systemhave over traditional content presentations such as a textbookor wiki page do users that create knowledge maps performbetter in the class (not a concept map as has been studiedcountless times in educational psychology) and to what ex-tent are large knowledge maps useful ie how big is too big
ACKNOWLEDGEMENTSColorado Reed would like to thank Kristin Stephens for herfantastic feedback throughout this project and Dan Allan andZach MacHardy for testing early versions of the knowledgemap creator
REFERENCES1 dagre javascript library
httpsgithubcomcpettittdagre Accessed2013-12-01
2 Duolingo httpwwwduolingocom Accessed2013-12-01
3 Khan academy httpwwwkhanacademyorgAccessed 2013-12-01
4 Metacademy httpwwwmetacademyorg Accessed2013-12-01
5 Palantir httpwwwpalantirtechcom Accessed2013-12-01
6 Touchgraph httpwwwtouchgraphcomnavigatorAccessed 2013-12-01
7 Batagelj V and Mrvar A Pajekanalysis andvisualization of large networks Springer 2004
8 Bostock M Ogievetsky V and Heer J D3Data-driven documents IEEE Trans Visualization ampComp Graphics (Proc InfoVis) (2011)
9 Canas A J Hill G Carff R Suri N Lott JEskridge T Gomez G Arroyo M and Carvajal RCmaptools A knowledge modeling and sharingenvironment In Concept maps Theory methodologytechnology Proceedings of the first internationalconference on concept mapping vol 1 (2004) 125ndash133
10 Chimani M and Gutwenger C The open graphdrawing framework (ogdf) httpwwwogdfnet
11 Eisner J Kornbluh M Woodhull G Buse RHuang S Michael C and Shafer G Visual navigation
through large directed graphs and hypergraphs InProceedings of the IEEE Symposium on InformationVisualization (InfoVisrsquo06) PosterDemo Session(Baltimore Oct 2006) 116ndash117
12 Gansner E R Koutsofios E North S C and VoK-P A technique for drawing directed graphs SoftwareEngineering IEEE Transactions on 19 3 (1993)214ndash230
13 Khan S The one world schoolhouse Educationreimagined Hachette Digital Inc 2012
14 Krzywinski M Birol I Jones S J and Marra M AHive plotsrational approach to visualizing networksBriefings in Bioinformatics 13 5 (2012) 627ndash644
15 Misue K Eades P Lai W and Sugiyama K Layoutadjustment and the mental map Journal of visuallanguages and computing 6 2 (1995) 183ndash210
16 Nielsen J Evaluating the thinking-aloud technique foruse by computer scientists In Advances inhuman-computer interaction (vol 3) Ablex PublishingCorp (1993) 69ndash82
17 Sugiyama K Tagawa S and Toda M Methods forvisual understanding of hierarchical system structuresSystems Man and Cybernetics IEEE Transactions on11 2 (1981) 109ndash125
18 Van Ham F and Perer A Search show contextexpand on demand Supporting large graph explorationwith degree-of-interest Visualization and ComputerGraphics IEEE Transactions on 15 6 (2009) 953ndash960
- Introduction
- Related Work
-
- Online Knowledge Graphs
- Visualization and Exploration of Directed Graphs
- Knowledge Map Creation
-
- Methodology
-
- Knowledge Map Creation and Exploration
- Knowledge Map Creation
-
- Discussion
- Future Work
- Acknowledgements
- REFERENCES
-
1 visualize the overall complexity of the targeted conceptie the depth and breadth of the graph and the number ofedges and nodes
2 locate specific concepts within the knowledge map
3 identify the prerequisites and outlinks for a concept theprerequisites of its prerequisites outlinks of its outlinksetc
4 focus on a particular concept
5 explore related concepts not shown on the current graph(inclusion of new data)
Metacademyrsquos primary form of interaction is allowing usersto mark concepts they have learned and subsequently removethem from the graph When removing the concepts the Meta-cademy system naively recalculates the graph layout with-out taking into account the previous layout The system theninstantaneously flashes to the new knowledge graph withoutshowing the user the corresponding transitions see Figure 3This dynamic form of interaction does not directly addressany of the objectives expressed by the students in our usabil-ity test Furthermore according to Misue et alrsquos theory ofmental map preservation for dynamic directed graphs [15]this transition destroys the userrsquos mental map of the system asit violates the following three mental-map-preservation prop-erties
1 orthogonal ordering maintaining the relative directionsbetween the nodes (these updates can shift the relative di-rections see eg the ldquorandom variablesrdquo and ldquoexpectationand variancerdquo nodes in Figure 3)
2 proximity model maintaining the relative distance of thenodes (these updates can change the relative distance of thenodes see eg the ldquorandom variablesrdquo and ldquoexpectationand variancerdquo nodes in Figure 3)
3 topology model graphical objects within a relative regionshould stay within that region (these updates can changethe regional placement of nodes this is not present in thesupplied example but it tends to occur with larger graphs)
While letting users remove nodes and edges associated withlearned concepts helps reduce the number of objects in thegraph this form of interaction does not address the identifiedobjectives of the users and comes at the cost of nullifying theuserrsquos previous mental map From these observations we didnot include this functionality within our system
To address the visualization and exploration objectives abovewe created two modes within our system (1) an overviewmode in which the entire graph is shown and (2) a focusedmode in which a particular concept is shown along withits immediate dependencies and outlinks The goal of theoverview mode is to allow users to visualize the overall com-plexity of the targeted concept as well as quickly trace the
and associated content for 30-60 minutes while verbalizing theirthoughts These maps showed the full prerequisite structure for agiven concept in machine learning or probabilistic artificial intelli-gence
paths of various concepts (objectives 1 and 3) as well asquickly find specific concepts within the knowledge map andexplore related concepts by loading them into the currentgraph (objectives 2 and 5) The focused mode inspired bythe focused graph view from [11] address objective 4 Inthe following paragraphs we describe these modes in greaterdetail
Both Metacademy and Khan Academy knowledge maps havea large number of long crisscrossing edges that make it dif-ficult to trace the paths within the knowledge map We ad-dressed this problem in our system by employing the follow-ing algorithm to reduce the number of visible long crisscross-ing edges in our graph
1 remove all transitive edges from the graph
2 compute the graph layout using dagre
3 set visible-outlinkslarr empty
4 for each node n in the graphset visible-outlinkslarr visible-outlinkscup outlinks(n) with path length lt Lcup shortest-outlink(n)
5 for all edges e not in visible-outlinks display an edge wispprotruding from the source and target of e
This algorithm is used to help accurately convey the relation-ships between the concepts while reducing the edge satura-tion within the graph The motivating philosophy behind thistechnique is to show users edges when they are interested inseeing them while still providing enough edges to maintainthe proper structure of the graph
In step 1 we remove the transitive edges in the graph becausethey do not affect the relative prerequisite structure of thegraph That is if we have edges (AB) (BC) and (AC) thenremoving (AC) from the graph will not affect the ABC or-dering and from a pedagogical perspective it is a redundantedge In step 2 we compute the edge and node placements us-ing dagre though any graph placement algorithm can be usedduring this step
In step 4 we remove all edges exceeding a length thresholdL while ensuring that each node has at least one outlink (theshortest outlink) By including the shortest outlink for eachnode we maintain the visual connectivity of the graph afterhiding other long edges Naively truncating all edges couldinduce artificially separated clusters of concepts that would bedetermined by the graph placement algorithm rather than thecontent We note that with a small enough threshold L eachnode would have one outlink and the graph would become atree Therefore if we simply did not include the non-visibleedges in the graph placement algorithm it would be possibleto obtain a tree layout with zero edge crossings This tree lay-out however could visually invalidate the relative prerequi-site structure of the graph Using the previous example withnodes A B and C we could have a tree with visible edges(BC) and (AC) and invisible edge (BC) which would resultin visually placing nodes A and B on the same level
Figure 3 One of Metacademyrsquos primary forms of interaction is allowing users to remove concepts they know (left) the original graph (middle) thedimmed node is marked as known (right) the node is removed from the graph In removing the node the Metacademy system naively redeterminesthe location of the edges and nodes without taking into account their previous locations In the text we argue that this form of interaction nullifies theuserrsquos mental map and as a result we have not included this feature in our system
The edge wisps from step 4 give a simple visible indication ofthe number of prerequisites and outlinks for a given conceptand are used in the interactive components discussed belowFigure 4 shows a medium-sized Metacademy knowledge mapof 39 nodes rendered directly from dagre (74 visible edgeswith 61 visible edge crossings) and the same knowledge mapobtained using the above algorithm (38 visible edges with 0visible edge crossings)
We employ the following interactive tools and techniques tohelp the user explore and understand the knowledge map gen-erated using the visible-edge-reduction-algorithm
bull hovering over nodes highlights the inlinksoutlinks fromthat node as well as the source (target) of the inlink (out-link) and shows a set of clickable icons that the user canuse to eg show the content associated with the node
bull clicking on a node triggers a series of animations that tran-sitions to the focused mode for that node (see below)
bull clicking on an edge shows a pop-up justification for theedge
bull a contractible side menu gives users the ability to searchfor a concept in the graph
The ldquofocused moderdquo shows the user a selected concept aswell its dependencies and outlinks We use a series of ani-mations to contract and expand the graph when a user entersthe focused mode the node is highlighted then centered onthe screen then the dependencies and outlinks are translatedso that they are adjacent to the focused node while the othernodes and edges fade out This sequence is based on similartransitions in the Open Graph Drawing framework [10] Theanimations and interactive tools were implemented using thed3 JavaScript library [8]
In the focused mode a small information box appears atthe bottom of the screen and indicates the number of hid-den nodes and edges The entire animation sequence takesslightly longer than one second Clicking on an adjacent nodewithin focused mode moves that node into focus while click-ing on the focused node runs the sequence of animations in
reverse and returns the user to the full graph Figure 5 showsa visualization of focused mode and the associated transition
Knowledge Map CreationIn this subsection we describe our visual knowledge map cre-ator While this tool is independent of a particular platformits current implementation is aligned for the Metacademy sys-tem We designed this visual editor using the d3 JavaScriptlibrary [8] and only included features that benefitted the fol-lowing operations addremove concepts addremove rela-tionships between concepts (edges) addremove associatedcontent and preview the knowledge map using the techniquesdiscussed in the previous subsection
Figure 6 shows the knowledge map creator The main viewconsists of a large graph editing surface and a small toolboxThe user can perform the following editing operations
bull add node shift + click on the editing surface
bull remove node click on a node and press delete or backspace
bull move node click + drag node
bull change node title shift + click on the node
bull edit content associated with node click on the small circlethat appears when hovering the node
bull expandcontract a nodersquos dependencies (outlinks) click theplusminus icon at the top (bottom) of the node
bull add directed edge shift + click on the source node and dragto the target node
bull remove edge click on a edge and press delete or backspace
bull scale graph scroll
bull translate graph click + drag on the editing surface
In addition the toolbox shown in Figure 7 provides the userwith the ability to optimize the graph location using dagreclear all nodes and edges from the graph preview the graphusing the knowledge map exploration system described in this
Figure 4 The left figure shows the knowledge map obtained directly using the dagre output the right figure shows the knowledge map obtained usingour visible-edge-reduction algorithm
Figure 5 The left figure shows the highlighted edges and nodes when hovering over a specific node The right figure shows the ldquofocused moderdquo thatoccurs after clicking the node Clicking the focused node in the focused mode returns to the full-graph display while clicking the adjacent nodes placesthe focus on them
paper and downloadupload a graph tofrom a JSON5 repre-sentation
The search bar at the bottom of the toolbox gives users theability to add preexisting concepts to graph When a user addsa preexisting concept to the graph the inlinksoutlinks of thatnode are contracted by default so that eg 20-50 prerequi-site nodes are not immediately placed into the graph Whenexpandingcontracting nodes a small information box at thebottom of the screen displays the number of hidden nodes andedges
Figure 6 This figure displays the graph editor described in the text
Figure 7 This figure shows the annotated toolbox for the graph editor
When hovering over a node a small circle appears on theouter radius of the node If the user clicks the small circle thena content editor overlays the graphs with roughly a 08 alphavalue Figure 8 This content editor lets the user provide asummary for the concept as well as provide links to learningresources and justifications for the prerequisites Presentlythis form is hard-coded html but in the near future we wouldlike to create a simple interface for specifying the types dataassociated with the knowledge map In this way we hope tomake this system usable outside of Metacademy
DISCUSSIONThe Metacademy system was designed so that individuals invariegated disciplines could both contribute content and ex-plore preexisting content Until now this process involved
5httpwwwjsonorg
Figure 8 This figure shows the content-editing form that overlays theknowledge map creator This form allows users to associate content withthe graph
exploring large static graphs and editing text files The con-tributions from this paper will help users interactively explorethe knowledge maps and associated content Furthermoreusers can now contribute content via a visual knowledge mapcreator
From watching users interact with our system during aresearch-poster session we found that several users had hadcomments along the line of ldquovisualizing the edges isnrsquot nec-essary until yoursquove chosen a reference node and you want tosee itrsquos relationshipsrdquo While this certainly is not a formal us-ability study we found it encouraging the users organicallyagreed with one of the driving forces for this work We maypursue a formal usability study in the future
This work and its inevitable progressions and exten-sions can be found at httpwwwmetacademyorgThe exact system described in this paper can befound at httpgithubcommetacademymetacademy|applicationtreegraph|editor
FUTURE WORKThe Khan Academy knowledge map inspired this paper andin the coming weeks we plan to make our system compatiblewith their data As mentioned in the previous section a for-mal user study would help validate or nullify our conjecturethat our knowledge map exploration and creation tools helpusers efficiently explore and create educational material
Beyond the realm of visualization we would like to explorehow users interact with these graphs in an online educationalsetting Some interesting questions to answer include whatadvantages and disadvantages does a knowledge map systemhave over traditional content presentations such as a textbookor wiki page do users that create knowledge maps performbetter in the class (not a concept map as has been studiedcountless times in educational psychology) and to what ex-tent are large knowledge maps useful ie how big is too big
ACKNOWLEDGEMENTSColorado Reed would like to thank Kristin Stephens for herfantastic feedback throughout this project and Dan Allan andZach MacHardy for testing early versions of the knowledgemap creator
REFERENCES1 dagre javascript library
httpsgithubcomcpettittdagre Accessed2013-12-01
2 Duolingo httpwwwduolingocom Accessed2013-12-01
3 Khan academy httpwwwkhanacademyorgAccessed 2013-12-01
4 Metacademy httpwwwmetacademyorg Accessed2013-12-01
5 Palantir httpwwwpalantirtechcom Accessed2013-12-01
6 Touchgraph httpwwwtouchgraphcomnavigatorAccessed 2013-12-01
7 Batagelj V and Mrvar A Pajekanalysis andvisualization of large networks Springer 2004
8 Bostock M Ogievetsky V and Heer J D3Data-driven documents IEEE Trans Visualization ampComp Graphics (Proc InfoVis) (2011)
9 Canas A J Hill G Carff R Suri N Lott JEskridge T Gomez G Arroyo M and Carvajal RCmaptools A knowledge modeling and sharingenvironment In Concept maps Theory methodologytechnology Proceedings of the first internationalconference on concept mapping vol 1 (2004) 125ndash133
10 Chimani M and Gutwenger C The open graphdrawing framework (ogdf) httpwwwogdfnet
11 Eisner J Kornbluh M Woodhull G Buse RHuang S Michael C and Shafer G Visual navigation
through large directed graphs and hypergraphs InProceedings of the IEEE Symposium on InformationVisualization (InfoVisrsquo06) PosterDemo Session(Baltimore Oct 2006) 116ndash117
12 Gansner E R Koutsofios E North S C and VoK-P A technique for drawing directed graphs SoftwareEngineering IEEE Transactions on 19 3 (1993)214ndash230
13 Khan S The one world schoolhouse Educationreimagined Hachette Digital Inc 2012
14 Krzywinski M Birol I Jones S J and Marra M AHive plotsrational approach to visualizing networksBriefings in Bioinformatics 13 5 (2012) 627ndash644
15 Misue K Eades P Lai W and Sugiyama K Layoutadjustment and the mental map Journal of visuallanguages and computing 6 2 (1995) 183ndash210
16 Nielsen J Evaluating the thinking-aloud technique foruse by computer scientists In Advances inhuman-computer interaction (vol 3) Ablex PublishingCorp (1993) 69ndash82
17 Sugiyama K Tagawa S and Toda M Methods forvisual understanding of hierarchical system structuresSystems Man and Cybernetics IEEE Transactions on11 2 (1981) 109ndash125
18 Van Ham F and Perer A Search show contextexpand on demand Supporting large graph explorationwith degree-of-interest Visualization and ComputerGraphics IEEE Transactions on 15 6 (2009) 953ndash960
- Introduction
- Related Work
-
- Online Knowledge Graphs
- Visualization and Exploration of Directed Graphs
- Knowledge Map Creation
-
- Methodology
-
- Knowledge Map Creation and Exploration
- Knowledge Map Creation
-
- Discussion
- Future Work
- Acknowledgements
- REFERENCES
-
Figure 3 One of Metacademyrsquos primary forms of interaction is allowing users to remove concepts they know (left) the original graph (middle) thedimmed node is marked as known (right) the node is removed from the graph In removing the node the Metacademy system naively redeterminesthe location of the edges and nodes without taking into account their previous locations In the text we argue that this form of interaction nullifies theuserrsquos mental map and as a result we have not included this feature in our system
The edge wisps from step 4 give a simple visible indication ofthe number of prerequisites and outlinks for a given conceptand are used in the interactive components discussed belowFigure 4 shows a medium-sized Metacademy knowledge mapof 39 nodes rendered directly from dagre (74 visible edgeswith 61 visible edge crossings) and the same knowledge mapobtained using the above algorithm (38 visible edges with 0visible edge crossings)
We employ the following interactive tools and techniques tohelp the user explore and understand the knowledge map gen-erated using the visible-edge-reduction-algorithm
bull hovering over nodes highlights the inlinksoutlinks fromthat node as well as the source (target) of the inlink (out-link) and shows a set of clickable icons that the user canuse to eg show the content associated with the node
bull clicking on a node triggers a series of animations that tran-sitions to the focused mode for that node (see below)
bull clicking on an edge shows a pop-up justification for theedge
bull a contractible side menu gives users the ability to searchfor a concept in the graph
The ldquofocused moderdquo shows the user a selected concept aswell its dependencies and outlinks We use a series of ani-mations to contract and expand the graph when a user entersthe focused mode the node is highlighted then centered onthe screen then the dependencies and outlinks are translatedso that they are adjacent to the focused node while the othernodes and edges fade out This sequence is based on similartransitions in the Open Graph Drawing framework [10] Theanimations and interactive tools were implemented using thed3 JavaScript library [8]
In the focused mode a small information box appears atthe bottom of the screen and indicates the number of hid-den nodes and edges The entire animation sequence takesslightly longer than one second Clicking on an adjacent nodewithin focused mode moves that node into focus while click-ing on the focused node runs the sequence of animations in
reverse and returns the user to the full graph Figure 5 showsa visualization of focused mode and the associated transition
Knowledge Map CreationIn this subsection we describe our visual knowledge map cre-ator While this tool is independent of a particular platformits current implementation is aligned for the Metacademy sys-tem We designed this visual editor using the d3 JavaScriptlibrary [8] and only included features that benefitted the fol-lowing operations addremove concepts addremove rela-tionships between concepts (edges) addremove associatedcontent and preview the knowledge map using the techniquesdiscussed in the previous subsection
Figure 6 shows the knowledge map creator The main viewconsists of a large graph editing surface and a small toolboxThe user can perform the following editing operations
bull add node shift + click on the editing surface
bull remove node click on a node and press delete or backspace
bull move node click + drag node
bull change node title shift + click on the node
bull edit content associated with node click on the small circlethat appears when hovering the node
bull expandcontract a nodersquos dependencies (outlinks) click theplusminus icon at the top (bottom) of the node
bull add directed edge shift + click on the source node and dragto the target node
bull remove edge click on a edge and press delete or backspace
bull scale graph scroll
bull translate graph click + drag on the editing surface
In addition the toolbox shown in Figure 7 provides the userwith the ability to optimize the graph location using dagreclear all nodes and edges from the graph preview the graphusing the knowledge map exploration system described in this
Figure 4 The left figure shows the knowledge map obtained directly using the dagre output the right figure shows the knowledge map obtained usingour visible-edge-reduction algorithm
Figure 5 The left figure shows the highlighted edges and nodes when hovering over a specific node The right figure shows the ldquofocused moderdquo thatoccurs after clicking the node Clicking the focused node in the focused mode returns to the full-graph display while clicking the adjacent nodes placesthe focus on them
paper and downloadupload a graph tofrom a JSON5 repre-sentation
The search bar at the bottom of the toolbox gives users theability to add preexisting concepts to graph When a user addsa preexisting concept to the graph the inlinksoutlinks of thatnode are contracted by default so that eg 20-50 prerequi-site nodes are not immediately placed into the graph Whenexpandingcontracting nodes a small information box at thebottom of the screen displays the number of hidden nodes andedges
Figure 6 This figure displays the graph editor described in the text
Figure 7 This figure shows the annotated toolbox for the graph editor
When hovering over a node a small circle appears on theouter radius of the node If the user clicks the small circle thena content editor overlays the graphs with roughly a 08 alphavalue Figure 8 This content editor lets the user provide asummary for the concept as well as provide links to learningresources and justifications for the prerequisites Presentlythis form is hard-coded html but in the near future we wouldlike to create a simple interface for specifying the types dataassociated with the knowledge map In this way we hope tomake this system usable outside of Metacademy
DISCUSSIONThe Metacademy system was designed so that individuals invariegated disciplines could both contribute content and ex-plore preexisting content Until now this process involved
5httpwwwjsonorg
Figure 8 This figure shows the content-editing form that overlays theknowledge map creator This form allows users to associate content withthe graph
exploring large static graphs and editing text files The con-tributions from this paper will help users interactively explorethe knowledge maps and associated content Furthermoreusers can now contribute content via a visual knowledge mapcreator
From watching users interact with our system during aresearch-poster session we found that several users had hadcomments along the line of ldquovisualizing the edges isnrsquot nec-essary until yoursquove chosen a reference node and you want tosee itrsquos relationshipsrdquo While this certainly is not a formal us-ability study we found it encouraging the users organicallyagreed with one of the driving forces for this work We maypursue a formal usability study in the future
This work and its inevitable progressions and exten-sions can be found at httpwwwmetacademyorgThe exact system described in this paper can befound at httpgithubcommetacademymetacademy|applicationtreegraph|editor
FUTURE WORKThe Khan Academy knowledge map inspired this paper andin the coming weeks we plan to make our system compatiblewith their data As mentioned in the previous section a for-mal user study would help validate or nullify our conjecturethat our knowledge map exploration and creation tools helpusers efficiently explore and create educational material
Beyond the realm of visualization we would like to explorehow users interact with these graphs in an online educationalsetting Some interesting questions to answer include whatadvantages and disadvantages does a knowledge map systemhave over traditional content presentations such as a textbookor wiki page do users that create knowledge maps performbetter in the class (not a concept map as has been studiedcountless times in educational psychology) and to what ex-tent are large knowledge maps useful ie how big is too big
ACKNOWLEDGEMENTSColorado Reed would like to thank Kristin Stephens for herfantastic feedback throughout this project and Dan Allan andZach MacHardy for testing early versions of the knowledgemap creator
REFERENCES1 dagre javascript library
httpsgithubcomcpettittdagre Accessed2013-12-01
2 Duolingo httpwwwduolingocom Accessed2013-12-01
3 Khan academy httpwwwkhanacademyorgAccessed 2013-12-01
4 Metacademy httpwwwmetacademyorg Accessed2013-12-01
5 Palantir httpwwwpalantirtechcom Accessed2013-12-01
6 Touchgraph httpwwwtouchgraphcomnavigatorAccessed 2013-12-01
7 Batagelj V and Mrvar A Pajekanalysis andvisualization of large networks Springer 2004
8 Bostock M Ogievetsky V and Heer J D3Data-driven documents IEEE Trans Visualization ampComp Graphics (Proc InfoVis) (2011)
9 Canas A J Hill G Carff R Suri N Lott JEskridge T Gomez G Arroyo M and Carvajal RCmaptools A knowledge modeling and sharingenvironment In Concept maps Theory methodologytechnology Proceedings of the first internationalconference on concept mapping vol 1 (2004) 125ndash133
10 Chimani M and Gutwenger C The open graphdrawing framework (ogdf) httpwwwogdfnet
11 Eisner J Kornbluh M Woodhull G Buse RHuang S Michael C and Shafer G Visual navigation
through large directed graphs and hypergraphs InProceedings of the IEEE Symposium on InformationVisualization (InfoVisrsquo06) PosterDemo Session(Baltimore Oct 2006) 116ndash117
12 Gansner E R Koutsofios E North S C and VoK-P A technique for drawing directed graphs SoftwareEngineering IEEE Transactions on 19 3 (1993)214ndash230
13 Khan S The one world schoolhouse Educationreimagined Hachette Digital Inc 2012
14 Krzywinski M Birol I Jones S J and Marra M AHive plotsrational approach to visualizing networksBriefings in Bioinformatics 13 5 (2012) 627ndash644
15 Misue K Eades P Lai W and Sugiyama K Layoutadjustment and the mental map Journal of visuallanguages and computing 6 2 (1995) 183ndash210
16 Nielsen J Evaluating the thinking-aloud technique foruse by computer scientists In Advances inhuman-computer interaction (vol 3) Ablex PublishingCorp (1993) 69ndash82
17 Sugiyama K Tagawa S and Toda M Methods forvisual understanding of hierarchical system structuresSystems Man and Cybernetics IEEE Transactions on11 2 (1981) 109ndash125
18 Van Ham F and Perer A Search show contextexpand on demand Supporting large graph explorationwith degree-of-interest Visualization and ComputerGraphics IEEE Transactions on 15 6 (2009) 953ndash960
- Introduction
- Related Work
-
- Online Knowledge Graphs
- Visualization and Exploration of Directed Graphs
- Knowledge Map Creation
-
- Methodology
-
- Knowledge Map Creation and Exploration
- Knowledge Map Creation
-
- Discussion
- Future Work
- Acknowledgements
- REFERENCES
-
Figure 4 The left figure shows the knowledge map obtained directly using the dagre output the right figure shows the knowledge map obtained usingour visible-edge-reduction algorithm
Figure 5 The left figure shows the highlighted edges and nodes when hovering over a specific node The right figure shows the ldquofocused moderdquo thatoccurs after clicking the node Clicking the focused node in the focused mode returns to the full-graph display while clicking the adjacent nodes placesthe focus on them
paper and downloadupload a graph tofrom a JSON5 repre-sentation
The search bar at the bottom of the toolbox gives users theability to add preexisting concepts to graph When a user addsa preexisting concept to the graph the inlinksoutlinks of thatnode are contracted by default so that eg 20-50 prerequi-site nodes are not immediately placed into the graph Whenexpandingcontracting nodes a small information box at thebottom of the screen displays the number of hidden nodes andedges
Figure 6 This figure displays the graph editor described in the text
Figure 7 This figure shows the annotated toolbox for the graph editor
When hovering over a node a small circle appears on theouter radius of the node If the user clicks the small circle thena content editor overlays the graphs with roughly a 08 alphavalue Figure 8 This content editor lets the user provide asummary for the concept as well as provide links to learningresources and justifications for the prerequisites Presentlythis form is hard-coded html but in the near future we wouldlike to create a simple interface for specifying the types dataassociated with the knowledge map In this way we hope tomake this system usable outside of Metacademy
DISCUSSIONThe Metacademy system was designed so that individuals invariegated disciplines could both contribute content and ex-plore preexisting content Until now this process involved
5httpwwwjsonorg
Figure 8 This figure shows the content-editing form that overlays theknowledge map creator This form allows users to associate content withthe graph
exploring large static graphs and editing text files The con-tributions from this paper will help users interactively explorethe knowledge maps and associated content Furthermoreusers can now contribute content via a visual knowledge mapcreator
From watching users interact with our system during aresearch-poster session we found that several users had hadcomments along the line of ldquovisualizing the edges isnrsquot nec-essary until yoursquove chosen a reference node and you want tosee itrsquos relationshipsrdquo While this certainly is not a formal us-ability study we found it encouraging the users organicallyagreed with one of the driving forces for this work We maypursue a formal usability study in the future
This work and its inevitable progressions and exten-sions can be found at httpwwwmetacademyorgThe exact system described in this paper can befound at httpgithubcommetacademymetacademy|applicationtreegraph|editor
FUTURE WORKThe Khan Academy knowledge map inspired this paper andin the coming weeks we plan to make our system compatiblewith their data As mentioned in the previous section a for-mal user study would help validate or nullify our conjecturethat our knowledge map exploration and creation tools helpusers efficiently explore and create educational material
Beyond the realm of visualization we would like to explorehow users interact with these graphs in an online educationalsetting Some interesting questions to answer include whatadvantages and disadvantages does a knowledge map systemhave over traditional content presentations such as a textbookor wiki page do users that create knowledge maps performbetter in the class (not a concept map as has been studiedcountless times in educational psychology) and to what ex-tent are large knowledge maps useful ie how big is too big
ACKNOWLEDGEMENTSColorado Reed would like to thank Kristin Stephens for herfantastic feedback throughout this project and Dan Allan andZach MacHardy for testing early versions of the knowledgemap creator
REFERENCES1 dagre javascript library
httpsgithubcomcpettittdagre Accessed2013-12-01
2 Duolingo httpwwwduolingocom Accessed2013-12-01
3 Khan academy httpwwwkhanacademyorgAccessed 2013-12-01
4 Metacademy httpwwwmetacademyorg Accessed2013-12-01
5 Palantir httpwwwpalantirtechcom Accessed2013-12-01
6 Touchgraph httpwwwtouchgraphcomnavigatorAccessed 2013-12-01
7 Batagelj V and Mrvar A Pajekanalysis andvisualization of large networks Springer 2004
8 Bostock M Ogievetsky V and Heer J D3Data-driven documents IEEE Trans Visualization ampComp Graphics (Proc InfoVis) (2011)
9 Canas A J Hill G Carff R Suri N Lott JEskridge T Gomez G Arroyo M and Carvajal RCmaptools A knowledge modeling and sharingenvironment In Concept maps Theory methodologytechnology Proceedings of the first internationalconference on concept mapping vol 1 (2004) 125ndash133
10 Chimani M and Gutwenger C The open graphdrawing framework (ogdf) httpwwwogdfnet
11 Eisner J Kornbluh M Woodhull G Buse RHuang S Michael C and Shafer G Visual navigation
through large directed graphs and hypergraphs InProceedings of the IEEE Symposium on InformationVisualization (InfoVisrsquo06) PosterDemo Session(Baltimore Oct 2006) 116ndash117
12 Gansner E R Koutsofios E North S C and VoK-P A technique for drawing directed graphs SoftwareEngineering IEEE Transactions on 19 3 (1993)214ndash230
13 Khan S The one world schoolhouse Educationreimagined Hachette Digital Inc 2012
14 Krzywinski M Birol I Jones S J and Marra M AHive plotsrational approach to visualizing networksBriefings in Bioinformatics 13 5 (2012) 627ndash644
15 Misue K Eades P Lai W and Sugiyama K Layoutadjustment and the mental map Journal of visuallanguages and computing 6 2 (1995) 183ndash210
16 Nielsen J Evaluating the thinking-aloud technique foruse by computer scientists In Advances inhuman-computer interaction (vol 3) Ablex PublishingCorp (1993) 69ndash82
17 Sugiyama K Tagawa S and Toda M Methods forvisual understanding of hierarchical system structuresSystems Man and Cybernetics IEEE Transactions on11 2 (1981) 109ndash125
18 Van Ham F and Perer A Search show contextexpand on demand Supporting large graph explorationwith degree-of-interest Visualization and ComputerGraphics IEEE Transactions on 15 6 (2009) 953ndash960
- Introduction
- Related Work
-
- Online Knowledge Graphs
- Visualization and Exploration of Directed Graphs
- Knowledge Map Creation
-
- Methodology
-
- Knowledge Map Creation and Exploration
- Knowledge Map Creation
-
- Discussion
- Future Work
- Acknowledgements
- REFERENCES
-
paper and downloadupload a graph tofrom a JSON5 repre-sentation
The search bar at the bottom of the toolbox gives users theability to add preexisting concepts to graph When a user addsa preexisting concept to the graph the inlinksoutlinks of thatnode are contracted by default so that eg 20-50 prerequi-site nodes are not immediately placed into the graph Whenexpandingcontracting nodes a small information box at thebottom of the screen displays the number of hidden nodes andedges
Figure 6 This figure displays the graph editor described in the text
Figure 7 This figure shows the annotated toolbox for the graph editor
When hovering over a node a small circle appears on theouter radius of the node If the user clicks the small circle thena content editor overlays the graphs with roughly a 08 alphavalue Figure 8 This content editor lets the user provide asummary for the concept as well as provide links to learningresources and justifications for the prerequisites Presentlythis form is hard-coded html but in the near future we wouldlike to create a simple interface for specifying the types dataassociated with the knowledge map In this way we hope tomake this system usable outside of Metacademy
DISCUSSIONThe Metacademy system was designed so that individuals invariegated disciplines could both contribute content and ex-plore preexisting content Until now this process involved
5httpwwwjsonorg
Figure 8 This figure shows the content-editing form that overlays theknowledge map creator This form allows users to associate content withthe graph
exploring large static graphs and editing text files The con-tributions from this paper will help users interactively explorethe knowledge maps and associated content Furthermoreusers can now contribute content via a visual knowledge mapcreator
From watching users interact with our system during aresearch-poster session we found that several users had hadcomments along the line of ldquovisualizing the edges isnrsquot nec-essary until yoursquove chosen a reference node and you want tosee itrsquos relationshipsrdquo While this certainly is not a formal us-ability study we found it encouraging the users organicallyagreed with one of the driving forces for this work We maypursue a formal usability study in the future
This work and its inevitable progressions and exten-sions can be found at httpwwwmetacademyorgThe exact system described in this paper can befound at httpgithubcommetacademymetacademy|applicationtreegraph|editor
FUTURE WORKThe Khan Academy knowledge map inspired this paper andin the coming weeks we plan to make our system compatiblewith their data As mentioned in the previous section a for-mal user study would help validate or nullify our conjecturethat our knowledge map exploration and creation tools helpusers efficiently explore and create educational material
Beyond the realm of visualization we would like to explorehow users interact with these graphs in an online educationalsetting Some interesting questions to answer include whatadvantages and disadvantages does a knowledge map systemhave over traditional content presentations such as a textbookor wiki page do users that create knowledge maps performbetter in the class (not a concept map as has been studiedcountless times in educational psychology) and to what ex-tent are large knowledge maps useful ie how big is too big
ACKNOWLEDGEMENTSColorado Reed would like to thank Kristin Stephens for herfantastic feedback throughout this project and Dan Allan andZach MacHardy for testing early versions of the knowledgemap creator
REFERENCES1 dagre javascript library
httpsgithubcomcpettittdagre Accessed2013-12-01
2 Duolingo httpwwwduolingocom Accessed2013-12-01
3 Khan academy httpwwwkhanacademyorgAccessed 2013-12-01
4 Metacademy httpwwwmetacademyorg Accessed2013-12-01
5 Palantir httpwwwpalantirtechcom Accessed2013-12-01
6 Touchgraph httpwwwtouchgraphcomnavigatorAccessed 2013-12-01
7 Batagelj V and Mrvar A Pajekanalysis andvisualization of large networks Springer 2004
8 Bostock M Ogievetsky V and Heer J D3Data-driven documents IEEE Trans Visualization ampComp Graphics (Proc InfoVis) (2011)
9 Canas A J Hill G Carff R Suri N Lott JEskridge T Gomez G Arroyo M and Carvajal RCmaptools A knowledge modeling and sharingenvironment In Concept maps Theory methodologytechnology Proceedings of the first internationalconference on concept mapping vol 1 (2004) 125ndash133
10 Chimani M and Gutwenger C The open graphdrawing framework (ogdf) httpwwwogdfnet
11 Eisner J Kornbluh M Woodhull G Buse RHuang S Michael C and Shafer G Visual navigation
through large directed graphs and hypergraphs InProceedings of the IEEE Symposium on InformationVisualization (InfoVisrsquo06) PosterDemo Session(Baltimore Oct 2006) 116ndash117
12 Gansner E R Koutsofios E North S C and VoK-P A technique for drawing directed graphs SoftwareEngineering IEEE Transactions on 19 3 (1993)214ndash230
13 Khan S The one world schoolhouse Educationreimagined Hachette Digital Inc 2012
14 Krzywinski M Birol I Jones S J and Marra M AHive plotsrational approach to visualizing networksBriefings in Bioinformatics 13 5 (2012) 627ndash644
15 Misue K Eades P Lai W and Sugiyama K Layoutadjustment and the mental map Journal of visuallanguages and computing 6 2 (1995) 183ndash210
16 Nielsen J Evaluating the thinking-aloud technique foruse by computer scientists In Advances inhuman-computer interaction (vol 3) Ablex PublishingCorp (1993) 69ndash82
17 Sugiyama K Tagawa S and Toda M Methods forvisual understanding of hierarchical system structuresSystems Man and Cybernetics IEEE Transactions on11 2 (1981) 109ndash125
18 Van Ham F and Perer A Search show contextexpand on demand Supporting large graph explorationwith degree-of-interest Visualization and ComputerGraphics IEEE Transactions on 15 6 (2009) 953ndash960
- Introduction
- Related Work
-
- Online Knowledge Graphs
- Visualization and Exploration of Directed Graphs
- Knowledge Map Creation
-
- Methodology
-
- Knowledge Map Creation and Exploration
- Knowledge Map Creation
-
- Discussion
- Future Work
- Acknowledgements
- REFERENCES
-
REFERENCES1 dagre javascript library
httpsgithubcomcpettittdagre Accessed2013-12-01
2 Duolingo httpwwwduolingocom Accessed2013-12-01
3 Khan academy httpwwwkhanacademyorgAccessed 2013-12-01
4 Metacademy httpwwwmetacademyorg Accessed2013-12-01
5 Palantir httpwwwpalantirtechcom Accessed2013-12-01
6 Touchgraph httpwwwtouchgraphcomnavigatorAccessed 2013-12-01
7 Batagelj V and Mrvar A Pajekanalysis andvisualization of large networks Springer 2004
8 Bostock M Ogievetsky V and Heer J D3Data-driven documents IEEE Trans Visualization ampComp Graphics (Proc InfoVis) (2011)
9 Canas A J Hill G Carff R Suri N Lott JEskridge T Gomez G Arroyo M and Carvajal RCmaptools A knowledge modeling and sharingenvironment In Concept maps Theory methodologytechnology Proceedings of the first internationalconference on concept mapping vol 1 (2004) 125ndash133
10 Chimani M and Gutwenger C The open graphdrawing framework (ogdf) httpwwwogdfnet
11 Eisner J Kornbluh M Woodhull G Buse RHuang S Michael C and Shafer G Visual navigation
through large directed graphs and hypergraphs InProceedings of the IEEE Symposium on InformationVisualization (InfoVisrsquo06) PosterDemo Session(Baltimore Oct 2006) 116ndash117
12 Gansner E R Koutsofios E North S C and VoK-P A technique for drawing directed graphs SoftwareEngineering IEEE Transactions on 19 3 (1993)214ndash230
13 Khan S The one world schoolhouse Educationreimagined Hachette Digital Inc 2012
14 Krzywinski M Birol I Jones S J and Marra M AHive plotsrational approach to visualizing networksBriefings in Bioinformatics 13 5 (2012) 627ndash644
15 Misue K Eades P Lai W and Sugiyama K Layoutadjustment and the mental map Journal of visuallanguages and computing 6 2 (1995) 183ndash210
16 Nielsen J Evaluating the thinking-aloud technique foruse by computer scientists In Advances inhuman-computer interaction (vol 3) Ablex PublishingCorp (1993) 69ndash82
17 Sugiyama K Tagawa S and Toda M Methods forvisual understanding of hierarchical system structuresSystems Man and Cybernetics IEEE Transactions on11 2 (1981) 109ndash125
18 Van Ham F and Perer A Search show contextexpand on demand Supporting large graph explorationwith degree-of-interest Visualization and ComputerGraphics IEEE Transactions on 15 6 (2009) 953ndash960
- Introduction
- Related Work
-
- Online Knowledge Graphs
- Visualization and Exploration of Directed Graphs
- Knowledge Map Creation
-
- Methodology
-
- Knowledge Map Creation and Exploration
- Knowledge Map Creation
-
- Discussion
- Future Work
- Acknowledgements
- REFERENCES
-
top related