mechanix: a sketch-based educational interface
TRANSCRIPT
Mechanix: A Sketch-Based Educational InterfaceTrevor Nelligan
Sketch Recognition LabTexas A&M University
Seth PolsleySketch Recognition LabTexas A&M University
Jaideep RaySketch Recognition LabTexas A&M [email protected]
Dr. Michael HelmsiDreem Lab
Georgia Institute ofTechnology
Dr. Julie LinseyiDreem Lab
Georgia Institute ofTechnology
Dr. Tracy HammondSketch Recognition LabTexas A&M University
ABSTRACTAt the university level, high enrollment numbers in classescan be overwhelming for professors and teaching assistantsto manage. Grading assignments and tests for hundreds ofstudents is time consuming and has led towards a push forsoftware-based learning in large university classes. Unfor-tunately, traditional quantitative question-and-answer mech-anisms are often not sufficient for STEM courses, wherethere is a focus on problem-solving techniques over find-ing the right answers. Working through problems by handcan be important in memory retention, so in order for soft-ware learning systems to be effective in STEM courses, theyshould be able to intelligently understand students sketches.Mechanix is a sketch-based system that allows students tostep through problems designed by their instructors with per-sonalized feedback and optimized interface controls. Opti-mizations like color-coding, menu bar simplification, and toolconsolidation are recent improvements in Mechanix that fur-ther the aim to engage and motivate students in learning.
Author KeywordsGuides; Computer-assisted Instruction (CAI); GraphicalUser Interfaces (GUI); Interaction Styles; User-centereddesign
ACM Classification KeywordsH.5.2 Information Interfaces and Presentation (e.g. HCI):K.3.1Computer Uses in Education
INTRODUCTIONSketching is one of the most important problem-solving tech-niques students can use. No matter their discipline, studentseverywhere draw diagrams, tables, and scratch work to help
Permission to make digital or hard copies of part or all of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the fullcitation on the first page. Copyrights for third-party components of this work mustbe honored. For all other uses, contact the Owner/Author. Copyright is held by theowner/author(s).IUI’15 Companion, March 29–April 1, 2015, Atlanta, GA, USAACM 978-1-4503-3308-5/15/03.http://dx.doi.org/10.1145/2732158.2732194
them illustrate and understand concepts. Students benefitfrom learning to draw engineering drawings by hand [12] [5],yet, as high enrollment in university courses and efforts toreduce costs is pushing many schools toward software-basedteaching tools, the natural interaction of sketching is oftenlost. STEM (Science, Technology, Engineering, and Math-ematics) fields are particularly good candidates for softwareteaching systems, owing to the complexity of content and dif-ficulty in finding qualified graders, but existing systems areoften insufficient for these courses. Because of an emphasison technique and understanding over calculating the correctanswer, STEM courses are difficult to grade through purelyquantitative question-and-answer mechanisms. It is impor-tant that teaching software in STEM courses be more flexibleand intuitive so that the focus is on student learning.
Mechanix is an interactive teaching tool designed to help stu-dents through the complete process of creating and solvingtruss and free body diagram problems [13] [4]. Mechanixuses a combination of sketch recognition and numerical ques-tions to evaluate student work and provide personalized feed-back [1]. In this way, students are free to work through aproblem by hand, but they still must adhere to the guidelinesof the question, as defined by the instructor, in order to re-ceive credit. Although this project has been under develop-ment for some time, Mechanix was recently deployed acrossclasses in three universities in a newly-improved form. Thenew interface includes an enhanced layout with more naturalgroupings of features and controls.
PRIOR WORKSTechnology-based solutions for teaching engineering, evensome systems that include sketching, are not new. SketchWorksheets is a domain-free sketch-based system for walk-ing students through any problem designed by the instructor,but it requires that instructors be comfortable with a speciallanguage in order to define facts about the sketch that maybe used to guide students [14]. Other packages such as Win-Truss [11], Bridge Architect, and McGraw Hill Connects en-gineering section provide software-based truss problem solv-ing. These systems do not include sketching, and so must fillin areas of the solution and ask students to provide missing
1
IUI 2015 • Poster & Demo Session March 29–April 1, 2015, Atlanta, GA, USA
53
pieces. Though similar in parts to Mechanix, none of thesesystems include the combination of sketch recognition withtrusses and free body diagrams in order to provide a completesolution for working students through an entire problem. Theusage of sketch recognition gives Mechanix several advan-tages that benefit both students and instructors.
MECHANIX OVERVIEWMechanix is built around providing instructors and studentswith an intuitive means of creating and solving truss andfree body diagram problems [2]. Because of the focus onsketching and natural interaction, the software relies heav-ily on sketch recognition algorithms to analyze drawings andperform actions intelligently. Mechanix identifies and distin-guishes among shapes without direct user intervention andprovides automatic, yet personalized, feedback and grading.Furthermore, even though Mechanix is primarily directed atcivil and mechanical engineers, it has been designed to besimple enough that anyone may use it with very little knowl-edge of its domain.
Mechanix is designed to emulate the pen and paper interac-tion that students are familiar with. In each interface mode,users are presented with a large canvas on which to draw, sothat they do not feel restricted by not having enough roomto draw in [9]. Users can keep on modifying their diagramsbased on Mechanixs evaluation until they have met all ofthe criteria. The software may also be used on many dif-ferent form factors with varying input devices: finger ontablets, mouse on desktop computers, digital pens/styli onother touch-based devices.
Mechanix is a modular client-server web application, but itis very easy for users to setup. Students and instructorsdownload their respective lightweight client to start usingMechanix, and all of the server-side configuration is set auto-matically. User logins, permissions, and sketch submissionsare all routed to the server, which verifies and stores informa-tion in a MySql database.
INTERACTIONAn important goal of the Mechanix project is to engage andmotivate students in learning. Mechanix includes many ad-vanced sketch recognition algorithms that run transparentlyto the user, wrapped under a carefully-designed interface withexpertly-tuned features like real-time evaluation. The user ex-perience begins with the interface.
InterfaceThe interface has evolved through various iterations to en-sure its effectiveness. We have used data points collated overmultiple terms in different universities to iron out the maindifficulties encountered by users in using the program. In re-working the design of the user interface, Gestalt Principles,a well-known set of design guides [10] [3], played a crucialrole in informing the structure of the interface. The overallmotif of the program has been altered to be more cohesive,and to incorporate the flat design philosophy that has becomeprevalent in recent years [7]. This design philosophy elim-inates superfluous elements and was helpful in distilling theprogram to its most critical elements.
Figure 1. The original Mechanix interface without color-coding or aunified design style.
Mechanix has three interface modes - an instructor interfacewhere instructor can upload the assignment problems and tu-torial examples, a student interface where students can workon the assignment problems, and a review mode available toinstructors that allows them to examine student submissions.
Instructors must first create assignments and problems in theinstructor interface. Rather than learning a complicated lan-guage or writing code, they can just draw. Places for filling inequations, values, and units are all provided. After problemsare submitted online, students from that instructors sectionare able to access the assignment and attempt the problemsthemselves from the student interface. Their submissions arecompared directly against the instructors by the server andsaved for later review by the instructor.
Figure 2. The new student user interface. Of particular note is theautomated color-coding of arrows so that students know immediatelywhether or not their arrows have been recognized. Other importantchanges can be seen as well, including the unified button styles.
RecognitionThough the workflow is straightforward, making Mechanixseem natural to users requires a great deal of underlyingrecognition algorithms. The building blocks of recognitionare points, strokes, and shapes. Mechanix generates pointsfrom movement over the input device. A stroke is a collectionof points from pen down to pen up, and a shape is a collectionof strokes that is recognized by Mechanix.
2
IUI 2015 • Poster & Demo Session March 29–April 1, 2015, Atlanta, GA, USA
54
Recognition starts with the generation of a stroke, which isthen sent to a shape recognizer, PaleoSketch [8], for classifi-cation as a primitive shape like line, arc, circle, or polyline.After primitive recognition, the groupings of shapes are sentto complex shape recognizers for higher-level recognition.These are partly based on geometric constraints. Mechanixhas specialized truss recognizers as trusses are the most im-portant figure in any statics course.
Great recognition is only meaningful if the user interactionbenefits from it. In the case of Mechanix, recognized shapesare used to drive intelligent behaviors that directly help in-structors and students. When creating forces, arrows andnodes are automatically recognized, so Mechanix can suggesta force name using the direction and node labels. The instruc-tor need not use any tool for this. Even after drawing a truss,nodes are immediately classified as such and recommendedlabels are made available. Students interact similarly as in-structors do. Truss nodes are automatically recognized andlabeled; arrows are color-coded. Color-coding acts as a vi-sual guide to further direct students and reinforce good work[6].
User actions are greatly simplified through this recognitionprocess. In other programs that focus less on organic inter-action, one might have to click a button to submit a drawingfor evaluation and only then know whether their drawing wascorrect. The system Mechanix employs is constantly evaluat-ing and updating the user on the results of those evaluations.For example, once a truss is identified, nodes are automati-cally labeled. Arrows may be drawn in many different ways,but once recognized, they are colored green and can be la-beled as a force. With these automatic responses, the programacknowledges the users actions as correct as soon as the userhas lifted their mouse/stylus/finger. This type of affirmativefeedback helps students stay encouraged and feel confidentthey are on the right track.
Figure 3. The interface being sketched on using a tablet, although anysupported input may be used in Mechanix.
FeedbackBecause Mechanix aims to engage and motivate studentsin learning, it offers a key feature of real-time feedback to
Figure 4. Shown here is the student interface for creative response prob-lems. In this mode, the instructor defines constraints for the system, andstudents must design a truss that meets the criteria. The trusses, forces,and equations are all evaluated automatically by Mechanix, which canprovide very beneficial feedback for students who are unsure about thetruss design or forces.
students. Personalized feedback is the most prominent dif-ference between student and instructor interaction. Color-coding and node labeling are important, but when students areattempting to solve equations and draw forces, they may needsome extra instruction to get them on the right track. Usingthe its recognition and comparative capabilities, Mechanixgenerates directed feedback to inform students what theyneed to do. If a force is missing, the student is told whatnode needs an additional force. If a numerical result is wrong,the relevant value will be highlighted, and students are eveninformed if their units are incorrect. The sketches use thresh-olds in determining student accuracy that provides them someforgiveness without being too inaccurate.
Each problem in Mechanix also has a dynamic checklist. Thedynamic checklist along with the user-requested feedback as-sists students to tackle a problem in a more structured order.The checklist updates as feedback is requested, and it helpsreinforce good problem-solving strategies.
Figure 5. One of the ways the interface provides feedback – the checklist.
3
IUI 2015 • Poster & Demo Session March 29–April 1, 2015, Atlanta, GA, USA
55
CONCLUSION AND DISCUSSIONMechanix is educational software directed at aiding studentsand instructors in learning and teaching free body diagramsin statics. It is uses advanced sketch recognition algorithmsto provide real-time grading and feedback. These capabil-ities are paired with carefully-crafted interface elements likeautomated node labeling and color-coding to provide studentswith positive feedback to help direct the problem-solving pro-cess. One of the most valuable tools Mechanix provides stu-dents is real-time feedback, specific to the work they havedone. This is helpful in all kinds of problems, but it canbe especially useful in problems where multiple answers areallowed. All of these interactions are important in STEMcourses as they reinforce problem-solving methodologies andencourage students to remain engaged in their engineeringcourses, characteristics that Mechanix seeks to exemplify.
ACKNOWLEDGMENTSThe authors would like to thank the NSF for funding this re-search under NSF EEC Grant No. 1129525. The authors alsowish to thank the members of the Sketch Recognition Labwho have provided significant contributions to Mechanix andsupport for this work, specifically Stephanie Valentine andDavid Turner. Additional thanks go to Dr. Matthew Green fortesting Mechanix with his students at LeTourneau University,the iDreem and Sketch Recognition Labs, and the ComputerScience and Engineering department at Texas A&M Univer-sity.
REFERENCES1. Atilola, O., Field, M., McTigue, E., Hammond, T., and
Linsey, J. Evaluation of a natural sketch interface fortruss fbds and analysis. In Frontiers in EducationConference (FIE), 2011 (Oct 2011), S2E–1–S2E–6.
2. Atilola, O., Valentine, S., Kim, H.-H., Turner, D.,McTigue, E., Hammond, T., and Linsey, J. Mechanix: Anatural sketch interface tool for teaching truss analysisand free-body diagrams. Artificial Intelligence forEngineering Design, Analysis and Manufacturing 28, 02(2014), 169–192.
3. Chang, D., Dooley, L., and Tuovinen, J. E. Gestalttheory in visual screen design: A new look at an oldsubject. In Proceedings of the Seventh World Conferenceon Computers in Education Conference on Computers inEducation: Australian Topics - Volume 8, CRPIT ’02,
Australian Computer Society, Inc. (Darlinghurst,Australia, Australia, 2002), 5–12.
4. Field, M., Valentine, S., Linsey, J., and Hammond, T.Sketch recognition algorithms for comparing complexand unpredictable shapes. In Proceedings of theTwenty-Second International Joint Conference onArtificial Intelligence - Volume Volume Three, IJCAI’11,AAAI Press (2011), 2436–2441.
5. Lane, D., Seery, N., and Gordon, S. The understatedvalue of freehand sketching in technology education.Engineering Design Graphics Journal 73, 3 (2009).
6. Michalski, R. The influence of color grouping on usersvisual search behavior and preferences. Displays (2014).
7. Page, T. Skeuomorphism or flat design: future directionsin mobile device user interface (ui) design education.International Journal of Mobile Learning andOrganisation 8, 2 (2014), 130–142.
8. Paulson, B., and Hammond, T. Paleosketch: Accurateprimitive sketch recognition and beautification. InProceedings of the 13th International Conference onIntelligent User Interfaces, IUI ’08, ACM (New York,NY, USA, 2008), 1–10.
9. Plimmer, B., Tang, G., and Young, M. Sketch toolusability: Allowing the user to disengage. People andComputers XX (Proceedings of HCI 2006) (2006).
10. Puustinen, M., Baker, M., and Lund, K. Gestalt: Aframework for redesign of educational software. Journalof computer assisted learning 22, 1 (2006), 34–46.
11. Sutton, M. G., and Jong, I.-C. A truss analyzer forenriching the learning experience of students. In ASEEAnnual Conference (2000).
12. Ullman, D. G., Wood, S., and Craig, D. The importanceof drawing in the mechanical design process. Computers& graphics 14, 2 (1990), 263–274.
13. Valentine, S., Vides, F., Lucchese, G., Turner, D., Kim,H.-h., Li, W., Linsey, J., and Hammond, T. Mechanix: Asketch-based tutoring system for statics courses. In IAAI(2012).
14. Yin, P., Forbus, K. D., Usher, J. M., Sageman, B., andJee, B. D. Sketch worksheets: A sketch-basededucational software system. In IAAI (2010).
4
IUI 2015 • Poster & Demo Session March 29–April 1, 2015, Atlanta, GA, USA
56