ui toolkits - github pagesuwdata.github.io/hcid520/15wi/lectures/02-uitoolkits.pdf · enable better...
TRANSCRIPT
UI Toolkits !
HCID 520 User Interface Software & Technology
http://www.cryptonomicon.com/beginning.html
Xerox Alto 1973
Command Line (UNIX shell, DOS prompt) Interaction driven by system User prompted for input when needed Text-based input and output !Event-Driven Interfaces (GUIs) Interaction driven by user System waits for user action and responds Pointing & text input, graphical output More complex architecture & development
Evolution of User Interfaces
Component Model
Encapsulate interactive components (widgets) Component library (button, slider, container) Interface built as a hierarchy of components !Widgets drawn by underlying graphics library !Input event generation and dispatch Historically mouse & keyboard, now touch, … !Bounds management & damage/redraw Model geometry, redraw updated regions only
Component Model
Java Swing Widgets
User Interface ComponentsLabel TextArea
ButtonsEach component is a clipped 2D canvas with its own coordinate system.
User Interface Components
Each component (widget) is an object with: !A bounding box !A paint method for drawing itself Drawn in the component’s coordinate system !Callbacks to process input events
public void paint(Graphics g) { g.fillRect(…); // interior g.drawString(…); // label g.drawRect(…); // outline }
Containment HierarchyWindow
Panel
TextArea Panel
Button Button
Label
Component LayoutWindow
Panel
TextArea Panel
Button Button
Label
Border Layout (direct placement)
NORTH
CENTER
SOUTH
strutssprings
“Struts and Springs” (simple
constraint-based layout)
Each container is responsible for positioning its contents.
Event-Driven Architecture
User input is modeled as events that must be processed by the system. !Hardware Events Mouse: move, button-down, button-up Keyboard: key-down, key-up !Inferred Events Mouse: click, double-click, enter, exit, drag Keyboard: key-press Window: move, resize
Events
Events encapsulate the information needed for components to react to input or changes. !Event Type (mouse moved, key down, etc) Event Target (the input component) Timestamp (when did event occur) Modifiers (Ctrl, Shift, Alt, etc) Event Content: Mouse: x,y coordinates, button pressed, # clicks Keyboard: which key was pressed
Anatomy of an Event
Event Dispatch Loop
Event Queue List of input events
Event Loop (runs in dedicated thread)
Remove next event from queue Determine event type Find proper component(s) Invoke callbacks on components Repeat, or wait until event arrives
Component Invoked callback method Update application state Request repaint, if needed
Mouse moved (t0,x,y)
Window
Panel
TextArea Panel
Button Button
Label
Event Queue Mouse moved (t0,x,y) Mouse pressed (t1,x,y,1) Mouse dragged (t2,x,y,1) Key typed (t3, ‘F1’) … (queue and dispatch incoming events in a dedicated thread)
/* callback for TextArea */ public void mouseMoved(e) { // process mouse moved event }
Event Dispatch
Go to http://mhcid.washington.edu !Open the JavaScript console, enter: ! var h = document.querySelector("#header"); h.addEventListener("click", function(event) { console.log(event); });!Click the header image at the top of the page. !Inspect the events in the JavaScript console.
Demo: Browser DOM Events
Event Capture Process event either as it moves down or up the component hierarchy? Default is up. Why? !
Event Minutiae
Window
Panel
TextArea Panel
Button Button
Label
Event Queue Mouse moved (t0,x,y) Mouse pressed (t1,x,y,1) Mouse dragged (t2,x,y,1) Key typed (t3, ‘F1’) … (queue and dispatch incoming events in a dedicated thread)
/* callback for TextArea */ public void mouseMoved(e) { // process mouse moved event }
Event Capture (Down)
Window
Panel
TextArea Panel
Button Button
Label
Event Queue Mouse moved (t0,x,y) Mouse pressed (t1,x,y,1) Mouse dragged (t2,x,y,1) Key typed (t3, ‘F1’) … (queue and dispatch incoming events in a dedicated thread)
Event Bubbling (Up)
Event Capture Process event either as it moves down or up the component hierarchy? Default is up. Why? !Throttling & Debouncing You don’t always want to process every event, both for performance and experience reasons. !Examples: Flurry of mouse drags > process last event Rapid enter/exit > pause to process exit
Event Minutiae
Model View Controller
Architecture for interactive applications Introduced by Smalltalk developers at PARC Partitions app to be scalable & maintainable
Model View Controller (MVC)
View
Controller
Model
Information the app is trying to manipulate Circuits: logic gates and connecting wires Drawing: geometry and color of shapes
Model
View
Controller
Model
Implements a visual display of the model Apps may have multiple views for one model Example: shape & numerical views of drawing
View
View
Controller
Model
Implements a visual display of the model !When model changes, each view must be notified so that it can update itself.
View
View
Controller
Model
Processes all input events from the user Query view to interpret events (e.g., selection) Call methods of model to make changes Model updates & notifies views
Controller
View
Controller
Model
Example Drawing Application
Blue circles: 3 Red squares: 2
The Controller handles input…
Blue circles: 3 Red squares: 2
… updates the Model …
Blue circles: 3 Red squares: 2
Click!
… and Views update in response.
Blue circles: 4 Red squares: 2
Controller must contact view to interpret what some user events mean. Example: Mouse(x, y) -> selected object !“Pattern of behavior in response to user events (controller issues) is independent of visual geometry (view issues).”
Relationship of View & Controller
View and controller are tightly intertwined Lots of communication between the two Occur in pairs: each view needs a controller Some architectures combine into a single class
Combining View & Controller
View
ControllerModel
Why MVC?
Putting MVC into one class does not scale Model may have more than one view Each view different, update on model change !Separation aids maintenance & extensibility Easy to change view or add a new view later Models can be extended; old views still work Flexibility of input handling when using separate controllers (e.g., mouse vs. touch)
Why MVC?
Adding Views Later
Blue circles: 3 Red squares: 2
Erase and Redraw Using background color to erase fails Draw shape in new position -> loses ordering !Move in model and then redraw everything Change position of shapes in model Model keeps shapes in a desired order Tell all views to redraw themselves in order Slow for large / complex drawings
Changing the Display
Move a red square…
Blue circles: 3 Red squares: 2
Erase at old position, draw at new
Blue circles: 3 Red squares: 2
Register “damage”, redraw region
Blue circles: 3 Red squares: 2
View informs window system of areas that need to be updated (i.e., damaged) !Does not redraw these areas yet… !Window system batches updates & clips them to visible portions of window !On next event cycle, window system invokes repaint, passes region that need updates !Critical for many 2D graphics systems, typically not used in 3D interfaces. Why?
Damage / Redraw
Cascading Style Sheets
Traditional GUI tools often make assumptions: Rectangular (non-overlapping?) components Difficult to style graphic elements !Many modern frameworks are more flexible, using a scenegraph to model the display. Ex: the browser document object model (DOM). !Decouple content from appearance using separate stylesheets to specify design.
Technical Limitations
A language for styling an HTML document W3C standard, first introduced in 1996 !Goal: separate content from presentation Specify layout, spacing, colors, fonts, … Share formatting across pages on a site Enable various formats for a single page !Why “cascading”? Multiple style definitions may apply to an element, ordered (“cascaded”) by priority.
Cascading Style Sheets
!p { margin-bottom: 1em; font-family: Helvetica, Arial; font-size: 16px; line-height: 24px; color: #333333;}
CSS Example
!p { margin-bottom: 1em; font-family: Helvetica, Arial; font-size: 16px; line-height: 24px; color: #333333;}
CSS ExampleSelector style p (paragraph) elements
!p { margin-bottom: 1em; font-family: Helvetica, Arial; font-size: 16px; line-height: 24px; color: #333333;}
CSS ExampleSelector
Property
!p { margin-bottom: 1em; font-family: Helvetica, Arial; font-size: 16px; line-height: 24px; color: #333333;}
CSS ExampleSelector
Property Value … or #333, or rgb(51,51,51)
Tag (select all elements of a certain type) a, p, h1, ul, li, etc !
CSS Selectors
Tag (select all elements of a certain type) a, p, h1, ul, li, etc !Id (select by unique identifier, prefix #) <div id="logo"> —> #logo !
CSS Selectors
Tag (select all elements of a certain type) a, p, h1, ul, li, etc !Id (select by unique identifier, prefix #) <div id="logo"> —> #logo !Class (select by shared identifier, prefix .) <div class="c1 c2"> —> .c1 !
CSS Selectors
Tag (select all elements of a certain type) a, p, h1, ul, li, etc !Id (select by unique identifier, prefix #) <div id="logo"> —> #logo !Class (select by shared identifier, prefix .) <div class="c1 c2"> —> .c1 !These selectors can be combined: div.headlines
CSS Selectors
For multiple selections, use a comma div.headlines, div.news { … }Selects both div.headlines and div.news elements !
CSS Selector Combinations
For multiple selections, use a comma div.headlines, div.news { … }Selects both div.headlines and div.news elements !For nested selections, use a space div.headlines ul li { … }Selects all li within a ul within a div.headlines !
CSS Selector Combinations
For multiple selections, use a comma div.headlines, div.news { … }Selects both div.headlines and div.news elements !For nested selections, use a space div.headlines ul li { … }Selects all li within a ul within a div.headlines !For direct descendants, use ‘>’ div.headlines > ul li { … }Now the ul must be the direct child of div.headlines
CSS Selector Combinations
CSS Box Model
block<div>!!inline<span>!inline-block!!none
CSS display property
<body> ! <div id=“left”> Sidebar column content (assume max 400px tall) </div> ! <div id=“right”> Main column content (variable height) </div> ! <div id=“footer”> Footer column content (assume 50px tall) </div>
</body>
Specify a 2-Column Layout
<body> —> select with body <div id=“left”> —> select with #left Sidebar column content (assume max 400px tall) </div> ! <div id=“right”> —> select with #right Main column content (variable height) </div> ! <div id=“footer”> —> select with #footer Footer column content (assume 50px tall) </div>
</body>
Specify a 2-Column Layout
Layout with no styling
Layout with absolute positioning, 250 pixel sidebar
What are some shortcomings of this layout? Why did we include body { position: relative }?
Layout with absolute positioning, without position:relative
position: relative sets the coordinate space for absolutely positioned elements. In this example the position [0,0] is relative to the browser window, not the body tag.
Layout with float: left and margin-left
What are some shortcomings of this layout? Why did we include #footer { clear: both }?
Layout with float: left and margin-left, without clear: both
clear: both tells the browser layout engine to clear any floats (both left and right) before placing the element. !Next, can we make the sidebar resize automatically in response to the window width?
Float layout using percentages
What if we want to ensure the width of the left sidebar never falls below a minimum size?
Here’s one attempt at setting a minimum width. Why is this unsatisfactory?
Float layout using percentages and minimum width
Float layout using percentages
Let’s go back to our previous design… How can we limit the maximum width of the page?
Float layout using percentages and body max-width
Use the max-width property on the body element.
Adding automatic margins
margin-left: auto; margin-right: auto centers the body.
margin (-left, etc) font padding (-left, etc) font-family border (-left, etc) font-size border-radius font-weight overflow (-x/-y) font-variant background-color line-height background-image text-transform position text-align float vertical-align (max-/min-) width, height display
Useful CSS Properties
CSS preprocessors (LESS / SASS) Add support for variables (e.g., $baseColor) Perform computation, unit conversion Enable better reuse, less repetition Now commonly used in web applications !HTML/CSS frameworks (e.g., Bootstrap) Provide better defaults “out of the box” More dynamic components (popup menus) Responsive design; mobile friendly
CSS Power Tools
UI Tools Discussion
Brief UI tool history, establish vocabulary in wide use today, and chart future trajectories. !At any point did they “miss the mark”? !
Myers, Hudson & Pausch
Brief UI tool history, establish vocabulary in wide use today, and chart future trajectories. !At any point did they “miss the mark”? !Web: subsuming many desktop applications !Design vs. Dev: In practice often done by teams with varied skills (interaction design, graphic design, implementation) !3D: Mostly limited to games? BUT fabrication now likely to drive new 3D design tools…
Myers, Hudson & Pausch
Not just for desktop GUI widgets & events Mobile applications (iOS, Android) Information visualization (d3.js, Protovis, Prefuse) Tangible user interfaces (Papier-Maché) Gestural user interfaces (Kinect API) Speech user interfaces (MS Speech API) Proxemic user interfaces (Proximity Toolkit) !Many exhibit similar tensions around expressiveness, efficiency and ease-of-use
Toolkits Abound…
Simplify & accelerate UI development Lower the threshold of specification difficulty Raise the ceiling of what can be expressed But hard to do both! !Establish path of least resistance Makes common / successful designs easy Makes unusual / ineffective designs harder Can limit & shape what we conceive / design !Enforce consistency of design Maintain familiar widgets and interactions
Toolkit Benefits & Shortcomings
Timeliness & Future Predictions Threshold/Ceiling, Expressivity vs. Usability Varied Display Sizes (from Walls to Watches) Personalization User Attention & Cognitive Load End User Programming Constraints / Model-Driven / Automatic Design Security
Discussion Question Topics
The authors of the main reading mention the possibility of tools that not only enable design systems that have high usability, but that actual promote or help ensure that they are. How might this be taken further in the future to help prevent designs that cause users frustration or discomfort (or error)? - Avery !The author suggests that there should be a balance between creating UIs that are accessible while also allowing for innovation to accommodate for ubiquitous computing. Shouldn't UI design and technology revolve around how a task is performed as opposed to creating technologies to fit a certain form factor or input/output interaction? - Garrick