Interact 2009 katsanos et al are 10 participants enough to evaluate scent
Download as PPT, PDF0 likes313 views
Ten participants provided information scent ratings that highly correlated with both the full dataset of 101 participants' ratings and actual users' navigation behavior, as measured by clicks and fixations on hyperlinks. Specifically, ratings from 10 participants were within 10% of the full dataset correlations for clicks (0.7% difference) and fixations (7.4% difference). While 10 ratings served as a cost-effective way to evaluate information scent, they should only be used as a rough indicator of how users distribute their attention, since the correlation between 10 ratings and fixations was not statistically significant.
![Analysis Methodology
Reference case = Scent-ratings from 101 participants
Select 10 random samples of different size N
N = 2, 5, 10, 15, 20, 25, 30, 40 and 50
[Samples-Ratings] VS [All 101 participants Ratings]
Average Spearman Correlation
How many raters are enough to represent the ratings
of the whole dataset? 5](https://image.slidesharecdn.com/interact2009katsanosetalare10partiicipantsenoughtoevaluatescent-140722092728-phpapp02/85/Interact-2009-katsanos-et-al-are-10-participants-enough-to-evaluate-scent-5-320.jpg)
![Analysis Methodology
Reference case = Behavioral data from 54 users
[Scent-ratings from samples of - 1st
phase] VS
[Measures of user’s navigation behavior - 2nd
phase]
Average Spearman Correlation
How many raters are enough to reach an acceptable
level of correlation with these two measures?
10](https://image.slidesharecdn.com/interact2009katsanosetalare10partiicipantsenoughtoevaluatescent-140722092728-phpapp02/85/Interact-2009-katsanos-et-al-are-10-participants-enough-to-evaluate-scent-10-320.jpg)
Interact 2009 katsanos et al are 10 participants enough to evaluate scent
- 1. Christos Katsanos | [email protected] Nikolaos Tselios | [email protected] Nikolaos Avouris | [email protected] Are Ten Participants Enough for Evaluating Information Scent of Web Page Hyperlinks? IFIP INTERACT | Uppsala, Sweden | 24-28 August, 2009
- 2. Purpose & Motivation 2 A critical factor in web navigation is information scent (Fu & Pirolli, 2007; Blackmon et al, 2005; Miller & Remington, 2004) user’s assessment of semantic relevance of navigation options in a webpage Often, participants are called to evaluate scent by providing ratings (Miller & Remington, 2004; Brumby & Howes, 2008) Remains unclear how many raters are required to obtain representative estimates of information scent.
- 3. The Study: First Phase 3
- 4. Design & Procedures Web-based survey Rate semantic relevancy of all links to the provided goal (1=poor relevance, 5=high relevance). 101 participants 8 navigation menus, 8 links each 4 6464 ratings
- 5. Analysis Methodology Reference case = Scent-ratings from 101 participants Select 10 random samples of different size N N = 2, 5, 10, 15, 20, 25, 30, 40 and 50 [Samples-Ratings] VS [All 101 participants Ratings] Average Spearman Correlation How many raters are enough to represent the ratings of the whole dataset? 5
- 6. Results 6 10 raters 84-90% total var. Error Bars = (rMEAN ± rSD)2 x2 raters still the same x3 raters +5% closer to whole dataset
- 7. First-phase: Conclusion 10 raters appear to be a cost-effective solution to evaluate information scent without expense in the quality of results 7 But how close are scent-ratings of 10 participants to observed navigation behavior?
- 8. The Study: Second Phase 8
- 9. Design & Procedures Eye-tracking user study Perform the same 8 navigation tasks used in first-phase 54 users (not involved in first-phase) Two measures of users’ behavior: clicks on each link fixations-adjusted-for-text-length on each link. 9 432 recordings
- 10. Analysis Methodology Reference case = Behavioral data from 54 users [Scent-ratings from samples of - 1st phase] VS [Measures of user’s navigation behavior - 2nd phase] Average Spearman Correlation How many raters are enough to reach an acceptable level of correlation with these two measures? 10
- 11. Results 11 Clicks on each link r10-raters is 0.7% different from r101-raters r101raters = 0.80, p<.01 Fixations on each link r10-raters is 7.4% different from r101-raters r101-raters = 0.40, ns Error Bars = rMEAN ± rSD
- 12. Second-phase: Conclusion 10 participants provide scent-ratings that are close to observed link-selection behavior (clicks) distribution of attention (fixations) 12 However, scent-ratings should be used only as a rough indicator of users’ distribution of attention rs = 0.40, ns
- 13. Summary & Questions Investigated the well-known debate of “how many users” in the context of information scent evaluation Scent-ratings of 10 participants appeared to be enough for a discount evaluation of information scent 13 More studies required in the context of highly specialized domains and/or varied user group composition Christos Katsanos | [email protected]
- 14. EXTRA SLIDES 14
- 16. Second-Phase: How many users are enough? 16 Clinks Count Observations Count
Editor's Notes
- #2: Hi my name is Christos Katsanos, member of the HCI Group of University of Patras in Greece. I am very glad to be here to present you our work with the title “AutoCardSorter: Designing the Information Architecture of a Web Site Using Latent Semantic Analysis”
- #3: The purpose of our work is to automate structural design of information spaces and therefore increase the efficiency & flexibility for practitioners
- #14: Summarizing we have proposed an approach that automates structural design of an information space. The validation study presented here depicted substantial effectiveness gain, with similar to a widely used user-based technique We argue that our approach is cheap, fast and easy and therefore has a greater possibility for wider adoption As a last note, ideally, the proposed approach should be complementary and by no means a substitute to user based techniques.