A/B TESTING IN WEB DESIGN FOR VALUE DISCOVERY

LANCE LEGEL
Entrepreneurship, Innovation, & Public Policy University of Colorado Boulder Spring 2013

CONTENTS
Overview Ch. 1 Design Principles 1.1 Optimize Information Extraction 1.1.1 Every millisecond counts 1.1.2 Pretty does not equal effective 1.1.3 Too much information kills 1.1.4 Importance dictates access 1.1.5 Minimize energy required from user 1.1.6 Use the best medium 1.2 Resonate with User Needs 1.2.1 Make people feel like they belong 1.2.2 Build trust by association 1.2.3 Cultivate social ecosystems 1.2.4 Talk like the customer 2. Strategy and Mechanics 2.1 Experimental Precision 2.1.1 Get qualitative and quantitative data before testing 2.1.2 Be significant (statistically) 2.2 Pricing 2.2.1 Customize with care 2.2.2 Form matters 3. References 10 11 9 5 6 7 8 4 3 2 1

OVERVIEW
In this research analysis, inspired by the lean startup movement, I organize a series of actionable, accessible, and auditable principles of design and strategy for testing websites. These principles are presented in a manner that is consistent with what is being preached: information points are modularized, carefully titled, and minimized. Any reader interested in learning more about a particular principle may immediately access the hyperlinked references. Thus, no single point is belabored, as this is a document meant for digital consumption. The science and practice of A/B testing is based on the notion that the greatest challenge for organizations - especially startups - is learning what exactly customers and stakeholders want most. It follows the belief of Henry Ford and Steve Jobs that customers dont really know what they want until they have it; but unlike both of those famous product designers, it follows the belief that a properly targeted minimum viable product that quickly and dirtily tests bold innovations can mean a world of difference from incremental optimization. Finally, the reader should be prepared for a rapid-fire series of insights, results from real-world tests, and recommendations, all of which are designed to answer the increasingly important question: How do I design A/B tests to best learn what my stakeholders actually want?

CHAPTER 1: DESIGN PRINCIPLES

1.1 Optimize Information Extraction
1.1.1 Every millisecond counts One central design principle from Google and Facebook is that every millisecond counts [2]. The general focus of saving as much of the customers time as possible is echoed in remarks by Steve Jobs to a hardware designer: Well, let's say you can shave 10 seconds off of the boot

time. Multiply that by five million users and thats 50 million seconds, every single day. Over a year, that's probably dozens of lifetimes. So if you make it boot ten seconds faster, you've saved a dozen lives. That's really worth it, don't you think?" [7]. 1.1.2 Pretty does not equal effective Many tests have found that what is seemingly uglier may yet be far more effective [1]. If users do not immediately understand how a product can solve their problem, and what is being asked of them to use the product, it doesnt matter how beautiful a website looks. This highlights why many analysts in the conversion of website visitors to paying customers have adopted a conversion trinity for guidance, from the perspective of the user [13]: Relevance: Are you relevant to my wants/needs/desires? Value: Why are you the best solution for me? Call to action: What would I do next?

1.1.3 Too much information kills Minimize distractions. Even if more information contributes to some perceived value-creating aspect - including every principle defined in this document - its inclusion is always risky [14]. More information at least suffers from a law of decreasing marginal returns. Realistically, people do not just see increasingly less value from more and more information they reflexively infer negative value in identifying a site or document that looks endlessly pointless, and will flee. Nobody likes paralysis by analysis, so it is incumbent on designers and writers alike to make this a central concern [3]. 1.1.4 Importance dictates access If one button sign up is most important to the conversion of web visitors to paying customers, then everything else should be designed to accommodate it. Having a button scroll

with the user like a navigation bar, so that it is accessible at the exact moment the visitor may be convinced, has been shown to increase conversions by 138% [4]. Alternatively, if discovered that the biggest issue for a digital startup is communicating exactly what its product does, perhaps an interactive minimalist demo is best, so that users can learn by playing. 1.1.5 Minimize energy required from user Adding free to a headline has been shown to increase cost per acquisition by almost 50% [ 4]. This is because users infer that they have little to no energy (i.e. money, time, focus) to provide in order to receive a reward. There have also been several examples where shorter forms that users are required to fill out generate higher conversions by double digits. This is because the rate of abandonment by users questioning their motivation decreases [1]. Intuitively, it is

important for users to be reassured that they will receive more value by proceeding than the value in time and energy they spend by investing. This argument is not to be made in words, but in information structures: when users can immediately recognize simple minimalist requirements for their contributions, they dont question - they act. If they see that a page only has one paragraph written in an graceful font, centered in a colorful box on a page with a picture or graphic that they immediately can identify as appealing, then they are more likely to actually read the text. 1.1.6 Use the best medium There are several media that designers can choose from to communicate and interact with users: text, pictures, art, graphs, videos, podcasts, surveys, demos, etc. Understanding precisely what combination of media is best depends on the context and resources available. For example, potential customers visiting online companies that sell physical products want to inspect the physical product closely to know exactly what they are getting. Thats why for companies of

this type large images are important MALL.CZ, an eCommerce site, increased sales by 9% through this simple A/B tested technique. One medium that is popular and well-accepted but not too challenging to employ is videos. Script in a video matters more than visual quality [5]; the script must answer all customer questions, and ideally be based on surveys of users about sources of confusion, needs to be solved, and product concerns; videos are not recommended to be longer than 2-3 minutes [5]. They may start by stating product purpose in one sentence, then explain the customers problem precisely in the customers words, show off product features as solutions to the customers problem, and finally instruct purchase and use of product, while addressing concerns (e.g. cost, ease of use). An effective video has been shown to increase conversions by 63% [5]. Of course videos are only one example to highlight among several others that should be A/B tested.

1.2 Resonate with User Needs

1.2.1 Make people feel like they belong When anybody goes someplace new, they appreciate feeling welcomed personally. The

converse of this instinct is that when a website feels cold, corporate, and generic, people are immediately disgusted and would prefer to leave. This is why people enjoy Googles doodles so much they celebrate interesting people and things in an artistic, clever, and fun way. Accordingly, these doodles are a reflection of an important and central design mantra for software engineers at Google: Focus on people - their lives, their work, their dreams. [15]. The motivation of making people feel happier upon visiting a website goes beyond just being a cool and admirable startup: it creates trust. Trust by users in a product is essential for the development of brand loyalty, and the success of a startup [16].

One specific focus that has been found to be useful for online companies is the integration of external advertisements with customized landing pages. This has been shown to increase

conversion rate by 115% [1]. Of course, every principle like this can be outperformed because of a variety of other factors that may be more important to the target audience in question. But the idea here is to provide integrated, seamless, personalized experiences for individuals on the Web, so that they feel more comfortable, confident, and trusting. 1.2.2 Build trust by association Indeed, the viability of any startup depends on building trust in the effectiveness of a product. The best way to earn trust is therefore through consistent creation of value for customers [17]. But how can a startup company that has not been consistently creating value earn the trust of new potential customers? There are several techniques that should be considered and tested. One is to use testimonials, where a quote by the very earliest customers is displayed to reassure new customers that the startup is a safe bet. WikiJob increased sales by 34% by A/B testing customer testimonials [18]. If there is no history of successfully serving customers, it is

important to associate as much as possible with brands that people do trust [1]. For example, a young student leading a startup would want to clarify the institution he is studying at, which must have checked his background and vetted him. A similar application of trust by association comes through logos of famous partners and journalism companies. Finally, beyond these more obvious means of gaining trust, the smartest startups can become industry thought leaders by publishing extensively on their methodology and research. Even though this would seem to have the risk of equipping competitors, it has the proven upshot of proving competence, which is vital to earning trust.

1.2.3 Cultivate social ecosystems Social interaction is central to human emotions and desires, and its absence can lead to sadness. While it is not the role of every startup to satisfy every human need, it can and should be tested whether social elements can significantly improve user experiences. Ecosystems may be the foundations for online businesses: e.g. Amazon provides an ecosystem of merchants and sellers; Facebook of friends, groups, and celebrities; MeetUp of local event organizers and attendees. Amazon increased sales by $2.7 billion because of the simple question Was this review helpful to you?, which enabled sorting of many reviews by helpfulness [8]. This follows from the fact that 63% of customers are more likely to purchase from sites with open reviews that they can consult and contribute [9]. Every case of adding information about what other people thought is based on a company building infrastructure to help website veterans supply the information that new website visitors most want to see. Application of external social networks can also be a means of viral growth, which is why social media strategy should be carefully organized and tested. Suppose the greatest source of growth is from users who share a startups website on Facebook. One company found that by making a social networking sharing bar constantly visible, they increased sharing by 3600% [6]. When encouraging users to share, it is important to do so in an intelligent way that considers why users want to share in the first place. A New York Times Insights study found that 94% of people who share carefully consider how the information may be valuable to others, while 49% share to inform others of products they care about, and potentially change opinions [19]; it found 5 main reasons for sharing: (i) bring valuable and entertaining content to others; (ii) define self to others; (iii) nourish relationships; (iv) self-fulfillment; and (v) promote causes or brands. It also defined 6 main types of sharers to appeal to: (i) altruists who want to be seen as helpful and

reliable, (ii) careerists who want to be seen as intelligent and valuable, (iii) hipsters who want to be seen as creative and unique, (iv) boomerangs who want to incite reaction and be validated, (v) connectors who want to make plans and build things happen, and (vi) selectives who want to be seen as especially resourceful and thoughtful [19]. Of course these are not mutually exclusive categories, but they may be helpful in guiding whether and how an encouragement to share should be made. For example, Amazon does not mention sharing until a user has already purchased a product, when it prepares a post to the users Facebook with I just bought... [20]. This makes sense because at the point the user has already decided they like the product. The reasoning is that if they have mustered the courage to buy the product, then they may also have the courage to share it with others. Targeted simple features like this carefully consider the psychology of users to be contextually relevant and valuable. They are the foundation for an overall pleasing experience with a company. 1.2.4 Talk like the customers It is important to discover and use key words that mean most to customer. This is well understood in the context of search engine optimization. But it is even more subtly important to convincing customers that you understand them. One company found a 161% increase in clickthrough-rate by changing a button from Request a Quote to Request Pricing, after surveys indicated that customers wanted to know more information about pricing [8]. The company did not even change any functionality, it just used words of the customer.

CHAPTER 2: STRATEGY AND MECHANICS

2.1 Experimental Precision
2.1.1 Get qualitative and quantitative data before testing Experienced A/B testers and optimizers of web design for customer conversion know that it is essential to primarily perform experiments not based on random aesthetic whims (Lets change the blue button to a red one!), but based on qualitative and quantitative data that is determined to be important to customers and industry [10]. In other words, before any tests are done, those who are leading the tests should carefully construct hypotheses of what may prove most useful for converting customers, based on an educated model of the customer. This data may be extracted through interviews, surveys, market research, video recording users on the current site, heatmaps of user activity on site areas, and many other useful techniques. For example, video recording 125 hours of user interaction led to 9.5% increase in sales just by fixing clear patterns of confusion [8]. The key, however, is not to even begin an A/B test unless there is concrete evidence justifying it. Of course, this is not to advocate for bureaucracy restricting the freedom of a startup to innovate, but rather for a pre-test research period that testers are strongly encouraged to undertake. This is because tests cannot just be run for a few hours (unless traffic is absolutely massive); each test may require weeks to ensure results are statistically significant. 2.1.2 Be significant (statistically) It is absolutely essential that tests are not stopped by gut, but rather, just like any legitimate scientist or public policy polling expert knows, designed to achieve statistical validity. The equation that determines this measures the variance (change in data) from a population, such that if that determines the minimum viable sample size for an acceptable variance in A/A tests: it measures how consistent the data is. Metrics from A/B tests are useless if they arent statistically

valid: 20% conversion improvement on a sub-population could really mean 10% decline over a longer period of time. For example, consider the following experimental conclusion from [26]: Last week I tossed a coin a hundred times. 49 heads. Then I changed into a red t-shirt and tossed the same coin another hundred times. 51 heads. From this, I conclude that wearing a red shirt gives a 4.1% increase in conversion in throwing heads. Of course, this conclusion is laughable, but it reflects how you cannot attribute statistical dependence cleanly to a single factor, and it is impossible to do so if the factor is independent of the forces the generate the data. There are many approaches for determining optimal sample size for A/B testing, including Bayesian statistics [28]. For example, for a website receiving 10,000 visitors per month, to receive a valid sample size it takes 3 full days of traffic to measure a conversion rate impact of 3%; 24 days for 1%; and 1.5 years for a 0.2% change [ 27]. Because of the long test durations for small-to-moderate sized websites, it is important that designers know how to prioritize the most interesting tests to actually run.

2.2 Pricing
2.2.1 Customize with care In 2000, a young Amazon faced a consumer backlash after a user disclosed that he was charged $4 less for a DVD once he anonymized his Internet connection [ 21]. The Wall Street Journal disclosed recently the nature of algorithms used by companies like Orbitz and Staples in price discriminating, based on factors like geographic income distribution, distance from store of a competitor, or whether or not the user is on a Mac [22,23]. According to a study by the Annenberg Public Policy Center at the University of Pennsylvania [ 24], 87 percent of people surveyed strongly objected to online dynamic pricing based on customer information; and nearly 66% of people falsely believed that this practice is illegal. Yet the practice is in

typically legal, so long as it does not violate the Robinson-Patman Act, which does state that
9

sellers must treat all competing customers on the same basis, but the act targets anticompetitive effects, which are unlikely on the highly competitive Internet [25]. It turns out that there are many examples of dynamic pricing in many traditional industries; haggling at flea markets and by car salesmen is very common, such that prices for the same exact product change as a function of the circumstance. However, people generally dont like being ignorant of games that are played on them even particularly nerdy people. There is one example of a search engine optimization blogger who discovered that he was quoted for a software product at twice the cost as his friend, who was looking at the offer from the exact same website at the exact same time. He was so upset that he filed complaints with the company [11]. The important point is that if customizing prices, it is essential to be very sensitive to how upset people may become. 2.2.2 Form matters Different forms of pricing that equal the same payment by customers can affect their willingness to purchase. For example, seeing $50 / month can be more attractive than seeing $600 / year. This probably has to do with the short-term bias that humans have in their imaginations, such that they can better imagine the value received in the upcoming month compared to the upcoming year.

10

REFERENCES
[1] Gardner, James. "12 Surprising A/B Test Results to Stop You Making Assumptions." Unbounce. 18 Sep. 2012. Web: http://unbounce.com/a-b-testing/shocking-results/. [2] Yang, Zizhuang. "Every Millisecond Counts." Facebook Engineering. 28 Aug. 2009. Web: https://www.facebook.com/note.php?note_id=122869103919. [3] Hilland, Anne. Social Proof A/B Test - Which Homepage, Targeting Small Business Owners, Got 24.79% More Email Opt-ins? 6 Apr. 2013 Which Test Won? Web: http://whichtestwon.com/archives/20992 [4] Patel, Neil. 11 Obvious A/B Tests You Should Try. Quicksprout. 14 Jan. 2013. Web: http://www.quicksprout.com/2013/01/14/11-obvious-ab-tests-you-should-try/ [5] Patel, Neil. Stop Creating Explainer Videos, Youre Doing It All Wrong! Quicksprout. 13 Aug. 2012. Web: http://www.quicksprout.com/2012/08/13/how-to-create-an-explainer-video-that-converts/ [6] Deswal, Siddharth. How AMD used A/B testing to achieve 3600% increase in social sharing. Visual Website Optimizer Blog. 2 Jul. 2012. Web: http://visualwebsiteoptimizer.com/split-testing-blog/amd-3600-social-sharing-increase/ [7] Hertzfeld, Steve. Saving lives: Steve wants us to make the Macintosh boot faster. Folklore. Aug. 1983. Web: http://www.folklore.org/StoryView.py?story=Saving_Lives.txt [8] Moth, David. Eight user testing case studies that achieved amazing results. Econsultancy. 22 Oct. 2012. Web: http://econsultancy.com/us/blog/10922-eight-user-testing-casestudies-that-achieved-amazing-results [9] Moth, David. 88% of consumers consult reviews when making a purchase. Econsultancy. 22 May 2012. Web: http://econsultancy.com/us/blog/9958-88-of-consumers-consultreviews-when-making-a-purchase [10] Patel, Neil. What Spending $252,000 On Conversion Rate Optimization Taught Me. Quicksprout. 8 Oct. 2012. Web: http://www.quicksprout.com/2012/10/08/what-spending252000-on-conversion-rate-optimization-taught-me/ [11] Brinker, Scott. When A/B Testing Prices, Proceed With Caution. Search Engine Land. 25 Jan. 2012. Web: http://searchengineland.com/when-ab-testing-prices-proceed-withcaution-108534

11

[12] Laja, Peep. Three Hard Truths About A/B Testing. ConversionXL. 12 Dec. 2012. Web: http://conversionxl.com/three-hard-truths-about-ab-testing/ [13] Eisenberg, Brian. The Conversion Trinity: The 3-Step Magic Formula To Increase Click Throughs and Conversions. Brian Eisenberg. 17 Jun. 2011. Web: http://www.bryaneisenberg.com/the-conversion-trinity-the-3-step-magic-formula-toincrease-click-throughs-conversions/ [14] Soucy, Kyle. Is Your Homepage Doing What It Should? Usable Interface. 2011. Web: http://www.usableinterface.com/articles/homepage.php [15] Factor, Sue. What makes a design Googley? Google Blog. 23 Apr. 2008. Web: http://googleblog.blogspot.com/2008/04/what-makes-design-googley.html [16] Young, Thomas. Building Trust Results in Customer Loyalty. Sales Training Plus. Web:
http://www.salestrainingplus.com/sales-marketing-information/sales-and-marketingarticles/55-building-trust-results-in-customer-loyalty

[17] Confino, Jo. Consumer trust is not something that can be bought. Guardian. 2 Sep. 2011. Web: http://www.guardian.co.uk/sustainable-business/consumer-trust-boots-reputation [18] Chopra, Paras. How WikiJob increased sales b y 34% by A/B testing customer testimonials. Visual Website Optimizer: A/B Split Testing, Case Studies. 11 Apr. 2011. Web: http://visualwebsiteoptimizer.com/split-testing-blog/customer-testimonialsincrease-sales/ [19] Bryan, Brett. The Psychology of Sharing: Why Do People Share Online? New York Times Customer Insights Group. Web: http://nytmarketing.whsites.net/mediakit/pos/ [20] Murphy, Jed. Amazon looks to users to socially-share their purchases. Digital Diary Interactive Thinking. 11 Nov. 2011. Web: http://jedmurphy.blogspot.com/2011/11/amazonlooks-to-users-to-socially-share.html

[21] Martinez, Michael. Amazon Error May End Dynamic Pricing ABC News. 29 Sep. 2000. [22] Mattioli, Dana. On Orbitz, Mac Users Steered to Pricier Hotels. Wall Street Journal. 23 Aug. 2012. Web:
http://online.wsj.com/article/SB10001424052702304458604577488822667325882.html

[23] Valentino-Devries, Jennifer, Jeremy Singer-Vine, and Ashkan Soltani. Websites Vary Prices, Deals Based on Users Information. Wall Street Journal. 24 Dec. 2012. Web:
http://online.wsj.com/article/SB10001424127887323777204578189391813881534.html 12

[24] Turow, Joseph, Lauren Feldman, and Kimberly Meltzer. Open to Exploitation: American Shoppers Online and Offline. Annenberg Public Policy Center at the University of Pennsylvania. Jun. 2005. Web:
http://www.annenbergpublicpolicycenter.org/Downloads/Information_And_Society/Turow_AP PC_Report_WEB_FINAL.pdf

[25] Ramasastry, Anita. Web sites change prices based on customers habits. CNN Law Center. 24 Jun. 2005. Web:
http://www.cnn.com/2005/LAW/06/24/ramasastry.website.prices/index.html

[26] Bowles, Cennydd. Statistical significance & other A/B test pitfalls. 16 Nov. 2009. Web:
http://www.cennydd.co.uk/2009/statistical-significance-other-ab-test-pitfalls/

[27] Moore, Robert. Prevent Analysis Paralysis By Avoiding Pointless A/B Tests. 12 Jan. 2013. Web: http://techcrunch.com/2013/01/12/current-conversion-rate-and-desiredconfidence-interval-will-help-you-avoid-analysis-paralysis-stop-running-stupid-tests/

[28] Goodman, Aaron. A Bayesian Approach to A/B Testing. Custora Blog. 4 May 2012. Web: http://blog.custora.com/2012/05/a-bayesian-approach-to-ab-testing/

13