A Funny Thing Happened On The Way To A/B Testing

TL;DR: I split tested the titles of my book: Here's what happened

So, if you were following my blog (thank you to the both of you), you’d know that I’ve been writing a book on the quirks of JavaScript. I split tested the book across two titles: Underhanded JavaScript: How To Be A Complete Arsehole with Bad JavaScript Code; and JavaScript Technical Interview Questions. This is the story of the twists and turns and missteps that happened when I wrote the book.

How The Book Came To Be

I was working on a virtual machine project when I complained about JavaScript to my friends. Specifically, I complained in a joking manner that JavaScript is so terrible for productivity that I could write a book on how shitty it was* Before JavaScript fanboys and fangirls interject - no, I happen to know JavaScript quite well, and on a fairly deep level - for reasons that will become obvious later. Knowing JavaScript does not negate the fact that it is a language full of pitfalls that impedes on my productivity because the amount of quirks I have to remember while writing JavaScript is very high. Also, please stop using "you don't know JavaScript, that's why you say it sucks" as a defence for JavaScript. . The required cognitive load while working on JavaScript is much higher compared to working on languages like Python or Go (I work extensively with the latter two).

At the same time, Hacker News went down, and I had nothing to while my time away. So I decided to actually write a blog post ranting about JavaScript as a procrastination from working on my VM. The more I wrote, the more it became apparent that it should be a book instead of just a couple of blog posts. The first day, I wrote about 2 and a half chapters on the quirks of JavaScript. I would call the book JavaScript Oddities. I decided to go with Leanpub mainly because I had originally started writing in Markdown.

Split Testing

Then, by sheer coincidence, I picked up a consulting gig as a favour for a friend on interviewing potential candidates about JavaScript, a couple of days after I've started working on the book. I had by then, Chapter 3 done. I suggested a few interview questions based on my chapter listings in my head. None of the candidates could answer them quite correctly* For example, I discovered that people have very strange ideas about variable hoisting in JavaScript. Hoisting simply means the variable declaration is moved to the top, when the control enters the function code. Nobody could give me a proper example of hoisting . That consulting gig ended after it was determined the questions I was asking were too difficult (I had not even went into my questions about the module pattern or AMD vs CommonJS or ES6 generators. I just asked some tricky questions based on the three chapters I had written). I had became that guy. You know, that guy at technical interviews who seemingly asks difficult questions just to show off his own technical knowledge and lord over the candidate? Yeah, that guy. The arsehole interviewer. I felt quite shitty for a number of days after that. Around that time too, I had an idea of hosting a contest for Underhanded JavaScript - in vein of The Underhanded C Contest. I talked about it with a number of JS developer friends. Being my usual unclear self, a number of them thought I was talking about my book. I wasn't but I saw that Underhanded JavaScript was a much much better title, so I changed the title of my book to Underhanded JavaScript. Then an idea struck: I could split test the title of the book - the content would be the same, but the titles would be different. I spent a couple of hours writing a new marketing copy for the landing page of JavaScript Technical Interview Questions.

Statistics - the Hypotheses

To perform an A/B test, I had to set up some targets before hand. Given an optimistic conversion rate of about 4%, and a 80% power, a standard deviation of 0.05, and a 95% confidence interval, I roughly calculated I only need about 700-ish visits to each of the pages before I could make any decision regarding which title I should choose. There is a good reason why I chose a 4% conversion rate. From my time working in the advertising industry, I had noticed that low-commitment conversions, such as submitting one's email address would have a rather high conversion rate, if compared to a high-commitment conversion, such as requiring one to type in credit card numbers. Those conversion rates are typically about 0.2-0.5% at best. I deliberately chose a lower power and a low difference-to-detect because I also know that given a 1 month timeframe, it's quite impossible to shove tens of thousands of impressions without spending considerable amounts of money. So I would choose something smaller that I could handle.

Running Ads with Adwords

So now I had a target of 700 visits that I needed, and I had two Leanpub book sites ready to go. I just need to send traffic to those sites. I fell back on my trusty tool, Google Adwords to drive traffic to the site. Like most tests I do with Adwords, I limit myself to about $100 initial budget to test the ideas and sign ups. It wasn't a good idea to run traffic from Adwords to the Leanpub pages. The result of spending $93 on it was 0 conversions. However, I did learn much from it.

Set Up

I set up two campaigns - one for each book, called "Underhanded" and "Interview" respectively, and set up a $1.50 maximum bid in both campaigns. It's generally bad advice to let Adwords optimize the campaigns but I let Adwords do the optimization for me since I didn't actually have much time to do that. Both campaigns had different sets of keywords that were bid for. For example, "Javascript wat", "Javascript this", "Javascript hack" were bid for in Underhanded, while "Javascript interview", "Javascript interview questions" and "Javascript career" were bid for in Interview. Very diverse and different audiences I was targeting. Both campaigns also each had a set of 10 ads. They weren't the best copy I have ever seen, but they were servicable. I didn't set up 10 ads to test the ad response. Instead, those ads were set up as to maximize the chances of an ad being shown to the audience. Here is a set of the best performing three ads for each campaign:
Interview Underhanded
Technical Interview Soon?
Prepare for that dream
Node.js developer position
leanpub.com/jsinterviewquestions
wat... JavaScript Code
Why is JavaScript so unexpected?
How to wreak havoc with bad JS
leanpub.com/underhandedjavascript
Looking for Node.js Jobs?
How would you fare in a technical
interview? Prepare yourself now.
leanpub.com/jsinterviewquestions
Javascript is weird
Or is it?
Learn the quirks of Javascript
leanpub.com/underhandedjavascript
Finding JavaScript Jobs?
Are you prepared for the technical
interviews? Prepare yourself now.
leanpub.com/jsinterviewquestions
How to cause trouble with JS
Learn more about JS quirks.
Stop writing bad Javascript code
leanpub.com/underhandedjavascript

Results

Here are the results from running this campaign. As previously mentioned, the amount of conversions in each page was 0.
Campaign Impressions Clicks CTR Cost Calculated CPC
Underhanded 11055 47 0.43% $48.42 $1.03
Interview 17037 37 0.22% $44.43 $1.20
A simple Z-test indicates that the differences in click-through-rate are statistically significant. However, this simply means that my ad copies for "Underhanded" are better than my ad copies for "Interview". The number of clicks are evenly distributed amongst the ads of the Interview campaign, and the click through rate for the ads are not any different from each other. However, for the Underhanded campaign, the "wat... Javascript Code" ad accounted for 64% of the clicks (30 out of 47). Interesting. Until I realized that the top performing keywords was "Javascript wat" However, the most important thing to note is that there are more impressions for the keywords for the Interview than the keywords for the Underhanded campaign. Although there wasn't enough space to experiment and gather data, from experience I can say that this means that there is a bigger audience searching for keywords related to JavaScript interviews than people searching for keywords related to JavaScript quirks. This is also shown by the calculated CPC. There are more people bidding on the keywords related to JavaScript technical interviews, and therefore the prices are slightly higher * a lot of other things go into calculating CPC, like quality score and the like and so to simplify the scenario, I'd just use this as a general rule of thumb Any how, no decision could be made from this Adwords test. I've had extremely successful Adwords campaigns in the past before, but also have had some really terrible ones. This was one of the okay ones, though it would seem I've lost some mojo from working in advertising.

Accidental Content Marketing

I was deep in Excel spreadsheets, preparing for a second round of Adwords ads when suddenly my blog post on why kettles are so expensive became insanely viral. For that I have Ben to thank. Suddenly there were 50,000 people reading my blog, many of them geeks who probably write code in JavaScript. In short, my target market. And so I got to work. I had only one link to Underhanded Javascript when the blog post became viral. I corrected that quickly by adding a second link to JavaScript Technical Interview Questions. The two links were always displayed in pairs. This was for a good reason. The reason for doing so was to basically immediately present the reader with a choice, since both links are next to each other. This would negate the memory effect when presenting readers with a choice. If a choice is presented with a lot of spacing in between them (time, or text), the reader may choose either the latter option, forgetting the previous option existed, or the reader may choose both, clicking on the former then later, clicking on the latter. This rather defies the purpose of a split test. In case you were wondering why I didn't just relegate this to a A/B testing tool or plugin, this is a personal blog, I write my thoughts here, mainly for me. I don't intend to squeeze every penny out of this blog because then that would just make blogging a job and a chore and I would no longer find it fun to share my thoughts* Blogging becoming a chore has happened to this blog many times over the past 10 years that I've been writing here, which is why there are many reboots to this blog . Anyway, the whole point was to present the reader with a choice, and hope that the reader chooses only one. But yet there was still another problem. Agarwal et al (2008) did a study on the effect of location on ads on a page and their click through rates. I would say this still holds true today, even with links* It's also the same reason why people pay top dollar for above-the-fold display ads - they work quite well . From past experience running a poor-man's A/B testing, I had noticed that in paired links, usually the first link gets clicked more often, especially when the second link is in parentheses. The solution was simple: for the first 2 days where the visitors were plentiful, I would switch the position of the links every 45 minutes or so, manually. That duration became longer and longer over the first two days, until I finally stopped changing the positions. This would ensure that roughly equal amounts of people will click through to the landing pages.

Statistics

How did it fare? Comparatively, the click through rate from my blog to either site could not be more different. Clickthroughs to JavaScript to Interview Questions were almost twice the clickthroughs to Underhanded JavaScript.

Conversion Rates

But that's not what we're after aren't we? We're after the conversions - in this case, the expression of interest to buy a book by entering an email address is considered a conversion. Even if Interview Questions has a higher click through rate than Underhanded JavaScript, if Underhanded JavaScript has a better conversion rate, then it'd be a better candidate. So what are the conversion rates like? Because Leanpub doesn't fire a conversion pixel when a expression of interest happens, I had to manually collect interests and match it to the period's visitor count. The numbers are not as accurate as I want them to be, but are within acceptable error range for me to make a decision. I also do not use an equal distance between the periods - simply because I don't feel that the date or day matters. As such, I use generic period numbers. Now weep as I show you some terrible numbers:
Period ConvUnderhanded(%) ConvUnderhanded(Totals) ConvInterview(%) ConvInterview(Totals)
0* All I could muster from friends family and fools, before I started active marketing n/a4n/a1
10.0045.0518
26.0684.8123
34.00105.4827
4* I released the slides of my JavaScript talk a day prior 12.001316.2133
510.71169.0935
610.001727.2738
70.001710.0041
86.24184.3542
98.00203.7043
1036.362414.2844
11* My article on floating points was published on Flippin Awesome the day prior 3.70256.6746
123.572720.0054
139.093011.7656
Average4.827.40
The average conversion rate of Underhanded is less than the average conversion rate of Interview. However, a simple chi-squared test shows that the differences in conversion rate is not statistically significant. Naturally, this is quite diappointing. I'm no Nathan Barry, but I do kinda want to see my book succeed. Having fewer than 100 people interested in a book is quite a slap in the face. I know this is bad because of the 90 or so people interested in the book, only a small proportion will actually buy the book. What's worse, I am unable to tell if one title is better than another. Perhaps, I thought, the willingness to pay would be an indicator to which title would do better. Alas, not quite. People who signed up for Underhanded JavaScript are willing to pay about 1.5x higher than the people who signed up for JavaScript Technical Interview Questions. While the expected value (the mean of the willingness to pay multiplied by the number of people) of Interview is slightly higher than the expected value of Underhanded, it's still too close to call. The signs are however, there. More people clicked on terms related to JavaScript Interview Questions, than did people click on links to Underhanded JavaScript. More people searched for terms related to JavaScript Interview Questions. My gut tells me that I should have chosen JavaScript Technical Interview Questions as the title of my book. If this were a scenario in my old job, I'd have gone with JavaScript Interview Questions as a title a long time ago. But I can't.

Not a Jedi, Nor a Buddhist Monk

The lack of data to make a call is the problem with many many many advertising campaigns and indeed projects I worked on - the numbers are too small to make a logical decision. When faced with such scenarios at my old work, I’d have just jumped with my gut and just picked one. I tried to do the same with this one but after a number of days, I have yet been unable to make a decision. You may ask why this is important. It’s because I had written the content of the book, and now I need the title to edit the content to fit the tone of the title.

Last night as I sat down and reviewed the data, I began to wonder where it had gone wrong, and I pondered upon my inability to make a decision. I realized that at my job, in the past, when confronted with a situation where I had not enough data to make a decision and the clock was ticking, I would always make a decision. The reason was because I wasn’t attached to the product.

This time it’s different. I had just spent the whole of last month writing a JavaScript book. That’s one entire month of my time, thinking of problems I had encountered in the past, and how to explain them in simple English. I had invested quite a bit in this, and I was very attached to it.

I am really attached to the idea of Underhanded JavaScript - there is nothing like the gleeful fun from the havoc that can be wreaked with bad JavaScript code. I’m attached to the structure of the book - JavaScript Interview Questions would require a different structure - one that doesn’t tell a story. However, I am also attached to the idea that JavaScript Interview Questions will sell more copies.

As much as I like the Buddhist and Jedi philosophy of having no attachments, I must confess I have strong difficulties adhering to it.

And so in a haze of disappointment and doubt, I thought up of a solution:

Por Que No Los Dos

And so, I’m happy to announce that Underhanded JavaScript and JavaScript Technical Interview Questions are going to be two separate titles! The content will be initially the same, but as both books go through editorial processes, they will eventually diverge into different books. Underhanded JavaScript, for example, will contain more non-sequitors and unrelated trivia about JavaScript and other things, while JavaScript Technical Interview Questions will be more focused on answering tough interview questions.

For now, you can get a sample copy of Underhanded JavaScript (PDF), and tell me if you like it. The sample copy of JavaScript Interview Questions will be coming shortly

Conclusion

I started writing a book. I then decided to split test the titles between Underhanded JavaScript and JavaScript Technical Interview Questions. The results were disappointing, and not enough data to make a conclusive answer. If this were other people’s projects, I’d have less attachment and would have decided which title to choose from. However, since I’m quite deluded and attached, I’ve decided to go with both.

But really, you should read the entire post. Then share it on Twitter, and Reddit and Hacker News.

Tell me what you think?

comments powered by Disqus