A Funny Thing Happened On The Way To A/B Testing

So, if you were following my blog (thank you to the both of you), you’d know that I’ve been writing a book on the quirks of JavaScript. I split tested the book across two titles: Underhanded JavaScript: How To Be A Complete Arsehole with Bad JavaScript Code; and JavaScript Technical Interview Questions. This is the story of the twists and turns and missteps that happened when I wrote the book.

How The Book Came To Be

At the same time, Hacker News went down, and I had nothing to while my time away. So I decided to actually write a blog post ranting about JavaScript as a procrastination from working on my VM. The more I wrote, the more it became apparent that it should be a book instead of just a couple of blog posts. The first day, I wrote about 2 and a half chapters on the quirks of JavaScript. I would call the book JavaScript Oddities. I decided to go with Leanpub mainly because I had originally started writing in Markdown.

Split Testing

That consulting gig ended after it was determined the questions I was asking were too difficult (I had not even went into my questions about the module pattern or AMD vs CommonJS or ES6 generators. I just asked some tricky questions based on the three chapters I had written). I had became that guy. You know, that guy at technical interviews who seemingly asks difficult questions just to show off his own technical knowledge and lord over the candidate? Yeah, that guy. The arsehole interviewer.

I felt quite shitty for a number of days after that. Around that time too, I had an idea of hosting a contest for Underhanded JavaScript - in vein of The Underhanded C Contest. I talked about it with a number of JS developer friends. Being my usual unclear self, a number of them thought I was talking about my book. I wasn’t but I saw that Underhanded JavaScript was a much much better title, so I changed the title of my book to Underhanded JavaScript.

Then an idea struck: I could split test the title of the book - the content would be the same, but the titles would be different. I spent a couple of hours writing a new marketing copy for the landing page of JavaScript Technical Interview Questions.

Statistics - the Hypotheses

There is a good reason why I chose a 4% conversion rate. From my time working in the advertising industry, I had noticed that low-commitment conversions, such as submitting one’s email address would have a rather high conversion rate, if compared to a high-commitment conversion, such as requiring one to type in credit card numbers. Those conversion rates are typically about 0.2-0.5% at best.

I deliberately chose a lower power and a low difference-to-detect because I also know that given a 1 month timeframe, it’s quite impossible to shove tens of thousands of impressions without spending considerable amounts of money. So I would choose something smaller that I could handle.

Running Ads with Adwords

It wasn’t a good idea to run traffic from Adwords to the Leanpub pages. The result of spending $93 on it was 0 conversions. However, I did learn much from it.

Set Up

Both campaigns had different sets of keywords that were bid for. For example, “Javascript wat”, “Javascript this”, “Javascript hack” were bid for in Underhanded, while “Javascript interview”, “Javascript interview questions” and “Javascript career” were bid for in Interview. Very diverse and different audiences I was targeting.

Both campaigns also each had a set of 10 ads. They weren’t the best copy I have ever seen, but they were servicable. I didn’t set up 10 ads to test the ad response. Instead, those ads were set up as to maximize the chances of an ad being shown to the audience. Here is a set of the best performing three ads for each campaign:

Results

A simple Z-test indicates that the differences in click-through-rate are statistically significant. However, this simply means that my ad copies for “Underhanded” are better than my ad copies for “Interview”. The number of clicks are evenly distributed amongst the ads of the Interview campaign, and the click through rate for the ads are not any different from each other. However, for the Underhanded campaign, the “wat… Javascript Code” ad accounted for 64% of the clicks (30 out of 47). Interesting. Until I realized that the top performing keywords was “Javascript wat”

However, the most important thing to note is that there are more impressions for the keywords for the Interview than the keywords for the Underhanded campaign. Although there wasn’t enough space to experiment and gather data, from experience I can say that this means that there is a bigger audience searching for keywords related to JavaScript interviews than people searching for keywords related to JavaScript quirks. This is also shown by the calculated CPC. There are more people bidding on the keywords related to JavaScript technical interviews, and therefore the prices are slightly higher * a lot of other things go into calculating CPC, like quality score and the like and so to simplify the scenario, I'd just use this as a general rule of thumb

Any how, no decision could be made from this Adwords test. I’ve had extremely successful Adwords campaigns in the past before, but also have had some really terrible ones. This was one of the okay ones, though it would seem I’ve lost some mojo from working in advertising.

Accidental Content Marketing

Suddenly there were 50,000 people reading my blog, many of them geeks who probably write code in JavaScript. In short, my target market. And so I got to work.

I had only one link to Underhanded Javascript when the blog post became viral. I corrected that quickly by adding a second link to JavaScript Technical Interview Questions. The two links were always displayed in pairs. This was for a good reason.

The reason for doing so was to basically immediately present the reader with a choice, since both links are next to each other. This would negate the memory effect when presenting readers with a choice. If a choice is presented with a lot of spacing in between them (time, or text), the reader may choose either the latter option, forgetting the previous option existed, or the reader may choose both, clicking on the former then later, clicking on the latter. This rather defies the purpose of a split test.

In case you were wondering why I didn’t just relegate this to a A/B testing tool or plugin, this is a personal blog, I write my thoughts here, mainly for me. I don’t intend to squeeze every penny out of this blog because then that would just make blogging a job and a chore and I would no longer find it fun to share my thoughts* Blogging becoming a chore has happened to this blog many times over the past 10 years that I've been writing here, which is why there are many reboots to this blog . Anyway, the whole point was to present the reader with a choice, and hope that the reader chooses only one. But yet there was still another problem.

Agarwal et al (2008) did a study on the effect of location on ads on a page and their click through rates. I would say this still holds true today, even with links* It's also the same reason why people pay top dollar for above-the-fold display ads - they work quite well . From past experience running a poor-man’s A/B testing, I had noticed that in paired links, usually the first link gets clicked more often, especially when the second link is in parentheses.

The solution was simple: for the first 2 days where the visitors were plentiful, I would switch the position of the links every 45 minutes or so, manually. That duration became longer and longer over the first two days, until I finally stopped changing the positions. This would ensure that roughly equal amounts of people will click through to the landing pages.

Statistics

Conversion Rates

So what are the conversion rates like? Because Leanpub doesn’t fire a conversion pixel when a expression of interest happens, I had to manually collect interests and match it to the period’s visitor count.

The numbers are not as accurate as I want them to be, but are within acceptable error range for me to make a decision. I also do not use an equal distance between the periods - simply because I don’t feel that the date or day matters. As such, I use generic period numbers. Now weep as I show you some terrible numbers:

The average conversion rate of Underhanded is less than the average conversion rate of Interview. However, a simple chi-squared test shows that the differences in conversion rate is not statistically significant.

Naturally, this is quite diappointing. I’m no Nathan Barry, but I do kinda want to see my book succeed. Having fewer than 100 people interested in a book is quite a slap in the face. I know this is bad because of the 90 or so people interested in the book, only a small proportion will actually buy the book. What’s worse, I am unable to tell if one title is better than another.

Perhaps, I thought, the willingness to pay would be an indicator to which title would do better. Alas, not quite. People who signed up for Underhanded JavaScript are willing to pay about 1.5x higher than the people who signed up for JavaScript Technical Interview Questions. While the expected value (the mean of the willingness to pay multiplied by the number of people) of Interview is slightly higher than the expected value of Underhanded, it’s still too close to call.

The signs are however, there. More people clicked on terms related to JavaScript Interview Questions, than did people click on links to Underhanded JavaScript. More people searched for terms related to JavaScript Interview Questions. My gut tells me that I should have chosen JavaScript Technical Interview Questions as the title of my book. If this were a scenario in my old job, I’d have gone with JavaScript Interview Questions as a title a long time ago.

But I can’t.

Not a Jedi, Nor a Buddhist Monk

The lack of data to make a call is the problem with many many many advertising campaigns and indeed projects I worked on - the numbers are too small to make a logical decision. When faced with such scenarios at my old work, I’d have just jumped with my gut and just picked one. I tried to do the same with this one but after a number of days, I have yet been unable to make a decision. You may ask why this is important. It’s because I had written the content of the book, and now I need the title to edit the content to fit the tone of the title.

Last night as I sat down and reviewed the data, I began to wonder where it had gone wrong, and I pondered upon my inability to make a decision. I realized that at my job, in the past, when confronted with a situation where I had not enough data to make a decision and the clock was ticking, I would always make a decision. The reason was because I wasn’t attached to the product.

This time it’s different. I had just spent the whole of last month writing a JavaScript book. That’s one entire month of my time, thinking of problems I had encountered in the past, and how to explain them in simple English. I had invested quite a bit in this, and I was very attached to it.

I am really attached to the idea of Underhanded JavaScript - there is nothing like the gleeful fun from the havoc that can be wreaked with bad JavaScript code. I’m attached to the structure of the book - JavaScript Interview Questions would require a different structure - one that doesn’t tell a story. However, I am also attached to the idea that JavaScript Interview Questions will sell more copies.

As much as I like the Buddhist and Jedi philosophy of having no attachments, I must confess I have strong difficulties adhering to it.

And so in a haze of disappointment and doubt, I thought up of a solution:

And so, I’m happy to announce that Underhanded JavaScript and JavaScript Technical Interview Questions are going to be two separate titles! The content will be initially the same, but as both books go through editorial processes, they will eventually diverge into different books. Underhanded JavaScript, for example, will contain more non-sequitors and unrelated trivia about JavaScript and other things, while JavaScript Technical Interview Questions will be more focused on answering tough interview questions.

For now, you can get a sample copy of Underhanded JavaScript (PDF), and tell me if you like it. The sample copy of JavaScript Interview Questions will be coming shortly

Conclusion

But really, you should read the entire post. Then share it on Twitter, and Reddit and Hacker News.

Tell me what you think?