Statistics - the HypothesesTo perform an A/B test, I had to set up some targets before hand. Given an optimistic conversion rate of about 4%, and a 80% power, a standard deviation of 0.05, and a 95% confidence interval, I roughly calculated I only need about 700-ish visits to each of the pages before I could make any decision regarding which title I should choose. There is a good reason why I chose a 4% conversion rate. From my time working in the advertising industry, I had noticed that low-commitment conversions, such as submitting one's email address would have a rather high conversion rate, if compared to a high-commitment conversion, such as requiring one to type in credit card numbers. Those conversion rates are typically about 0.2-0.5% at best. I deliberately chose a lower power and a low difference-to-detect because I also know that given a 1 month timeframe, it's quite impossible to shove tens of thousands of impressions without spending considerable amounts of money. So I would choose something smaller that I could handle.
Running Ads with AdwordsSo now I had a target of 700 visits that I needed, and I had two Leanpub book sites ready to go. I just need to send traffic to those sites. I fell back on my trusty tool, Google Adwords to drive traffic to the site. Like most tests I do with Adwords, I limit myself to about $100 initial budget to test the ideas and sign ups. It wasn't a good idea to run traffic from Adwords to the Leanpub pages. The result of spending $93 on it was 0 conversions. However, I did learn much from it.
|Technical Interview Soon?
Prepare for that dream
Node.js developer position
How to wreak havoc with bad JS
|Looking for Node.js Jobs?
How would you fare in a technical
interview? Prepare yourself now.
Or is it?
Are you prepared for the technical
interviews? Prepare yourself now.
|How to cause trouble with JS
Learn more about JS quirks.
ResultsHere are the results from running this campaign. As previously mentioned, the amount of conversions in each page was 0.
|0* All I could muster from friends family and fools, before I started active marketing||n/a||4||n/a||1|
|11* My article on floating points was published on Flippin Awesome the day prior||3.70||25||6.67||46|
Not a Jedi, Nor a Buddhist Monk
The lack of data to make a call is the problem with many many many advertising campaigns and indeed projects I worked on - the numbers are too small to make a logical decision. When faced with such scenarios at my old work, I’d have just jumped with my gut and just picked one. I tried to do the same with this one but after a number of days, I have yet been unable to make a decision. You may ask why this is important. It’s because I had written the content of the book, and now I need the title to edit the content to fit the tone of the title.
Last night as I sat down and reviewed the data, I began to wonder where it had gone wrong, and I pondered upon my inability to make a decision. I realized that at my job, in the past, when confronted with a situation where I had not enough data to make a decision and the clock was ticking, I would always make a decision. The reason was because I wasn’t attached to the product.
As much as I like the Buddhist and Jedi philosophy of having no attachments, I must confess I have strong difficulties adhering to it.
And so in a haze of disappointment and doubt, I thought up of a solution:
But really, you should read the entire post. Then share it on Twitter, and Reddit and Hacker News.
Tell me what you think?