To avoid big A/B testing mistakes, perhaps you shouldn't use phrases like "To counter that noise it is important to first “prove the Null hypothesis.” To prove the Null hypothesis you..."
One of the most important underlying statistical principles in inference tests is that the null hypothesis can never be proven. Any data you collect can only reject the null hypothesis or fail to reject it.
Good point. Not sure where I picked up that phrasing - especially since the wikipedia article linked by the phrase specifically says:
> It is important to understand that the null hypothesis can never be proven. A set of data can only reject a null hypothesis or fail to reject it. For example, if comparison of two groups (e.g.: treatment, no treatment) reveals no statistically significant difference between the two, it does not mean that there is no difference in reality. It only means that there is not enough evidence to reject the null hypothesis (in other words, the experiment fails to reject the null hypothesis).
I'm going to update the post to fix that mistake. Thanks.
Edit: I updated that section. This part and the next part of the series are the ones I'm most anxious about because of all the math and the questions that have right and wrong answers.
Here is better phrasing or a good way to think about hypothesis testing:
"The goal of the test is to determine if the null hypothesis can be rejected. A statistical test can either reject (prove false) or fail to reject (fail to prove false) a null hypothesis, but never prove it true (i.e., failing to reject a null hypothesis does not prove it true)." (Wikipedia)
I might also change some of the wording of these:
"The Null Hypothesis states that if you don’t change anything than nothing will be different."
The null hypothesis states that there are no differences in X (where X is the metric you are using to evaluate the performance of the different experiences, or recipes. X could be things like Conversion Rate, Click Rate, Time on Page, Revenue per Visitor etc) between the control experience and any number of test experiences.
"If you divide your traffic and see significant differences in your metrics you have failed the Null hypothesis."
I would just say "...you would reject the null hypothesis." In this case, you're basically saying that there is evidence to suggest that the differences in X (where X is the variable you're measuring) are not due to chance / noise / natural variation alone. By saying "you would reject the null hypothesis", the implication is that you would accept the alternative hypothesis. Remember, the null is that there are no differences in X and the alternative is that there are differences in X (where X is your success metric).
"Failing the Null hypothesis means that either you have not collected enough results to even out the noise or there is something wrong with the algorithm that you are using to segment your traffic."
By "Failing the Null hypothesis," do you mean "rejecting it" or "failing to reject it?"
If you mean "rejecting it", I'm not sure how to read what you wrote.
If you mean "failing to reject it," you're basically saying here that if you don't reject the null hypothesis, it is due to one of 3 things (or a combination of the 3):
1) Your sample size is not large enough
This is a tricky boat to get into. Any difference in the metric your are testing for will reach statistical significance with a large enough sample size. If you're interested more in this, go look at the math formulas.
You should decide ahead of time how long you want to run a test for, knowing how much traffic the page you're testing will receive, and how many experiences / groups / segments you are splitting this traffic into.
If you don't reach significance, it could just be that the differences you were testing for are too small.
Knowing the standard deviation / variance in the metric you're testing can help you understand which one of the above it is (sample size is too small or differences you are tested are too small)
2) You don't have enough evidence to suggest that any differences you see are due to anything other than variation / chance / noise.
This can happen a lot when the changes / differences you're testing are too small.
3) You have a Type II error
You have failed to reject a false null hypothesis. The test should have told you there was a difference, but for whatever reason, the sample data you saw did not provide you with evidence to do so.
Oh - but I do also want to say that you did a great job on 99% of the article. You hit on pretty much all the major considerations, biases and issues one could run into when testing.
Whenever you get frustrated about a tool, a process, a company, a product, or an interaction you have (basically anything that is not a personal relationship), write it down.
Keep a list of these. Come back to them once a week and think about what you would like to be different or how you would make your experience better. Cross off any frustrations that you would not pay money to fix.
Refine this list by determining if:
a) Others (people) have similar frustrations to you
b) Others would pay money to fix or reduce their frustrations
That should get you started in the right direction.
I have a Notebook and a Evernote notebook about this. I write down all my ideas on these notebooks. My passion is so big about startups, that sometimes I do this when I have a Patient. I stop the treatment and with the help of my nurses I write the ideas on the list.
But the ideas are not revolutionary. It seems that everything is already done. I know that always there is a unsolved problem but, I don't know if it's my fault or it's normal, it's too hard to come up with a great idea nowadays.
Very good advice, but I'll also recommend you educate yourself further. Take 90k and save it, it is easy to burn money on a business. -
read: http://www.paulgraham.com/selfindulgence.html
Also keep in mind, most great products / services are not revolutionary but evolutionary. Google, Facebook both improved on existing ideas, Apple's ipod was not an original Apple idea/tech either : http://www.dailymail.co.uk/news/article-1053152/Apple-admit-...
If I'm looking to start consulting on the side, is it reasonable to charge lower rates at first to attract business and get some experience (something in the $50/hr range)
While I want to be paid fairly for what I'm worth, I also don't want to overcharge clients while I figure out what I'm doing (processes and services etc).
You probably won't get a straight answer to this so I'll say the tough words.
Anyone who's established will be easier to philosophize about why you shoudl charge more.
At the end of the day, no one pays your bills for you but yourself. Getting any work is more important than getting good work. Getting great work is more important than getting good work.
I don't know if I am doing it the best way but I did start ($50/hr in 1999) doing absolutely anything anyone would pay me for at that equivalent rate.
Also, figuring out your processes and services is complete BS. You never stop developing and improving them. I've trashed my entire setup and rebuilt/programmed it from scratch 5 times in 10 years only to finally settle on a happy-ish medium of Fogbugz and Freshbooks.
You're likely not going to have a business situation which requires an employee to code a solution in 5 minutes. Even if you do, the team + collaborative environment in that situation vs in this present interview situation will make a major difference in the employee's performance.
You obviously want to hire people that can handle stress and perform well under it, but it doesn't make a lot of sense to create these unrealistic situations to test how well a candidate might perform on the job.
I'd recommend informing the candidate upfront that the interview will be taped. Then give him the problem and come back in an hour. Make the candidate provide you with all the scratch sheets / work and use these in conjunction with your tape to evaluate his/her performance.
I realize this may not be realistic, but it is possible and I think it would lead to some interesting and useful results.
I just want to say that I am incredibly touched by the volume, sincerity, and generosity of responses. This community is incredible - so proud to be a part of it.
Mattman, I can't offer much in terms of advice or connections that have already been posted, but I will pray for you and your situation.
Location: NYC / NJ
Remote: Yes
Willing to relocate: Open to it
Technologies: Marketing (FB/IG, Adwords), Analytics (Tableau, Looker, Segment, SQL, Python/Pandas)
Résumé/CV: https://www.linkedin.com/in/mattacurtis/
Email: matt.a.curtis@gmail.com