This work lacks grounding because the measure of interest, Alexa rank, is not situated in measures that are familiar.
The author, dmor, is not familiar with the semantics of the Alexa rank. Nor are most readers. We know loosely that low rank is good. But what does going from rank 1000 to 100 really mean? How hard is that? Is it a traffic increase of 10x? 100x? What exponential base does it follow?
So this chart shows the YC companies that have the largest Alexa rank delta. Frankly, the rankings look plain wrong to me, based upon what I know of these companies.
Other commentators have suggested that, given the power law distribution, we actually care about the delta log-rank. A priori, I would agree. But then we get a new ranked list, and we still don't know if we're actually learning something or our log-metric is messed up. That's actually quite pernicious, if the results look vaguely correct at a coarse level, so we trust the results at a fine-grained level, and don't realize that the methodology and results are wrong.
We're all just blowing hot air because we don't really know what Alexa ranks mean. Only SEOs who work with Alexa ranks frequently, understand the undocumented warts, and have a gut sense of what they mean, can interpret this.
But right now, we really have no point of reference and the table leaves us without having gained any insight.
Perhaps I'm actually happy that the results are so clearly wrong and not helpful. At least that way, no one will trust them. It would be much more insidious if the results were commonly thought to be instructive, but in fact were misleading.
While I understand your point, I disagree with the conclusion. Simply because a system is complicated that doesnt mean that we should give up. You create a first order approximation and then iterate (very much like how it seems dmor is trying to do). If you know something about how the Alexa ranking system works that we do not, then by all means share it. Simply saying that the author does not know and therefore all conclusions are useless is unhelpful and probably incorrect. To be honest, this reminds me of the elderly who dont understand computers and just give up without trying.
When you don't have deep (insider) access to key metrics, you have to rely on other indicators of success. Alexa provides one piece of data that is semi-meaningful: if nothing else, I'd wager it's roughly correlated (on average) with market cap and/or exit price. (dmor: this would be a cool graph!)
Plus, these companies are in drastically different markets. Their key metrics are probably quite different, or (at the very least) not directly comparable to growth in value / profitability.
Alexa provides one piece of data that is semi-meaningful:
That's my point. This measure is semi-meaningful.
How meaningful is it? Very? Somewhat? Well, we don't know.
The way you determine how meaningful it is, is by connecting the dots with another measure. For example, we have this measure that is cheap to acquire but perhaps inaccurate (Alexa). Can we connect it with another measure that is harder to acquire but more accurate (e.g. exit value)?
if nothing else, I'd wager it's roughly correlated (on average) with market cap and/or exit price. (dmor: this would be a cool graph!)
I would wager that too. But we don't know until you run the numbers.
Grounding your measures is what separates cool hacks from data you can actually draw meaningful inferences from. I think dmor is trying to do something real here, which is why I think it useful to help her actually push the ball forward and really make something much more valuable.
I agree that exit value / market cap is a good auxiliary measure. I think I also suggested this in another comment.
It sounds like the value may be in who is on the list and growing. Maybe the order is not as important as the fact that these are all companies worth watching?
Let me say first of all that I have great respect for the way you have engaged with criticism on this post and on your previous ones. Even when I disagree with your approach or whatever, I appreciate your curiosity and eagerness to learn.
Some ways that you could make weaker claims but with more confidence in your results:
* Have an unordered list of companies that are growing. This was your suggestion. I don't know how interesting that is.
* Group companies into five or ten buckets, based upon Alexa ranks. (i.e. 1=low traffic bucket, 5=high traffic) Find the top three movers-and-shakers within each bucket. This makes the weaker assumption that Alexa ranks deltas are comparable within each bucket, as opposed to your original assumption that they are comparable across the entire spectrum of Alexa ranks.
There are a handful of other things you can do that are more complicated. For example, if you can correlate Alexa ranks with Compete unique visitor estimates or some other number (company's exit value). Compete estimates are also biased, but at least people have better intuition of what unique visitor numbers mean.
Thank you, especially for hearing my earnestness through all the noise - I'm glad it comes through. Without revealing too much, I can tell you that I have a much varied and interesting data set to use to rank companies for the next full YC index. I expect there will be feedback on the method for that one too, but with every iteration it is getting massively better.
Crazy idea - javascript startups can put on their site to report their actual data to me?
I basically agree with this, putting an order on this list might have irked some people, but a list of companies to follow in no particular order might have worked better.
The author, dmor, is not familiar with the semantics of the Alexa rank. Nor are most readers. We know loosely that low rank is good. But what does going from rank 1000 to 100 really mean? How hard is that? Is it a traffic increase of 10x? 100x? What exponential base does it follow?
So this chart shows the YC companies that have the largest Alexa rank delta. Frankly, the rankings look plain wrong to me, based upon what I know of these companies.
Other commentators have suggested that, given the power law distribution, we actually care about the delta log-rank. A priori, I would agree. But then we get a new ranked list, and we still don't know if we're actually learning something or our log-metric is messed up. That's actually quite pernicious, if the results look vaguely correct at a coarse level, so we trust the results at a fine-grained level, and don't realize that the methodology and results are wrong.
We're all just blowing hot air because we don't really know what Alexa ranks mean. Only SEOs who work with Alexa ranks frequently, understand the undocumented warts, and have a gut sense of what they mean, can interpret this.
But right now, we really have no point of reference and the table leaves us without having gained any insight.
Perhaps I'm actually happy that the results are so clearly wrong and not helpful. At least that way, no one will trust them. It would be much more insidious if the results were commonly thought to be instructive, but in fact were misleading.