> What about benchmarks game is arbitrary in your opinion?
For example, look at the history of D being in the game vs. not. Generally speaking, all of http://benchmarksgame.alioth.debian.org/play.html#languagex is incredibly arbitrary. I don't think that the maintainers of the game have some sort of moral obligation to support things they don't want to, but it is an arbitrary line which influences what kind of benchmark it is.
As another, the domain of the examples, which are largely related to numeric computing and/or memory usage. Which is fine, but isn't going to give me information about other use cases.
> None of the tasks boil down to "Can you write non-idiomatic code to theoretically make this as fast as something else?" -- if you don't want them to!
Yes, they do. Because one man is not an island: people use the raw numbers of the Benchmark Game to claim "X is faster than Y", even if the implementation of X and Y are totally different than what their implementation would be.
> that's been the choice of the Rust community.
I'm actually not thinking of Rust here. I'm thinking more of languages like Haskell, who have implementations that look significantly more like C than like Haskell.
But regardless of if the implementations are idiomatic or not, it means that The Benchmark Game is useless for evaluating if a program I write in a language is going to generally be faster or not than if I did it in another. Because this isn't the code that I'd write: it's code that takes advantage of every little last corner to get as small a number as possible.
>>… being in the game vs. not … but isn't going to give me information about other use cases.<<
1) That might be a credible criticism if the benchmarks game claimed to be some kind of exhaustive comparison.
In fact (if it wasn't already completely obvious that the benchmarks game has nothing to say about things it does not show) over & over again the web pages state -- "These are not the only compilers and interpreters. These are not the only programs that could be written. These are not the only tasks that could be solved. These are just 10 tiny examples."
2) You seem to have assumed that because you personally don't know the reasons, there were no reasons.
>>…totally different than what their implementation would be.<<
What does the benchmarks game have to say about those "totally different" programs that are not shown? Nothing.
Do you think there's an idiomatic style for programs written as though performance matters or does idiomatic just mean programs written as though performance doesn't matter?
>>…useless for evaluating if a program I write in a language is going to generally be faster or not than if I did it in another.<<
Does the benchmarks game website claim to be useful for that purpose? It sounds like magical thinking to me --
"… but the question is still asked - Will my program be faster if I write it in language X? - and there's still a wish for a simpler answer than - It depends how you write it!"
You can state those things all you like, in as large a font as you like, but it doesn't stop people from comparing languages based on the benchmarks game.
It's magical thinking to say otherwise: just because something doesn't claim to be X (or claims to not be X) doesn't stop popular opinion from thinking X. Maybe popular opinion is wrong, but it exists and it's what we have to contend with.
A few comments up you are saying that Steve's problems with the benchmarks game are invalid and the game isn't considered for the sort of comparison Steve (and many others) dislike because it says all over the website that
> These are not the only compilers and interpreters. These are not the only programs that could be written. These are not the only tasks that could be solved. These are just 10 tiny examples.
etc.
But now you are saying that obviously people will do naive comparisons using the benchmarks game. This validates the dislike of widely publicised one-dimensional benchmarks like the benchmarks game (NB this applies to a lot of benchmarks on the internet, the benchmarks game is just a particularly famous example, please don't get too defensive again).
Sure a discussion is taking place, but essentially any discussion about the benchmark game degenerates into either an argument about why the benchmark game doesn't cause one-dimensional comparisons (with the pro-Benchmarks-Game side consistently being overly defensive), or an argument about why using every unsafe corner of language X to basically mimic the fastest C program is perfectly reasonable, idiomatic and common in the real world.
For example, look at the history of D being in the game vs. not. Generally speaking, all of http://benchmarksgame.alioth.debian.org/play.html#languagex is incredibly arbitrary. I don't think that the maintainers of the game have some sort of moral obligation to support things they don't want to, but it is an arbitrary line which influences what kind of benchmark it is.
As another, the domain of the examples, which are largely related to numeric computing and/or memory usage. Which is fine, but isn't going to give me information about other use cases.
> None of the tasks boil down to "Can you write non-idiomatic code to theoretically make this as fast as something else?" -- if you don't want them to!
Yes, they do. Because one man is not an island: people use the raw numbers of the Benchmark Game to claim "X is faster than Y", even if the implementation of X and Y are totally different than what their implementation would be.
> that's been the choice of the Rust community.
I'm actually not thinking of Rust here. I'm thinking more of languages like Haskell, who have implementations that look significantly more like C than like Haskell.
But regardless of if the implementations are idiomatic or not, it means that The Benchmark Game is useless for evaluating if a program I write in a language is going to generally be faster or not than if I did it in another. Because this isn't the code that I'd write: it's code that takes advantage of every little last corner to get as small a number as possible.