The idea that you get a lot of just marginal significant results due to "P hacking" , i.e. minor fiddling with excluding outliers or picking a certain hypothesis test over another is probably true. Then there is positive publication bias i.e. only publishing positive findings.
However looking in the abstract doesn't really address either of these .. as even absent either P hacking or positive bias you would still expect the abstract to contain the selected highlights (i.e. positive findings) from the paper. It is the bit of the paper where you really should have positive selection bias!
If the paper has lots of negative tests (and most biological papers will report lots of negative control p values) these aren't picked up. A better way to see this problem (and I believe it is a problem) is to look at the whole paper and view the peak of margin results) compared to the whole set of p values.
That's a common problem with concluding things from automated literature analysis. Often the decision to scan only abstracts isn't an intentional experimental-design decision, but made due to "data of convenience": the researcher has easy access to a machine-processable set of plaintext abstracts, but not easy access to a similarly easy to work with set of full papers. Therefore, abstracts are analyzed!
Another potential confound for longer-term analyses is that the form of abstracts is not constant over the years: abstracts in the 1970s and 2010s aren't written in the same ways, and have different norms for what to include and how to include it. Among other things, the form of abstracts has gotten somewhat more structured/boilerplate, which is one reason I suspect they are finding an increase in all hits for their boilerplate search query.
>absent either P hacking or positive bias you would still expect the abstract to contain the selected highlights (i.e. positive findings) from the paper.
Sure, provided that the reported p-values for positive findings have been corrected appropriately for any multiple comparisons. Abstracts should summarise but not mislead...
However looking in the abstract doesn't really address either of these .. as even absent either P hacking or positive bias you would still expect the abstract to contain the selected highlights (i.e. positive findings) from the paper. It is the bit of the paper where you really should have positive selection bias!
If the paper has lots of negative tests (and most biological papers will report lots of negative control p values) these aren't picked up. A better way to see this problem (and I believe it is a problem) is to look at the whole paper and view the peak of margin results) compared to the whole set of p values.