Home
Verify
Use cases
The problem
Records
About
Docs
Download
Contact
Verify a report
The problem
A collection of public posts on what researchers keep running into. Select a note to read it.
+
−
Fit
List view
solves
partly
no line: not solved
K-Veritas
Can't reproduce
Lost code
Numbers changed
Cherry-picked runs
Reviewers can't check
Benchmark claims
Fabricated results
AI-written
“But for GPT based papers, what you’re actually testing could change without you knowing. There’s no way to know if a paper is reproducible at all.”
Hacker News · 2023-12
“I could not reproduce the results; in fact, I could not get anywhere close to the reported scores.”
OpenReview · ICLR 2026 review · 2025-10
“The results are not reproducible with the provided code. After fixing minor errors, I attempted to replicate the results using the code in the appendix, but I could not achieve the reported outcomes.”
OpenReview · ICLR 2025 review · 2024-11
“The number of times I have spoken to someone and they can't reproduce their results because of some combination of code and config drift that they can't resolve... infuriating”
Bluesky · 2026-09
“It seems to me that he is also not able to reproduce his own result and his idea does not even beat the baseline.”
Stack Exchange · 2024-07
“We are unable to reproduce the results because there are several implementation details are not mentioned in the paper. ... We now feel angry and start suspecting the authenticity of their reported results.”
Stack Exchange · 2014-07
“Most papers out there are not reproducible because either the code or the data is not publicly available. Somehow this is perfectly acceptable.”
Hacker News · 2023-11
“If you're lucky, someone releases code+data associated with their published paper. If you're really lucky, that code and data is in the same state as it was in the published paper. If you're really really lucky, someone besides the author can get it to run.”
Hacker News · 2024-07
“The provided implementation is not accessible. (Error: "The repository is expired") In the current state, results are not reproducible since key hyperparameters are missing.”
OpenReview · NeurIPS 2024 review · 2024-07
“In the new RECLAIM preprint, the best agent reproduced 41% of results in the tier with code, data and weights available. Without code, the best rate was 15%.”
X · 2026-09
“In the year of our lord 2026, if your paper still has “data and code available from the authors upon reasonable request” there should be an automatic poop emoji next to your OrcID icon.”
Bluesky · 2026-08
“Now, the paper has spent a long time in review, and that grant has expired, the cloud account has been closed, and all of the data has been deleted (although I still have the code).”
Stack Exchange · 2022-07
“The paper wasn't very good- all the results were bad. Overnight, my advisor rewrote all the results of the paper, partly juicing the results, and partly obscuring the problems, all while glossing over the limitations.”
Hacker News · 2026-01
“The experimental results will be different from my thesis results if I repeat the experiment (because I have to collect the dataset again from websites, and the image dataset will change).”
Stack Exchange · 2022-06
“Eventually the first author responded with just a few lines saying: “The hyperparameters to use are ....” People ran with those parameters and replicated the results. However, the parameters he gave disagree with those reported in the paper.”
Stack Exchange · 2015-08
“Even the original author sometimes couldn’t train the same model and get similar results!”
Blog · 2018-03
“I've been burnt way too many times by fancy new methods that claimed improvement, where I spent a ton of effort to implement them, and they ended up being poop.”
Hacker News · 2025-04
“Not denying that these advances are impressive, but it is important to consider that this is a cherry-picked result. ... What you won’t see is how many others they had to try to get this result.”
Hacker News · 2026-05
“Because the variance can be uniformly high, making it difficult to properly judge the improvement of one method vs the baseline method: did you actually improve, or did you just get a few lucky seeds?”
Hacker News · 2022-12
“The "Searched best seed" related rows in Tables 1 and 2 should be removed. Optimized seed results are rather uninformative, since the seed is not a tunable hyperparameter.”
OpenReview · ICLR 2025 review · 2024-11
“In robotics manipulation, we see many cherry picked demos but no standardized benchmarks.”
X · 2026-04
“Friendly paper writing advice: If you report results on 4 scenes from a 6 scenes dataset, you make it pretty obvious that you cherry picked scenes, presumably based on your results.”
Bluesky · 2024-11
“It seems that this could be done very easily: for example, one could cherry pick the items of the data set to make it seem like an algorithm has 90%+ accuracy while in reality it's about 60%.”
Stack Exchange · 2017-04
“If I need to report both the accuracies, can I select the seeds that give good validation accuracy and a good testing accuracy? What if I use the best test accuracy I got and report that in the paper?”
Stack Exchange · 2022-02
“He says this is okay because the numbers in the paper don't matter as much as the high level message.”
Stack Exchange · 2016-10
“Reviewers are just people reading papers, if it's hard to reproduce a paper's results, they can't verify that they are correct.”
Hacker News · 2019-02
“They can't verify that the data collected actually was collected without literally watching over the shoulder of the researcher while the researcher was collecting it, which is obviously not feasible.”
Hacker News · 2018-10
“In rebuttal we did exactly what reviewers asked, but AC rejected saying: "it is not certain whether this new experiment performed in a short period of time was done accurately." If can reject by simply not trusting rebuttal, why having it?”
X · 2022-05
“Writing neurips reviews has killed my remaining faith in the AI community. Do people not care whether things are actually useful? Or have any integrity or willingness to transparently validate?”
Bluesky · 2026-06
“Common structure: evaluators can't access ground truth from the layer below. - Benchmark judges can't verify answers - AI reviewers can't check if experiments ran”
Bluesky · 2026-02
“In the results section, I can ask some questions about why this or that is so, but how can I judge if the simulation was really performed or these are just fabricated graphs?”
Stack Exchange · 2019-01
“Do most journals / conferences just "trust" that people who submit the paper really implemented the theory and got those exact results?”
Stack Exchange · 2014-06
“Everybody has incentives to manipulate benchmark results to show their models in the best light.”
Hacker News · 2026-06
“It is *incredibly* easy to game the LLM benchmarks. Training on test set is for the rookies.”
X · 2024-09
“A model was fine-tuned on the test set and managed to hit #2 on the Hugging Face leaderboard.”
X · 2024-04
“in 2026 every single AI company will claim SOTA on random benchmarks”
X · 2025-12
“You are guaranteed to fight toxic, low quality reviews and spend time beating meaningless “SOTA” benchmarks on random datasets.”
Bluesky · 2025-05
“I’ve noticed that the benchmark scores I get are consistently different from the ones reported in their tech reports or papers — sometimes by a wide margin.”
Hugging Face forum · 2025-05
“When I was a graduate student I discovered some fabricated results and was told in no uncertain terms that if I pressed the matter that my career would be over before it began. So I kept my mouth shut.”
Hacker News · 2019-08
“Peer review does not even attempt to control for fabricated experimental results or other data.”
Hacker News · 2021-03
“Create paper and figures with AI. Sell authorship. Submit to a journal with a bribed editor. Fake peer review reports.”
X · 2026-04
“Only 4 papers have been retracted. Papers are full of fake data, yet they still get published.”
Bluesky · 2025-09
“My supervisors are secretly suggesting me to somehow put some imaginary values in at this point, but that feels heartbreaking.”
Stack Exchange · 2022-05
“The paper fails to include an LLM usage statement. I have strong concerns that this paper was partly written by an LLM, and the authors did not declare its use.”
OpenReview · ICLR 2026 review · 2025-10
“> LLM writes an entire paper > two LLMs reviewed and gave 8 (accept, good paper) (if you read the review, it LITEARLLY REPEATS THE ABSTRACT) When will we entirely disregard these conferences for good?”
X · 2025-11
“I recently came across an ICLR submission whose reviews all sound like an LLM, have extremely high scores and confidence 5. More importantly than low-quality reviews, I suspect collusion between the reviewers and the authors to game the system.”
X · 2023-11
“In my AC lot of NeurIPS 2026 papers: - 5 out of 8 submissions have 2+ hallucinated references and will likely be desk rejected. Citations are easy to verify and hallucinations can be caught. You can only imagine the slop elsewhere...”
X · 2026-08
“Just had my first encounter with a review generated by ChatGPT instead of a person and I am horrified.”
Bluesky · 2025-11