Back in undergrad, one of my toxicology courses spent a full class on just this topic (calling bullshit on scientific studies). We went through 5 different papers from prestigious journals and identified the issues that make conclusions shaky.
Fascinating stuff. Now whenever I see some extraordinary claim in a paper I automatically assume it's wrong until proven otherwise. Sadly I'm often correct.
In my CS MSc program, my research group had a weekly paper reading group. This seems to be a common thing. What I'm not sure about is how common the general outcome was. We had one faculty member in the group that was excellent at identifying methodology errors and raising them as discussion points. I'd guess about 1/2 of the papers we read, he was able to find something to pick on. Sometimes small, sometimes enough to bring the whole paper into question. Very great learning experience and helped me hone my bullshit detector.
Fascinating stuff. Now whenever I see some extraordinary claim in a paper I automatically assume it's wrong until proven otherwise. Sadly I'm often correct.