We provide a service that provides independent criticism of research in the social sciences.
Early research established the feasibility of paying independent scientists to criticize research. In a pilot - the “Red Team Challenge” - we offered financial incentives to five scientists to find errors in a submission-ready manuscript. Critics found 107 issues, some of which were so substantial that the author delayed submission and ran an additional experiment. Since then, we have developed a database of professional critics and have conducted projects to provide rapid, independent criticism on a wide range of projects. We have also recently begun to provide independent analyses of scientific fraud.
Our service works as follows. Any individual who wishes to receive independent expert feedback or critical evaluation of work in the social sciences can contact us with their aim and budget. We work together to determine an appropriate time frame and to find independent critics who are able to complete the work. Critics sign a confidentiality agreement and a contract specifying terms and expected output. Critics then provide independent evaluations of the research - whether a research plan, manuscript, or published paper. We review critics' comments for professionalism, add our own evaluations (when applicable), and generate a professional final report. When applicable, we also maintain a transparent record of the entire process of criticism.
Goals and intentions
A core value in science is “organized skepticism”, whereby scientific claims are subjected to systematic and institutionalized criticism. This value is so fundamental because systematic criticism allows society to judge whether claims are trustworthy. Yet, the most common approach to criticism, pre-publication peer review, is slow, not transparent, unreliable, and is only available at the end of research projects (once it is too late to correct mistakes). These problems persist because criticism is largely monopolized by scientific journals, who have incentives to rely on volunteer labor and keep review processes closed and unavailable to scrutiny.
Our goal is to build a service that allows anyone to solicit trustworthy scientific criticism at any time, where criticism is independent of journals, and where critics are financially compensated. Anyone should be able to use our service to acquire critical evaluations of research. Such criticism can help researchers to to improve the quality of their planned research, gain confidence in whether they can trust published research, or overcome groupthink by receiving feedback from "devil's advocates" that originate from diverse intellectual backgrounds.