In July 2023, the School of Economics at UEA, in association with the Institute for Replication (I4R), had the pleasure of hosting the Norwich Replication Games on the University of East Anglia campus. The Institute for Replication was created to form a permanent network of scientists interested in reproducibility and replicability, including graduate students, academics, and individuals working for governmental agencies and research institutes. The institute aims to promote and generate reproductions and replications of experimental and (mostly) non-experimental studies.
Reproducibility and replicability are vital to science as it helps make science a self-correcting system, allowing us to assess the reliability and validity of the empirical result. Put differently, the purpose of scientific publishing is the dissemination of robust research findings, exposing them to the scrutiny of peers. Replication can improve credibility within empirical research, representing a key input in economic policy design.
The Norwich Replication Games aimed to promote and generate replications and disseminations, strengthen the visibility of researchers at the University of East Anglia and the network of social scientists, and advance the training of the next generation of young economists. The event was a great success with over 45 national and international participants joining together to replicate 11 papers. The findings of the workshop will be summarised in a published meta-paper where all participants of the workshop will co-author the paper. Additionally, Economic Inquiry has issued a call for papers for a special issue, "On the Reproducibility and Replicability in Economics" which will be co-edited by Abel Brodeur, Michalis Drouvelis and Farasat Bokhari.
To find out more about the Institute for Replications, and any future Replication Games, please visit https://i4replication.org/ and for further information on the special issue, see https://weai.org/news/view/50
The Institute for Replication (I4R) was created to form a permanent network of scientists interested in reproducibility and replicability, including graduate students, academics and individuals working for governmental agencies and research institutes. I4R’s aim is to promote and generate reproductions and replications of experimental and (mostly) non-experimental studies.
Reproducibility and replicability are vital to science as it helps make science a self-correcting system, allowing to assess the reliability and validity of empirical result. Put differently, the purpose of scientific publishing is the dissemination of robust research findings, exposing them to the scrutiny of peers. Replication can lead to improved credibility within empirical research, which represents a key input in economic policy design. For instance, the reproduction attempt of Carmen Reinhart and Kenneth Rogoff’s result (and discovery of a spreadsheet error) that rising levels of government debt are associated with much weaker rates of economic growth highlights the key role played by reproducibility in enhancing our understanding of the institutional environments most conducive to free and flourishing societies.
In Spring 2022, I4R organized its first summer school for graduate students in Montreal. In October 2022, I4R organized their first Replication Games in Oslo, Norway. The goal of this workshop (Games) is to bring social scientists together, enable them to learn from one another, and replicate studies. 40 participants will be teaming up to replicate 10 studies (i.e., 10 teams, each replicating a study).
Our goal is to bring the Replication Games to Norwich, UK. The Norwich Replication Games will take place on Monday 10th July 2023 and participants will include faculty members, post-docs and PhD students affiliated with UK and European universities. Graduate students are teamed up with faculty members and senior researchers, ensuring the training of the next generation of young economists. There are no fees for participants.
The 2023 Norwich Games have three specific objectives:
1. To promote and generate replications and dissemination of new knowledge
While the Replication Games are a novel and unique way that I4R developed to generate and promote replications, they are inspired by well-known established events such as the Science/Engineering Olympics. Science Olympics are usually a one-day events where students/faculty compete or collaborate to apply their knowledge of science and engineering. The 2023 Games at the University of East Anglia will also be made up of small teams who work together to apply their knowledge to replicating recently published studies. The replications will then be publicly disseminated, leading to knowledge generation. We are hoping for approximately 40 participants who will replicate 10 studies published in leading economics, finance and political science journals. Our findings will benefit society at large, since strengthening the credibility of science is a natural defense against populism and ignorance.
2. The visibility of faculty members is strengthened through networking and working with other researchers
The visibility of faculty members at the University of East Anglia is strengthened through networking and working with other researchers. Networking of social scientists from the University of East Anglia and international and Canadian researchers is strengthened through both knowledge exchange and professional networking at the workshop. Knowledge exchange in terms of emerging methods to conduct sensitivity analysis, availability of data sources, and the latest research findings contribute to the attendee’s ability to answer policy relevant questions. Professional networking opportunities are numerous and include: introducing graduate students to researchers in their field and introducing junior researchers to more established researchers. Last, participants will all be granted co-authorship to a large meta-paper combining the work of all workshop participants.
3. To advance the training of the next generation of young economists and leadership development
The 2023 Games in Norwich advances training of social scientists by encouraging both graduate students and junior faculty to collaborate and work on a replication exercise with established researchers. Each team will be composed of junior and senior participants. Of note, we expect a large number of UEA PhD students will participate. Graduate students who participate in this workshop will thus benefit from the feedback they receive from both faculty members specialized in their respective field from UK and abroad. Graduate students also benefit from exposure to existing and emerging methods used in their discipline, as the studies to be replicated were published in leading journals in 2022.
We expect a large number of participants for Norwich’s Games, including academics from the national and international scenes. Reserved slots will be allocated to UEA faculty and graduate students if we receive too many applications. This will be a great opportunity for UEA students to collaborate and network with top international researchers.
To facilitate the dissemination of research replicated at the workshops, I4R has a permanent website and discussion paper series. With permission of the participants, replications will be made available after the workshop at the I4R website.
Based on the Oslo and Calgary Games, the 2023 Norwich Games will provide a valuable and successful workshop for social scientists interested in reproductions and replications. While the location will be Norwich instead of Oslo or Calgary, the organising committee and I4R’s leadership remain. Their involvement provides continuity and efficiency. The organizing committee consists of Profs. Abel Brodeur (uOttawa), Anna Dreber (SSE), Fernando Hoces (UC Berkeley) and Edward Miguel (UC Berkeley).
The workshop programme will be similar to the other Games. Researchers will be grouped into teams of 3-5 based on research interests. Each team will have at least one faculty members and will be assigned a study to replicate. Teams may either conduct a robustness replication, which is the ability to duplicate the results of a prior study using the same data but different procedures as were used by the original investigator - it basically involves conducting sensitivity analysis - or recode the study using the raw or intermediate data. They may also do a combination of robustness/recoding.
The goal is not to show that the results are not replicable. The goal is instead to test if the results are replicable to recoding and/or robustness checks. If the results are replicable, great news. If they are not, then it is important to publicly document this.
Before the workshop teams have to familiarise themselves with the data/codes/readme file. They also have to carefully read the paper. A virtual meeting with the organisers before the workshop will allow each team to discuss a gameplan; choosing whether to conduct a robustness or recoding replication or a combination. During the workshop, each team will be working on their replication. The organisers will be providing assistance to each team. Lastly, each team will complete the replication and write a report in the weeks following the workshop. The workshop is scheduled for 10 July 2023.
The Games will be on Monday 10 July 2023 at the University of East Anglia, Norwich, UK. The Games will be held in room 2.03 and 2.04 in the Thomas Paine Study Centre.
To summarise the findings of this workshop, a meta-paper will be written and published. All workshop participants will co-author this large paper. The study and all replications will be made publicly available through I4R’s discussion paper series, and our aim is to publish it in a top general interest journal. The papers to be replicated were/will be published in top journals in 2022-2023, and we expect many teams to publish their single replication/comment in those journals as comments (or in other outlets interested in replications).
Several journals are now publishing special issues for reproducibility and replicability in Economics, including this call for papers by the Economic Inquiry.