Environmental Economics

The cromulent economics blog

recent posts

About the authors

Replication vs Duplication: What’s the difference?

Your thoughts on this are appreciated.

There's been a good bit of news lately on the issue of replication in various disciplines.  Psychology may or may not be in the midst of a replication crisis. Experimental economics bats around .600 when is comes to replicability in a very small sample, which is good for baseball, but unclear just how good it is for the state of science in experimental economics.   

Replication is something that has interested me for a while.  A lot of my own research ideas come from trying to figure out and replicate what others have done (copying or standing on the shoulders of giants?).  So for the final project in my PhD Applied Econometrics class this year (1st year PhD class), I decided to have them perform a replication:

Final Project: Replication, review and extension of existing results

1. Identify a published peer-reviewed journal article for which you would like to replicate, review and extend the results.

  • The article should be published in a respected economics journal. If you don’t know what qualifies as a respected economics journal, talk to faculty members who are experienced at publishing, or try Googling “journal rankings in economics.”
  • The article must have estimable econometric results (don’t choose a theoretical paper).
  • The article must have data available for replication purposes. Most journals have a formal data policy (for example, here is the data policy for the American Journal of Agricultural Economics: http://www.oxfordjournals.org/our_journals/ajae/for_authors/general.html). Some journals require authors to make data available to interested readers, others do not. YOU MUST FIND AN ARTICLE FOR WHICH DATA IS AVAILABLE. As an example, the AJAE provides links to data with their on-line content (http://ajae.oxfordjournals.org/content/98/1.toc).
  • And most important, the article should interest you. Don’t just pick the first article you find that has data available and you think you can use to satisfy the requirement. READ. A LOT. And then find an article that you’re truly interested in understanding.

2. Write a review of the chosen article in which you:

  • Replicate, or at least attempt to replicate, the main result(s) of the paper. To replicate the results, you should write and original program that re-estimates at least one model from the paper. DO NOT USE PROGRAMS PROVIDED BY THE AUTHORS, YOU MUST WRITE AN ORIGINAL PROGRAM TO REPLICATE THE RESULTS OF SOMETHING IN THE PAPER. You should do your best to replicate the main finding by estimating the model. If you can replicate the results, report your results. If you come close, report that. If you can’t replicate the results, explain why (is it because you can’t figure it out, or is the paper wrong, or some combination of those?).
  • Critique the paper (comment on the motivation, theory, model, estimation, interpretation, data, any issues you find with the data…). Reference other published papers as appropriate.
  • Give a brief description of how you would extend the results of the paper given the chance. What would you do different? What could you do better? What would you do next?

Seems like a reasonably interesting project for a first year PhD students, but the way I have described the project brings a question to my mind:  Is this replication or duplication?  And is there a difference? And yes I know that is two questions.

I understand replication in the lab sciences (including experimental economics).  Simplified: Rerun the experiment under the conditions laid out in the original and see if the results hold up.  This even goes so far as questions in valuation studies which John and I regularly take on.  We can always run another survey and see if the results hold up.  But how does replication work in a truly applied setting, where the experiment (or study conditions) is not replicable.  If we simply take the authors' data set and rerun the same models that the author runs, is that replication, or duplication?  Sure we can learn something about the assumptions made in collecting data, cleaning data, deriving an estimable model from theory, model specification, statistical assumptions, etc, but do we really learn anything about the replicability of the results by simply redoing exactly what the authors did with the same data? 

Is that replication or is that duplication?

And does it matter?

Thanks in advance for your thoughts.

 

Posted in
  1. Ironman Avatar
    Ironman

    Duplication is be the situation where you have the same people repeating the same study in all of its aspects (subjects, data, methods, procedure, etc.)
    Replication would be the situation where you have different people repeating the same study in all of its aspects. Replication makes it possible to determine the influence of the investigators or analysts upon the study’s results – kind of like the Gage R&R studies that are commonly done in manufacturing settings. The ability to successfully replicate a study’s results should provide confidence in the original study’s methods and also the abilities of both sets of investigators.

    Reply
  2. Benjamin Anderson Avatar
    Benjamin Anderson

    Since students are using a pre-existing dataset provided by the authors, this seems somewhere in between duplication and replication. Maybe “partial replication”?
    Perhaps a full replication could entail re-running experiments, re-sampling a population for a survey, re-constructing a dataset, etc. along with the data analysis. Doing this would require the researcher to make choices on the appropriate sample as well as variable measurement, even if it is done with pre-existing, secondary data. A partial replication would focus on the modeling assumptions and estimation methods under the assumption that the underlying data is valid. What is done under a “partial replication” is also completed under a full replication, but the data collection process is not.
    One thing that remains unclear to me is whether we should care about replication for internal validity reasons or for external validity purposes. The experimental economics replications seem focused on internal validity (can the original results be found using a similar sample and without significant changes to experimental design). Much of the debate on the psychology replications seems to focus on external validity and whether failing to replicate using different samples and/or different experimental designs reveals anything meaningful about the original study.
    The first (replication) part of the assignment seems focused on assessing internal validity, the second (critique) portion seems to be venturing into concerns of external validity. I think most replications in economics follow a similar structure and can be a useful tool in assessing the robustness of results to (minor) changes in the modeling assumptions or methodology.

    Reply
  3. David Pannell Avatar
    David Pannell

    I think “replication” is a reasonable word for this, but it’s not replication in the sense used by, say, biological sciences. What you are asking them to replicate is the statistical analysis, not the generation of the data. It would help to be a bit more specific and say “statistical replication” rather than just “replication” I think.
    I don’t really like “duplication” in this context. It doesn’t convey any sense of checking or testing the original research.
    An alternative word could be “audit” (as used by Steve McIntyre on the Climate Audit web site). I think this conveys a good sense of what the activity is about. I guess auditing is potentially a bit broader, including detailed checking of the data as well. But then again, that should probably be done as part of statistical replication.

    Reply
  4. Ironman Avatar
    Ironman

    I think auditing would be more expansive than either replication or duplication, since that would more likely entail challenging data, methods, assumptions, etc. – things that can impact results. You might be introducing something new in that process (say sensitivity analysis) that gets more into novel analysis rather than limiting the scope of the effort to simple replication or duplication. It’s really something that you would get into after a successful replication.

    Reply
  5. Timtaylor Home Avatar
    Timtaylor Home

    Checking the results in the paper is one thing, thinking about whether the models are the best ones to use (given that some of the papers may be older and new techniques may now be available)…

    Reply
  6. willwheels Avatar
    willwheels

    I like the language used in the replication section of the Journal of Applied Econometrics (http://onlinelibrary.wiley.com/journal/10.1002/(ISSN)1099-1255/homepage/News.html#replication): wide versus narrow replication. Taking the author’s data and re-running their models is a narrow replication. Once you start extending, you are conducting a wide replication.
    I’d also note that if your students pick an article from the JAE’s list, they could submit to their replication section.

    Reply
  7. willwheels Avatar
    willwheels

    p.s. Have you considered letting them use the original code, perhaps at the half-way point and after checking with you? In my experience, not everything is documented in the article and the code can really help.

    Reply

Leave a Reply

Discover more from Environmental Economics

Subscribe now to keep reading and get access to the full archive.

Continue reading