Intercoder Reliability Standards: Reproducibility

Data Set

Being able to clearly and accurately describe what data one is analyzing is one of the most important aspects of reproducibility with regard to content analysis. If other researchers are not able to determine what it was that the researcher analyzed, it will not be possible for them to attempt to reproduce the results. It is important that the researcher is very clear about how much data was collected. One also needs to be specific about what was analyzed. For instance, a researcher may be analyzing fashion magazines. In this case, he or she would need to let the reader know if they were analyzing the entire magazine, just the advertisements, or just articles. One would want to make sure they know the exact date range of magazines analyzed and how many different editions came out during that time period. If there was any data that was excluded from analysis or removed for any reason, the researcher would need to report this as well. The researcher should explain why he or she decided to exclude this information. The goal is to provide enough information for someone else to collect the exact same data set.

It is also important that the researcher explain how he or she collected the data set. Explaining how the data were collected allows future researchers to follow the same process. Other researchers should be able to follow the exact same process that a researcher went through in collecting data. This helps to ensure that they will collect the exact same data. For example, in the fashion magazine example, the researcher would need to report whether he or she just collected each magazine issue as it came out, or if he or she contacted the publisher and asked for all of the past issues at once. This may not seem important, but the more detail the researcher can provide about the data set and how it was collected, the easier it will be for other researchers to reproduce the study and collect the same data.

...