Posts in reliability

The times, they are a-changin’ (3/3)

enero 2nd, 2012 Posted by aside to camera, blog, empirical research, methods, reliability No Comment yet

In the first post of this threefold series, I praised e-journals and suggested that specialization might lead tooverall improvements in journal quality. Let us welcome TC3, a promising addition to the short list of specialized journals within Translation and Interpreting Studies. In the second one, I sketched some advantages of a few digital resources, and of using on-line tools to build a community of practice. Here I would like to address what I think amounts to a precondition for the success of any attempt of building a community of practice, namely setting high and common standards. 

I will go straight to the point: I would like to argue that combining reproducible research with open access may probably be the best option to bring about these standards.

Now you see it, now you don’t

You try to repeat a fascinating piece of research reported in a journal article. You follow it step by step, veeeery carefully, only soon to realize that a lot of missing bits make it unlikely that you are actually replicating the original test by any rigorous accounts, and settle for approximation instead of comparability. Your results are different, but you cannot really hypothesize why. —A colleague asks you for the way you carried out certain details in an old research project. It takes you long to find the appropriate materials, and even longer to reconstruct some of the rationals behind decisions that were obviously taken but perhaps never stated as explicitly as it seems to have become appropriate now. Yeap, sometimes some of us cannot even make sense of our own old materials.

Yet another typical case. You would like to find support for a given hypothesis and the setting you establish to put it to the test actually fits quite well in that of a previous piece of research, so that their data might really be of use to you. But you will never know because many of the conditions were implicit in the published paper, or had not been recorded. You may become unnecessarily stuck with a reduced number of subjects. Testing your new hypothesis in their old materials might have also been a sound previous step to carry out your planned research project, but you can’t because it seems very difficult to get ahold of these materials. You cannot even challenge the interpretation of data that was offered then by reanalyzing data now from a different perspective on the same issue, on the same materials, because they are not available.

These are only some of the problems we have to deal with in our daily research efforts to study translators’ and interpreters’ cognition. Process research methods have not yet been standardized enough, and the information structure and contents of research reports are still lacking field-wide guidelines (where field allows for several readings). In PETRA we are trying to contribute to method standardization by looking at ways to profile texts and subjects. But perhaps starting the other way round, i.e. setting standards for research reporting in future efforts, might simply be a more practical way to reach both goals. That is, setting report standards might prompt agreements on some methodological standards.

A simple strategy to draft tentative report guidelines might be letting information standards emerge from the demands of the readers of research reports. A fuzzy network of researchers who mutually recognize each other as pertaining to the field of translation and interpreting process research might, after a while, reach agreements on initial, tentative report and methodological standards by exchanging information of their works. They might simply do so by spotting new, additional information demands when trying to re-use the information to reach their own goals, until these information needs are streamlined and settled. In any case, it should be an additive and realistic process of study of past projects to be applied only to new ones.

Reproducible research

A publication is not the research project itself, but merely a report of that piece of research. That is why one single research project may lead to many publications. Current information and communication technologies allow for the exchange of all research-related information in digital formats of any kind (first post, again). Today’s digital scholarship comprises the publications plus the complete environment and the full set of instructions applied to carry out the project. Simply because we can? I don’t think so. Rather, we will benefit from accessing the whole lot. For instance, providing additional materials might enlarge the possibilities of other researchers to suggest more solid observations by re-using somebody else’s tools (e.g. questionnaires) and even some data, when possible.

Replicating research projects often yields different results in all fields, not just in ours. Statistics became a discipline to come to the rescue in such scenarios. However, often many differences in replicated results may be ascribed to lack of information in the research report. As I wrote above, digital media let us share a wealth of information which would make our job easier by letting us re-use each others’ data. The information should include—as many, when not in full—actual materials whenever possible, in order to make sure that future approaches will be able to analyze the real data, and not only filtered reports, which would become just one part of the information provided. In other words, publications would become points of access to further information on a research project, including raw materials, which altogether are seen as a compendium. A reproducible-research compendium for translation and interpreting process research might include

1. The research paper

  • Full text (e.g. a PDF)
  • Full bibliographic citation with current publication status (e.g. a BibTeX file)
  • Supporting bibliography (with abstracts, when possible)

2. The experimental setting

  • Explanatory documentations for each factor, if not standardized
  • A list of the parameters, settings, and platforms under which the project was carried out that lead to the published (also unpublished?) results
  • Original texts as presented to subjects
  • Subjects profiles [and those of other parties, such as evaluators, when applicable]

3. The Data

  • Raw output data (log files, video recordings, user activity data, questionnaires, tests, etc.)
  • Criteria used for data cleaning
  • Notes on data cleaning and preparation process
  • Prepared data (files after cleaning)
  • Explanatory documentations for each part of the process of manipulating data

4. Results

  • All the results including high resolution figures, complete tables, full statistical analyses when applicable, etc.
  • Explanatory documentations for each part of the analysis.
Such a compendium might raise concerns about data privacy and the release of potentially confidential information. This may simply be sorted out by choosing a combination of the toughest standards. For instance, we might request for proof that papers in compendiums have cleared their status regarding their publishers’ policies for self-archiving. At the same time, we definitely need to look for ways to record information that will guarantee anonymity while providing a maximum of data with the consent of the participants. At PETRA we are now exploring the possibilities of on-line data collection through a website (+ computer application) to let subjects participate from their usual working environments. Apart from our main goals, we are also interested in finding out whether there are any effects of going distant and on-line on the behavior of participants in general and on ecological validity, in particular. This includes the availability of participants, their willingness to take part in the project, and other factors such that might be affected by the research situation.

Establishing standards for a compendium such as the one sketched above is probably still far too ambitious a goal, but that’s what goals are for. Several items in the checklist above need further elaboration. Variability is not easy to reduce when studying such complex tasks as those carried out by translators and interpreters. Clarifying the contents and extent of such items is as good a starting point as any to advance to standardization. In any case, we need to set the tone and fine-tune our efforts to jointly perform the score. Staying committed to just the information provided by papers, books and book chapters is like trying to stick to your old vinyl records. We all love vinyl records, but just try to carve out a ringtone for your phone from your old 33 rpm’s or to stream them over from your stereo into your computer or your tv. Just find that song by what’s-his-face-again (musical categories have really fuzzy edges) or choose to listen music from twenty different records in a row without having to attend the device after each song. In translation and interpreting process research, we seem to be sticking to scholarly vinyls. However, project documentation initiatives such as TransComp are steps in the right direction.

Open access

Once an initial standard has been established, the next step is broader dissemination. Open access offers several advantages. From a technical scope, it allows for comparison, fosters rigor in both methods and reports, and may ease junior researchers into replicating projects. If the how-tos are completely available, repeating a research project is a suitable and manageable goal for MA candidates. A wider scope on the advantages of open access may also take into account the benefits of extending the community of practice to new members and promoting good practices at once. Gatekeeping should become less tricky when more supported by both evidence and general agreement. Also, compendiums might make science more transparent to the public, and accountability is one of the sources of legitimacy for scientific endeavors.

New research projects offering compendiums could be made available in one or several dedicated website repositories, only when abiding to standards or properly challenging them. New candidates to enlarge, reduce or modify the standards for research reports would need to prove the relevance of their proposal for new or established research goals and also their fit and interaction with the rest of variables already in the standard. Successive versions of this standard would accommodate new information demands. In other disciplines there are frequent splits in standards for competing frameworks, so we may expect standards to vary in different types of projects. That's ok. Probably, new communication venues might specifically target standard-abiding research reports.

There may be other strategies, but reproducible research and open access seem both to be taking roots in neighboring disciplines, including linguistics, reading research, writing research, cognitive science, and psychology, so perhaps it is worth to give them a try, ‘cause los tiempos están kambiando.
by R. Muñoz

Three good reasons for carrying out a pilot study

junio 3rd, 2011 Posted by blog, empirical research, functionality check, pilot study, preliminary results, reliability, The sorcerer's apprentice No Comment yet

Both in experimental and in descriptive research, a pilot study is a small-scale version or trial run of the main study. A pilot study has to be carried out under the same conditions of the main study. Otherwise carrying out a pilot study would not make much sense, since the main goal is to trace possible sources of error to avoid those in your main study. There are at least three good reasons for carrying out a pilot study before you carry out the full-scale experiment:


    1. Functionality check of the study design by testing


a) research tools and methods regarding adequacy

When choosing tools and methods for data collection, you should always consider your study aim(s): What do you really want to measure in your study? By which means can you achieve these aims? With a pilot study, you can check whether your methods and tools are optimal for your purposes.

b) study feasibility

A pilot study will show you if your study design works the way you planned. Maybe you will have to introduce some modifications, for example, regarding the experimental setting.

    1. Collecting preliminary results

A pilot study will give you some first tentative results which may show at least the potential trends in the future outcome of the main study. This will let you read more about the possible ways to interpret these results.

  1. Increasing research reliability

By doing a pilot study, the reliability of your research project will increase. Other researchers and (maybe even funding) institutions maybecome more interested in your project.

Take your time for a pilot study. It is one of the best investments of time and effort in order to make an excellent main study trial.

Further reading

Altman, D., Burton, N., Cuthill, I., Festing, M., Hutton, J. & L. Playle (2006). Why do a pilot study. National Centre for the Replacement, Refinement and Reduction of Animals in Research. URL:

Gilbert, N. (2001). The importance of pilot studies. Social Research Update, 35, URL:

Neunzig, W. (2002). Estudios empíricos en traducción: apuntes metodológicos. In F. Alves (Hrsg.): O proceso de traducão. Cadernos de Traducão, 10, 75-96.


by P. Klimant

Recent Comments