Questions and answers about observer bias in RAMP

Refer to the previous post for background on observer bias in RAMP.

Q: What are the options when grappling with cognitive/expectation and sampling biases in manipulative fisheries research experiments under sometimes challenging conditions at sea?

A: Begin by training and calibrating observation. We all recognize vitality when we see animals with high vitality. This recognition is based on rapid visual assimilation of information about several traits including injury, activity, and responsiveness. We cannot separate our cognitive impression of vitality level from the act of observing individual traits and scoring their presence or absence. Presence or absence of reflex actions is scored relative to control animals which have a set of reflex actions consistently present. Reflex actions range from clearly seen through weakening stages to clearly absent. As the animal becomes more stressed and impairment increases, the interaction of impression and scoring observations contributes bias. 

If observers are trained to clearly recognize a suite of real reflex actions in the species of interest, then correctly recognizing the impairment or absence of those reflex actions should be a realistic accomplishment. An experiment to test for the effect of observer bias and variability in scoring reflex actions could be conducted in the lab or field if enough fish and observers are available. Stress some fish (air exposure) to produce replicates over a range of RAMP impairment scores and have the observers sample reflex actions. Blind the study treatments from observers. Estimates for observer bias from stress studies with different species will be useful for improving observer training by identifying protocols that need to be more defined and less subject to observer opinions. Alternatively, Benoît et al. (2010) modeled observer bias as a random factor. 

Q: How can we achieve a blinded experimental design if the experimenter who assigns or is aware of experimental treatments also scores reflex impairment on board (commercial) vessels?

A: Perform some fish experiments on observer bias outlined above and decide how important observer bias is after training with well-defined protocols for testing individual reflex actions. The bias problem may be mitigated by training using clear definitions of present or absent for reflex actions. I will assume that the vessel captain is conducting the experimental fishing treatments. So the captain could be given treatment conditions by the scientist and then could conduct fishing by assigning treatments randomly without the knowledge of the scientist observer. However tow time, soak time, or haul time and catch volume will be apparent to observers. 

Q: Is an observer influenced in his/her ability to score reflexes if, apart from knowing the treatment, also the condition of an organism is evident even before the scoring begins? Is there any option to minimise this?

A: We cannot separate the correlation between overall impression of vitality and scoring reflex actions. However, we can be trained to clearly recognize the presence of reflex actions. Any impairment through weakness, delay, or loss of action is scored absent.  The key method for minimizing observer bias for reflex actions is to clearly establish what the suite of reflex actions look like when they are consistently present in control animals. If presence of a reflex action is difficult or inconsistent to determine then it is not a good candidate for testing. Any deviation from control appearance in action strength or delayed time for action can be considered impaired and scored absent. The goal is to eliminate variability in detection of presence for reflex actions. By sharpening the decision criteria, bias and variability can be reduced. This idea can be tested using the outlined experiment design.

Q: Seeing that vitality assessments of discarded fish in Europe are now being developed in several places is there a need to also quantitatively evaluate the ability of different observers to score reflexes consistently? What would be the best setup for such a training exercise? 

A: As mentioned above, a stress experiment can be conducted to quantify observer bias and consistency.  With enough replicate fish and observers, an air stress experiment could produce fish with varying levels of reflex action impairment. These fish could be sampled by observers with defined criteria and using an experimental design for testing the effects of observer variability and bias. The effect of training could also be evaluated using this design.