Observer bias and RAMP

Cognitive bias (The Daily Omnivore, 2012)

Subjective scores for animal behavior can be biased by observer opinions about experimental treatment differences and resulting outcomes (Tuyttens et al. 2014). The research paper title expresses a fundamental bias of human perception and belief: “Observer bias in animal behaviour research: can we believe what we score, if we score what we believe?” The problem is to separate belief from observation. This may be accomplished by clearly defining and adhering to consistent protocols for behavior observation and analysis.

RAMP relies on subjective scoring for presence or absence of reflex actions or injury types. Control fish have a suite of reflex actions that are consistently and clearly present when tested for. When an observer begins to notice the weakening or complete loss of a reflex action, that action is scored as absent (impaired). There will be variation among observers in the decisions about when reflex actions are impaired and bias will vary with experimental protocol. 

Because RAMP is an aggregate vitality impairment index summed from control reflex actions and potential injury types, a RAMP score includes the observer bias for each included reflex action and injury. Close correspondence of RAMP scores and mortality is noted at low and high scores because observers clearly know when fish are active and when fish are severely injured and impaired. Relationship of mortality and RAMP is more variable at intermediate levels of impairment and mortality in part because observer opinion about impairment is more variable. To reduce observer bias, RAMP for a species must be designed to include reflex actions and injury types that can be clearly separated into present or absent scores. Also experimental treatments can be administered without informing observers.   

Vitality of a stressed fish is readily observed. We are primarily seeing the activity, responsiveness, and injury presented by the animal. The most widely used vitality index in commercial fisheries is for the halibut fisheries of the northeast Pacific Ocean (AFSC Observer Manual 2015), based on Appendices S-X for trawl, pot, and longline fisheries.  For trawl and pot fisheries, three levels of vitality (excellent, poor, and dead) are scored by observing injury types and spontaneous activity, startle response to touch, and operculum clamping. For longline fisheries, vitality is scored by observing injury types. Mortality rates are assigned to vitality impairment scores using tagging experiments (Williams 2014).

Vitality impairment codes (Benoît et al. 2010).

Benoît et al. (2010) constructed a fishery vitality index with four levels of impairment (excellent, good, poor, moribund) that are scored by observing injury types, spontaneous body movement, startle to touch, and operculum clamping. Their vitality index and the halibut vitality index use the progressive increase of injury and impairment of activity to score vitality impairment. Benoît et al. (2010) corrected for observer bias by using a random effects term in their statistical model. 

Reflex actions scored for presence or absence in RAMP for snapper (McArley & Herbert 2014).

The RAMP vitality index alters impairment scoring to only include presence or absence of a larger number of injury and reflex actions. This shift attempts to introduce more information about activity and injury types that may be associated with mortality and to reduce decisions about degree of impairment for individual activity and injury traits. Impairment is observed as a progressive increase in the number of reflex actions that become absent and the number of injury types that become present when compared to control animals. Because observer bias can be introduced in scoring, observer protocols must be well defined with clear rules for presence or absence of traits. Observer judgements about correspondence between experimental treatments and outcomes could also be eliminated by careful experimental design.