Elements of vitality testing and delayed mortality in fisheries

Conceptual diagram outlining elements for vitality testing and delayed mortality in fisheries. Fish are captured and environment sampled. Fish become stressed which is measured as impairment from control health by observing reflex actions and injury types. Stressed fish are held for captive observation to determine delayed mortality. Bias and error can be introduced by initial impressions of vitality before testing reflex actions and injury, by differing observer scoring opinions, and by holding conditions that are stressful for the fish. 

Scoring vitality impairment is most difficult when observer decision is used. Training observers is a key part of RAMP development. Reflex actions (RA) are clearly present in control animals, and observers do not need decisions to score present. As impairment increases, scoring RA requires increasing observer decisions about whether sampled RA are present. The decision can be based on how control RA appear to trained observers. Each observer will have different opinions that can be influenced by their initial impressions of the animal and of the stressor treatments the animal has been exposed to.

Initially after stress induction, RA impairment increases and mirrors stress levels, while mortality is not evident. When animals reach a critical impairment level, replicates begin to show mortality, which increases rapidly over small changes in RA score. At highest levels of impairment decisions are less frequent as the animal ceases general movement and responsiveness.

Questions and answers about observer bias in RAMP

Refer to the previous post for background on observer bias in RAMP.

Q: What are the options when grappling with cognitive/expectation and sampling biases in manipulative fisheries research experiments under sometimes challenging conditions at sea?

A: Begin by training and calibrating observation. We all recognize vitality when we see animals with high vitality. This recognition is based on rapid visual assimilation of information about several traits including injury, activity, and responsiveness. We cannot separate our cognitive impression of vitality level from the act of observing individual traits and scoring their presence or absence. Presence or absence of reflex actions is scored relative to control animals which have a set of reflex actions consistently present. Reflex actions range from clearly seen through weakening stages to clearly absent. As the animal becomes more stressed and impairment increases, the interaction of impression and scoring observations contributes bias. 

If observers are trained to clearly recognize a suite of real reflex actions in the species of interest, then correctly recognizing the impairment or absence of those reflex actions should be a realistic accomplishment. An experiment to test for the effect of observer bias and variability in scoring reflex actions could be conducted in the lab or field if enough fish and observers are available. Stress some fish (air exposure) to produce replicates over a range of RAMP impairment scores and have the observers sample reflex actions. Blind the study treatments from observers. Estimates for observer bias from stress studies with different species will be useful for improving observer training by identifying protocols that need to be more defined and less subject to observer opinions. Alternatively, Benoît et al. (2010) modeled observer bias as a random factor. 

Q: How can we achieve a blinded experimental design if the experimenter who assigns or is aware of experimental treatments also scores reflex impairment on board (commercial) vessels?

A: Perform some fish experiments on observer bias outlined above and decide how important observer bias is after training with well-defined protocols for testing individual reflex actions. The bias problem may be mitigated by training using clear definitions of present or absent for reflex actions. I will assume that the vessel captain is conducting the experimental fishing treatments. So the captain could be given treatment conditions by the scientist and then could conduct fishing by assigning treatments randomly without the knowledge of the scientist observer. However tow time, soak time, or haul time and catch volume will be apparent to observers. 

Q: Is an observer influenced in his/her ability to score reflexes if, apart from knowing the treatment, also the condition of an organism is evident even before the scoring begins? Is there any option to minimise this?

A: We cannot separate the correlation between overall impression of vitality and scoring reflex actions. However, we can be trained to clearly recognize the presence of reflex actions. Any impairment through weakness, delay, or loss of action is scored absent.  The key method for minimizing observer bias for reflex actions is to clearly establish what the suite of reflex actions look like when they are consistently present in control animals. If presence of a reflex action is difficult or inconsistent to determine then it is not a good candidate for testing. Any deviation from control appearance in action strength or delayed time for action can be considered impaired and scored absent. The goal is to eliminate variability in detection of presence for reflex actions. By sharpening the decision criteria, bias and variability can be reduced. This idea can be tested using the outlined experiment design.

Q: Seeing that vitality assessments of discarded fish in Europe are now being developed in several places is there a need to also quantitatively evaluate the ability of different observers to score reflexes consistently? What would be the best setup for such a training exercise? 

A: As mentioned above, a stress experiment can be conducted to quantify observer bias and consistency.  With enough replicate fish and observers, an air stress experiment could produce fish with varying levels of reflex action impairment. These fish could be sampled by observers with defined criteria and using an experimental design for testing the effects of observer variability and bias. The effect of training could also be evaluated using this design.

Observer bias and RAMP

Cognitive bias (The Daily Omnivore, 2012)

Subjective scores for animal behavior can be biased by observer opinions about experimental treatment differences and resulting outcomes (Tuyttens et al. 2014). The research paper title expresses a fundamental bias of human perception and belief: “Observer bias in animal behaviour research: can we believe what we score, if we score what we believe?” The problem is to separate belief from observation. This may be accomplished by clearly defining and adhering to consistent protocols for behavior observation and analysis.

RAMP relies on subjective scoring for presence or absence of reflex actions or injury types. Control fish have a suite of reflex actions that are consistently and clearly present when tested for. When an observer begins to notice the weakening or complete loss of a reflex action, that action is scored as absent (impaired). There will be variation among observers in the decisions about when reflex actions are impaired and bias will vary with experimental protocol. 

Because RAMP is an aggregate vitality impairment index summed from control reflex actions and potential injury types, a RAMP score includes the observer bias for each included reflex action and injury. Close correspondence of RAMP scores and mortality is noted at low and high scores because observers clearly know when fish are active and when fish are severely injured and impaired. Relationship of mortality and RAMP is more variable at intermediate levels of impairment and mortality in part because observer opinion about impairment is more variable. To reduce observer bias, RAMP for a species must be designed to include reflex actions and injury types that can be clearly separated into present or absent scores. Also experimental treatments can be administered without informing observers.   

Vitality of a stressed fish is readily observed. We are primarily seeing the activity, responsiveness, and injury presented by the animal. The most widely used vitality index in commercial fisheries is for the halibut fisheries of the northeast Pacific Ocean (AFSC Observer Manual 2015), based on Appendices S-X for trawl, pot, and longline fisheries.  For trawl and pot fisheries, three levels of vitality (excellent, poor, and dead) are scored by observing injury types and spontaneous activity, startle response to touch, and operculum clamping. For longline fisheries, vitality is scored by observing injury types. Mortality rates are assigned to vitality impairment scores using tagging experiments (Williams 2014).

Vitality impairment codes (Benoît et al. 2010).

Benoît et al. (2010) constructed a fishery vitality index with four levels of impairment (excellent, good, poor, moribund) that are scored by observing injury types, spontaneous body movement, startle to touch, and operculum clamping. Their vitality index and the halibut vitality index use the progressive increase of injury and impairment of activity to score vitality impairment. Benoît et al. (2010) corrected for observer bias by using a random effects term in their statistical model. 

Reflex actions scored for presence or absence in RAMP for snapper (McArley & Herbert 2014).

The RAMP vitality index alters impairment scoring to only include presence or absence of a larger number of injury and reflex actions. This shift attempts to introduce more information about activity and injury types that may be associated with mortality and to reduce decisions about degree of impairment for individual activity and injury traits. Impairment is observed as a progressive increase in the number of reflex actions that become absent and the number of injury types that become present when compared to control animals. Because observer bias can be introduced in scoring, observer protocols must be well defined with clear rules for presence or absence of traits. Observer judgements about correspondence between experimental treatments and outcomes could also be eliminated by careful experimental design.