Concern about potential observer bias in measures for vitality

A paper by Benoît et al. 2010 summarized concerns about potential observer bias for scoring fish vitality.  The writers scored fish vitality using a "semi-quantitative measure based on four code levels of activity and injury":

A similar vitality scoring system with three levels of code; excellent, poor, and dead, is used to measure Pacific halibut bycatch viability in NE Pacific fisheries; see Appendices Q, R, S, T, U, and V in the 2012 Fisheries Observer Manual.  Manire et al. (2001) and Hueter et al. (2006) used a condition scoring system for sharks consisting of five levels of code; good, fair, poor, very poor, and moribund. Enever et al. 2009 used a condition scoring system for skates consisting of three levels of code; good, moderate, poor.

On their vitality method, Benoît et al. 2010 state:

"Because the scoring is based on a rapid visual and tactile assessment by observers, a large number of individual fish from a variety of fishing sets can be sampled. To the extent that observers obtain samples that are representative of the activities of a fishery, the observed frequency distribution of fish among vitality levels will reflect the overall survival potential of discarded fish, integrating over all of the various factors contributing to that survival. Furthermore, these studies typically yield sufficient sample sizes and contrasts in the factors affecting survival to allow for correlative analyses over the relevant ranges of a number of those factors. The principal disadvantage of these studies is the somewhat subjective nature of the scoring criteria, which could lead to differences between observers in their vitality assessments. A second disadvantage is the need for additional studies to relate vitality scores to eventual survival."

Their solution to potential observer bias is to use an appropriate statistical model for vitality:

"The study addresses the two disadvantages of vitality-score studies listed above. We propose the use of a mixed effects multinomial linear model based on proportional-odds (e.g., Agresti 2002, pp. 513–515), which is appropriate for modelling the ordinal vitality data and is a useful approach for addressing observer scoring subjectivity. We use this model to evaluate the effect of relevant factors believed to influence the survival potential (e.g., gear type, fish size, set duration, handling time) for eleven fish taxa. We also present analyses of an experimental study aimed at relating vitality codes to eventual survival."

From the perspective of RAMP, sources for potential observer bias are addressed by shifting measured variables from general observations of activity and injuries to observations of specific reflex actions (and injuries where appropriate). These specific reflex actions can be consistently quantified as present or absent and combined in a quantitative reflex impairment score, RAMP. Studies of potential observer bias for scoring reflex actions can be made by comparing RAMP scores among teams of observers in systems of interest.  

A second disadvantage for vitality scores is that they are not direct measures for animal fitness. This seeming deficit is a key need in all research on measures for vitality and stress.  Scores for animal activity, injury, physiology, or reflex actions can be used directly to compare the relative effects of stressors on stress induction. Ultimately these vitality scores and other measures of stress can and should be linked to fitness outcomes that cannot be observed directly, i.e., delayed mortality or recovery.  Tagging animals and tracking their fates in open or closed systems can supply information for linking vitality measures to fitness outcomes over a wide range of temporal and spatial scales in individuals and populations.