A new family of dissimilarity metrics for discrete character matrices that include inapplicable characters and its importance for disparity studies

1. Introduction

The study of morphological diversity, or disparity, has provided invaluable insight into the evolution of clades, especially when changes in disparity are discordant with changes in taxonomic or ecological diversity [1–4]. Disparity studies are often based on sets of continuous characters, such as measurements or landmarks, that describe the shape of particular traits [1,5]. As taxonomic scope increases, however, the number of potential traits that are found in only a subset of taxa increases. Discrete character data are useful in such situations because, unlike continuous characters, it is possible to code for absence, and to code characters that only apply to a subset of taxa (i.e. hierarchical characters that describe variation in a trait that is not shared among all taxa in the study). For taxa where hierarchical characters cannot be coded, it is typical to treat the entries as missing data. Conflating hierarchical characters and missing data, however, creates a conceptual problem. In such situations, some characters cannot be coded because the primary feature, such as feathers, is absent, and secondary characters, such as the colour or composition of the feathers, make no sense without the presence of the primary character [5,6]. In contrast, with missing data, some characters cannot be coded because while the character describes a feature the taxon has, it was not possible to collect the data needed to code the character accordingly.

The treatment of hierarchical characters as missing data also creates a practical problem. Specifically, it is possible for a pair of taxa that share more primary characters to be assigned a larger dissimilarity than a pair of taxa with fewer shared primary characters because of the undue influence of the unshared secondary characters. For a simple running example of this, see the ‘Metrics’ section below.

Only a few papers have previously approached the problem of reranked dissimilarities when hierarchical characters are present. Kendrick & Proctor proposed to weight primary characters according to the number of secondary characters [7,8]. This weighting scheme was applied by adding a weight of unity to the primary character for each secondary character that described it. Although originally described in the context of the simple matching coefficient and Jaccard’s similarity, this weighting scheme can easily be implemented using Gower’s coefficient (electronic supplementary material, appendix 1). Williams [9] demonstrated that Kendrick & Proctor’s proposal was numerically sound but suggested reducing the contribution of each primary character in proportion to the number of secondary characters describing it; this avoids large, essentially arbitrary, weighting factors [9]. Gower [10] codified the approach (eqn 9 and 10, p. 863) as part of his paper introducing a generalized coefficient of similarity/dissimilarity that has since become referred to as ‘Gower’s coefficient.’ Dissatisfied that Gower’s coefficient produced non-Euclidean matrices with mixed character types, Wills [11] developed a generalized Euclidean distance (GED) coefficient. Missing data (and thus also inapplicable characters) are accommodated by replacing incalculable per-character dissimilarities with the average per-character dissimilarity for each pair of taxa.

In addition to the example provided by Kendrick & Proctor, we know of only a few empirical attempts to apply any of these strategies [12–14]. This may be because it can be difficult to define primary and secondary characters, or because authoritative sources have recommended that workers avoid secondary characters entirely (e.g. [15]). These reasons are unsatisfying: the need to make explicit homology statements in character creation frequently results in inapplicable characters, and coding schemes that avoid the creation of inapplicable characters overweight absences and create logical dependencies [16,17].

Inspired by previous efforts, we introduce a new family of metrics that proportionally weight the primary characters by the secondary characters that describe them using a tuning parameter α. The parametrized family includes Gower’s approach [10, eqn 9, p. 863] for when α = 1. We show that it is a metric (electronic supplementary material, appendix 2) and examine its effectiveness on an empirical dataset [18]. We find that the inclusion and treatment of hierarchical characters as missing data results in the reranking of pairwise dissimilarity estimates. When group association correlates with morphological similarity (as might be expected for taxonomic groups), the effect on relative within-group disparity estimates appears to be minimal, but between-group distances may be altered significantly. Application of the family of metrics proposed herein allows for the inclusion of hierarchical characters while maintaining the power of primary characters to distinguish between groups. The tuning parameter α allows the user to determine how much hierarchical characters influence downstream disparity estimates within and between groups.

2. Metrics

The development of similarity and dissimilarity metrics emerged from the fields of numerical taxonomy and numerical ecology. In numerical taxonomy, the basic discrete data type is the presence or absence of traits (coded as ‘1’ and ‘0’, respectively); in numerical ecology, the basic discrete data type is the presence or absence of species at a locality (coded as ‘1’ and ‘0’, respectively). The two simplest metrics for describing similarity in these data are the simple matching coefficient, found by dividing the number of shared occurrences/character states (whether both absent or both present) over the total number of sites/characters, and Jaccard’s similarity, found by dividing the number of positive matches (only those that are both present) over the number of sites/characters where at least one of the pair under comparison has been coded positively. Numerous variations of each of these exist [15,19,20]. Jaccard’s similarity and variations thereof are popular in numerical ecology because species may be absent from the same locality for different reasons [20]; they also may have utility in assessing information dissemination through networks [21]. However, Jaccard’s similarity is usually not appropriate for morphological matrices, especially for those composed of mixed character types (see below). Thus for the purpose of this study, we restrict our discussion to metrics that count all shared character state comparisons and have an explicit mechanism for handling missing data that can be applied to hierarchical characters.

To give the mathematical context to our proposed approach, we introduce a standard notation, following Gower [10] and Wills [5]. We then present Gower’s general coefficient of similarity [10] and Wills’s GED metric [11] with an example that demonstrates for both the reranking problem associated with treating hierarchical characters as missing data. We then introduce a new family of metrics that extends Gower’s [10] weighting solution to the reranking problem.

(a) Notation

Assume there are t taxa, and associated sequences of v traits, {X₁, X₂, …, X_t}. When comparing the ith and jth sequences, at the kth position, X_ik, X_jk, resp., we write:

for binary, continuous and ordered characters where range(k) is the size of the set of values that the kth character can assume. Note that range(k) = 1 for unranged continuous and ordered characters. For binary and unordered multistate characters, we define S_ijk = 0 when the values matches and S_ijk = 1 for mismatches.

Secondary characters describe variation within a trait—that is, there is another character that describes whether that trait is present or absent. A tertiary character is similarly defined, referring back to a secondary character.

(b) Previous metrics

We write Gower’s general coefficient [10, eqn 1, p. 859]:

where W_ijk is the weight of character k and δ_ijk = 1 if the ith and jth sequences are comparable for character k, otherwise, it is 0. Note that Lloyd [22, eq. 6] recommends setting δ_ijk to be the maximal value of S_ijk when ordered characters are included.

We write Wills’s GED metric [11, eqn 2, p. 469]:

For the latter, when either X_ik or X_jk are absent, the average dissimilarity between all characters for taxa i and j is computed and substituted for all undefined S_ijk [11, eqn 3, p. 471]. Note that some implementations of this coefficient have substituted the average per-character dissimilarity across the entire character matrix, which can lead to illogical values, specifically estimates for pairs of taxa that share no comparable characters [22].

(c) Running example

In order to demonstrate how previous metrics can rerank distances when hierarchical characters are present, we define a running example of a matrix of binary characters and compute the dissimilarity matrix under the GED and Gower’s metrics, with and without secondary characters (table 1). For both metrics, the rank order of pairwise dissimilarities is different than if there were no secondary characters; for example, d_GED(t₁, t₂) > d_GED(t₁, t₃), despite having more primary characters in common.

(d) Proposed scalable metric

Following Gower’s suggestion to scale the contribution of the primary character to avoid such reranking [10, eqn 9, p. 863], we define a family of metrics with a parameter α that can range from 0 to 1. In order to make it clear how this family of metrics compares to Gower’s suggestion, we first define a similarity measure. For the simple case shown in the running example above, let s be the number of secondary characters that agree between the ith and jth taxa and k be the number of primary characters that agree out of the remaining n primary characters.

The intuition is that when α = 0 the secondary characters contribute nothing to the comparisons of the two taxa. When , the primary character contributes weight according to the fraction of shared secondary characters. When α = 1, the primary character is not counted in the weight separately, and this yields Gower’s weighting solution [10, eqn 9, p. 863]. The dissimilarity metric is defined as d_α(i, j) = 1 − S_α(i, j). When tertiary characters are present, the above approach is applied recursively, first scaling the secondary characters by their associated tertiary characters, and then scaling the primary characters by their associated (and possibly rescaled) secondary characters. Actual empirical datasets will have more than one set of secondary characters contingent on a primary character. The generalization of the above metric for such datasets, as well as a proof that the metric properties hold, are in electronic supplementary material, appendix 2.

3. Empirical example

We explore the effectiveness of our family of metrics on a character matrix dataset describing variation in myriapods [18]. This matrix was modified as described in electronic supplementary material, appendix 3. The final matrix consisted of 47 taxa, two of which are Symphyla, 16 are Diplopoda (millipedes) and 29 are Chilopoda (centipedes). There are 205 characters, 129 of which are primary, 71 are secondary and 5 are tertiary. To this dataset, we applied the following treatments: Wills’s GED metric to just the primary characters (figure 1), Wills’s GED metric to the entire dataset but treating inapplicable characters as missing data (figure 1), Gower’s coefficient to just the primary characters (figures 1 and 2), Gower’s coefficient to the entire dataset but treating inapplicable characters as missing data (figures 1 and 2), and our family of metrics for α = 0, 0.1, 0.5, 0.9 and 1 applied to the entire dataset (figure 2).

• Download figure
• Open in new tab
• Download powerPoint

• Download figure
• Open in new tab
• Download powerPoint

Not surprisingly, when GED is applied to the entire matrix as opposed to just the primary characters, the dissimilarity between Lithobius forficatus and other taxa increases because more characters are contributing to the dissimilarity estimate (figure 1a,b). However, this increase is greater for the dissimilarity between Lithobius forficatus and other centipedes because most of the secondary characters that are present could contribute the maximum-possible dissimilarity (i.e. they are applicable). By contrast, the contribution of most secondary characters to the dissimilarity between Lithobius forficatus and species from other subclades will be the average across the character states (because they are inapplicable). The trend is not as strong for Gower’s coefficient (figure 1c,d) because the dissimilarities are standardized against the total number of comparable characters and secondary characters that are inapplicable do not contribute at all. This effect is further minimized using the family of metrics proposed herein (figure 2).

Interestingly, the reranking is more extensive for dissimilarities between Lithobius forficatus and other species than for Narceus americanus and other species (figure 2b,c). This is particularly notable in the case of the Symphyla species, whose relative dissimilarities to Lithobius forticatus decrease as the influence of secondary characters is increased (i.e. α is increased). This reflects the differences in the number of shared primary and secondary characters among the three groups, which is also evident in the shift in relative distances within the PCO analyses (compare figure 2a with electronic supplementary material, appendix 4).

Table 2 shows the relative disparity estimates within and between subclades for three treatments (Gower’s coefficient applied to just primary characters, Gower’s coefficient applied to the entire dataset and treating inapplicable characters as missing, and our proposed metric with α = 0.5); ratios are used to make values comparable across treatments. Regardless of the treatment, the disparity within Chilopoda (centipedes) is twice as large as the disparity within Diplopoda (millipedes), and this ratio does not change significantly across treatments (electronic supplementary material, appendix 4). There is some variation in the relative disparity of two Chilopoda subclades (Scolopendromorpha and Geophilomorpha) across treatments, but subsampling indicates that the difference is not highly significant (electronic supplementary material, appendix 4). By contrast, if secondary characters are treated as missing data (Gower, middle column of table 2), the disparity between Chilopoda and Diplopoda relative to the disparity within either group decreases significantly; this effect is not as strong if secondary characters are treated appropriately (our metric, right column of table 2; see also electronic supplementary material, appendix 4).

4. Discussion

Application of the family of metrics proposed herein allows for the inclusion of hierarchical characters while maintaining the power of primary characters to distinguish between groups. Furthermore, the tuning parameter α allows the user to assess the influence of the secondary characters on the ranked dissimilarities and subsequent disparity estimates. When α = 0, secondary characters are given no weight at all, and the estimated dissimilarities are based entirely on unshared primary characters. As α increases, secondary characters are given increasingly more weight relative to the subset of primary characters they describe. When α = 1, the influence of the primary character on the dissimilarity is determined entirely by the unshared secondary characters. For users who would like the unshared secondary characters to modify but not eclipse the influence of the primary character they describe (which is, by definition, shared), we recommend .

We chose the myriapod character matrix for the example above because it described variation at a high taxonomic level and included a large number of hierarchical character sets in order to do so. However, we encountered a number of obstacles in the discovery of existing datasets to which the proposed metric can be applied. First, of course, not all workers employ a hierarchical coding strategy (also called contingent or reductive coding, see Brazeau [17]). In such cases, it may be possible to recode the matrix, but this can be time-consuming and may require some taxonomic expertise. Second, because inapplicable characters have been treated almost exclusively as missing data in phylogenetic and disparity analyses, they are frequently not distinguished from one another in character matrices (in such cases, the presence of inapplicable characters is known only when the authors state that they are coded the same way). Third, even when inapplicable characters are coded separately (usually using ‘–’ or ‘N’), there is often little to no information about contingency (i.e. which primary characters the secondary characters describe), or primary characters may be excluded altogether. In such cases, it may be possible to add primary characters based on the description of secondary characters, and it may be possible to determine contingency by observing character state distributions and looking for correlations (see electronic supplementary material, appendix 3 for an example). However, both strategies frequently require some taxonomic expertise. Best practice is to explicitly indicate contingency in the character description; Cotton [23] provides an example of concise notation that could provide a standard going forward.

Ironically, Cotton [23] also provides examples of common but problematic coding practices. In particular, it is not uncommon for secondary characters to be contingent on different states of the same primary character, or for secondary characters to be contingent on multiple primary characters. Neither is strictly hierarchical, and will produce unexpected results if the proposed metric is applied to a matrix that includes them. Both practices also decrease the efficacy of hierarchical coding to make explicit homology statements.

Finally, the results of this study have implications for the treatment of hierarchical characters in phylogenetic analysis. In the total evidence tree of Fernández et al. [18], Symphyla are basal to Diplopoda (millipedes); however, their morphology-only tree [18, fig. S3] places Symplyla basal to Chilopoda (centipedes). Symphyla shares some primary characters with each group, so the treatment of hierarchical characters does have an effect on their relative dissimilarity. Interestingly, if secondary characters are treated as missing data, the average between-group disparity is close to even, but if secondary characters are excluded or treated appropriately using the metric proposed herein, the disparity between Symphyla and Chilopoda is relatively larger than between Symphyla and Diplopoda (table 2; see also electronic supplementary material, appendix 4). These results indicate more morphological similarity between the latter two groups, providing phyletic support for the total evidence tree, and suggesting that the treatment of inapplicable characters as missing data in phylogenetic analysis may be producing undesirable results.

Data accessibility

Nexus file and character type table for empirical example, and R code with functions for implementing Gower’s coefficient, Wills’s GED and the metric proposed herein are included with the electronic supplementary materials and available from the Dryad Digital Repository at: https://doi.org/10.5061/dryad.r3k7m3c [24]. Modified myriapod character matrix with notes is also available from MorphoBank (http://morphobank.org/permalink/?P3189).

Authors' contributions

M.J.H. conceived of the study, developed the new metric, modified the empirical dataset, wrote the functions to implement the metrics in R and helped draft the manuscript. K.S. developed the new metric, wrote the mathematical proof, rendered figures in Python and helped draft the manuscript. All authors gave final approval for publication.

Competing interests

We declare we have no competing interests.

Funding

We would like to thank the Simons Foundation and the National Science Foundationfor funding (K.S.).

Footnotes

Electronic supplementary material is available online at https://dx.doi.org/10.6084/m9.figshare.c.4302683.