Statistical tests for comparing matrices

I have been remiss. Quite a few years ago I found myself in the position of wanting to compare two genetic covariance matrices. At the time it was before the Flury hierarchy had been suggested by Pat Phillips (Phillips & Arnold 1999. Evolution 53: 1506-1515), so I found myself in a position of needing to invent my own. Later, apparently along with others I decided I wasn’t particularly enamored with the Flury hierarchy. This resulted in two publications (Goodnight & Schwartz 1997 Biometrics 53: 1026-1039; Calsbeek &. Goodnight 2009 Evolution 63: 2627-2635), the first of which is not particularly well known. The first publication also suffered from not having a good software implementation. With the appearance of R this has now been rectified. In any case I would like to remind people of these statistical methods for comparing covariance matrices.

First off, there is nothing wrong with the Flury hierarchy, I just don’t particularly find it intuitively useful. As I understand it the Flury hierarchy is a model selection approach, whereas the methods I will discuss are parametric statistical tests. I recommend you read Philips and Arnold’s papers and make your own decision. So enough preamble.

We had just done an experiment in which we sent a population a population bottleneck, and we had measured several traits. We wanted to know if the derived population and the ancestral population had the same genetic structure, aka, the same genetic covariance matrices. For a single trait we know exactly how to do this. You “simply” measure the additive genetic variance in the two populations and do an F test to see if they are the same or different. I put simply in quotations because measuring additive variance is never easy.

When we get to a multivariate settings things become more complicated. Again, we will likely use a MANOVA to measure an additive genetic covariance matrix for each population. We would then like to compare these to see if they are the same or different. The good news is that genetic covariance matrices are square and generally easy to work with. The bad news is that when we go multivariate there are several ways that matrices can be different. In Goodnight and Schwartz (1998) we decided there are three ways of interest. The matrices can be of different dimension, they can be of different size, and they can be of different shape. These are really independent ways of being different, so it makes sense to develop three tests. The way we tested these was using bootstrapping.

The bootstrap: Bootstrapping is an interesting statistical procedure that was popularized in the 80s by Brad Efron (Efron 1979, The Annals of Statistics 7:1-26) (I took a workshop he offered somewhere around 1985). The basic idea is that if you have a data set you can create new pseudo data sets by randomly sampling with replacement from the original data. If enough of these bootstrap data sets are generated they will actually provide a distribution for the data. This at first seems counter intuitive, but as long as your data set is relatively large it works very well. To use this as a statistical test you need to decide what your null hypothesis is, and then figure out a random sampling scheme that makes that null hypothesis true. For example with a t-test, the null hypothesis is that the two populations have the same mean. You can make that null hypothesis true in several ways. You could simply combine the data from the two populations. Then randomly assign them back to the two populations without regard to original source. As a result there will be no true difference between the populations. If you randomly create several thousand of these pairs of populations you will get a distribution of observed differences in the means when you know the true difference is actually zero. You can then take the actual difference between the two populations and simply ask what percentage of the bootstrap differences that are more extreme than the difference in the actual data. That percentage is your probability of the observed difference occurring by chance. There are more sophisticated approaches, but this gives the idea.

In our particular test we had an ancestral population and a population derived from two generations of brother sister mating. We wanted to see if the two populations were the same or different. Our null hypothesis was that their covariance matrices were the same (this is important!), and we decided to use data from the ancestral population as our source for the bootstrap data.

Dimension: A genetic covariance matrix can be thought of as enclosing a space. Thus a univariate “matrix” is a single vector of a length that is equal to the variance. A two-trait covariance matrix defines a plane, a three trait matrix a cube, and so on.

Screen Shot 2015-04-08 at 5.47.22 PM

Figure 1; a one dimensional vector, and two and three dimensional matrices.

There are two things that can happen to the additive genetic variance after a population goes through a bottleneck. First it can disappear, that is, it can go to zero. Second, it can become so highly correlated with other traits that it becomes a linear combination of these traits. In graphic terms, in the three-trait case, that would be the equivalent of one of the vectors lying exactly in the plane of the other two vectors.

3 colinear vectors

Figure 2: in this matrix trait z is a linear combination of traits y and x. As a result all three lie in a single plane, and the resulting matrix is a two dimensional matrix.

Consider trying to compare two matrices with three variances. One is like the three dimensional matrix in figure 1, and the second has only two dimensions as in figure 2. It won’t work to compare these. As an analogy it is like asking which is bigger, a box or a sheet of paper. The three dimensional matrix has an extra dimension along which it can evolve that is qualitatively different from the two dimensional structure.

The way we tested this was to find the largest sub-matrix that had valid variances that were not linear combinations of other vectors. We then tested the absolute value in the difference in rank ( |RpopA-RpopB|) as our test statistic measured against the bootstrap populations were there was no true difference in rank. In this data set the difference in rank was not significant.

Screen Shot 2015-04-07 at 12.19.17 PM

Difference in size: As mentioned above, matrices can be considered to be planes, volumes or hyper volumes. It turns out that the determinant is a measure of the space enclosed by the matrix. For example, in a two-trait matrix the determinant is the area of the matrix, in a three trait matrix it is the volume, etc. Thus two matrices of the same dimension, regardless of shape, can be compared by comparing the determinants. The analogy is having two oddly shaped vases. We can compare them by asking how much water they hold. In this case shape is of no consequence, only the size of the space enclosed.

The important caveat is that they must be the same dimension. Again, the same question: which is larger the volume of a box, or the area of a sheet of paper. And again it is a meaningless question. We chose to resolve this by doing an “orthogonal projection” of the larger dimension matrix on the smaller dimension matrix. That is, we searched the matix pairs for a set of traits that had valid variances in both matrices. We did the analysis on this pair of sub matrices.

The next question, is how to compare the two determinants. It turns out that there is a good test, the multivariate Bartlett’s test that can be used. Bartlett’s test, has two problems. First, it is very sensitive to the assumption of multivariate normality, and second, it is not structured for use with MANOVA derived data. Still we can use the basic statistic and combine it with the bootstrap data, and it works perfectly well. One of the very useful features of bootstrap tests is that they make no assumptions about the distribution of the data. Also, if properly designed, they work well with virtually any experimental design. Interestingly, since, the standard test was not developed for use with MANOVA the parametric multivariate Barlett’s test was way to optimistic and the bootstrap ended up doing a much better job. A final modification is that we had an a priori interest in whether the derived genetic covariance matrix was significantly larger than that in the ancestral population. Thus, we multiplied the Bartletts statistic by 1 if the derived population was larger than the ancestral population and -1 if it was smaller, giving us the signed Bootstrap Bartlett’s test that allowed for both one tailed and two tailed tests.

Screen Shot 2015-04-08 at 6.05.37 PM

Shape: For shape we decided to go with a test similar to the Mantel test. Many rightly complain about the classic Mantel test for numerous reasons. However, the basic idea is useful. The idea is that you calculate a correlation between the pairwise elements of the two matrices. That is you pair up the elements of the two matrices and simply calculate the correlation among them. The problems with the traditional Mantel test for this application are three fold. First, the traditional Mantel has a null hypothesis that the two matrices are independent, whereas our null hypothesis is that the two matrices are identical. The bootstrap solves this by allowing us to generate a distribution of Mantel correlations among pairs of matrices that have a true correlation of 1.

Second, the Mantel test is meant to compare correlation matrices, which have 1s on the diagonal, whereas this is not true for a covariance matrix. In the classic Mantel test this diagonal is excluded, whereas in ours it is not. Third, all of the elements of a correlation matrix are between -1 and 1, whereas covariance matrices can have vastly different variances for different traits, which can inappropriately skew the results. This last we solved by standardizing the elements to the average of the diagonals of the two matrices. The final equation is somewhat ugly, so I refer you to the paper if you want the details. The results indicate that females, but not males, have a significant change in the shape of their covariance matrix. That is the population bottleneck significantly changed some of the variances and covariances among traits in the two populations, even though it did not change the total amount of additive genetic variance.


So, the point of this is simply to suggest one possible way to compare genetic covariance matrices. One of the reasons I really enjoy multivariate math (I can’t believe I said that) is that very simple ideas, like the variance of a trait, suddenly become so much richer, and can change in so many more ways as we move into a multivariate setting. Obviously simple multivariate math in a pale comparison with the real world, but this only serves to make the diversity of the real world even more easily understood.

The other reason I wanted to put this up is that I have an R program that does these analyses, along with random skewers and selection skewers, which I will talk about next time. I am not an R developer, so I would be more than pleased if somebody were to take this script and turn it into something that didn’t actually need to be adjusted for the needs of every data set. If you do choose to finish developing this, please let me know!

Here is the program: 

Writeup on how to use the program:  Matrix comparison writeup

The program:Bootstrap command

Relevant example data sets:

balanced stock females

stock data female

population 3 females


2 Responses to “Statistical tests for comparing matrices”

  1. there are several papers covered in this post.

    Goodnight, C. J. and J. M. Schwartz (1997). “A bootstrap comparison of genetic covariance matrices.” Biometrics 53: 1026-1039.

    Calsbeek, B. and C. J. Goodnight (2009). “Empirical comparison of G matrix test statistics: finding biologically relevant change.” Evolution 63: 2627-2635.

    The program was written for the Calsbeek paper

  2. Julián Fernández says:


    I used your program for one paper. How can I cite you? Have you published this anywhere?



Leave a Reply