You want to use all the data available to you to derive the best possible estimates of empirical statistics, such as here, covariance. In the condition with less trials, your estimate will be less accurate, more noisy, but that's the best available to you.
@John_Mosher and @pantazis, please chime in.