1. Introduction

The recent developments in RNA-sequencing (RNA-seq) technology have led to a rapid increase in gene expression data in the form of counts. RNA-seq enables some applications not achievable by microarrays, including discovering novel transcripts and alternative splicing. One fundamental question in functional genomics, however, remains the regulation of gene expression under the different conditions (treatments, genotypes, environments, or developmental stages). Therefore, identifying differential expression (DE) is still a key task in transcriptomics research.

The measurement of expression in RNA-seq is the number of sequenced fragments that can be mapped to a genomic unit of interest, which can be a gene, an exon, or any region of interest. We use gene thereafter for convenience. One major difference in the DE detection methods for RNA-seq and microarray data is that data from microarrays are often modeled as a Gaussian distribution after proper preprocessing. These methods do not take into account the discrete nature of RNA-seq data and thus may not be directly applicable. There are nonetheless some common features between the two technologies. For example, the advancement in technology does not eliminate biological variability, which early applications of RNA-seq often neglected(Hansen and others, 2011). In fact, the much higher dynamic range of RNA-Seq data makes the heteroskedasticity, i.e. the gene-specific dispersion, even more important than it was with microarrays.

In both types of gene expression data, there are two sources of variation: technical variation, which represents the measurement error inherent to the technology and sample preparation, and biological variation, which represents the heterogeneity among samples in the same treatment group or population. If all subjects within the same group have an identical expression level for a given gene, i.e. lacking biological variation, the counts for that gene can be modeled as a Poisson random variable, as done byMarioni and others (2008) and Wang and others (2010). The earliest tests for DE essentially tested for whether the difference of average counts in two groups exceeds what is expected by Poisson sampling error. However, since gene expression is a stochastic process, biological replicates in the same treatment group do not share identical expression levels. The presence of biological variation leads to the “over-dispersion” problem, e.g. the read counts show variation greater than expected from Poisson random variables.

A common approach to address the over-dispersion problem is to use the negative binomial (NB) model. The NB model is a Gamma–Poisson mixture and can be interpreted as the following: the Gamma distribution models the unobserved true expression levels in each biological sample, and conditioning on the expression level, the measurement from the sequencing machine follows a Poisson distribution(Robinson and Smyth, 2007; Anders and Huber, 2010; Hardcastle and Kelly, 2010). The variance from an NB distribution depends on the mean μ in the relationship:

(1.1)

where the first term represents variance due to Poisson sampling error and the second term represents variance due to variation between biological replicates. The parameter ϕ is referred to as the dispersion parameter. Note that ϕ is the reciprocal of the shape parameter in the Gamma distribution, and thus is the squared coefficient of variation (CV). Therefore, ϕ represents the variation of a gene’s expression relative to its mean.

Most statisticians agree that the over-dispersion problem needs to be addressed. The difference is how the dispersion is modeled, estimated, and used in inference.Robinson and Smyth (2008) assume a common dispersion for all genes and use information from all genes to estimate a global ϕ. This stabilizes the estimation for ϕ but a common dispersion means an identical CV for all genes, while it is known that some genes are more tightly controlled (e.g. housekeeping genes) and other genes vary much more relative to their means (e.g. immune-modulated and stress-induced genes(Pritchard and others, 2001)). The gene-specific biological variation is reproducible across technologies(Hansen and others, 2011) and is not reflected by a common dispersion. As a result, genes that are naturally more variable are more likely to be reported as DE due to an underestimation of their dispersion.Anders and Huber (2010) introduce DESeq, another method based on the NB model. Instead of assuming the second term in(1.1) to be proportional to μ² with a dispersion ϕ, they let the variance be a smooth function of the mean (with proper offset accounting for the sequencing depth). However, the model inAnders and Huber (2010) still assumes that conditioning on the mean expression, the variance is constant. That is, two genes with the same mean expression level also has the same variance. There have been many updates of the DESeq package and the current version uses the greater value between the empirical gene-specific dispersion and the mean-dependent fitted value. Robinson and Smyth (2007) modify the common dispersion model and introduce an empirical Bayes shrinkage estimate using weighted conditional log-likelihood. This new estimate shrinks the sample dispersion for each gene toward a common prior instead of shrinking them completely to the common dispersion. The method is implemented in the widely used R/Bioconductor package edgeR(Robinson and others, 2010), with recent extensions to multifactor experiments(McCarthy and others, 2012). Hardcastle and Kelly (2010) use NB distribution for the counts in a Bayesian hierarchical model setting. Non-parametric prior distributions for the NB parameters are empirically determined. This is a much more computationally intensive method and, possibly for this reason, not as widely applied as edgeR and DESeq.

Since these publications, many RNA-seq studies have emerged and the accumulated RNA-seq datasets with a reasonable number of replicates encourage us to re-evaluate the estimation of dispersion in RNA-seq. We find that existing methods often capture the overall level of dispersion but do not adequately reflect the heterogeneity in dispersion among genes. In this paper, we propose a new empirical Bayes method to shrink the dispersion parameter. We demonstrate using real data-based simulation that the proposed method provides a better estimate for the gene-specific dispersion and improves DE detection. The rest of the paper is organized as follows. In Section2, we describe the datasets that provide both the motivation for this research and the basis for the evaluation of DE detection. In Section3, we demonstrate the limitation of current estimation of the dispersion parameter and motivate our model for the prior distribution of the dispersion parameter. In Section4, we present the hierarchical model, estimation, and testing procedure. In Section5, we compare the DE detection using the proposed method with the leading alternatives. Finally,in Section6, we discuss the interpretation of the dispersion, the connection with microarray data, and future directions.

2. Data description

Two real RNA-seq datasets are used in the paper for estimating simulations parameters and showing motivational plots. The first is fromCheung and others (2010), which quantifies the expressions of lymphoblastoid cell lines from 41 CEU individuals in International HapMap Project(Gibbs and others, 2003). The second is from Blekhman and others (2010), which includes six liver samples from both males and females. These datasets are referred to as “Cheung data” and “Gilad data”, respectively, thereafter. For both datasets, the sequence reads are summarized into counts within genes from Ensembl 61 annotation. The gene counts are obtained from the ReCount webpage presented byFrazee and others (2011). Samples in both datasets include biological replicates, so the variance in observed data is a result of both biological variation and technical variation.

We also use data from the MicroArray Quality Control Project (MAQC)(Shi and others, 2006) phase III, also known as Sequencing Quality Control. Two biological samples, human brain and universal human reference sample, are assayed using seven lanes each. The MAQC data use samples from the same library preparation protocols, and thus represent data with technical replicates only. This dataset provides an extreme case in which the biological variation reaches the lower bound.

3. Motivation

The NB distribution is widely used in modeling sequence count data because the Gamma–Poisson mixture provides a natural hierarchy for the generation of sequencing counts. The Gamma distribution is a flexible continuous distribution that characterizes the expression rate for a gene. Given the expression rate in a biological sample, the sequencing technology provides a count from a Poisson process with the mean proportional to the expression rate and an offset related to sequencing depth and possibly other factors affecting the detection efficiency (Hansen and others, 2012). When there are a large number of replicates, the dispersion parameter can be estimated from the data and over-dispersion is a well-addressed problem in the generalized linear model framework. The challenge in RNA-seq data is that there are often very limited replicates, such that the sample estimate of gene-specific dispersion is unstable. In this case, it is often useful to combine information from other genes to improve the estimation. DESeq and edgeR are the most widely used methods that provide shrinkage estimates of the dispersions by borrowing information across genes. To assess how well the dispersions are estimated from these methods, we conduct a simulation based on real data. First the gene-specific means and dispersions are estimated from the Cheung data. Pseudo-datasets of 20 000 genes, in two treatment groups with four replicates each, are constructed under the NB model with parameters taken from the real data. The mean expressions are directly sampled from estimated gene-specific means non-parametrically. The dispersions are generated from log-normal(−1.72,1.07²), which well approximates the empirical distribution of the dispersion observed in the Cheung data (supplementary material available at Biostatistics online, Figure S1).

DESeq function estimateDispersions with the default settings (sharing Mode="maximum" and fitType="parametric") and edgeR functions estimateCommon Disp followed by estimateTagwiseDisp with the default settings (trend="movingave", prop.used=0.3, method="grid") are applied to the simulated counts to obtain estimated dispersions. Figure1(a) and(b) shows the estimated versus true dispersions from both methods. Clearly, the estimates do not correlate well with the truth. Although both edgeR and DESeq estimate the central magnitude of the dispersions well, they fail to capture the variations in dispersions among genes. Changing the settings in the edgeR function estimateTagwiseDisp leads to different estimates (supplementary material available at Biostatistics online, Figure S4), but does not appear to substantially increase the accuracy of the estimated dispersion. The near horizontal clusters of points show that the dispersion of many genes are estimated to be approximately the same, which indicates over-shrinkage of the dispersions.

Since the dispersion parameter represents biological variation, having a good estimate of ϕ is crucial to finding the DE that is beyond natural biological variation. In order to account for gene-specific dispersion, we take advantage of real RNA-seq datasets with replicates and note that the distribution of the logarithm of sample dispersion, , is approximately Gaussian as shown in Figure2. In the next section, we describe a novel shrinkage estimator for ϕ using a log-normal prior and an NB likelihood. In Section5, we show that the proposed method greatly improves the estimation of ϕ and leads to better detection of the DE.

4. Methods

Denote the observed counts from gene g in sample i by Y _gi, and the unobserved expression rate by θ_gi. We assume the following hierarchical model:

Here the Gamma distribution is parameterized with mean and dispersion (ϕ_g is the reciprocal of the shape parameter in the common parameterization); k(i) denotes the treatment group of sample i; and s_i represents the normalizing factor, such as the relative library size, for sample i. This normalization factor could also be a gene-specific factor, taking into account the varying sequencing efficiency due to the guanine-cytosine content and/or length (Hansen and others, 2012; Risso and others, 2012), but would nonetheless be treated as a known constant. Based on the hierarchical model, the marginal distribution of Y _gi given μ_g,k(i) and ϕ_g is NB with mean ν_gi=μ_g,k(i)s_i and dispersion ϕ_g. In the simplest setting where there are only two groups, the goal of DE detection is to test, for each gene, whether the mean expressions are identical in both groups, e.g. μ_g,1=μ_g,2, as noted by other researchers(Robinson and Smyth, 2007, 2008; Anders and Huber, 2010; Robinson and others, 2010).

Estimating ϕ_g is a crucial step in DE detection and shrinkage estimators have been shown to be useful in typical RNA-seq experiments with a small number of replicates(Anders and Huber, 2010; Hardcastle and Kelly, 2010; Robinson and others, 2010). As discussed in previous works (Robinson and Smyth, 2007), there is no conjugate prior for ϕ_g to ease the computation of a posterior distribution. Thus, we decide to use a prior that approximates the ϕ_g in real RNA-seq data. We choose the log-normal distribution, motivated by Figure2.

For gene g, the conditional posterior distribution of ϕ_g given all observed counts and means, p(ϕ_g|Y _gi,ν_gi,i=1,…,n) satisfies (detailed derivation can be found in supplementary material available at Biostatistics online):

(4.1)

where ψ(.) is the log gamma function and n is the number of samples. To obtain a point estimate of ϕ_g, we could compute the posterior mean by using numerical methods such as importance sampling. It is, however, too computationally intensive to be practical for RNA-seq data. Instead, we compute the posterior mode by maximizing an approximate of (4.1) and denote the estimate by .

Specifically, we substitute ν_gi by where , and n_k(i) is the number of samples in the same treatment group as i. The estimate is the same as the sample estimate of true concentration in Anders and Huber (2010). We plug in the hyper-parameters m₀,τ² estimated from the data, and maximize the approximated Equation(4.1) using the Newton–Raphson method. In practice, we use “optimize” function in R , which provides excellent computational performance.

The hyper-parameters m₀,τ² are estimated from the data by pooling information from all genes (details in the next subsection). The estimate is therefore an empirical Bayes estimate shrunken toward the common prior. It can also be viewed as an estimate that maximizes a penalized pseudo-likelihood, as we are maximizing an approximate of the penalized likelihood (4.1), with penalty , by replacing the nuisance parameter ν_gi by its estimate . The first term in the penalty, , penalizes values that deviate far from the common prior m₀ and the second term adds an additional penalty for large values of ϕ_g. As mentioned in Section1, ϕ_g has the interpretation of the squared CV. Based on RNA-seq data from human populations(Cheung and others, 2010; Pickrell and others, 2010), we observe that around 70% of the genes have a CV less than 0.5. The CV in inbred animal models or established cell lines are likely even lower. Thus, penalizing larger ϕ_g appears a desirable feature.

Estimating hyper-parameters. We start with a simple dispersion estimate for each gene motivated by the method of moments. Define a new random variable ; we have , which is identical for every sample in a certain treatment group. Using the sample mean to estimate the first moment of z_gi and pooling data from different treatment groups together, one can obtain an estimator (detailed derivation in supplementary material available at Biostatistics online).

Next we estimate m₀ and τ using the empirical distribution of . We use the median of ’s, , to estimate m₀. The estimation of τ² is less straightforward as the sample variance of overestimates τ². The sample variance of comprises two parts: τ², the variance representing the heterogeneity of ’s among genes, and , the variation due to estimating . Even when τ²=0 (all genes share common dispersion, ϕ_g=ϕ₀∀g), we still observe non-zero . Therefore, it is important to take into account this inflation in sample variance of . We use an ad hoc adjustment that behaves well in practice evaluated by extensive simulation. Specifically, we create a pseudo-dataset with τ²=0 by simulating from , where is used as common dispersion for all genes. We then compute the sample dispersion for each gene, and estimate as . From a number of pseudo-datasets, we obtain the mean as the baseline. Lastly, we estimate τ² as . Here c₀ is the lower bound for . In practice, we use c₀=0.01.

Note that is only used in estimating the hyper-parameters and not directly used to obtain the shrinkage estimate . Although each is a rather crude estimate of the dispersion, especially for a small number of samples, the large number of genes in RNA-seq data allows us to obtain stable estimates of the hyper-parameters. Once the prior is established in the empirical Bayesian method, our shrinkage estimate directly comes from maximizing(4.1).

Test statistic and false discovery rate. With all parameters estimated, a hypothesis testing for the comparison of any two groups can be carried using a Wald test or exact test.Bullard and others (2010) show that the likelihood ratio test and the exact test perform similarly. Our simulation show that the exact test and the Wald test provide similar performance (supplementary material available at Biostatistics online, Figure S9). We choose to use the Wald test for its simplicity.

The Wald test statistic for two-group comparison is constructed as

where , is the estimated variance for (detailed derivation in supplementary material available at Biostatistics online), and similarly defined. When there is no need for normalization (s_j=1∀j), reduces to , an estimate of the variance in(1.1), standardized by the sample size in group 1.

It is not trivial to derive the null distribution for the test statistic and the asymptotic distribution is irrelevant for most RNA-seq data. However, the empirical distribution of t_g is Gaussian-like (Figure5(a)) and we can estimate a local false discovery rate (FDR) based on the empirical null (Efron, 2004) estimated from the data.

We implemented the proposed method in an R package titled “DSS”, standing for Dispersion Shrinkage for Sequencing. The package is available from Bioconductor.

5. Results

We compare results from the proposed method with two most widely used software packages, edgeR (version 2.6.2) and DESeq (version 1.8.2) using their latest release versions. For DESeq, we call the functions estimateDispersions and nbinomTest with default settings. For edgeR, we call functions estimateCommonDisp followed by estimateTagwiseDisp to estimate the dispersion parameter, and exactTest to test for DE.

First, we show an improvement in estimating the gene-specific dispersion parameter ϕ_g. All three methods use the common NB model, and estimating ϕ_g is the key challenge and what determines the ability to rank the truly DE genes above the null genes. In Figure1(c), we show the estimates plotted against true ϕ_g. It is clear that the shrinkage estimates from DSS track the real dispersion much better than those from edgeR or DESeq. Note that the dispersion parameter estimates are shrunk more for genes with lower counts, as expected, since for those genes there is little information about ϕ in the data.

To quantify the improvement,we further use a similar approach as done inRobinson and Smyth (2007) to compare ϕ_g estimation. The mean squared error (MSE) of the estimates, in the scale of ϕ_g/(1+ϕ_g), were calculated from 50 simulations. Instead of using fixed mean expressions as inRobinson and Smyth (2007), we use a more realistic approach and randomly sample from mean expressions computed from the Cheung data. DSS has the lowest MSE in both settings (Figure3), and in all strata of mean expression (supplementary material available at Biostatistics online, Figure S3). We next compare the ability to detect the DE under a variety of situations. To evaluate DE detection under known truth, and to conduct the comparison under the most realistic scenario encountered in RNA-seq experiments, we simulated data semi-parametrically based on real RNA-seq datasets. In all simulations, the mean expressions were randomly sampled from the means calculated from the source dataset, Cheung or Gilad. The dispersion parameters were generated from parametric distributions, in which the parameters were computed from the source dataset (details in supplementary material available at Biostatistics online). To demonstrate robustness to distributional assumption, we simulated ϕ_g from both log-normal and Gamma distribution. In all simulations, 5% of all genes were true DE genes as this proportion is expected to be low in most gene expression experiments(Smyth, 2004). The logarithm of fold changes for the DE genes were randomly sampled from N(0,1).

First, we evaluate the ability to rank true DE genes above non-DE genes. Since DE detection is often used as a hypothesis generating tool, and the goal is to have as many true positives as possible in the top-ranked genes, we compare the percentage of true DEs (i.e. 1−false discovery proportion) in the top-ranked genes up to top 1000. DSS shows a much higher proportion of true positives among the top-ranked genes when the dispersion is based on biological replicates as seen in the Gilad and Cheung data (Figure4). All three methods provide identical performances when the biological variations are near zero transcriptome wide, as expected in technical replicates such as MAQC data in supplementary material available at Biostatistics online, Figure S11.

To further evaluate the performance of DSS in ranking genes under different dispersion patterns, we performed additional simulations with dispersion from different distributions. We found that in general when the variation in dispersions is large, DSS provides better performance. The average dispersion level does not cause much differences among the results from the three methods. This is consistent with the findings that DESeq and edgeR provide reasonable estimates for the average dispersion, but underestimate the variations in dispersion due to over-shrinkage. The results for these simulations are presented in supplementary material available at Biostatistics online (Section10, Figures S6 and S7).

In addition to better sensitivity/specificity, another important task in DE detection is to provide a measure of statistical significance to guide the choice of a cut-off. In genomics, this is typically done using the FDR or some variant of the FDR. The Wald statistic in DSS is Gaussian-like in the center (Figure5(a)) and satisfies the assumption in local FDR calculation(Efron, 2004). Based on the connection between the local FDR and the FDR(Efron, 2004), we convert the local FDR to the scale of the classical FDR for comparison. For edgeR and DESeq, we use the FDR reported from the packages. Figure5(b) plots the reported (dashed lines) and true FDRs (solid lines) for the top-ranked genes from the simulation. The simulation settings are the same as those used in Figure4(b). The curves are averaged from 50 simulations. For DSS, the reported FDR curve is close to the true FDR. In contrast, both edgeR and DESeq give misleading FDR estimations. The underestimation of the FDR in the top-ranked genes gives an over-optimistic certainty in the reported DE. Supplementary material available at Biostatistics online, Figure S8, shows the comparison of the FDR under more simulation settings.

6. Discussion

We present a new shrinkage estimator for the dispersion parameter in the NB model for RNA-seq data. We show that this new estimator better captures the variation in gene-specific dispersion and, as a result, leads to a better detection in the DE compared with two most popular DE detection softwares DESeq and edgeR. In addition, the nominal FDR based on the local FDR method is much closer to the actual FDR. We do not include another method, baySeq, in the comparisons because the computational time becomes formidable for the large number of simulations we perform.

We find that a good estimate of ϕ_g is critical in DE detection when biological replicates are involved. This is not surprising as we have seen in the past decade that improving the estimation of the gene-specific variance in microarray data has greatly improved DE detection by using moderated Wald tests, including the most widely used methods LIMMA(Smyth, 2004) and SAM(Tusher and others, 2001). In fact, we find that ϕ_g has a similar interpretation as the gene-specific variance in microarray data. Let Y represent an untransformed expression level in microarray data. It is often assumed that , where μ and σ² are the mean and variance for the log scale expression measure, respectively. In the original scale, the squared CV CV (Y)²=e^σ2−1≈σ² for σ²≪1, which is satisfied in most genes (McCall and others, 2011). Thus, σ² is approximately the squared CV in the untransformed scale. In comparison, RNA-seq data are often modeled by the NB distribution, i.e. a Gamma–Poisson mixture, in which the Gamma distribution models the biological variation and the Poisson distribution models the variation due to the counting process in sequencing. The dispersion parameter ϕ_g is the squared CV of the Gamma distribution, i.e. ϕ_g represents the gene-specific biological variation, and is directly linked to in log-transformed microarray expression data.

This interpretation explains several findings in our analysis of RNA-seq data with biological replicates. It has been well established that genes have different biological variance, and thus we should not expect a constant ϕ_g for all genes. In experiments with a small number of replicates, an empirical Bayes method that combines the information from the observed sample and shrinks toward a common prior of ϕ_g helps stabilize the estimate of ϕ_g. However, the extent of shrinkage should not be uniform as the amount of information about ϕ_g varies between genes. The variation of counts from genes with low expression is dominated by the Poisson counting error, and provides less information of ϕ_g (supplementary material available at Biostatistics online, Figure S10) ; thus the estimates of ϕ_g for these genes should be shrunken more heavily . DSS does exactly what we desire, shrinking ϕ_g less for high-expression genes and more for low-expression genes (Figure1(c)).

It has been reported that the biological variation, thus the dispersion parameter, may depend on the expression level. DESeq and EdgeR allow trended estimation of the dispersion depending on the mean level. To account for this dependence, we also provide an option in DSS by letting the hyper-parameter m₀ be a smooth function of expression. The details of the trended estimation is provided in supplementary material available at Biostatistics online.

Some datasets are generated to investigate technical reproducibility and do not capture the biological variation in a population. Even in these cases with technical replicates only, such as the MAQC data, we find that analysis assuming a not-too-small dispersion provides improvement in DE detection (supplementary material available at Biostatistics online, Figure S12). When biological replicates are from established cell lines or inbred animal models, the variance among replicates may be lower than the variance in a random human population. However, the result on MAQC data with technical replicates suggests that even when there is no strong evidence for over-dispersion in the data, it is beneficial to analyze the data under a positive dispersion model.

We find that most of the improvement obtained in DSS is due to the different dispersion estimate, as passing the DSS estimates to edgeR/DESeq yields very similar results as in DSS (supplementary material available at Biostatistics online, Figure S9). Both the edgeR and the DESeq methods have been expanded to now accommodate multiclass comparisons. Our test is currently limited to two-class comparison and it is our immediate plan to extend the dispersion estimators to multifactor designs. With an estimate of the dispersion, one can use generalized linear models as done in McCarthy and others (2012).

In all comparisons presented here, we use semiparametric simulation based on real RNA-seq datasets for lack of real data example with known truth. Although the NB model is the current predominant choice for RNAseq data, most models only approximate reality. Real data with a reasonable number of known DE, for example, using spike-in genes, will enable better comparison.

Supplementary material

Supplementary material is available at http://biostatistics.oxfordjournals.org.

Funding

This research was supported by National Science Foundation (DBI-1054905).

Supplementary Material