1. Hypothesis Testing
2. Multiple Testing Considerations

In biological research we often ask:

Perhaps more correctly, we are asking:

We might be comparing outcomes among treatment groups:

• Cell proliferation in response to antibiotics in media
• mRNA abundance in two related cell types
• Allele frequencies in two populations
• Methylation levels across genomic regions

How do we decide if our experimental results are “important”?

Asking if the outcomes could occur by chance alone if there really were no differences in outcomes.

• Is it normal variability?
• What would the data look like if our experiment had no effect?
• What would our data look like if there was an effect?

Every experiment is considered as a random sample from all possible repeated experiments.

Most experiments involve measuring something:

• Discrete values e.g. read counts, number of colonies
• Continuous values e.g. C_t values, fluorescence intensity

Every experiment is considered as a random sample from all possible repeated experiments.

Many data collections can also be considered as experimental data sets

In the 100,000 Genomes Project a risk allele for T1D has a frequency of \(\pi = 0.07\) in European Populations.

• Does this mean, the allele occurs in exactly 7% of Europeans?

In our in vitro experiment, we found that 90% of HeLa cells were lysed by exposure to our drug.

• Does this mean that exactly 90% of HeLa cells will always be destroyed?

• Experimentally-obtained values represent an estimate of the true effect
• More formally referred to as population-level parameters
• Every experiment is considered a random sample of the complete population
• Repeated experiments would give a different (but similar) estimate

All population parameters are considered to be fixed values, e.g. 

• Allele frequency (\(\pi\)) in a population
• The average difference in mRNA levels

All classical statistical testing involves:

1. a Null Hypothesis (\(H_0\)) and
2. an Alternative Hypothesis (\(H_A\))

Why do we do this?

• We define \(H_0\) so that we know what the data will look like if there is no effect
• The alternate (\(H_A\)) includes every other possibility besides \(H_0\)

An experimental hypothesis may be

\[ H_0: \mu = 0 \quad Vs \quad H_A: \mu \neq 0 \]

Where \(\mu\) represents the true average difference in a value (e.g. mRNA expression levels)

For every experiment we conduct we can get two key values:

1: The sample mean (\(\bar{x}\)) estimates the population-level mean (e.g. \(\mu\))

\[ \text{For} \quad \mathbf{x} = (x_1, x_2, ..., x_n) \] \[ \bar{x} = \frac{1}{n}\sum_{i = i}^n x_i \]

This will be a different value every time we repeat the experiment. This is an estimate of the true effect

For every experiment we conduct we can get two key values:

2: The sample variance (\(s^2\)) estimates the population-level variance (\(\sigma^2\))

\[ s^2 = \frac{1}{n-1} \sum_{i = 1}^n (x_i - \bar{x})^2 \]

This will also be a different value every time we repeat the experiment.

• Comparing expression levels of FOXP3 in T_reg and T_h cells (\(n = 4\) donors)
• The difference within each donor is obtained as \(\Delta \Delta C_t\) values
• \(\mathbf{x} =\) (2.1, 2.8, 2.5, 2.6)

• Comparing expression levels of FOXP3 in T_reg and T_h cells (\(n = 4\) donors)
• The difference within each donor is obtained as \(\Delta \Delta C_t\) values
• \(\mathbf{x} =\) (2.1, 2.8, 2.5, 2.6)

where \(\mu\) is the average difference in FOXP3 expression in the entire population

Now we can get the sample mean:

\[ \begin{aligned} \bar{x} &= \frac{1}{n}\sum_{i = i}^n x_i \\ &= \frac{1}{4}(2.1 + 2.8 + 2.5 + 2.6) \\ &= 2.5 \end{aligned} \]

This is our estimate of the true mean difference in expression (\(\mu\))

And the sample variance:

\[ \begin{aligned} s^2 &= \frac{1}{n - 1} \sum_{i = 1}^n (x_i - \bar{x})^2\\ & = \frac{1}{3}\sum_{i = 1}^4 (x_i - 2.5)^2\\ &= 0.0867 \end{aligned} \]

• Every time we repeat an experiment, we obtain a different value for \(\bar{x}\)
• Would these follow any specific pattern?

\[ \mathbf{\bar{x}} = \{\bar{x}_1, \bar{x}_2, \dots, \bar{x}_m \} \]

This represents a theoretical set of repeated experiments with a different sample mean for each.

We usually just have one experiment (\(\bar{x}\)).

• Every time we repeat an experiment, we obtain a different value for \(\bar{x}\)
• Would these follow any specific pattern?
• They would be normally distributed around the true value!

\[ \mathbf{\bar{x}} \sim \mathcal{N}(\mu, \frac{\sigma}{\sqrt{n}}) \]

where:

• \(\mu\) represents the true population mean
• \(\sigma\) represents the standard deviation in the population (probably unknown)
• Recall that the standard deviation, \(\sigma\), is the square root of the variance, \(\sigma^2\)

We know what our experimental results (\(\bar{x}\)) will look like.

\[ \bar{x} \sim \mathcal{N}(\mu, \frac{\sigma}{\sqrt{n}}) \]

If we subtract the population mean:

\[ \bar{x} - \mu \sim \mathcal{N}(0, \frac{\sigma}{\sqrt{n}}) \]

NB: We almost always test for no effect \(H_0: \mu = 0\)

• One final step gives us a \(Z\) statistic where \(Z \sim \mathcal{N}(0, 1)\)

\[ Z = \frac{\bar{x} - \mu}{\sigma / \sqrt{n}} \sim \mathcal{N}(0, 1) \]

• If \(H_0\) is true \(Z\) will come from \(\mathcal{N}(0, 1)\)
• If \(H_0\) is NOT true \(Z\) will come from some other distribution
• We compare our results (i.e. \(Z\)) to \(\mathcal{N}(0, 1)\) and see if our results are likely or unlikely
• In reality we usually don’t know \(\sigma\)

• We are usually testing for an unknown population value (i.e. \(\mu\))
• Commonly we are testing that an average difference is zero (\(\mu = 0\))
• The alternative is that \(\mu \neq 0\)
• We use our experiment to draw conclusions about \(\mu\)

\[ H_0: \mu = 0 \quad \text{vs} \quad H_A: \mu \neq 0 \]

If \(H_0\) is true, where would we expect \(Z\) to be?
If \(H_0\) is NOT true, where would we expect \(Z\) to be?

Would a value \(Z > 1\) be likely if \(H_0\) is TRUE?

Would a value \(>2\) be likely if \(H_0\) is TRUE?

• The area under all probability distributions adds up 1
• The area to the right of 2, is the probability of obtaining \(Z >2\)
• This is 0.023
• Thus if \(H_0\) is true, we know the probability of obtaining a \(Z\)-statistic \(>2\)

In our qPCR experiment, could the \(\Delta \Delta C_t\) values be either side of zero?

In our qPCR experiment, could the \(\Delta \Delta C_t\) values be either side of zero?

• Gene expression could go up or down after our treatment
• This means we also need to check the values for the other extreme
• This distribution is symmetric around zero:

\[p(|Z| > 2) = p(Z > 2) + P(Z < -2)\]

• Known as a two-sided test

This is the most common way of determining how much evidence we have against \(H_0\)

\[ Z = \frac{\bar{x} - \mu}{\sigma / \sqrt{n}} \]

1. We first calculate \(Z\), and compare to \(\mathcal{N}(0, 1)\)
2. We obtain the probability of obtaining a \(Z\)-statistic at least as extreme as our value
   • If \(H_0\) is true, \(Z\) will come from \(\mathcal{N}(0, 1)\)
   • If \(H_0\) is NOT true, we have no idea where \(Z\) will be.
   • It can be anywhere \(-\infty < Z < \infty\)

Definition

A \(p\) value is the probability of observing data as extreme, or more extreme than we have observed, if \(H_0\) is true.

To summarise:

1. We calculate a test statistic \(Z\) using \(\mu = 0\) and \(\sigma\)
2. Compare this to \(\mathcal{N}(0, 1)\) to find the probability (\(p\)) of observing data as extreme, or more extreme, than we observed if \(H_0\) is true
3. If \(p\) is low (e.g. \(p<0.05\)), we reject \(H_0\) as unlikely and infer \(H_A\)

In reality, we will never know the population variance (\(\sigma^2\)), just like we will never know \(\mu\)

• If we knew these values we wouldn’t need to do any experiments
• We can use our sample variance (\(s^2\)) to estimate \(\sigma^2\)

Due to the uncertainty introduced by using \(s^2\) instead of \(\sigma^2\) we can no longer compare to the \(Z \sim \mathcal{N}(0, 1)\) distribution.

Instead we use a \(T\)-statistic

\[ T = \frac{\bar{x} - \mu}{s / \sqrt{n}} \]

Then we compare to a \(t\)-distribution

The \(t\)-distribution is very similar to \(\mathcal{N}(0, 1)\)

• Bell-shaped & symmetrical around zero
• Has fatter tails \(\implies\) extreme values are more likely
• The parameter degrees of freedom (\(\nu\)) specifies how “fat” the tails are
• As \(\nu \rightarrow 0\) the tails get fatter

At their simplest:

\[ \text{df} = \nu = n - 1 \]

As \(n\) increases \(s^2 \rightarrow \sigma^2\) and \(\implies t_{\nu} \rightarrow Z\)

For our qPCR data, testing \(\mu = 0\):

\[ \begin{aligned} T &= \frac{\bar{x} - \mu}{s / \sqrt{n}} \\ &= \frac{2.5 - 0}{0.2943920289 / \sqrt{4}} \\ &= 16.984 \end{aligned} \]

• Our degrees of freedom here are \(\nu = 3\)
• By comparing to \(t_3\) we get

\[p = 0.00044\]

• Thus, the probability of observing data as (or more) extreme as this is very low if \(H_0\) was true (1 in ~2300 experiments).
• We reject \(H_0\) and infer that our alternative \(H_A\) is more likely

1. Define \(H_0\) and \(H_A\)
2. Calculate sample mean (\(\bar{x}\)) and variance (\(s^2\))
3. Calculate \(T\)-statistic and degrees of freedom (\(\nu\))
4. Compare to \(t_{\nu}\) and obtain probability of observing \(\bar{x}\) if \(H_0\) is true
5. Fail to reject or reject \(H_0\) if \(p < 0.05\) (or some other value)

1. Define \(H_0\) and \(H_A\)
2. Calculate sample mean (\(\bar{x}\)) and variance (\(s^2\))
3. Calculate \(T\)-statistic and degrees of freedom (\(\nu\))
4. Compare to \(t_{\nu}\) and obtain probability of observing \(\bar{x}\) if \(H_0\) is true
5. Fail to reject or reject \(H_0\) if \(p < 0.05\) (or some other value)

• This applies to most situations
• Assumes errors (deviations of our data around the mean) are normally distributed

For \(H_0: \mu_A = \mu_B\) Vs \(H_A: \mu_A \neq \mu_B\)

1. Calculate the two sample means: \(\bar{x}_A\) and \(\bar{x}_B\)
2. Calculate the two sample variances \(s_A^2\) and \(s_B^2\)
3. Calculate the pooled denominator: \(\text{SE}_{\bar{x}_A - \bar{x}_B}\)
   • Formula varies for equal/unequal variances
4. Calculate the degrees of freedom
   • Formula varies for equal/unequal variances

\[ T = \frac{\bar{x}_A - \bar{x}_B}{\text{SE}_{\bar{x}_A - \bar{x}_B}} \]

If \(H_0\) is true then

\[ T \sim t_{\nu} \]

1. We compare our test-statistic to this distribution
2. Are we likely to see this value (or more extreme) under \(H_0\)?
3. Fail to reject or Reject \(H_0\)

\(H_0: \mu_A = \mu_B\) Vs \(H_A: \mu_A \neq \mu_B\)

• Used for any two measurement groups which are not Normally Distributed.
• Assigns each measurement a rank
• Compares ranks between groups
• Determines probability of observing differences in ranks
• Also known as the Mann-Whitney Test
• Requires higher sample sizes than \(t\)-tests

• Used for \(2 \times 2\) tables (or \(m \times n\)) with counts and categories
  • Are relative proportions of one variable different depending on the value of the other variable?
• Analogous to a Chi-squared (\(\chi^2\)) test but more robust to small values
• Commonly used to test for enrichment of an event within one group above another group (GO terms; TF motifs; SNP Frequencies)

An example table

\(H_0:\) No association between allele frequencies and location
\(H_A:\) There is an association between between allele frequencies and location

• We find the probability of obtaining tables with more extreme distributions, holding row and column totals fixed
  • Sum the probabilities of each table more extreme than the observed pattern (in both directions)

## Reject null hypothesis of no association between 
## allele frequencies and loctaion

## P-value =  0.0305

A \(p\) value is the probability of observing data as (or more) extreme if \(H_0\) is true.

We commonly reject \(H_0\) if \(p < 0.05\)

How often would we incorrectly reject \(H_0\)?

A \(p\) value is the probability of observing data as (or more) extreme if \(H_0\) is true.

We commonly reject \(H_0\) if \(p < 0.05\)

How often would we incorrectly reject \(H_0\)?

About 1 in 20 times, we will see \(p < 0.05\) if \(H_0\) is true

• Type I errors are when we reject \(H_0\) but \(H_0\) is true
• Type II errors are when we fail to reject \(H_0\) when \(H_0\) is false

What are the consequences of each type of error?

What are the consequences of each type of error?

Type I: Waste $$$ chasing dead ends
Type II: We miss a key discovery

• In research, we usually try to minimise Type I Errors
• Increasing sample-size reduces Type II Errors

1. Imagine we are examining every human gene (\(m=\) 25,000) for differential expression using RNASeq
2. Imagine there are 1000 genes which are truly DE

How many times would we incorrectly reject \(H_0\) using \(p < 0.05\)

1. Imagine we are examining every gene (\(m=\) 25,000) for differential expression using RNASeq
2. Imagine there are 1000 genes which are truly DE

How many times would we incorrectly reject \(H_0\) using \(p < 0.05\)

We effectively have 25,000 tests, with 24,000 times \(H_0\) is true

\(\frac{25000 - 1000}{20} = 1200\) times

Could this lead to any research dead-ends?

This is an example of the Family-Wise Error Rate (i.e. Experiment-Wise Error Rate)

Definition

The Family-Wise Error Rate (FWER) is the probability of making one or more false rejections of \(H_0\)

In our example, the FWER \(\approx 1\)

What about if we lowered the rejection value to \(\alpha = 0.001\)?

We would incorrectly reject \(H_0\) once in every 1,000 times

\(\frac{25000 - 1000}{1000} = 24\) times

The FWER is still \(\approx 1\)

• If we set the rejection value to \(\alpha* = \frac{\alpha}{m}\) we control the FWER at the level \(\alpha\)
• To ensure that \(p\)(one or more Type I errors) = 0.05 in our example:
  
  \(\implies\) Reject \(H_0\) if \(p < \frac{0.05}{25000}\)

What are the consequences of this?

• If we set the rejection value to \(\alpha* = \frac{\alpha}{m}\) we control the FWER at the level \(\alpha\)
• To ensure that \(p\)(one or more Type I errors) = 0.05 in our example:
  
  \(\implies\) Reject \(H_0\) if \(p < \frac{0.05}{25000}\)

What are the consequences of this?

• Large increase in Type II Errors
• BUT what we find we are very confident about \(\implies\) we don’t waste time and money on dead ends!

• An alternative is to allow a small number of Type I Errors in our results \(\implies\) we have a False Discovery Rate (FDR)
• Instead of controlling the FWER at \(\alpha = 0.05\), if we control the FDR at \(\alpha = 0.05\) we allow up to 5% of our list to be Type I Errors

Most common procedure is the Benjamini-Hochberg

What advantage would this offer?

• An alternative is to allow a small number of Type I Errors in our results \(\implies\) we have a False Discovery Rate (FDR)
• Instead of controlling the FWER at \(\alpha = 0.05\), if we control the FDR at \(\alpha = 0.05\) we allow up to 5% of our list to be Type I Errors

Most common procedure is the Benjamini-Hochberg

What advantage would this offer?

• Lower Type II Errors
• 5% chance we chase a dead end

For those interested, the BH procedure for \(m\) tests is (not-examinable)

1. Arrange \(p\)-values in ascending order \(p_{(1)}, p_{(2)}, ..., p_{(m)}\)
2. Find the largest number \(k\) such that \(p_{(k)} \leq \frac{k}{m}\alpha\)
3. Reject \(H_0\) for all \(H_{(i)}\) where \(i \leq k\)

Controlling the FDR at \(\alpha = 0.05\)