statistical_testing.Rmd 5.71 KB
Newer Older
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
---
title: "Statistical testing for high-throughput experiments"
author: "Thomas Schwarzl<br/>.... based on <br/>Modern Statistics for Modern Biology - Susan Holmes, Wolfgang Huber"
output:
  slidy_presentation:
    theme: paper
---

# design an experiment already with the analysis in mind

_"To consult the statistician after an experiment is finished is often merely to ask him to conduct a post mortem examination. He can perhaps say what the experiment died of"_ - R.A. Fisher


# start with the analysis as soon as you have acquired the first data

Thomas Schwarzl's avatar
Thomas Schwarzl committed
16
Don't wait until everything is collected and it's too late to troubleshoot
17 18


Thomas Schwarzl's avatar
Thomas Schwarzl committed
19
# Start writing the paper while you're analyzing the data
20

Thomas Schwarzl's avatar
Thomas Schwarzl committed
21
once you're writing, you realize what you should have done to properly support them
22 23 24 25

# Types of experiments

- measurements have limited precision and accuracy
Thomas Schwarzl's avatar
Thomas Schwarzl committed
26
 - preliminary data to estimate them
27 28
- directly or indirectly measurement 
- side effects of treatment conditions
Thomas Schwarzl's avatar
Thomas Schwarzl committed
29
- interfering signals or "background noise"
30 31 32 33 34
- limited sample sizes 




Thomas Schwarzl's avatar
Thomas Schwarzl committed
35 36
# Types of experiments
### controlled experiment
37 38 39 40 41

- (model) system under study
- the environmental conditions
- the experimental readout.

Thomas Schwarzl's avatar
Thomas Schwarzl committed
42
_e.g. we could have a well-characterized cell line growing in laboratory conditions on defined media, temperature and atmosphere, we will administer a precise amount of a drug, and after 72h we measure the activity of a specific pathway reporter._
43

Thomas Schwarzl's avatar
Thomas Schwarzl committed
44 45
# Types of experiments
### study
46 47 48 49 50

important conditions that may affect the measured outcome are not under control of the researcher, usually because of ethical concerns or logistical constraints. 

_e.g. in an ecological field study, this could be the weather, the availabilty of nutrition resources or the activity of predators_

Thomas Schwarzl's avatar
Thomas Schwarzl committed
51 52
# Types of experiments
### observational study
53 54 55

_e.g. in a clinical trial, this might be the assignment of the individual subjects to groups. Since there are many possibilities for confounding _

Thomas Schwarzl's avatar
Thomas Schwarzl committed
56
correlation is not causation!
57

Thomas Schwarzl's avatar
Thomas Schwarzl committed
58 59
# Types of experiments
### signal / noise ratio
60

Thomas Schwarzl's avatar
Thomas Schwarzl committed
61
_"Generally speaking, a well-designed experiment is one that is sufficiently powered and one in which technical artifacts and biological features that may systematically affect measurements are balanced, randomized or controlled in some other way in order to minimize opportunities for multiple explanations for the effect(s) under study."_ - Bacher and Kendziorski 2016
62

Thomas Schwarzl's avatar
Thomas Schwarzl committed
63
# Bias and noise
64

Thomas Schwarzl's avatar
Thomas Schwarzl committed
65 66
<center>
![](img/TargetVariance.png){height=150}
67

Thomas Schwarzl's avatar
Thomas Schwarzl committed
68
noise: "averages out" if we just perform enough replicates
69 70 71



Thomas Schwarzl's avatar
Thomas Schwarzl committed
72
![](img/TargetBias.png){height=150}
73

Thomas Schwarzl's avatar
Thomas Schwarzl committed
74 75
bias: remains, becomes more apparent with enough replicates
</center>
76

Thomas Schwarzl's avatar
Thomas Schwarzl committed
77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105
# Confounding factors / Batch effects
<center>
![](img/chap10-chap10-r-confounding-1-1.png){height=500}
</center>

# Confounding factors / Batch effects
![](img/batcheffect.png){height=500}

# Batch effect vs confounding 

Confounding need not only be between a biological and a technical variable, it can also be more subtle. For instance, the biomarker might have nothing to do with the disease directly – it might just be a marker of a life style that causes the disease (as well as other things), or of an inflammation that is caused by the disease (as well as by many other things), etc.

# Effect size and replicates
<center>
![](img/chap10-Design-effectsize-1.png){height=500}
</center>



# Blockbox

Block what you can, randomize what you cannot.
(George Box, 1978)

![](img/chap10-Design-blockbox-1.png){height=400}

# Replicates

![Figure 13.2 from Book](img/chap10-Design-comparesamplesize-1.png)
106

Thomas Schwarzl's avatar
Thomas Schwarzl committed
107
# Biological Replicates vs technical replicates
108

Thomas Schwarzl's avatar
Thomas Schwarzl committed
109
- A person is weighed on milligram precision scales, with 20 replicates. He follows the diet, and four weeks later, he is weighed again, with 20 replicates.
110

Thomas Schwarzl's avatar
Thomas Schwarzl committed
111
- Ten people weigh themselves once on their bathroom scales and report the number. Four weeks later, they weigh themselves and report again.
112 113


Thomas Schwarzl's avatar
Thomas Schwarzl committed
114 115 116 117 118 119 120 121 122 123 124 125 126 127 128
# How many replicates do I need

The package pwr 

*pwr.2p.test*, *pwr.chisq.test*, *pwr.f2.test* 
 
```{r, eval = F}
library("pwr")
str(pwr.t.test)
```

If you call the function with a value for power and effect size, it will return the sample size needed, or if you specify the sample size and effect size, it returns the power.

```{r, eval=F}
pwr.t.test(n = 15, d = 0.4, sig.level = 0.05, type = "paired")
129 130
```

Thomas Schwarzl's avatar
Thomas Schwarzl committed
131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162
d is effect size (Cohen's d) - difference between the means divided by the pooled standard deviation


Test	 | small | medium	| large
-------|-------|--------|-------------
tests for proportions (p) |	0.2	| 0.5 | 0.8
tests for means (t)	| 0.2	| 0.5	| 0.8
chi-square tests (chisq) | 0.1 | 0.3 | 0.5
correlation test (r) | 0.1 | 0.3 | 0.5
anova (anov) | 0.1 | 0.25 | 0.4
general linear model | (f2)	0.02 | 0.15 | 0.35



# Fold-changes

- fold changes and proportions are ratios. 
- denominator is a random variable (as it changes from lab to lab and probably from experiment to experiment), which can create high instability and very unequal variances between experiments
 - *transformations!*

# Regular and catastrophic noise

- Regular noise can be modelled by simple probability models such as
  - independent normal distributions
  - Poisson
  - or mixtures such as gamma–Poisson or Laplace. 
- to take such noise into account in our data analyses and to compute the probability of extraordinarily large or small values.


In the real world, this is only part of the story: measurements can be completely off scale (a sample swap, a contamination or a software bug), and they can go awry all at the same time (a whole microtiter plate went bad, affecting all data measured from it). Such events are hard to model or even correct for – our best chance to deal with them is data quality assessment, outlier detection and documented removal.


163

Thomas Schwarzl's avatar
Thomas Schwarzl committed
164
# Mean-variance relationships and variance-stabilizing transformations
165

Thomas Schwarzl's avatar
Thomas Schwarzl committed
166
etc