{ "cells": [ { "cell_type": "markdown", "id": "e3ed112d-7cf4-4b16-8250-f82c1ea06256", "metadata": {}, "source": [ "# Statistical Formulas\n", "\n", "These formulas are similar to many such sections in statistics textbooks and provide guidance on how perform basic statistical hypothesis testing and other basic statistical tasks." ] }, { "cell_type": "markdown", "id": "5ce020ed-7f61-4891-8df1-5319cfc3bcb7", "metadata": {}, "source": [ "## Probability\n", "\n", "The following rules of probability will be utilized for the entire course:\n", "\n", "- For any event $A$, $0\\leq P(A) \\leq 1$\n", "- The sample space $S$ has probability $P(S) =1$\n", "- For disjoint event sets $A$ and $B$, \n", "\n", "$$P(A \\text{ or } B) = P(A) + P(B)$$\n", "\n", "- In general, for event sets $A$ and $B$,\n", "\n", "$$P(A \\cup B) = P(A) + P(B)-P(A\\cap B)$$\n", "\n", "- $P(A \\text{ does not occur}) =1-P(A)$\n", "- For a discrete probability density function $p(x)$:\n", " \n", " - $0\\leq p(x_i) \\leq 1 \\hspace{.25cm}\\text{for all}\\hspace{.25cm} 1\\leq i \\leq n$\n", " \n", " - $\\sum p(x_i)=1$\n", "\n", "- For any continuous probability density function $f(x)$:\n", "\n", " - $f(x) \\geq 0 \\hspace{.25cm}\\text{for all}\\hspace{.25cm} x\\in(-\\infty,+\\infty)$\n", "\n", " - $\\displaystyle\\int\\limits_{-\\infty}^{+\\infty} f(x)dx=1$" ] }, { "cell_type": "markdown", "id": "d4b67a5c-c7e8-4126-a430-89c8fe0c4e31", "metadata": {}, "source": [ "## Exploring Data: Distributions and Descriptives\n", "\n", "Look for overall pattern (shape, center, spread) and deviations (outliers).\n", "\n", "- Mean:\n", "\n", "$$\\bar{x}=\\frac{x_1+x_2+...+x_n}{n}=\\frac{1}{n}\\sum{x_i}$$\n", "\n", "- Standard deviation:\n", "\n", "$$s=\\sqrt{\\frac{1}{n-1}\\sum{(x_i-\\bar{x})^2}}$$\n", "\n", "- Median: Arrange all observations from smallest to largest. The median $M$ is located $\\frac{(n + 1)}{2}$ from the beginning of this list. We also use the notation $\\tilde{x}$ for the median.\n", "- Quartiles: The first quartile $Q1$ is the median of the observations whose position in the ordered list is to the left of the location of the overall median. The third quartile $Q3$ is the median of the observations to the right of the location of the overall median.\n", "- Standardized value of $x$:\n", "$$z=\\frac{x-\\mu}{\\sigma}$$" ] }, { "cell_type": "markdown", "id": "421170e8-9a5b-4f8e-8be6-359a739e2f6c", "metadata": {}, "source": [ "### Five Number Summary\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
StatisticSymbol
Minimummin
1st QuartileQ1
Medianmed
3rd QuartileQ3
Maximummax
" ] }, { "cell_type": "markdown", "id": "7e1a3392-234b-43bb-8bb7-0ebbfc076e3f", "metadata": {}, "source": [ "## Exploring Data: Relationships\n", "\n", "Look for overall pattern (form, direction, strength) and deviations (outliers, influential observations).\n", "\n", "- Correlation (conceptual form using $z$-scores):\n", "\n", "$$\\begin{align*} \n", "r=&\\frac{1}{n-1}\\sum z_{x_i}z_{y_i} \\implies\\\\\n", "r=&\\frac{1}{n-1}\\sum\\left(\\frac{x_i-\\bar{x}}{s_x}\\right)\\left(\\frac{y_i-\\bar{y}}{s_y}\\right)\n", "\\end{align*}$$\n", "\n", "- Least-squares regression line:\n", "\n", "$$\\begin{array}{lc}\n", "&\\hat{y} = a + bx \\\\[.25cm]\n", "\\text{with slope}& b = r\\cdot\\frac{s_y}{s_x} \\\\[.1cm]\n", "\\text{and intercept }& a = \\bar{y}- b\\bar{x}\\\\[.1cm]\n", "\\text{thus containing}&(\\bar{x},\\bar{y})\\\\\n", "\\end{array}$$\n", "\n", "The slope $b$ of the regression line (or line of best fit) is the standard change (deviation) in $y$ over the standard change (deviation) in $x$ multiplied by the strength (correlation) of the relationship between the $x$- and $y$-variables. \n", "- Residuals: for any data point $(x_i,y_i)$:\n", "\n", "$$\\text{Residual} = \\text{Observed }y_i-\\text{Predicted }y_i$$\n", "or$$y_i-\\hat{y_i}=y_i-a-bx_i$$\n", "\n", "which gives the vertical distance between the actual $y$-value and the line of best fit. The ``line of best\" is the ``least squares line\" that minimizes the sum of the squared residuals, e.g. minimizes the total vertical distance between the actual data and the line of best fit (using calculus)." ] }, { "cell_type": "markdown", "id": "259f756f-3b98-487a-8356-e54879fcc9e8", "metadata": {}, "source": [ "## Sampling Distributions\n", "\n", "- $\\bar{x}$ has mean $\\mu$ and standard deviation $\\frac{\\sigma}{\\sqrt{n}}$\n", "- $\\bar{x}$ has a Normal distribution if the population distribution is Normal.\n", "- Central limit theorem: $X$ is approximately Normal when $n$ is large.\n", "\n", "## Basics of Inference\n", "\n", "- $z$ confidence interval for a population mean ($\\sigma$ known, SRS from Normal population):\n", "\n", "$$\\mu\\in\\bar{x}\\pm z^*\\frac{\\sigma}{\\sqrt{n}} \\hspace{1cm}\\text{with }z^*\\text{ from } N(0,1)$$\n", "\n", "- Sample size for desired margin of error $m$:\n", "\n", "$$n=\\left(\\frac{z^*\\sigma}{m}\\right)^2$$\n", "\n", "- $z$ test statistic for $H_0 : \\mu = \\mu_0$ ($\\sigma $ known, SRS from Normal population, rarely used in modern practice):\n", "\n", "$$z=\\frac{\\bar{x}-\\mu}{\\frac{\\sigma}{\\sqrt{n}}}\\hspace{.5cm}$$\n", "\n", "with $p$-values from $N(0,1)$." ] }, { "cell_type": "markdown", "id": "f424b619-842f-472a-b81a-0e00ff6c1930", "metadata": {}, "source": [ "### Inference About Means\n", "\n", "- The $t$ confidence interval for a population mean (SRS from Normal Population):\n", "\n", "$$\\mu\\in\\bar{x}\\pm t^*\\frac{s}{\\sqrt{n}}$$\n", "\n", "with $t^*$ from the $t$-distribution with degrees of freedom $n-1$.\n", "\n", "- $t$ test statistic $H_0:\\mu=\\mu_0$ (SRS from Normal Population):\n", "\n", "$$t=\\frac{\\bar{x}-\\mu_0}{\\frac{s}{\\sqrt{n}}}$$\n", "\n", "with $P$-values from the $t$-distribution with degrees of freedom$n-1$.\n", "\n", "- Matched pairs: To compare the responses to the two treatments, apply the one-sample $t$ procedures to the observed differences.\n", "\n", "- Two-sample confidence interval for $\\mu_1-\\mu_2$ (independent SRSs from the Normal Populations):\n", "\n", "$$(\\mu_1-\\mu_2)\\in (\\bar{x}_1-\\bar{x}_2)\\pm t^*\\sqrt{\\frac{s_1^2}{n_1}+\\frac{s_2^2}{n_2}}$$\n", "\n", "with conservative $t^*$ from $t$ with $df=\\min(n_1-1,n_2-1)$, or use software.\n", "- Two-sample $t$ test statistic for $H_0 : \\mu_1 = \\mu_2$ (independent SRSs from Normal populations):\n", "\n", "$$t=\\frac{\\bar{x}_1-\\bar{x}_2}{\\sqrt{\\frac{s_1^2}{n_1}+\\frac{s_2^2}{n_2}}}$$\n", "\n", "with conservative $p$-values from $t$ with $df=\\min(n_1-1,n_2-1)$, or use software." ] }, { "cell_type": "markdown", "id": "5274ee69-988a-4d20-a492-80e7074ed8b8", "metadata": {}, "source": [ "### Inference About Proportions\n", "\n", "- Sampling distribution of a sample proportion: when the population and the sample size are both large and $p$ is not close to $0$ or $1$, $\\hat{p}$ is approximately $N\\left(p,\\sqrt{p(1-p)/n}\\right)$.\n", "\n", "- Large-sample $z$ confidence interval for $p$:\n", "\n", "$$\\text{prop}\\in\\hat{p}\\pm z^*\\sqrt{\\frac{\\hat{p}(1-\\hat{p})}{n}}$$\n", "\n", "with $z^*$ from $N(0,1)$. Plus Four Method greatly improves accuracy: same formula after adding four imaginary observations: two success and two failures.\n", "\n", "- The $z$ test statistic for $H_0: p=p_0$ (large SRS):\n", "\n", "$$z=\\frac{\\hat{p}-p_0}{\\sqrt{\\frac{p_0(1-p_0)}{n}}}$$\n", "\n", "with $p$-values from $N(0,1)$}$.\n", "\n", "- Sample size for desired margin of error $m$: $$n=\\left(\\frac{z^*}{m}\\right)^2p^*(1-p^*)$$where $p^*$ is a guessed value for $p$, or $p^*=0.5$\n", "\n", "- Large-sample $z$ confidence interval for $p_1-p_2$ where\n", "\n", "$$(p_1-p_2)\\in(\\hat{p}_1-\\hat{p}_2) \\pm z^*\\text{SE}$$\n", "\n", "with $z^*$ from $N(0,1)$ and standard error: \n", "\n", "$$\\text{SE}=\\sqrt{\\frac{\\hat{p}_1(1-\\hat{p}_1)}{n_1}+\\frac{\\hat{p}_2(1-\\hat{p}_2)}{n_2}}$$\n", "\n", "Plus 4 Method greatly improves accuracy: use the same formulas after adding one success and one failure to each sample.\n", "\n", "- Two-sample $z$ test statistic for $H_0 : p_1=p_2$ (large independent SRSs):\n", " \n", "$$z=\\frac{\\hat{p}_1-\\hat{p}_2}{\\sqrt{\\hat{p}(1-\\hat{p})\\left(\\frac{1}{n_1}+\\frac{1}{n_2}\\right)}}$$\n", "\n", "where $\\hat{p}$ is the pooled (overall) proportion of successes." ] }, { "cell_type": "markdown", "id": "68c8c9cf-fe11-436d-bd44-d5bba8265f7d", "metadata": {}, "source": [ "## The $\\chi^2$ Test\n", "\n", "- Calculating cells of Expected Matrix:\n", "\n", "$$\\text{expected count}=\\frac{\\text{row total }\\times\\text{ column total}}{\\text{table total}}$$\n", "\n", "- $\\chi^2$ test statistic for testing whether the row and column variables in an $r\\times c$ table are unrelated (expected cell counts not too small):\n", "\n", "$$X^2=\\sum{\\frac{(\\text{observed count}-\\text{expected count})^2}{\\text{expected count}}}$$\n", "\n", "with $p$-values from the $\\chi^2$ distribution with \n", "\n", "$$df=(r-1)(c-1)$$" ] }, { "cell_type": "markdown", "id": "8cd3ca0d-f1b3-4ea3-bf6b-626151828b49", "metadata": {}, "source": [ "## Inference for Regression\n", "\n", "- Conditions for regression inference: $n$ observations on $x$ and $y$. The response $y$ for any fixed $x$ has a Normal distribution with mean given by the true regression line $y=\\alpha+\\beta x$ and standard deviation $\\sigma$. Parameters are $\\alpha, \\beta, \\sigma$.\n", "\n", "- Estimate $\\alpha$ by the intercept $a$ and $\\beta$ by the slope $b$ of the least-squares line.\n", "\n", "- Estimate $\\sigma$ by the regression standard error: $$s=\\sqrt{\\frac{1}{n-2}\\sum{\\text{residual}}^2}$$\n", "\n", "- A $t$ confidence interval for regression slope $\\beta$ can be calculated by hand as follows:\n", "\n", "$$b\\pm t^* \\text{SE}_b\\hspace{1cm} t^* \\text{ from } t(n-2)$$\n", "\n", "where \n", "\n", "$$SE_b=\\frac{\\sqrt{\\frac{\\sum (y_i - \\hat{y}_i)^2}{n - 2}}}{\\sqrt{\\sum(x_i - \\bar{x})^2}}$$\n", "\n", "However, best practice strongly indicates *using software to calculate all standard errors in regression*.\n", "\n", "- Testing for no correlation, $H_0:\\rho=0$:\n", "\n", "$$t=r\\cdot\\sqrt{\\frac{n-2}{1-r^2}}\\hspace{1cm} t^* \\text{ from } t(n-2)$$\n", "\n", "where $\\rho$ is the parameter that $r$ estimates." ] }, { "cell_type": "markdown", "id": "e8aeef47-5c78-4ba5-ac2d-1ce710c8879e", "metadata": {}, "source": [ "## ANOVA\n", "\n", "- ANOVA tests whether $k$ populations have the same mean based on independent SRS's from $k$ normal populations. The $p$-values come from the $F$ distribution with $k-1$ and $N-k$ degrees of freedom, where $N$ is the total observations in all samples.\n", "- Describe the data using the $k$ sample means $(\\bar{x}_i)$ and standard deviations $(s_i)$ and side-by-side graphs of the samples. The overall sample size is $N=n_1+n_2+\\ldots+n_k$, and the grand mean is $(\\bar{x})$, the airthmetic average of all $N$ observations.\n", "- The $F$ test statistic is given by\n", "\n", "$$F=\\frac{\\text{MSB}}{\\text{MSW}}\\hspace{1cm}\\text{or}\\hspace{1cm}F=\\frac{\\text{MSG}}{\\text{MSE}}$$ \n", "\n", "where MSB is *between group* mean sum of squares (or MS Factor for TI calculators):\n", "\n", "$$\\text{MSB}=\\frac{n_1(\\bar{x}_1-\\bar{x})^2+...+n_k(\\bar{x}_k-\\bar{x})^2}{k-1}$$\n", "\n", "and MSW is *within group* mean sum of squares and (or MSE for mean sum of squares for the *error* on TI):\n", "\n", "$$\\text{MSW}=\\text{MSE}=\\frac{\\sum\\limits_{i=1}^k \\sum\\limits_{j=1}^{n_i}(x_{i_j}-\\bar{x}_i)^2}{N-k}$$\n", "\n", "However, with some algebra, this reduces to a more computationally friendly version:\n", "\n", "$$\\text{MSW}=\\frac{(n_1-1)s_1^2+...+(n_k-1)s_k^2}{N-k}$$" ] }, { "cell_type": "markdown", "id": "d2a67da5-060f-4c6d-a69a-15a3e055f870", "metadata": {}, "source": [ "### ANOVA Post Hoc Testing\n", "\n", "After finding significant differences between the sample means, we employ a \\textit{post hoc} test to ferret out significantly different group means. Tukey's HSD (Honestly Significant Difference) is the most common. However, Tukey's HSD is liberal and tends to err on the side of finding significance differences. Using Scheffe's or Dunnet's may be preferable where a more conservative or flexible approach is appropriate.\n", "\n", "- Tukey's HSD must be computed for each individual pair of group means and depends upon the harmonic mean (see below). The HSD value is given by:\n", "\n", "$$\\text{HSD}_{ij}=q^*\\sqrt{\\frac{\\text{MSW}}{n_{ij}}}$$\n", "\n", "where $q^*$ is a value from the Studentized Range Statistic (found in a table based upon $\\alpha$ and degrees of freedom).\n", "\n", "- We flag as significantly different any pair of groups $i$ and $j$ such that\n", "\n", "$$|\\bar{x}_i-\\bar{x}_j|>HSD_{ij}$$\n", "\n", "### Harmonic Mean\n", "\n", "The harmonic mean $H$ is the reciprocal of the arithmetic mean of the reciprocals. For 2 real numbers $r$ and $s$ we have\n", "\n", "$$H_{rs}=\\left(\\frac{\\frac{1}{r}+\\frac{1}{s}}{2}\\right)^{-1}$$\n", "\n", "which simplifies to\n", "\n", "$$H_{rs}=\\frac{2rs}{r+s}$$\n", "\n", "In the HSD calculation, $n_{ij}$ refers to the harmonic mean of group sizes $n_i$ and $n_j$:\n", "\n", "$$n_{ij}=\\frac{2n_in_j}{n_i+n_j}$$" ] }, { "cell_type": "code", "execution_count": null, "id": "1d3b8333-43a4-467f-b5c5-1ccb289ecec3", "metadata": {}, "outputs": [], "source": [] }, { "cell_type": "code", "execution_count": null, "id": "b35daec9-545a-4c69-89bc-896b51deda21", "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "R", "language": "R", "name": "ir" }, "language_info": { "codemirror_mode": "r", "file_extension": ".r", "mimetype": "text/x-r-source", "name": "R", "pygments_lexer": "r", "version": "3.6.1" } }, "nbformat": 4, "nbformat_minor": 5 }