---

{% raw %}

<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">

<div class="text_cell_render border-box-sizing rendered_html">

<h4 id="New-to-Plotly?">New to Plotly?<a class="anchor-link" href="#New-to-Plotly?">&#182;</a></h4><p>Plotly's Python library is free and open source! <a href="https://plot.ly/python/getting-started/">Get started</a> by dowloading the client and <a href="https://plot.ly/python/getting-started/">reading the primer</a>.

<br>You can set up Plotly to work in <a href="https://plot.ly/python/getting-started/#initialization-for-online-plotting">online</a> or <a href="https://plot.ly/python/getting-started/#initialization-for-offline-plotting">offline</a> mode, or in <a href="https://plot.ly/python/getting-started/#start-plotting-online">jupyter notebooks</a>.

<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">

<div class="text_cell_render border-box-sizing rendered_html">

<h3 id="Normality-Tests">Normality Tests<a class="anchor-link" href="#Normality-Tests">&#182;</a></h3>

<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">

<div class="text_cell_render border-box-sizing rendered_html">

<p>In statistics, normality tests are used to determine whether a data set is modeled for Normal (Gaussian) Distribution. Many statistical functions require that a distribution be normal or nearly normal.</p>

<p>There are several methods of assessing whether data are normally distributed or not. They fall into two broad categories: <em>graphical</em> and <em>statistical</em>.

<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">

<div class="text_cell_render border-box-sizing rendered_html">

<h3 id="Test-Dataset">Test Dataset<a class="anchor-link" href="#Test-Dataset">&#182;</a></h3>

<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">

<div class="text_cell_render border-box-sizing rendered_html">

<p>Let's first develop a test dataset that we can use throughout this tutorial.</p>

<p><em>The tutorial below imports <a href="http://www.numpy.org/">NumPy</a>, <a href="https://plot.ly/pandas/intro-to-pandas-tutorial/">Pandas</a>, and <a href="https://www.scipy.org/">SciPy</a>.</em></p>

<span class="kn">import</span> <span class="nn">scipy</span>

<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">

<div class="text_cell_render border-box-sizing rendered_html">

<p><em>Generate Gaussian Data</em></p>

<span class="nb">print</span><span class="p">(</span><span class="s1">&#39;mean=</span><span class="si">%.3f</span><span class="s1"> stdv=</span><span class="si">%.3f</span><span class="s1">&#39;</span> <span class="o">%</span> <span class="p">(</span><span class="n">np</span><span class="o">.</span><span class="n">mean</span><span class="p">(</span><span class="n">gauss_data</span><span class="p">),</span> <span class="n">np</span><span class="o">.</span><span class="n">std</span><span class="p">(</span><span class="n">gauss_data</span><span class="p">)))</span>

<div class="output_area">

<div class="output_subarea output_stream output_stdout output_text">

<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">

<div class="text_cell_render border-box-sizing rendered_html">

<p>We can see that the mean and standard deviation are reasonable but rough estimations of the true underlying population mean and standard deviation, given the small-ish sample size.</p>

<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">

<div class="text_cell_render border-box-sizing rendered_html">

<h3 id="Histogram-Plot">Histogram Plot<a class="anchor-link" href="#Histogram-Plot">&#182;</a></h3><p>A simple and commonly used plot to quickly check the distribution of a sample of data is the histogram.</p>

<p>In the histogram, the data is divided into a pre-specified number of groups called bins. The data is then sorted into each bin and the count of the number of observations in each bin is retained.</p>

<span class="n">py</span><span class="o">.</span><span class="n">iplot</span><span class="p">([</span><span class="n">trace</span><span class="p">],</span> <span class="n">filename</span><span class="o">=</span><span class="s1">&#39;normality-histogram&#39;</span><span class="p">)</span>

<div class="output_area">

<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">

<div class="text_cell_render border-box-sizing rendered_html">

<p>We can see a Gaussian-like shape to the data, that although is not strongly the familiar bell-shape, is a rough approximation.</p>

<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">

<div class="text_cell_render border-box-sizing rendered_html">

<h3 id="Quantile-Quantile-Plot">Quantile-Quantile Plot<a class="anchor-link" href="#Quantile-Quantile-Plot">&#182;</a></h3>

<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">

<div class="text_cell_render border-box-sizing rendered_html">

<p>Another popular plot for checking the distribution of a data sample is the quantile-quantile plot, Q-Q plot, or QQ plot for short.</p>

<p>This plot generates its own sample of the idealized distribution that we are comparing with, in this case the Gaussian distribution. The idealized samples are divided into groups (e.g. 5), called quantiles. Each data point in the sample is paired with a similar member from the idealized distribution at the same cumulative distribution.</p>

<span class="n">qqplot_data</span> <span class="o">=</span> <span class="n">qqplot</span><span class="p">(</span><span class="n">gauss_data</span><span class="p">,</span> <span class="n">line</span><span class="o">=</span><span class="s1">&#39;s&#39;</span><span class="p">)</span><span class="o">.</span><span class="n">gca</span><span class="p">()</span><span class="o">.</span><span class="n">lines</span>

<span class="n">py</span><span class="o">.</span><span class="n">iplot</span><span class="p">(</span><span class="n">fig</span><span class="p">,</span> <span class="n">filename</span><span class="o">=</span><span class="s1">&#39;normality-QQ&#39;</span><span class="p">)</span>

<div class="output_area">

<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">

<div class="text_cell_render border-box-sizing rendered_html">

<p>Running the example creates the QQ plot showing the scatter plot of points in a diagonal line, closely fitting the expected diagonal pattern for a sample from a Gaussian distribution.</p>

<p>There are a few small deviations, especially at the bottom of the plot, which is to be expected given the small data sample.</p>

<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">

<div class="text_cell_render border-box-sizing rendered_html">

<h3 id="Statistical-Normality-Tests">Statistical Normality Tests<a class="anchor-link" href="#Statistical-Normality-Tests">&#182;</a></h3>

<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">

<div class="text_cell_render border-box-sizing rendered_html">

<p>There are many statistical tests that we can use to quantify whether a sample of data looks as though it was drawn from a Gaussian distribution.</p>

<p>Each test makes different assumptions and considers different aspects of the data.</p>

<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">

<div class="text_cell_render border-box-sizing rendered_html">

<h3 id="Shapiro-Wilk-Test">Shapiro-Wilk Test<a class="anchor-link" href="#Shapiro-Wilk-Test">&#182;</a></h3>

<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">

<div class="text_cell_render border-box-sizing rendered_html">

<p>The <a href="https://en.wikipedia.org/wiki/Shapiro%E2%80%93Wilk_test">Shapiro-Wilk test</a> evaluates a data sample and quantifies how likely it is that the data was drawn from a Gaussian distribution, named for Samuel Shapiro and Martin Wilk.</p>

<p>In practice, the Shapiro-Wilk test is believed to be a reliable test of normality, although there is some suggestion that the test may be suitable for smaller samples of data, e.g. thousands of observations or fewer.</p>

<span class="n">py</span><span class="o">.</span><span class="n">iplot</span><span class="p">(</span><span class="n">swt_table</span><span class="p">,</span> <span class="n">filename</span><span class="o">=</span><span class="s1">&#39;shapiro-wilk-table&#39;</span><span class="p">)</span>

<div class="output_area">

<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">

<div class="text_cell_render border-box-sizing rendered_html">

<p>Running the above example calculates the statistic and p-value.</p>

<p>The p-value is interested and finds that the data is likely drawn from a Gaussian distribution.</p>

<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">

<div class="text_cell_render border-box-sizing rendered_html">

<h3 id="Anderson-Darling-Test">Anderson-Darling Test<a class="anchor-link" href="#Anderson-Darling-Test">&#182;</a></h3>

<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">

<div class="text_cell_render border-box-sizing rendered_html">

<p><a href="https://en.wikipedia.org/wiki/Anderson%E2%80%93Darling_test">Anderson-Darling Test</a> is a statistical test that can be used to evaluate whether a data sample comes from one of among many known data samples, named for Theodore Anderson and Donald Darling.</p>

<p>It can be used to check whether a data sample is normal. The test is a modified version of a more sophisticated nonparametric goodness-of-fit statistical test called the <a href="https://en.wikipedia.org/wiki/Kolmogorov%E2%80%93Smirnov_test">Kolmogorov-Smirnov test</a>.</p>

<span class="n">py</span><span class="o">.</span><span class="n">iplot</span><span class="p">(</span><span class="n">andar_table</span><span class="p">,</span> <span class="n">filename</span><span class="o">=</span><span class="s1">&#39;anderson-darling-table&#39;</span><span class="p">)</span>

<div class="output_area">

<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">

<div class="text_cell_render border-box-sizing rendered_html">

<p>Running the example calculates the statistic on the test data set and the critical values are tabulated.</p>

<p>Critical values in a statistical test are a range of pre-defined significance boundaries at which the H0 can be failed to be rejected if the calculated statistic is less than the critical value. Rather than just a single p-value, the test returns a critical value for a range of different commonly used significance levels.</p>

<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">

<div class="text_cell_render border-box-sizing rendered_html">

<h3 id="D'Agostino's-$K^{2}$Test">D'Agostino's $K^{2}$Test<a class="anchor-link" href="#D'Agostino's-$K^{2}$Test">&#182;</a></h3>

<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">

<div class="text_cell_render border-box-sizing rendered_html">

<p>The <a href="https://en.wikipedia.org/wiki/D%27Agostino%27s_K-squared_test">D'Agostino's $K^{2}$ test</a> calculates summary statistics from the data, namely kurtosis and skewness, to determine if the data distribution departs from the normal distribution, named for Ralph D’Agostino.</p>

<span class="n">py</span><span class="o">.</span><span class="n">iplot</span><span class="p">(</span><span class="n">normt_table</span><span class="p">,</span> <span class="n">filename</span><span class="o">=</span><span class="s2">&quot;D&#39;Agostino-test-table&quot;</span><span class="p">)</span>

<div class="output_area">

<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">

<div class="text_cell_render border-box-sizing rendered_html">

<p>Running the above example calculates the statistic and p-value.

The p-value is interpreted against an alpha of 5% and finds that the test dataset does not significantly deviate from normal.</p>

<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">

<div class="text_cell_render border-box-sizing rendered_html">

<h4 id="Conclusion">Conclusion<a class="anchor-link" href="#Conclusion">&#182;</a></h4><p>We have covered a few normality tests, but this is not all of the tests that exist. It is recommended to use all possible tests on your data, where appropriate.</p>

<p><strong><em>How to interpret the results?</em></strong></p>

<li>Start looking into the use of nonparametric statistical methods instead of the parametric methods.</li>

<li>If some of the methods suggest that the sample is Gaussian and some not, then perhaps take this as an indication that your data is Gaussian-like.</li>

<li>In many situations, you can treat your data as though it is Gaussian and proceed with your chosen parametric statistical methods.</li>