.MCAD 304020000 1 74 63 0
.CMD PLOTFORMAT
0 0 1 1 1 0 0 1 1 
0 0 1 1 1 0 0 1 1 
0 1 0 0 1 1 NO-TRACE-STRING
0 2 1 0 1 1 NO-TRACE-STRING
0 3 2 0 1 1 NO-TRACE-STRING
0 4 3 0 1 1 NO-TRACE-STRING
0 1 4 0 1 1 NO-TRACE-STRING
0 2 5 0 1 1 NO-TRACE-STRING
0 3 6 0 1 1 NO-TRACE-STRING
0 4 0 0 1 1 NO-TRACE-STRING
0 1 1 0 1 1 NO-TRACE-STRING
0 2 2 0 1 1 NO-TRACE-STRING
0 3 3 0 1 1 NO-TRACE-STRING
0 4 4 0 1 1 NO-TRACE-STRING
0 1 5 0 1 1 NO-TRACE-STRING
0 2 6 0 1 1 NO-TRACE-STRING
0 3 0 0 1 1 NO-TRACE-STRING
0 4 1 0 1 1 NO-TRACE-STRING
0 1 1 21 15 0 0 3 
.CMD FORMAT  rd=d ct=10 im=i et=3 zt=15 pr=3 mass length time charge temperature tr=0 vm=0
.CMD SET ORIGIN 0
.CMD SET TOL 0.001000000000000
.CMD SET PRNCOLWIDTH 8
.CMD SET PRNPRECISION 4
.CMD PRINT_SETUP 1.200000 0.979167 1.200000 1.200000 0
.CMD HEADER_FOOTER 1 1 *empty* *empty* *empty* 0 1 *empty* *empty* *empty*
.CMD HEADER_FOOTER_FONT fontID=14 family=Arial points=10 bold=0 italic=0 underline=0 colrid=2006071309
.CMD HEADER_FOOTER_FONT fontID=15 family=Arial points=10 bold=0 italic=0 underline=0 colrid=2006071309
.CMD DEFAULT_TEXT_PARPROPS 0 0 0
.CMD DEFINE_FONTSTYLE_NAME fontID=0 name=Variables
.CMD DEFINE_FONTSTYLE_NAME fontID=1 name=Constants
.CMD DEFINE_FONTSTYLE_NAME fontID=2 name=Text
.CMD DEFINE_FONTSTYLE_NAME fontID=4 name=User^1
.CMD DEFINE_FONTSTYLE_NAME fontID=5 name=User^2
.CMD DEFINE_FONTSTYLE_NAME fontID=6 name=User^3
.CMD DEFINE_FONTSTYLE_NAME fontID=7 name=User^4
.CMD DEFINE_FONTSTYLE_NAME fontID=8 name=User^5
.CMD DEFINE_FONTSTYLE_NAME fontID=9 name=User^6
.CMD DEFINE_FONTSTYLE_NAME fontID=10 name=User^7
.CMD DEFINE_FONTSTYLE fontID=0 family=Times^New^Roman points=10 bold=0 italic=0 underline=0 colrid=-1
.CMD DEFINE_FONTSTYLE fontID=1 family=Times^New^Roman points=10 bold=0 italic=0 underline=0 colrid=-1
.CMD DEFINE_FONTSTYLE fontID=2 family=Arial points=12 bold=0 italic=0 underline=0 colrid=-1
.CMD DEFINE_FONTSTYLE fontID=4 family=Arial points=10 bold=0 italic=0 underline=0 colrid=-1
.CMD DEFINE_FONTSTYLE fontID=5 family=Courier^New points=10 bold=0 italic=0 underline=0 colrid=-1
.CMD DEFINE_FONTSTYLE fontID=6 family=System points=10 bold=0 italic=0 underline=0 colrid=-1
.CMD DEFINE_FONTSTYLE fontID=7 family=Script points=10 bold=0 italic=0 underline=0 colrid=-1
.CMD DEFINE_FONTSTYLE fontID=8 family=Roman points=10 bold=0 italic=0 underline=0 colrid=-1
.CMD DEFINE_FONTSTYLE fontID=9 family=Modern points=10 bold=0 italic=0 underline=0 colrid=-1
.CMD DEFINE_FONTSTYLE fontID=10 family=Times^New^Roman points=10 bold=0 italic=0 underline=0 colrid=-1
.CMD UNITS U=1
.CMD DIMENSIONS_ANALYSIS 0 0
.CMD COLORTAB_ENTRY 0 0 0
.CMD COLORTAB_ENTRY 128 0 0
.CMD COLORTAB_ENTRY 0 128 0
.CMD COLORTAB_ENTRY 128 128 0
.CMD COLORTAB_ENTRY 0 0 128
.CMD COLORTAB_ENTRY 128 0 128
.CMD COLORTAB_ENTRY 0 128 128
.CMD COLORTAB_ENTRY 128 128 128
.CMD COLORTAB_ENTRY 192 192 192
.CMD COLORTAB_ENTRY 255 0 0
.CMD COLORTAB_ENTRY 0 255 0
.CMD COLORTAB_ENTRY 255 255 0
.CMD COLORTAB_ENTRY 0 0 255
.CMD COLORTAB_ENTRY 255 0 255
.CMD COLORTAB_ENTRY 0 255 255
.CMD COLORTAB_ENTRY 255 255 255
.TXT 2 2 3 0 0
Cg a72.000000,72.000000,895
{\rtf\ansi \deff0{\colortbl;\red0\green0\blue128;}{\fonttbl{\f0
\fcharset0\fnil Arial;}{\f1\fcharset2\fnil Symbol;}}\plain\cf1\fs24 
\pard MULTTEST.MCD     --  Chris Evans   -- 25.v.95\par \par This is a 
Mathcad 5.0 PLUS demo. file using the maths power \par of Mathcad to 
illustrate the so-called "Multiple tests problem"\par i.e. that 
conducting multiple inferential significance tests on a \par variety 
of hypotheses within one experiment raises the risk of \par a single 
falsely significant answer in all the tests above the \par chosen risk 
for any one hypothesis test (the alpha, {\f1 a} or "upper\par limit of 
the p values indicating a significant result").\par \par A succinct 
way of putting it is that the "experimentwise" alpha\par may be much 
higher than the chosen "testwise" alpha.\par \par I am using the 
terminology of "experiment" because it is \par conventional in the 
literature but the same applies to non-\par experimental studies.\par 
\par If you are viewing this file with Mathbrowser you should be able
\par to change the various parameters below to suit your own \par 
interests and situations.}
.TXT 51 0 5 0 0
Cg a72.000000,72.000000,276
{\rtf\ansi \deff0{\colortbl;\red0\green0\blue128;}{\fonttbl{\f0
\fcharset0\fnil Arial;}{\f1\fcharset2\fnil Symbol;}}\plain\cf1\fs24 
\pard The problem can be modelled if you consider what a significant 
result\par means: that the results observed were less likely than your 
testwise {\f1 a}\par to have occurred under the null hypothesis of a 
random relationship in\par the population.\par \par It's conventional 
to set that testwise {\f1 a} at .05}
.EQN 16 0 6 0 0
{0:testwise_alpha}NAME:.05
.TXT 0 20 7 0 0
Cg a52.000000,52.000000,126
{\rtf\ansi \deff0{\colortbl;\red0\green0\blue128;}{\fonttbl{\f0
\fcharset0\fnil Arial;}{\f1\fcharset2\fnil Symbol;}}\plain\cf1\fs24 
\pard {\fs20 This is how you define a value for a variable\par in 
Mathcad.  You can change that if want to\par consider another possible 
testwise }{\f1\fs20 a}}
.TXT 8 -20 9 0 0
Cg a72.000000,72.000000,130
{\rtf\ansi \deff0{\colortbl;\red0\green0\blue128;}{\fonttbl{\f0
\fcharset0\fnil Arial;}}\plain\cf1\fs24 \pard That means that {\i if 
the null hypothesis is true}, the probability of {\i \par not }getting 
a falsely significant result is 1 - testwise alpha}
.EQN 7 0 10 0 0
{0:p_test_not_false}NAME:1-{0:testwise_alpha}NAME
.TXT 0 26 12 0 0
Cg a46.000000,46.000000,46
{\rtf\ansi \deff0{\colortbl;\red0\green0\blue128;}{\fonttbl{\f0
\fcharset0\fnil Arial;}}\plain\cf1\fs24 \pard {\fs20 This is how you 
use variables for calculations}}
.TXT 3 0 13 0 0
Cg a46.000000,46.000000,57
{\rtf\ansi \deff0{\colortbl;\red0\green0\blue128;}{\fonttbl{\f0
\fcharset0\fnil Arial;}}\plain\cf1\fs24 \pard {\fs20 This is how you 
check a result\par (note the simple = not :=)}}
.EQN 1 -26 11 0 0
{0:p_test_not_false}NAME={0}?_n_u_l_l_
.TXT 5 0 15 0 0
Cg a72.000000,72.000000,672
{\rtf\ansi \deff0{\colortbl;\red0\green0\blue128;}{\fonttbl{\f0
\fcharset0\fnil Arial;}}\plain\cf1\fs24 \pard Imagine that the null 
hypothesis {\i is }true for all the individual tests that\par are 
carried out, {\i and }that the tests are independent of one another
\par (the so-called "general null hypothesis"; often extraordinarily 
implausible\par but then so are all null hypotheses: they are models 
of a hypothetical \par that give an handle on possible improbabilities 
or perhaps improbable \par possibilities!)\par \par In that case the 
tests are independent, so the probabilities of joint \par outcomes are 
the products of the probabilities of the separate\par outcomes.\par 
\par So, the probability of getting two appropriately non-significant 
results \par given the general null hypothesis is the square of the 
testwise\par probability}
.TXT 34 23 24 0 0
Cg a49.000000,49.000000,35
{\rtf\ansi \deff0{\colortbl;\red0\green0\blue128;}{\fonttbl{\f0
\fcharset0\fnil Arial;}}\plain\cf1\fs24 \pard {\fs20 This sets up a 
very simple function}}
.EQN 1 -23 20 0 0
{0:p_ok}NAME({0:n}NAME):({0:p_test_not_false}NAME)^({0:n}NAME)
.EQN 4 0 21 0 0
{0:p_ok}NAME(2)={0}?_n_u_l_l_
.TXT 3 0 25 0 0
Cg a72.000000,72.000000,534
{\rtf\ansi \deff0{\colortbl;\red0\green0\blue128;}{\fonttbl{\f0
\fcharset0\fnil Arial;}{\f1\fcharset2\fnil Symbol;}}\plain\cf1\fs24 
\pard Now it's possible to consider the more general case of n 
independent \par tests.  I've suggested trying one to forty but you 
can change this.  I've\par also started the calculations for a way out 
of the problem which rapidly\par emerges, i.e. that the probability of 
getting one falsely significant result,\par given the general null 
hypothesis goes up very rapidly with the number\par of tests carried 
out.\par \par The way out of the problem is called the Bonferroni 
correction: simply\par to divide the usual testwise {\f1 a} by the 
number of tests to be carried out, {\i n}}
.EQN 22 0 22 0 0
{0:n}NAME:1;40
.EQN 0 14 55 0 0
{0:m}NAME:1;40
.EQN 3 -14 26 0 0
{0:p_one_or_more_false}NAME({0:n}NAME):1-{0:p_ok}NAME({0:n}NAME)
.EQN 1 38 51 0 0
{0:p_one_or_more}NAME({0:n}NAME):{0:m}NAME${0:p_one_false}NAME({0:m}NAME)
.EQN 3 -38 33 0 0
{0:Bonferroni_tw_alpha}NAME({0:n}NAME):({0:testwise_alpha}NAME)/({0:n}NAME)
.EQN 4 0 45 0 0
{0:Bonferroni_p_test_not_false}NAME({0:n}NAME):1-{0:Bonferroni_tw_alpha}NAME({0:n}NAME)
.EQN 4 0 43 0 0
{0:Bonferroni_p_ok}NAME({0:n}NAME):({0:Bonferroni_p_test_not_false}NAME({0:n}NAME))^({0:n}NAME)
.EQN 4 0 35 0 0
{0:Bonferroni_p_one_or_more_false}NAME({0:n}NAME):1-{0:Bonferroni_p_ok}NAME({0:n}NAME)
.EQN 6 0 23 0 0
{0:p_ok}NAME({0:n}NAME)=
.EQN 0 13 63 0 0
{0:p_one_or_more_false}NAME({0:n}NAME)=
.EQN 0 25 61 0 0
{0:n}NAME=
.EQN 0 9 59 0 0
{0:Bonferroni_tw_alpha}NAME({0:n}NAME)=
.EQN 0 20 57 0 0
{0:Bonferroni_p_one_or_more_false}NAME({0:n}NAME)=
.TXT 98 -67 41 0 0
Cg a72.000000,72.000000,351
{\rtf\ansi \deff0{\colortbl;\red0\green0\blue128;}{\fonttbl{\f0
\fcharset0\fnil Arial;}{\f1\fcharset2\fnil Symbol;}}\plain\cf1\fs24 
\pard This shows very clearly how the probability of getting one 
falsely significant\par result goes up with n and how the Bonferroni 
correction method keeps it very\par steady, close to the testwise {\f1 a} 
set before applying the Bonferroni "correction" \par to set a new 
testwise {\f1 a}.  It also shows how the Bonferroni "corrected" 
testwise\par {\f1 a} gets very small as the {\i n }goes up}
.EQN 12 1 49 0 0
&&(_n_u_l_l_&_n_u_l_l_)&{0:p_one_false}NAME({0:n}NAME)@&&(_n_u_l_l_&_n_u_l_l_)&{0:n}NAME
0 0 1 1 1 0 0 1 1 
0 0 1 1 1 0 0 1 1 
0 1 0 0 1 1 NO-TRACE-STRING
0 2 1 0 1 1 NO-TRACE-STRING
0 3 2 0 1 1 NO-TRACE-STRING
0 4 3 0 1 1 NO-TRACE-STRING
0 1 4 0 1 1 NO-TRACE-STRING
0 2 5 0 1 1 NO-TRACE-STRING
0 3 6 0 1 1 NO-TRACE-STRING
0 4 0 0 1 1 NO-TRACE-STRING
0 1 1 0 1 1 NO-TRACE-STRING
0 2 2 0 1 1 NO-TRACE-STRING
0 3 3 0 1 1 NO-TRACE-STRING
0 4 4 0 1 1 NO-TRACE-STRING
0 1 5 0 1 1 NO-TRACE-STRING
0 2 6 0 1 1 NO-TRACE-STRING
0 3 0 0 1 1 NO-TRACE-STRING
0 4 1 0 1 1 NO-TRACE-STRING
0 1 1 54 26 10 0 3 
.TXT 34 -2 48 0 0
Cg b73.125000,73.125000,1167
{\rtf\ansi \deff0{\colortbl;\red0\green0\blue128;}{\fonttbl{\f0
\fcharset0\fnil Arial;}{\f1\fcharset2\fnil Symbol;}}\plain\cf1\fs24 
\pard I think this Mathcad demonstration shows very clearly the issues 
in the multiple tests problem.  It's not much of a challenge for 
Mathcad or for Mathbrowser but it does illustrate the interactivity 
and flexibility of the system.\par \par \par                                  
        {\b\ul REFERENCES\par \par }A readable reference to the 
Bonferroni method in a psychological context is:\par      Grove, W. 
M., & Andreason, N. C. (1982). Simultaneous tests of many\par       
hypotheses in exploratory research. Journal of Mental and Nervous 
Diseases, \par      170(1), 3-8.\par \par a very readable and short 
summary is:\par      Bland, J. M., & Altman, D. G. (1995). Multiple 
significance tests: the Bonferroni \par      method. British Medical 
Journal, 310, 170.\par \par for a trenchant (frankly overstated in my 
opinion) version of the very complex questions that arise about 
whether or not we should wish to keep the experimentwise {\f1 a} 
constant see:\par      Rothman, K. J. (1990). No adjustments are 
needed for multiple comparisons. \par      Epidemiology, 1, 43-46.\par 
\par \par \par Chris Evans\par Psychotherapy Section,\par Department 
of Mental Health Sciences,\par St. George's Hospital Medical School,
\par Cranmer Terrace,\par London SW17 0RE\par C.Evans@sghms.ac.uk\par }