Table of Contents
Validity refers to the accuracy of the findings of a research study. Reliability, on the other hand, refers to the internal consistency of the results. Findings need to be reliable before they can be valid, but they can be reliable without being valid. Below is a listing of threats to validity.
Internal Threats
The following is a chart describes potential internal
threats to validity. The list was compiled with the help of Gall, Borg, and
Gall (1996: pp. 466-473).
|
Threat to Validity
|
Meaning
|
| 1. History |
Experimental treatments occur over time, allowing other
events to impact subjects besides the treatments. Counter by controlling
for all factors other than the treatment itself.
|
| 2. Maturation |
During the course of the experiment, physical and/or
psychological changes in the subject are likely to take place. Counter
this by using a control group that varies from the experimental group
only in treatment.
|
| 3. Testing
(aka, test-wise) |
Students are likely to become test-wise from any pre-test
given. Counter by not giving a pre-test or by varying the pre-test from
the post-test.
|
| 4. Instrumentation |
Varying the measurement instrument from pre-test to
post-test may under or over estimate learning gains. Counter by not
pre-testing or ensuring that the instruments are as similar as possible
(counter to the previous suggestion). Make experimenter blind to who
is in which group.
|
| 5. Statistical Regression |
Tendency for subjects who score at either extremes on
a test to score closer to the mean on the second test. Again, counter
by testing only once.
|
| 6. Differential Selection |
Effect of not randomly selecting participants. Counter
by random selection.
|
| 7. Experimental
Mortality (aka, attrition) |
Some subjects may drop out of the study during the course
of it. Counter by randomly assigning and making both groups equally
desirable.
|
| 8. Selection-maturation interaction | Having two sample groups that vary by age or some other maturation factor. Counter through random assignment or making both groups are as identical as possible. |
| 9. Experimental Treatment Diffusion | If the treatment condition is seen as highly desirable, the control group may try to access it, particularly if they are in close proximity to the experimental group. Counter this by isolating the treatment groups as much as possible and making sure that the control group is getting a modertately desirable treatment. |
| 10. Compensatory Rivalry by the Control Group | The John Henry effect involves the control group trying harder in order to outperform the experimental group. Counter this by isolating the groups and making the subjects blind to which treatment they receive. |
| 11. Compensatory Equalization of Treatments | If the control group is given a compensatory treatment (e.g., a placebo), it may react positively just from the psychological effect of that treatment. Counter this by not using placebo treatments. |
| 12. Resentful Demoralization of the Control Group | The control group may become resentful if they perceived that a positive treatment is being withheld from them. Counter by keeping groups blind to each other's treatment. |
External Threats
The following is a chart describes potential external
threats to validity. The list was compiled with the help of Gall, Borg, and
Gall (1996: pp. 466, 473-478).
-
Population Validity - The extent to which one can generalize from the experimental sample to a defined population... (and) the extent to which personological variables interact with treatment effects" (p. 466).
-
Ecological Validity - I interpret it to mean the extent to which one can generalize from an experimental setting to a real-life setting. The following are some specific threats to ecological validity from Gall, Borg, & Gall (1996: pp. 475-478):
|
Threat to Validity
|
Meaning
|
| 1. Explicit description |
Researchers sometimes do not explain their methods,
procedures, and constructs explicitly enough for other researchers to
replicate or build on their studies.
|
| 2. Multiple treatment interference |
Using more than one treatment per group weakens the
researcher's ability to generalize the findings. Was it the first treatment,
the second, the second followed by the first, or the first followed
by the second that caused the effect?
|
| 3. Hawthorne Effect |
The potential effect on the subjects caused by the extra
attention they get from the researcher. Just knowing that you're being
observed can cause you to behave differently.
|
| 4. Novelty and disruption effects |
Differences might be detected simply because the treatment
is different from the boring day-to-day conditions.
|
| 5. Experimenter Effect |
The treatment might be effective only because of the
particular experimenter, proctor, teacher, etc. used to administer the
treatment.
|
| 6. Pretest sensitization |
The pretest can interact with the treatment (e.g., primes
the subject to learn the material)
|
| 7. Posttest sensitization |
Sometimes, the results could be dependent on the learning
that takes place during the posttest. This one is not so well defined
or researched as pretest sensitization.
|
| 8. Interaction of history and treatment | We should not generalize beyond a certain time period because a novel approach today may not be one 10 years from now. |
| 9. Measurement of the dependent variable | Generalizability can also be limited to the type of measurement instrument used. For example, a treatment might only be effective on recall tasks and not recognition tasks. |
| 10. Interaction of measurement time and treatment | Posttest scores measured at two points in time may result in different findings about the effects of the treatment.This can be a problem with delayed re-testing to measure long-term retention. |
Representative Design - Snow (1974) advocates the use of representative research designs and criticizes the artificiality and lack of generalizability of most empirical research. A representative design attempts to reflect real-life environments and contextual factors in an effort to improve the generalizability of the results. Unfortunately, most empirical research is designed with the idea of protecting internal validity (i.e., reducing error) at the expense of external validity (i.e., generalizability).
The following chart lists the most common experimental designs shows potential threats to internal and external validity. It was adapted from Gall, Borg, & Gall (1996: p. 486):
|
Design
|
Sources of Invalidity
|
|
|
|
Model*
|
Internal
|
External
|
|
| Single-group designs | |||
| 1. One-shot
case study |
X O |
History, maturation, selection, mortality
|
Interaction of selection and X
|
| 2. One-group pretest-posttest design | O X O |
History, maturation, testing, instrumentation, interaction
of selection and other factors
|
Interaction of testing and X; interaction of selection
and X
|
| 3. Time Series Design | O O O X O O O |
History
|
Interaction of testing and X
|
| Control-group designs with random assignment | |||
| 4. Pretest-posttest control-group design | R O X
O R O _ O |
None
|
Interaction of testing and X
|
| 5. Posttest-only control-group design | R X O
R _ O |
Mortality
|
None
|
| 6. Solomon four-group design | R O X O R O _ O R _ X O R _ X O |
None
|
None
|
* X = experimental treatment, O = observation (test) of dependent variable, R = random assignment, _ = nothing
Multi-Trait Multi-Method (MTMM) Matrix
Coming soon
http://trochim.human.cornell.edu/kb/mtmmmat.htm
MMTM lesson developed by William M.K. Trochim - part of the Research Methods
Knowledge Base
Campbell, D.T., & Fiske, D.W. (1959). Convergent and discriminant validity in the multitrait-multimethod matrix. Psychological Bulletin, 56, 81-105.
Fiske, D.W. (1973). Can a personality construct be validated empirically? Psychological Bulletin, 80, 89-92.
Fiske, D.W. (1976). Can a personality construct have a singular validational pattern? Rejoinder to Huba and Hamilton. Psychological Bulletin, 83, 877-879.
Gall, M.D., Borg, W.R., & Gall, J.P. (1996). Educational research: An introduction (6th ed.). London: Longman.
Huba, G.J., & Hamilton, D.L. (1976). On the generality of trait relationships: Some analyses based on Fiske's paper. Psychological Bulletin, 83, 868-876.
Kavanagh, M.J., MacKinney, A.C., & Wolins, L. (1971). Issues in managerial performance: Multitrait-multimethod analysis of ratings. Psychological Bulletin, 75, 34-49.
Marsh, H.W., & Smith, I.D. (1982). Multitrait-multimethod analyses of two self-concept instruments. Journal of Educational Psychology, 74(3), 430-440.
Reichardt, C.S., & Coleman, S.C. (1995). The criteria for convergent and discriminant validity in a multitrait-multimethod matrix. Multivariate Behavioral Research, 30(4), 513-538.
Snow, R.E. (1974). Representative and quasi-representative designs for research on teaching. Review of Educational Research, 44, 265-291.
