Internal Validity

These lecture notes are additional comments on the experiments lecture.  A lot of relevant material that you should know is in the experiments and experimental design chapters of your text book.

What is internal validity?  
This refers to whether the bivariate association between the independent variable (IV) and the dependent variable (DV) is due to the causal effect of IV on DV.  The other possibility is that some extraneous variable produced the statistical association when, really, the IV does not cause the DV.  NOTE: There is still an internal validity question if the statistical association between IV and DV is zero in your research.  In that case, the internal validity question is whether it is really true that the IV did not cause the DV, or did some extraneous variable create a zero statistical association when really the IV does affect the DV.  

In the lecture I gave, the internal validity question is: does the tutoring program raise students' grades?  A similar but perhaps more socially relevant internal validity question today is: if I see differences in children's test scores between schools, is it correct to assume that the schools whose children have higher test scores are better (in the sense of teaching the children more) than schools whose children have lower test scores?  IV is which school the child attends, DV is amount learned.  Other examples of internal validity questions: (1) Is medicine A better than medicine B as a treatment for a given disease?  IV is medicine (A vs B), DV is some measure of disease remission.  (2) If people recommend more severe penalties for a man (versus a woman) or a racial minority (versus a white) for some offense, is that due to differential treatment by sex or race?  IV is sex or race of applicant, DV is proposed penalty for a criminal offense.  Or, relatedly, if Black people or people with criminal records are less likely to be called after applying for a job, is that due to discrimination by race or background?  IV is race or criminal background, DV is whether called for a job.

In these examples, you want to know whether some other factors influenced the result and made it appear that the IV caused the DV when it really did not.  (Or made it appear that the IV did not influence the DV when it really did.)  The other factors that might have influenced the result are called extraneous variables.  Your book lists lots of kinds of extraneous variables.  They include intrinsic and transitory characteristics of the subjects (sex, race, age, intelligence, aggressiveness, tiredness etc.), external environmental factors that might influence the DV that are tied to time or place, and external factors that are inadvertently linked to the administration of the IV (such as how long a treatment takes).

Example of a factor inadvertently linked but properly identified with a control group: A study of the effects of stress on mice.  Stress treatment was to shock the mice.  Control not to shock the mice.  Since shocking the mice necessarily involved picking them up out of their cages, a third treatment was added in which mice were removed from their cages but not shocked.  Hypothesis was that shocking the mice would make them ill.  The surprising result to the experimenters in this case was that the mice left in the cages got more ill, while there was no difference between the mice who were shocked and those who were removed from cages and not shocked.  Experimenters concluded two things: (1) Being lifted out of your cage by a huge monster was probably pretty stressful to a mouse whether or not it was subsequently shocked; (2) some level of stress if healthy for a mouse's body.  Notice that if they had NOT added the third control group, they would have falsely concluded that being shocked produced the difference between the "stressed" and "not stressed" treatments, when really it was being picked up and handled that made the difference.

Examples of subject characteristics distorting results: Schools are routinely given "report cards" based on their children's test scores.  But children of highly educated parents have enormous advantages in schooling relative to children of less educated parents, and all these "report cards" do is tell us which schools serve the well off and which serve the poor.  A fairer measure of a school's quality is a before/after measure: how much do the children know in June compared to how much they knew the previous September.  Measuring the change in children's test scores reveals a very different set of rankings of which are the good schools and which are the less good schools.  But even a change scores measure could be influenced by extraneous variables.  A stronger test of school quality is to randomly assign children to schools and then see how well they do.

Any extraneous variable (EV) that is going to distort the statistical relationship between IV and DV must (mathematically) be statistically associated with both the IV and the DV.  Experimental design forces a zero bivariate association between the IV and any EV's that are held constant.  Within the limits of sampling variation, experimental design also forces a zero association between the IV and any EV's that are randomized – it does this by making the average value of the EV the same in all treatment groups.

Experimental design is the strongest way to eliminate EV's as possible distorting factors in an IV-DV relationship.  In your experimental design paper, you are asked to come up with an inventory of extraneous variables that might plausibly affect your dependent variable, and then talk through which ones are controlled by being held constant, which ones are controlled by being randomized, and which ones remain possibly uncontrolled.  You need to give specific examples of variables relevant to your particular experiment.  You will not get credit for a vague general answer, you need to demonstrate your knowledge of experimental design and the logic of holding constant and randomization by classifying various possible extraneous variables correctly.  If you try to dodge the answer by being vague or saying little, you will not get full credit.

I should note that a paper that does everything in the experiment right EXCEPT has errors in the "internal validity" section will still get a B.  The internal validity section is distinguishing the highest grades from the rest, but you can still do OK (if not great) even if you mess it up.