Sociology 357 Piliavin

ARTICLE ANALYSIS ASSIGNMENT

DUE DATES: See summary sheet

READ THIS HANDOUT CAREFULLY! You must do this analysis by answering the specific questions listed. Keep your answers as brief as possible using an "outline" style rather than an elaborate writing style whenever possible.

Criteria for Article Selection

The articles reviewed for this assignment must report the results of someone's research in an area of social research. The research should have been carried out by the author(s). The article must be directed at a scholarly audience.

Your review must be on an article reporting structured research, that is, one with variables, statistical analyses, relationships among variables, etc. The article may be about any social science topic you choose. Check with me if you have any doubts about your topic. Research in sociology, political science, psychology, education, or social work are fine. (But remember you need research articles; not all articles in any field are research articles.)

The following types of articles may NOT be used:

Purely theoretical papers which discuss concepts and propositions, but report no empirical research;
Statistical or methodological papers where data may be analyzed but the bulk of the work is on the refinement of some new measurement, statistical or modelling technique;
Review articles, which summarize the research of many different past researchers, but report no original research by the author;
Popularizations or abridged reports, commonly found in popular newsstand magazines such as Psychology Today or books of readings designed for use by undergraduates;
Extremely short reports with less than four pages devoted to methods and findings.

Most research reports begin with sections on theory and reviews of others' research, so skim the whole article or read the abstract, if there is one, to determine whether the author reports actual research he or she has done. Sociology, as is true of all scientific fields, is becoming increasingly complex in its statistical analyses. I therefore strongly suggest that you use articles no more recent than the 1970's. A working rule is: if you can't understand the statistical analyses presented in the results section, don't choose the article.

All articles must receive my OK. No two students may review the same article. It is OK to use articles you have to read for another class, if they meet all of the above criteria, but you may not use the articles in Golden.

Where and How to Find an Article

You must use scholarly articles for this assignment; these are found in professional journals, not general circulation magazines. The University of Wisconsin subscribes to a large number of such journals,in both physical and electronic form. Recent issues of most of the physical journals are kept in the periodicals room of Memorial Library. Past issues are bound in hardcover by volume and kept on the first and second floors of the south stacks of Memorial Library. Bound volumes of some journals are in the reserve room of Helen C. White library and in the Social Science Library. To find the call number of a specific journal, look up the journal's title in MADCAT, or in the list of journals in the periodicals room.

If you want to find articles about a particular topic, use the data bases available through the Library home page. Another place to get citations of articles in a topic area is in the bibliographies of other books or articles in the topic area. If you are having trouble finding an article, go to the second floor of Memorial Library and ask a librarian for help or come see me.

I suspect that most of you will go first to full text databases. If you get an article from one of these, choose the PDF format if it is available. If it is not, MAKE SURE to print out all of the tables and figures. You sometimes have to do this separately in non-PDF files.

If your interests are wide, general, eclectic, or uncertain, you may prefer to locate a supply of journals in the stacks or the reserve room and flip through them until you spot an article that looks interesting to you. The major general sociology journals are American Journal of Sociology, American Sociological Review, and Social Forces. Some other journals in sociology are: Journal of Marriage and the Family,Criminology, Crime and Delinquency, Social Psychology Quarterly, Sociology and Social Research, Social Problems, Journal of Political and Military Sociology, Journal of Sport and Social Issues, Sociology of Sport Journal. There are dozens of other specialized journals.

Final approval will be given only on the basis of the photocopy or printout of the whole article; I will write approval on the copy itself. When you have found the article(s) you want, photocopy it, and write right on the photocopy the journal name, volume number, issue number, month, year of publication, and pages. The author's name and the article's title should be on the first page; if they are not, copy these down too. (You should get into the habit of writing the full citation on everything you photocopy. This saves having to return to the library for the information when you later decide to use the material in a term paper or, worse, not being able to find it.) Don't save ten or twelve cents by omitting the last page of the references. Do write your own name on copies you turn in to me. If you wish to save money, check out the journal(s) themselves and bring them to me.

Example of a student article analysis, with the article

Outline for Your Article Review

PLEASE NUMBER THE SECTIONS OF YOUR REVIEW TO CORRESPOND TO THE NUMBER OF MY QUESTIONS. It is not in your interest for me to have to guess what you're writing about. Answer the questions as briefly as possible. This is not a literary essay. An "outline" style, tables, and other devices to keep your answers brief while complete are all acceptable.

Introduction

NOTE: Make sure that the full citation is either printed or written on your photocopy or you will not get credit for the review. Attach the photocopy to your review. I simply cannot grade your review without the photocopy.

What is the problem or question(s) this research concerns? You should be able to identify the central focus. If there are additional secondary problems, identify these too. (1-4 sentences)
What is the source of the data? (That is, questionnaire, intensive interview, documents, existing statistical information, observations, laboratory manipulations, field manipulations, etc.) In some studies there are two or more sources of data. Give a brief overview of how the data were acquired. (2-5 sentences)
Briefly, what do the key findings turn out to be? (1-5 sentences)

External Validity

Give the following information about the sampling procedures in outline form, saying "not given," if it is not:
1. Definition of the population of theoretical or substantive interest; a) What is the population of theoretical or substantive interest; that is, to whom does the author seem to want to be able to generalize? Your answer to this should be based on what the author says in the introduction to the article, not in the methods section.
2. Geographic areas, organizational units (e.g. what state, University, county), or other primary sampling units and how these were chosen. Was the sampling of these units probability or nonprobability?;
3. The sampling units (e.g. people, organizations, sentences). These may or may not be the same as the units of analysis;
4. Sampling frame, or operationalization of the actual population studied; by what rule or list were units of analysis located?
5. Method of selecting the units of analysis from the sampling frame. Was the sampling of these units probability or nonprobability?;
6. What kind of sample (e.g., convenience, stratified random, etc.) does this seem to be?
7. Response rate (e.g., to a mailed survey) and sample size; if analyzed sample size is different from initial sample size (e.g., cases were dropped for missing data) explain why.
8. Does the author discuss any shortcoming in the sample or the sampling procedures? If so, what does s/he say?
If you feel that this outline does not adequately demonstrate your understanding of the sampling, or that there is something important about the sampling that does not fit in this outline, write an additional paragraph that provides any necessary extensions or clarifications. (Do not, however, omit the outline.)

Often articles that use one of the well-known large national probability samples do not give much information about the sample because they assume that professionals will recognize the sample title and already know the basic information. Check with me if you suspect this is the situation with your article. You may need to track down an earlier article to get the details.
Evaluate the sampling procedures.

Do the geographic or other restrictions imposed on the actual population (b, d above) seem justified in light of the purposes of the research and practical constraints?
Were the units of analysis selected in a way that allows generalization to the desired population? Why or why not?
Are you aware of anything in the research procedures that added any implicit restrictions to the sample (e.g. interviewing only during the day)?
Does information available in the article (e.g. frequency distributions) suggest that the sample is reasonably representative, or does it point to problems or biases? e) Overall, how good do you feel the sampling was?

Generalization

Strictly speaking, to what population can the results of this research be generalized?
To what population would you feel reasonably confident the results probably apply? Why?
At what point would you be very hesitant to apply these results?

Construct Validity of Measures of Variables

Preferably using a chart, list ALL of the operationalized variables in this research and the concepts or variables of theoretical or substantive interest they are intended to represent. You should discuss all the operationalized variables, but it will be often easiest to write your answer by starting with the concepts, and explaining how each is measured. Sometimes there are several measures for one concept or variable. Do NOT "dump" all the measurement details here. This is just a summary that lists all the measured variables and what their logical relation is to the purposes of the research. DO NOT talk about how one variable relates to other variables here.

It is hard to explain this question clearly, because how to do it depends very much on what your article is like. Probably the best explanation is an example. For the horn-honking article, the answer would be: The independent variable is status of frustrator. This is operationalized as the type of car and the driver's clothes. The dependent variable is aggression, which is operationalized as latency of honking and number of honks. The frustrating situation is operationalized with a car being blocked at a green light. Sex of subject was a control variable.

Different articles have different logical structures, and the best way to do your article is to describe what is happening in it. Some have no distinction between the concepts and the operationalizations; everything is just operationalization. Others have complicated and convoluted steps getting from the original concepts to the measured variables.

This is where you should tell me if the units of analysis are different from the sampling units. Sometimes there are several different units of analysis in one article. Measurement is dependent on these units.
Select the most complicated or difficult variable in the article. That is, choose the variable that must have been hardest for the author(s) to figure out how to measure, or how to make the conceptual-operational link. Call this variable "a" in your outline. Then do the following detailed analysis.
a) Give the name of the variable
1) State the concept and give a brief summary of what (if anything) the author(s) say about issues or problems in measuring this, how others have measured it, why they are measuring it this way, etc. (NOT the measurement details themselves.)
2)What is the measure of this concept that is used in the data analysis? (E.g., in Ransford there is a scale for racial dissatisfaction, dichotomized "conceptiually".) If there are several, state all of them. Then explain the following:
1. How the relevant variables were originally measured on the units of analysis. That is, what were the initial items of information obtained and what were their attributes? (In the Ransford article, this would be the questions and answer formats that make up the scale.)
2. Explain how the original measured variables were combined or modified to create the specific operational variable that was used in the statistical analysis. (In the Ransford article, this would be that the original questions were summed and then dichotomized, using a conceptual split.) This is where you describe index or scale construction. (Often the original measured variable was not modified; if this is true, just say "does not apply.") Discussions of how cases were grouped or regrouped belong here, too.
3) Summarize any discussion by the author(s) of why this is a good measure or of what its problems are, including statistics like factor analyses or reliability coefficients, but NOT material on the bivariate relations with other variables.
4) Give your own evaluation of how good you think this measurement is, explaining your reasons.

Now select the second most difficult variable to measure. Call it "b". Then do all of the above steps again for that variable.

NOTE: The format of the above questions works best when the variable that gets into the statistics is a composite of several original measured variables. In some articles, what is more interesting is to start with a concept that has several related measures (each of which might be fairly simple) which are then analyzed to see which is "best," in which case you might want to discuss them as a group and treat the matter of choosing among them in d). I suggest you ask me if there is any doubt in your mind about which two variables would be good choices. I should note that in some articles, all of the variables are pretty straightforward. In this case, just pick any two of them. You will not be graded down because your article is less complicated. However, I do expect people with very uncomplicated variables to analyze them perfectly, while I might decide that a mistake in analyzing some complicated variable is not that bad. (If there are both simple and complicated variables in your article and you choose to talk about the simple ones, I will assume you do not understand operationalization, which is not in your best interests.)

Internal Validity

Identify two of the most important bivariate hypotheses (explicit or implicit) or questions of the research. For each hypothesis or question, list those findings which are most centrally relevant to it. If there are only a few relevant findings, list all of them, but if there are many, list only the few that you think are most important.

NOTE: A finding is the actual number(s) from the statistics, not just the author's word summary. Often a particular hypothesis is supported by several different findings which show that the bivariate relation holds true after other variables have been statistically controlled, or when the research design is altered, or when the variables are measured in different ways. If so, you would list several different findings as relating to the same hypothesis or question, but if there are many different relevant numbers for the same hypothesis, you would pick out only the few most important ones.

When articles list more than two hypotheses or goals, it can be difficult to decide which is most important. Think about the central purpose or argument of the article (usually found in the introduction). Four common approaches lead to long lists of implicit or explicit hypotheses or questions.

They are all variations on the same general idea. In this case, pick the two variations that seem most central in the discussion.
The author actually believes in only one or two of the hypotheses, and the others are set up as alternates to be proved wrong. In this case, pick the ones the author seems to believe in.
The argument has a series of logical steps and there are hypotheses about each step. In this case, all the steps do matter, but pick out the ones that seem to you or the author to be most central in this article.
the article does not really have a central point and there is just a laundry list of hypotheses, questions or topics. In this case, pick out the ones that you or the author think are most interesting.

If there are additional findings that you or the author found interesting or surprising, list them here. (Again, a finding is not just the verbal summary, but the number that backs it up.) If you already wrote a lot for (1), you may just say "no additional findings" here.

NOTE: If your article has only a few statistics, you may end up writing about all of them, but if your article has a lot of statistics, do NOT write about everything. Instead, try to figure out what is really important. I do want you to learn to read the numbers, and you may ask me for help translating them.

In this section, you will evaluate the internal validity of the data. It is OK to make summary statements that are true for all findings, where appropriate, but be very sure to discuss the findings separately where necessary.

Is the conclusion supported by an appropriate bivariate statistical result? That is, look at the statistic copied above to be sure it is actually relevant to the hypothesis it is supposed to be related to. Sometimes in a bad article, the relevant finding is actually not reported! (Remember that a bivariate association of zero supports a hypothesis of no effect.)
Is there adequate justification given or implied for the presumed direction of causality, i.e. for why A causes B instead of B causing A? If yes, say why in one sentence. If no, say in one sentence what you think the problem is.
List the potential extraneous variables that have been controlled for in any multivariate statistical tests. (This is simply a matter of being able to read your tables.) If multivariate statistical tests (e.g. regression) were not done, just say so. ) Ask if you have a question.
What kinds of extraneous variables are simply irrelevant for this finding and could not possibly be a problem? (Examples: on stage effects for research on historical documents, maturation or other time-tied variables for research that is conducted in one short period.) Just list general classes of variables.
Which potentially significant extraneous variables have been controlled in the design of the research, by holding constant, by randomization, or by some other method? Just list general classes of variables, mentioning specifically only those which would otherwise be a special problem (e.g. organismic variables in a within-subjects experiment).
Are there any other possible problems or extraneous variables that the author discusses, giving reasons why they should not be problems? Summarize the discussion.
Are there other possible problems or extraneous variables that the author thinks have not been adequately eliminated? Summarize the discussion.
Are there any remaining possible problems or extraneous variables that you can see that have not already been discussed above? Are there variables that should be controlled that were not? Could a different designed have eliminated problems? Are there things you can see as problems that you wouldn't know how to fix? If yes to any of these, discuss your concerns. I am referring to simple random error here; you need to identify variables that are potential threats to internal validity.
Overall, how much internal validity do you attach to the findings? Why? (Be sure to say whether your answer varies from finding to finding.)

Overall Evaluation

Give your overall evaluation of the methods used in this article: what things were done well? what were done poorly? How much trust do you put in the findings?
Look at this article's "packaging," that is, the theoretical introduction and the discussion or interpretation at the end. Do you feel that the actual methods and results support the theoretical and interpretive claims of the author? Why?
What possible ethical issues might have arisen in the process of doing this research? Do you think the researcher's ethical decisions were all justified, or are some questionable? Why?
To sum up, what do you feel you've learned worth knowing from this article? (If your answer is "nothing", explain why.) (Please note: this question is about the article and refers to the quality of information it contains.)
Tell me anything you would like me to know about your experiences doing this analysis, or any suggestions you have for future revisions of this assignment.

*** END OF REVIEW ***

Some Remarks on Grading Standards

The key to this assignment is to apply the methodological concepts you have learned in this course to the evaluation of a research article. You demonstrate your ability by specifically linking the procedures discussed in the article to the concepts. Think of it as a take-home final, not as an opinion essay. You have the burden of proof to demonstrate that you know what you are doing. In particular:

Never answer just "yes" or "no"; always explain your answer.
Never state some general methodological term or principle without linking it up specifically to something in the article (or to something missing in the article).
Never give a vague or evasive answer in which you avoid sticking your neck out (hoping you won't be marked "wrong"); if you don't commit yourself to a specific answer, I will assume you do not know what it is. But try to say what is needed as briefly as possible. Long-winded, rambling answers are evidence that you do not know precisely what is important.

Questions of "fact" will be graded by comparing what the article says with what you said it said, along with your ability correctly to use the relevant methodological terms. Questions requiring evaluation will be graded according to these criteria:

you take some position
you defend your position by talking about your article in ways that raise issues that we discussed in class.

If the article fails to give some information the review asks for, you get credit by saying that the article fails to give the information. Note that this failure should then become part of your evaluation of the relevant section. (I will try to avoid approving articles that are missing too much of the relevant information.)
If the article is unclear or ambiguous, or if you are ambivalent in your evaluation of something, it is fine to give an answer that expresses these problems.
Don't blindly assume the author is using the correct methodological terms for what s/he did.For example, Ransford describes his sample as "disproportional stratified" (p. 298 of Golden reader). But if you carefully read the paragraph on p. 298 and the extended description of the sample on pp. 309-310, you will discover that the sample was not stratified at all: three clusters (Watts, South Central, Crenshaw) were chosen purposively; blocks were chosen randomly within clusters; and households were chosen purposively within blocks, after a random start on block corner and an overall quota of 8 households per block. The use of the term "random methods," rather than "random sample," is the sort of thing you'll see when the procedures are less than ideal.

Top

Questions? Comments? Please contact jpiliavi @ssc.wisc.edu

Sociology 647

Sociology 965

Sociology Homepage