Written Report and Evaluation

PLEASE FOLLOW THIS FORMAT EXACTLY.
You can and should write all of this up to "III" before you get your computer analysis back. You will not have time to do all of it after that.

About Truthfulness

Science depends on researchers telling the truth about what really happened in their research, not what they wish had happened. At the same time, students worry that they will be graded down if they tell the truth. So, for each question, I insist that you tell the truth about what really happened in the research, but then follow it with an opportunity to explain what you now think you should have done. If there was a mistake and your self-criticism gives a correct statement about what you should have done, you will receive full credit as if you had done things right in the first place.

Stylistic Notes

Use descriptive phrases as labels in all tables and in writing about the questions, not question numbers. Be consistent, so that the reader can easily work between the tables, the text, and the questionnaire. Tables may be embedded in the text or prepared on separate pages, which can be interleaved with the text or attached together at the end. Each table should have a number and a title; everything should have meaningful labels. Cut-up computer printouts are not adequate tables. (Note my exception for the correlation matrix.) It is OK to prepare the tables by hand, as typing them is extremely difficult unless you have a word processor with table options.

OUTLINE

Title page. Title of report, author(s), date. Put partner's name in parentheses at the bottom of the page if you worked with someone but wrote reports separately.
Abstract. Write one paragraph that summarizes the central concept and its measurement, data collection procedures, and findings. You may include this on the title page if you wish.
Body of paper.

Introduction. Write a paragraph stating your concept and why it is interesting. This can involve theory, policy, or pure curiosity reasons. Then state your "obvious" bivariate hypothesis, and why you think it is obvious that your concept will be related in that way to the independent variable. (Note: Citations to readings are not needed, but go here if something you read went into your thinking on this project.) REMEMBER THAT THIS EXERCISE IS ALL ABOUT MEASURING YOUR CONCEPT.
Methods of research. (Note: To aid grading, number each section of this discussion as it is numbered here.)

Sampling.

Describe your sampling procedures including when, where, and how you selected your subjects. Describe the kinds of people your group got into your sample, by sex, age, other interesting characteristics. Be sure to discuss any differences among team members in this.
Tell the reader in a few sentences what to think about the external validity of the sample. We know you do not have a probability sample, so strictly speaking, your external validity is low. But less strictly speaking, do you feel that you probably have a good representation of the population of interest, or do you feel there are clear biases? Explain a little.
Evaluation: why you think your sampling was good, given your resources and limitations, or what you now believe should have been done differently. Please note, this evaluation is in terms of what was actually possible in this assignment, and is not about the standards you believe professionals should adhere to.

NOTE:

see me to get a new set of instructions.

Independent Variables.

Most of these variables are simple and unproblematic. Just list them, noting anything that would not be obvious. (For example, the categories of sex are obvious, but the categories of religion are not, so you should say what categories you used.) If you have a more complicated independent variable, state the question(s) you used to measure it, and anything you did in the way of recoding or forming an index.
Either assure me that there were no problems with the independent variable questions, either in the questionnaire administration or the coding, or describe any problems that turned up. For example, are there missing data? Were respondents confused by wording?
Briefly evaluate your measures, stating any changes that should be made. (Note: relation to dependent variable goes elsewhere, as does discussion of wishing you had included other variables. This is just evaluation of the measures.)

Dependent variable.

Discuss a bit how you defined and revised your concept; be sure to mention the dimensions of your concept you tried to capture in your questions. That is, what thinking about the nature of your concept led into the specific open- and closed-ended questions you used to measure it?
Briefly list the closed-ended items or refer to the questionnaire copy. Based on your own second thoughts or the comments of your subjects, not yet on the computer printout, evaluate the closed-ended items in terms of their clarity and lack of bias. Are they OK, or should some be revised? In this connection, discuss any problems of multiple answers or missing data, and what you did about them. (If there are questions that seem problematic here, keep that in mind later when deciding which to discard in forming an index.)
Explain how you coded your open-ended question. Were there particular problems you ran into? Did partners disagree about the coding? Do you feel confident that you were able to capture the essence of your concept using this measure?

Results: You must present six tables, as described in the example.

Frequency Distributions for Independent Variables.
These are presented in Table 1. A brief discussion should note if these data indicate problems with low variability on a variable you would otherwise be interested in studying, and whether the sample seems fairly representative or fairly biased in terms of these variables.
Validity of Index and Open-ended question

Present and refer to Table 2, the frequency distribution table for dependent variables.

Either discuss any problems of variability evident among your closed-ended questions, or explain why there are none. Do not discuss every number or every question. Explain the problem with the problematic variables or, if none have problems, pick the question with the lowest variability and explain why it is variable enough not to be a problem.
Similarly, discuss the distribution of responses to your open-ended question.
Summarize what these tables show about the distribution of attitudes in your sample. That is, what do these people seem to think about your concept? To summarize is to paint a verbal picture in a two to five sentences; it is not to quote all the numbers.

Discuss the reliability analysis and matrix of correlations among the closed-ended questions measuring your central concept. Exactly what you say will depend on what the numbers look like. You'll either be explaining why all the items seem pretty good, or why you think some are not so good. You should not discuss every single number in the table, but you should let me know what you are looking at in the table. The original full table of correlations among all closed questions intended to measure the concept -- Table 3 -- should be presented, with the item-total correlations and alphas. You may paste in the matrix of correlations from the printout, and the five columns below, which include the item-total correlations. Don't forget to indicate the alpha, and give the table a title.
Say which items you would keep for your final index, and write one or two sentences verbally summarizing why, based on the discussions in the first part of this section.
Discuss the relationship between your open-ended question and your Index. Do they seem to be measuring the same thing? If not, discuss which one you think has the problems, and why. What does this say about the validity of each of the measures? This discussion should be based on Table 4.

Independent-dependent variable relationships.

Tests of Obvious Hypothesis, which is a test of construct or predictive validity.

Tests of hypothesis using the index measure of the concept. If your independent variable is categorical, you will have a table of the means of the index for each category of the independent variable and an analysis of variance test (F) for the test of your hypothesis. You will present your results in the form shown in Table 5. If your independent variable is continuous, you will have a correlation coefficient showing the relationship between your independent variable and your Index. Simply present this correlation with its associated p-value in the text. If you are having trouble figuring this out, please see me.
Tests of hypothesis using the open ended measure of the concept. Regardless of whether your independent variable is continuous or categorical, you will have a contingency table (cross-tabulation) with a chi-squared analysis relating your open-ended question to the independent variable. If your independent variable originally was continuous, or if there were small sample sizes for some of the categories, I will have grouped the scores into two or three categories using a recode statement, which you can find in your syntax file printout. Again, see me if you are confused about this. You will present your results in the form shown in Table 6.
Discuss your results, indicating whether your hypothesis is confirmed or rejected. If the results for the Index and the open-ended question differ, try to explain why this might be and what it means. What do these results mean for the validation of your measures? If you had more than one obvious hypothesis, and the results differ, you must again discuss what this means for the hypotheses and for your measures.

Relation of measures of concept to other independent variables, if any. These will be relationships between your measures of your concept and other variables you just "threw in" for fun. These will again be correlations or differences of means for the Index, and chi-squared analyses for the open-ended question. Briefly summarize what these show. These should also be presented in your Tables 5 and 6 (or in the text if the analyses involve correlation coefficients). If analyses were done, you must report them, however briefly, even if they are not interesting. Also present any other analyses you may have done (such as controlling for third variables) here.

Discussion

If you view this as a pretest of your measures in preparation for a larger study, what do you conclude? How good are your measures? If you were to pursue this research, how would you change your measurement? Does it seem worthwhile to pursue this research? Or, do you now think this is a dead end?
Explain the conclusions you draw from testing your hypotheses and from the frequency distributions. This is your chance to explain the wider significance of your work or to speculate a bit. In this section, you don't have to worry so much about sample bias, small samples, or what you did well, and instead talk about what was interesting to you and why.

Appendices (points will be lost if these are missing)

Attach your code book, syntax file, and all computer analyses to the report. If you are writing individual papers, make sure that at least one member of the group does this.
Attach a copy of the code sheet that was used for entering your data into the computer, or a xerox of it.
Submit all the completed questionnaires when you submit the assignment. Make sure there are code numbers on each questionnaire to correspond to the identification numbers on the data sheet.
All tables, 1-6, if they are not imbedded in the text.

. Group process report. Pick the category that applies to you and answer the relevant questions.

No partner.

How did you feel about working alone? Would you do it again, or would you prefer a group?
How much effort did you have to put into this project?
How well prepared did you feel in terms of course materials and understanding what to do.
Tell me if there is anything I should know about you or your life that you want me to know, especially if it might affect your grade or my ability to be fair in grading your work.

Had partner, wrote separate papers.

Compare you and your partner(s) in the effort you put into the project.
Compare you and your partner in the extent to which you studied course materials and knew what to do for the assignment.
Who developed the questions?
Who prepared the tables from the printout?
Who figured out how to interpret the statistical results?
Did you start trying to work together before deciding to write separate papers? How far did you get?
Were there some things you found necessary to discuss in preparation for writing your papers? What?
How did the group process work out? Was it a positive or negative experience? Would you do things differently in the future?
Tell me anything else I should know that might affect your grade or your partners', or that I should know to be fair in grading your work.

Wrote joint paper.

Do you stand by the paper as written, or is there something you feel should have been said differently? Any corrections you offer at this point will be factored into your grade.
Compare you and your partner(s) in the effort you put into the project.
Compare you and your partner in the extent to which you studied course materials and knew what to do for the assignment.
Who developed the questions?
Who prepared the tables from the printout?
Who figured out how to interpret the statistical results?
How did you go about getting the writing done?
How did the group process work out? Was it a positive or negative experience? Would you do things differently in the future?
Tell me anything else I should know that might affect your grade or your partner's, or that I should know to be fair in grading your work.

Top

Questions? Comments? Please contact jpiliavi @ssc.wisc.edu