Thursday, December 16, 2010

Among the fallacies used by man, one of the most dangerous to science is that known as post hoc•, ergo propter hoc: after this, therefore caused by this. We may joke, with a tinge of seriousness, "If I take an umbrella, it won't rain." We may even seriously say that delinquents are delinquent because of a lack of discipline in the schools or that religious education makes children more virtuous. It is very easy to assume that one thing causes another simply because it occurs before the other, and because one has such a wide choice of possible "causes." Then, too. many explanations often seem plausible. It is easy to believe, for instance, that the learning of children improves because we institute a new educational practice or teach in a certain way. We assume that the improvement in their learning was due to the new spelling method, to the institution of group processes into the classroom situation, to stern discipline and more homework (or little discipline and less homework). We rarely realize that children will usually learn something if they are given the opportunity to learn. The social scientist and the educational scientist constantly face the problem of the post hoc fallacy. The sociologist who seeks the causes of delinquency knows that he must exercise extreme care in studying this problem. Slum conditions, broken homes, lack of love —each, or all, of these conditions are possible causes of delinquency. The psychologist seeking the roots of adult personality faces an even subtler problem: hereditary traits, child-rearing practices, educational in-fluences, parental personality, and environmental circumstances are all plausible explanations. The educational scientist, with the goal of understanding the basis of successful school achievement, also faces a large number of reasonable possi¬bilities: intelligence, aptitude. motivation, home environment, teacher personality, pupil personality, and teaching methods. 378 EX POST FACTO RESEARCH 379 The danger of the post hoc assumption is that it can, and often does, lead to wrineous and misleading interpretations of research data, the effect being par¬n=larly serious when the scientist has little or no control over time and indepen¬.AL--r variables. When he is seeking to explain a phenomenon that has already ,=-urred, he is confronted with the unpleasant fact that he does not have real mmtrol of the possible causes. Hence he must pursue a course of research action ,kiferent in execution and interpretation from that of the scientist who experiments. 3rinition E-- post facto research' is systematic empirical inquiry in which the scientist does ,we have direct control of independent variables because their manifestations tome already occurred or because the), are inherently not manipulable. Inferences ap--,ut relations among variables are made, Without direct intervention, from w,wcomilant variation of independent and dependent variables. Assume that an investigator is interested in the relation between sex and aeativity in children. He measures the creativity of a sample of boys and girls mid tests the significance of the difference between the means of the two sexes. Tbw mean of boys is significantly higher than the mean of girls. He concludes that bvvs are more creative than girls. This may or may not be a valid conclusion. The ue!E-ation exists, true. With only this evidence, however, the conclusion is doubtful. 7"Nbe question is: Is the demonstrated relation really between sex and creativity? Sm'e many other variables are correlated with sex. it might have been one or mme of these variables that produced the difference between the creativity scores d:he two sexes. Basic Difference between Experimental Research and Ex Post Facto Research he basis of the structure in which the experimental scientist operates is simple. e-ie hypothesizes: If x, then y; if frustration, then aggression. Depending on cir¬_instances and his personal predilections in research design, he uses some -ethod to manipulate or measure x. He then observes y to see if concomitant --ration, the variation expected or predicted from the variation in x, occurs. If it this is evidence for the validity of the proposition. x — y, x --), y meaning then y." Note that the scientist here predicts from a controlled x to y. To him achieve control, he can use the principle of randomization and active --ipulation of x and can assume, other things equal, that y is varying as a result --ne manipulation of x. In ex post facto research, on the other' y is observed, and an x, or several -fie complete definition of ex post facto research used in this book is somewhat different from often accepted. The term was originally used by Chapin and Greenwood to mean a quasi-experi¬in which art attempt is made to control independent variables by matching and symbolic means. this meaning is not broad enough for our purposes, "ex post facto" is here expanded to include -search that has the characteristics discussed in the text. 380 TYPES OF RESEARCH x's, are also observed, either before, after, or concomitant to the observation of bserva' There is no difference whatever in the basic logic: it can be shown that the a. ment structure and its logical validity are the same in experimental and ex facto research .2 And the basic purpose of both is also the same: to establish thA empirical validity of so-called conditional statements of the form: If p, then q. The essential difference is direct control of p, the independent variable. In expe-. mental research, p can be manipulated, which is rather direct "control." -~ rol." Whew Clark and Walberg had teachers give one group of. subjects massive reinforc4" ment and other teachers give another group moderate reinforcement, they west directly manipulating or controlling the variable reinforcement. Similarly, whet Bandura and Menlove showed one group a movie with a single model, anothL-- group a movie with multiple models, and a third group a "neutral" movie, the's were directly manipulating the variable modeling. In addition, subjects can be assigned at random to the experimental groups. In ex post facto research, direct control is not possible: neither experiment; manipulation nor random assignment can be used by the researcher. These are two essential differences between experimental and ex post facto approaches Owing to lack of relative control of x and other possible x's. the "truth" of the hypothesized relation between x and y cannot be asserted with the confidence of the experimental situation. Basically, ex post facto research has, so to speak, ar. inherent weakness: lack of control of independent variables. The most important difference between experimental research and ex pos: facto research, then, is control. In the experimental case, the investigator at leas*_ has manipulative control: he has at least one active variable. If an experiment is "true" experiment, he can also exercise control by randomization. He can assign subjects to groups at random, or he can assign treatments to groups at random. In the ex post facto research situation, this kind of control of the independent vari¬ables is not possible. The investigator must take things as they are and try to disentangle them. Take a well-known case. When an experimenter paints the skins of rats with carcinogenic substances (x), adequately controls other variables, and the rats ultimately develop carcinoma (y), the argument is compelling because x (and other possible x's, theoretically) is controlled and y is predicted. But when an investi¬gator finds cases of lung cancer (y) and then goes back among the possible multi¬plicity of causes (x,, X2 x„) and picks cigarette-smoking (say x3) as the culprit, he is in a more difficult and ambiguous situation. Neither situation is sure, of course; both are probabilistic. But in the experimental case the investigator can be more sure— considerably more sure if he has adequately made "other things equal--that the statement If x, then 3, is empirically valid. In the ex post facto case, however, the investigator is always on shakier ground because he cannot say, with nearly as much assurance, "other things equal." He cannot control the independent variables by manipulation or by randomization. In short, the proba 'The basic logic is set forth in: F. Kerlinger, "Research in Education." In R. Ebel, V. Noll, and R. Bauer, eds., Encyclopedia of Educational Research, 4th ed. New York: Macmillan, 1969, pp. 1127-1144 (pp. 1133-1134). EX POST FACTO RESEARCH 381 • ity that x is "really" related to y is greater in the experimental situation than it is - :he ex post facto situation, because the control of x is greater. Self-Selection and Ex Post Facto Research k an ideal social scientific research world, the drawing of random samples of jests, e cts, and the random assignment of subjects to groups and treatments to Vmps, would always be possible. In the real world, however, one, two, or even a at three of these possibilities do not exist. It is possible to draw subjects at ran- a Sel S a I d &rn= in both experimental and ex post facto research: But it is not possible, in ex I pest facto research, to assign subjects to groups at random or to assign treatments no groups at random. Thus subjects can "assign themselves" to groups, can '"iclect themselves" into the groups on the basis of characteristics other than those which hich the investigator may be interested. The subjects and the treatments imne. as it were, already assigned to the groups. Self-selection occurs when the members of the groups being studied are in *c groups, in part, because they differentially possess traits or characteristics amneous to the research problem, characteristics that possibly influence or are afterwise related to the variables of the research problem. Examples of self-4eection may aid understanding. In the well-known research on cigarette-smoking and cancer, the smoking flunts of a large number of people were studied. This large group was divided into tese who had lung cancer— or who had died of it—and those who did not have it. The dependent variable was thus the presence or absence of cancer. Investigators pm*ed the subjects' backgrounds to determine whether they smoked cigarettes, and if so, how many. Cigarette-smoking was the independent variable. The inves¬Wors found that the incidence of lung cancer rose with the number of cigarettes ;coked daily. They also found that the incidence was lower in the cases of light mL*-ers and nonsmokers. They came to the conclusion that cigarette-smoking lung cancer.' This conclusion may or may not be true. But the investigators u.~.-.ot come to this conclusion, although they can say that there is a statistically upancant relation between the variables. The reason they cannot state a causal connection is that there are a number id xher variables, any one of which, or any combination of which, may have mL,ed lung cancer. And they have not controlled other possible independent lies. They cannot control them, except by testing alternative hypotheses, a educe to be explained later. Even when they also study "control groups" of - e who have no cancer, self-selection may be operating. Maybe tense, anxious :re doomed to have lung cancer if they marry blonde women, for instance. It :ul scientific investigators will usually not say "cause." The word "cause" is used here to make more emphatic and because authoritative sources so use it: see The,'Vezv York Tbnes, Dec. 6, I ],.where the Surgeon General of the United States Public Health Service is directly quoted "the weight of evidence at present implicates smoking as the principal etiological (causa¬,I in the increased incidence of lung cancer." 382 TYPES OF RESEARCH may just happen that this type of man also smokes cigarettes heavily. The cigare-ar. smoking is not what kills him —he kills himself by being born tense and anxious_ and possibly by marrying a blonde. Such men are selected into the sample M. investigators only because they smoke cigarettes. But such men select themseli'm into the sample because they commonly possess a temperament that happens in have cigarette-smoking as a concomitant. Self-selection can be a subtle business. There are two kinds: self-selectNa into samples and into comparison groups. The latter occurs when subjects are W lected because they are in one group or another: cancer and no cancer, college a* no college, underachievement and no underachievement. That is, they are selected because they possess the dependent variable in greater or lesser degree. S, 1f-selection into samples occurs when subjects are selected in a nonrandom fashxw into a sample. The crux of the matter is that when assignment is not random. there is always a loophole for other variables to crawl through. When we put subjects into groups-in the above case and in similar cases, or they "put themselves" into groups. M the basis of one variable, it is possible that another variable (or variables) ca:-related with this variable is the "real" basis of the relation. The usual ex post facer study uses groups that exhibit differences in the dependent variable. In some longitudinal-type studies the groups are differentiated first on the basis of the independent variable. But the two cases are basically the same, since group mem¬bership on the basis of a variable always brings selection into the picture. For example, we may select college freshmen at random and then follo-a them to determine the relation between intelligence and success in college. The students selected themselves into college, so to speak. One or more of the charac¬teristics they bring with them to college, other than intelligence —socioeconomic le%el, motivation, family backgroVnd—may be the principal determinants of coi¬iepe success. That we start with the independent variable, in this case intelligence. does not change the self-selective nature of the research situation. In the sampling sense. the students selected themselves into college, which would be an importan: iactor if we were studying college students and noncollege students. But if we are xxierested only in the success and nonsuccess of college students, self-selection nzz- college is irrelevant, whereas self-selection into success and nonsuccess p;-`cps is crucial. That we measure the intelligence of the students when the} mar college and follow them through to success and nonsuccess does not change ermcr the selection problem or the ex post facto character of the research. In sum. ime ;rodents selected themselves into college and selected themselves to succeed zr ax to succeed in college. Lir-ze-Scale Ex Post Facto Research study of research examples will help us evaluate ex post facto research -ehavioral sciences. Several examples have been given in the context of 7 -oblems. Now, we focus on the inherent nature of ex post facto research. 1 '~ and 6, we necessarily concentrated on experimental research, because EX POST FACTO RESEARCH 383 analysis of variance and research design have been developed in an experimental framework. This does not mean that experimental research is necessarily more important or even more frequent in the behavioral sciences. Indeed, it is probably w, exaggeration to say that a large proportion of research in sociology, education, jmthropology, and political science has been ex post facto. Though psychologists me much more inclined to be experimental in their approach— many more psycho¬'legical than sociological variables, for example, are manipulable —it is probably Sze to say that a substantial proportion of psychological studies, perhaps half or nore than half, are ex post facto. The Authoritarian Personality Study4 Tv Authoritarian Personality Study was actually a series of studies which to her constitute an important and influential contribution to social scientific, pnicularly psychological. research. The general hypothesis of the study was that prdtical, economic, and social beliefs are related to deep-seated personality zharacteristics. Another hypothesis was that adult personality is derived from mr~.v childhood experiences. In short, attitudes and beliefs were related to under¬im-ni: personality trends. The investigators, among other things, studied anti Semi—ms— as part of a general characteristic called ethnocentrism. Later, the investigators ct:ended their thought and work to a still larger construct, authoritarianism, which ibn- conceived to be a broad personality syndrome that determines in part idlinocentrism, social attitudes, and certain other behaviors. The authoritarian pa~onality was conceived to be conventional, cynical, destructive, aggressive, pmer-centered. and ethnocentric. While this is an inadequate summary of the basic problems of a very complex mody. it is sufficient for the present purpose. The study had to be ex post facto — aftough there have been later experimental studies in which authoritarianism has limen manipulated and in which high authoritarianism and low authoritarianism, %w instance. have been attribute variables —because authoritarianism, as defined, * i nonmanipulable variable. One of the major results of the study was informa -we on the relation between authoritarianism and prejudice. It is obvious that ,kneg one studies such variables one is studying already existing sets of personal ty, -',aracteristics and attitudes. The subjects are ready-made authoritarians or wr._t_thoritarians (with gradations between) and come to the research with al well-forMUlated attitudes. One can conceive, somehow, of manipulating • .ariables. but the manipulation, as indicated previously, changes their na At any rate, whenever one studies the relations between variables that "a]- %dorno et al., The Authoritarian Personality. New York: Harper & Row, 1950. An extensive ,,f this book has been published: R. Christie and M. Jahoda, eds., Studies in the Scope and "The Authoritarian Personality." New York: Free Press, 1954. Study of the latter volume is for the intermediate or advanced student of social scientific research. See, especially, the H. Hyman and P. Sheatsley, "The Authoritarian Personality-A Methodological Cri¬p. 50-122. More recent research reviews and critiques are: J. Kirscht and R. Dillehay. Dimen- 11horitarianism: A Review of Research and Theory. Lexington: University of Kentucky Press, Messick and D. Jackson, "The Measurement of Authoritarian Attitudes," Educational and :cal Measurement, XVIII (1958), 241-253; H. Titus and E. I lollander, "The California F Scale )logical Research: 1950-1955," Ps.ychologi cal Bulletin. LIV (1957), 47-64. 384 TYPES OF RESEARCH ready exist" in the individuals studied, or whenever one studies the determi-, of such variables, one is deeply embedded in ex post facto research and its I lems. Social Class Influences on Learning Studies' An extensive set of investigations into social-class influences on learning, the tails of which do not concern us here, has greatly affected modern educa These studies are ex post facto studies and, as such, are laden with interpret- difficulties. One of the principal aims of research in such studies is to attempt explain differences in school achievement between middle-class and lower- children. An investigator notes that there are striking differences in ~'c' achievement. Can these differences be attributed in part to social class mem ship? He examines the collective achievement scores of middle-class and lo class children and consistently notes significant differences: middle-class chill do better in school than lower-class children. He may then come to the conclusus that social class is a determinant of school achievement. I The ex post facto character of such research is clear. The investigator star-% with the dependent variable, school achievement (or "learning"), and among dft many possible influential independent variables, he selects social class. NatoraQ he may pick other independent variables as well, variables such as intelligence amf motivation, both of which are also related to school achievement and to socai class. This makes no difference. It is not a matter of complexity; it is a matter at control. The social class researcher has no power to manipulate social class, nor has he the power of randomization. In this case, the relation between social class and school achievement seems well-established. Even the relation between social class and measured intelligence seems to be well-established. Yet these "established" relations may be spurious. and they are more likely to be spurious in ex post facto than in experimental research, other things equal. The major determinants of the difference between the two groups is school achievement may be intelligence and motivation. Middle-class children inay tend to have higher measured intelligence and higher motivation for school Work than lower-class children. It may be these two variables that are the major determinants of school achievement —not social class. Social class membership -1h2pWris" to be a correlate of these two variables. It is, so to speak, part of the awreiational baggage of intelligence, motivation, and school achievement rathe¬*an a determinant of school achievement in its own right. "rlv—r area large number of studies of social class and its relation to a number of variables, includ¬mg %u_i educational variables as curriculum choice, testing, grades, and educational motivation. Two pffrnm=E references are A. Davis, Social-Class Influences upon Learning. Cambridge: Harvard Univet. im ",m 1W-: and W. Warner, R. Havighurst, and M. Loeb, Who Shall Be Educated? New York: Amcw ik Row. 1944. More recent, extensive, anti studies are the Coleman and Wilson MiPmax, j Coleman et al., Equality of Educational Opportunity. Washington, D.C.: U.S. Government 1111nomg 0111liKe. 1%-6: A. Wilson, "Educational Consequences of Segregation in a California Com lw t - An L'S. Commission on Civil Rights, Racial Isolation in the Public Schools. Washington, D.C.: CAL IL-motmummt Printing Office, 1967, vol. 2, pp. 165-210. The Wilson report's central focus was w o :=w& rwe. and school achievement. EX POST FACTO RESEARCH 385 I'deology, Consensus, and Operational Beliefs Studies :1olitical scientists have been interested in certain key questions, the answers to :hick have deep significance in a democracy. Two of these are: Does consensus exist among Americans on basic democratic beliefs? If such consensus exists, do Americans subscribe to specific operational beliefs and behaviors implied by the basic beliefs? Three important studies addressed to these and other related ques¬tions have been done in the last decade or so, and the answers to the questions are not simples In one of the most recent of these studies, Free and Cantril obtained the re-sponses of two probability samples of more than 3000 Americans to a variety of questions on political beliefs. They found distinct and sharp differences between what they called the ideological spectrum and the operational spectrum of beliefs. Americans, for instance, respond quite differently to abstract statements of beliefs than they do to more specific operational statements. Evidently they are ideologi¬cally conservative and operationally liberal .7 A somewhat different conclusion, however, comes from two other large-scale studies. Prothro and Grigg sampled (randomly), in a northern city and a southern city, the responses of registered voters to abstract and specific statements, like -Democracy is the best form of government" (abstract) and "A Negro should not ~e allowed to run for mayor of this city" (specific). They found high agreement among the voters with the abstract statements and much less agreement on the specific statements. Here people seem to be ideologically (abstractly) liberal and operationally conservative. In the third, and perhaps most important and sophisticated study, McClosky administered to his respondents a wide variety of statements on personal back-ground, personality traits, and political, economic, and social values, attitudes, and beliefs. His respondents were of two kinds: leaders or political influentials (N = zWO+) and the general electorate in the United States (N = 1500+) selected to be representative (in a Gallup poll). Like Prothro and Grigg, McClosky found greater support for general, abstract statements of democratic beliefs than for the speci¬fic application of the beliefs. For instance, 89 percent of the electorate sample agreed with the statement, "I believe in free speech for all no matter what their views might be," but 50 percent agreed with "A book that contains wrong political views cannot be a good book and does not deserve to be published." More¬over, McClosky found that the leaders were distinctly more democratic in their outlook and rejected antidemocratic sentiments more than the general electorate did. We note two important points about these studies. One, they are clearly ex 1L. Free and H. Cantril, The Political Beliefs of Americans. New Brunswick, N.J.: Rutgers University Press, 1967; H. McClosky, "Consensus and Ideology in American Politics," American Political Science Rtviezv. LVIII (1964), 361-382; J. Prothro and C. Grigg, "Fundamental Principles of Democracy: Bases of Agreement and Disagreement," journal of Politics, XXII (1960), 276-294. 7 Although Free and Cantril's results can be questioned on the basis of the way they measured liberalism and conservatism, this is not our concern here. We are interested mainly in the general nature ofthe research. Ili h 386 TYPES OF RESEARCH post facto. There is no experimental manipulation whatever. While the sar-,:-,, was random—except, perhaps, in the McClosky study—there was no poss-_ of random assignment. People were asked for their responses to question statements, and these responses were related to each other, mostly in perce-crossbreak form. That is, the relations between independent and dependent abler, as reflected in the responses that people brought with their to the st;_ were analyzed. The beliefs of people, as expressed in their responses, were us-1w,, taken as dependent variables and related to other responses or sociological There was literally no intervention by the researchers. Small-Scale Ex Post Facto Research Regional Differences in Prejudice' In a well-executed study of a social problem that is difficult to probe experirr)--T. tally, Pettigrew asked the question: Is anti-Negro prejudice more closely rela-= to social factors and less so to personality factors in the South than in the North' lz effect, this amounts to contrasting sociological and psychological explanations at prejudice. To test hypotheses derived from this question, Pettigrew administerec authoritarianism, anti-Semitism. and anti-Negro scales to random samples ef white adults in four northern and four southern towns. One hypothesis predicted a simple difference between northern and southe-T anti-Negro prejudice, and the southern sample did have a significantly highe7.- mean score than the northern sample. There was no significant mean difference OT the authoritarianism measure, which was used as a control test. Pettigrew rea¬soned that, since the authoritarianism scale is presumed to measure "externalizirz personality potential," and since the two regions did not differ on this scale but dic differ on the anti-Negro scale, the hypothesis that externalizing personality factors are of equal importance in the North and the South, and that social-cultural factors are more important in the South than in the North, was supported. Freedom and Equality Study9 great deal of theoretical and empirical work has been done on prejudice. It can safely be said that because of this work we understand a good deal about the psychology and sociology of prejudice, its stereotypes. how it operates. and how it --s supported. Much of the research has been ex post facto. For example, the many ~-udies of sterotypes of minority-group members have been largely and perforce ex post facto. To understand phenomena like prejudice and stereotypes, after all, we have to know their incidence and their relations to other variables. Of course. further understanding of such phenomena is enhanced when we can change them. The former knowledge requires ex post facto research, while the latter requires an T. Pettigrew, -Regional Differences in Anti-Negro Prejudice," Journal of Abnormal and Social Psychology. LI X (1959), 28-36. `M. Rokeach, Beliefs. Attitudes, and Values. San Francisco:.1 ossey- Bass, 1968, pp. 168-178. EX POST FACTO RESEARCH 387 txperimental approach. The Pettigrew study just cited is an excellent example of to former demand. The research we are about to examine is an ex post facto part -C theoretically based research that is both ex post facto and experimental. With¬out the knowledge yielded by the ex post facto work, it is doubtful that the later experimental work would have been possible. Rokeach has had a number of groups, including a national sample, rank order rwo sets of what he calls terminal and instrumental values. Two of the terminal -values, freedom and equality, have been particularly significant because they are evidently keys to fundamental differences in social and political value outlooks. Rokeach has found, for instance. that different groups reliably and significantly rank these two values (embedded among others) quite differently. In Table 22. 1, TABLF. 22.1 COMPOSITE RANKS OF Freedom AND Equality OF DIFFERENT GROUPS (1), AND -ri[EIR FREQUENCY OF MENTION AND COMPARABLE RANKS IN FOUR SAMPLES OF POLITICAL WRI-I-INGS (11) Unemployed Unemployed Calvinist Policemen (50) Whites (141) Negroes (28) Students (75) Freedom la 3 10 8 Equality 12 9 1 9 Socialists Hitler Goldwater Lenin Freq." Rank Freq. Rank Freq. Rank Freq. Rank Freedom +66 1 —48 16 +85 1 —47 17 Equality +62 2 —71 17 —10 16 +88 1 11 is the highest rank, 12 the lowest. 'Frequency is defined as number of favorable mentions minus number of unfavorable mentions. composite ranks of the two values given by different groups are given (I). The results are dramatic. Policemen and unemployed blacks are very, very different in their social value outlook. To the policemen, freedom is highly important, while equality is not, but to the blacks equality is all-important and freedom is not. Quite an upset of certain of our traditional ideas' In contrast, Calvinist students rank both values rather low. Rokeach seems to have hit upon a fundamental difference in value outlook. His results are evidently not spurious. Study the bottom half of Table 22.1 (11). He ~-7J a colleague (James Morrison) counted the number of times that freedom and (Ilit " v were mentioned favorably and unfavorably in the writings of socialists, :ler, Goldwater. and Lenin. Again, there are striking contrasts, which need no ,-aboration. This is another excellent example of ex post facto research and its

No comments:

Total Pageviews