RESUMO
A variety of ways to detect questionable research practices in small sample social science surveys have been discussed by a variety of authors. However, some of those approaches (e.g., GRIM test, SPRITE test) do not work well for results obtained from larger samples. Here several approaches for detecting anomalies in larger samples are presented and illustrated by an analysis of 78 journal articles in the area of criminology, 59 by Dr. Eric Stewart, published since 1998 with similar methods and/or authors. Of all 59 articles, 28 (47.5%, p < .001, d = 0.94) had two or more major anomalies compared to none of the 19 control group articles. It was also found that the larger the role of Dr. Stewart in article authorship, the greater the number of anomalies detected (p < .001, d = 1.01) while for his coauthors, there were few significant relationships between their roles and total anomalies. Our results demonstrate that extensive problematic results can remain undetected for decades despite several levels of peer review and other scientific controls; however, use of two types of control groups and the use of statistical methods for measuring and evaluating anomalies can improve detection.
RESUMO
Social science is commonly used in debates about controversial issues, especially for those concerning human sexuality. However, caution must be exercised in interpreting such social science literature, because of a variety of methodological and theoretical weaknesses that are not uncommon. Families are complex structurally and over time; such data are not easily analyzed. Merely determining the number of, for example, sexual minority families has been a difficult task. While some new theories are popular with social scientists, for example, sexual minority theory, they are often used to the exclusion of other, equally valid theories and often are not well tested empirically. Some types of families remain relatively unexamined. Social scientists can be biased by their own values, which are reflected in weak use of theory and in a variety of methodological problems. Eight studies are presented as examples of probable confirmation bias, in which methods and theory were modified in unusual ways that may have affected the outcomes and conclusions. Suggestions for improving social science include greater attention to effect sizes rather than statistical significance per se, deliberately minimizing the politicization of science, developing a culture of humility with respect to social science, deliberately reducing common biases, and maintaining a deeper curiosity about social science than is often seen. Scientists must be open to seeing their best "sacred cow" ideas or theories disproven or modified with increases in research on such issues. Summary: In controversial areas of social science, there can be numerous threats to the validity of science. Here, some of the more common risks for social science research and theory are examined, with several specific illustrations of how bias appears to have crept into social science, often as confirmation bias. Recommendations are made for reducing bias in future research.
RESUMO
Recent research on transgender children who have had support from their parents for their transitioning has concluded that their mental health is virtually no different than that of nontransgender children. Such research has been extensively cited, over 370 times in the past three years. Most of the hundreds of reviews received the stated results of the studies with little caution. However, the research featured numerous statistical errors and omissions, the implications of which would likely lead neutral observers to conclude that the mental health of transgender children, even when supported by their parents, was poorer than that of the groups of control children. In particular, levels of anxiety as reported by both parents and their transgender children appear to be significantly higher, and the transgender children's reports of self-worth appear to be significantly lower. Although reports regarding depression are not as significantly different, the effect sizes were generally in a similar direction as the other outcomes, being less favorable for the transgender children. Such issues highlight the need for careful examination of statistical research, even when published in highly regarded medical journals. As with other research, findings from the early stages of controversial research may often be premature. Further research is needed to explore factors underlying these results. SUMMARY: Some scholars have believed that if transgender children were supported by their parents before the children reached puberty, the generally higher rates of mental illness experienced by many transgender persons might be prevented or alleviated. Dr. Kristina Olson of the Department of Psychology at the University of Seattle was the first scholar to have studied groups of transgender children who were being supported by their parents and to have compared them to a control group of children and to siblings of the same transgender children. Her conclusion was that there were minimal, if any, differences in anxiety, depression, and self-worth among the groups of children; her research has since been cited extensively as having found just that. We reanalyzed her raw data and found that, to the contrary, the transgender children, even when supported by their parents, had significantly lower average scores on anxiety and self-worth. Often, a significantly higher percentage of transgender children, compared to controls, featured preclinical or clinical levels of anxiety. Parental support of transgender children may temporarily reduce levels of poor mental health for some transgender children, but it does not appear to eliminate those problems for all transgender children. Our findings should serve as a warning against accepting research at a surface level, which can lead to acceptance of invalid information and pursuit of ineffective interventions.
