Brian D. Ray, Ph.D.
June 1, 2020
Excerpt:
Social science research (e.g., education research) almost never sees “the perfect study.” While teaching research methods at the university graduate and undergraduate levels across seven years, I explained to my students that many studies in education are exploratory, descriptive, and correlational in design. Some studies include designs that allow for more conclusions that address cause and effect. But researchers are not allowed (morally or legally) to execute a cause-and-effect design like this: randomly assigning one-third of 1,000 students in New Orleans to state or public schooling, one-third to private institutional schooling, and one-third to private homeschooling, letting things roll for 13 years to see how all the students “come out in the end” on academic achievement or amount of abuse experienced at the hands of adults and peers, then making a bunch of causal claims. Researchers are not allowed, by law or ethics, to force parents to put their children into these three types of schooling (i.e., educational treatment) and so may not conduct this type of study.
37 Years Ago
When I began doing research—about 37 years ago—on homeschool families, students, and groups, other scholars and I had a lot of questions. I wanted to know why parents chose homeschooling. What were their demographics? And how were the children performing academically?
When I first set out to measure homeschool educational attainment back in 1985, there were several challenges. How do I study a “hard-to-reach population?” What is a way to gain participation when there is no master list from which to randomly select families or students? How do I ethically engage their participation? How can I statistically control for background demographic traits so that reasonable comparisons can be made to public school student norms? How will I get test scores when there is no research budget for new testing? How can I do this on a shoestring of a budget and with very little manpower to assist me? (And especially when thousands of old-fashioned, multipage survey questionnaires would be going out and coming back via the US mail and all the data would be entered by hand, one question and click at a time, into a spreadsheet.)
As in any study, there were problems and challenges to executing my studies the ideal way. With hard-to-reach populations, there are typically even more obstacles. To learn something rather than nothing, researchers must come up with solutions and compromise between the ideal approach and the realistic or practical approach. I had to do that; just as most other social scientists have had to do. I had a lot of conversations and brainstorms with other scholars, leaders of homeschool organizations, academic achievement testing services, government officials, and homeschool families. It was and is complicated. It tries the mind. It pits the curious scientist against reality while trying to get questions answered. It was exhausting at times. It was thrilling at times. I thoroughly enjoyed it (and still do).
The Perfect Study
It is highly unlikely that an investigator will ever get to do a “random-assignment true experiment” regarding the “effects” of schooling type (e.g., public institutional school compared to private institutional school compared to homeschooling). Research opportunities and life are just not that way. Research scholars must work within budgetary, timeline, and ethical constraints to design and execute the best study that they can to meet their objectives. So, for example, since a scholar may not do a “true experiment,” the alternative might be some kind of “quasi-experimental” or non-experimental design, such as a matched-pair study (or a retrospective, explanatory study; or a cross-sectional, predictive study; or a cross-sectional, explanatory study).
Every study related to homeschooling is based on a particular research design, with an underlying theoretical framework; and each specific study has limitations and delimitations, based on its methods (which are often based on the aforementioned factors: budgetary, timeline, and ethical constraints), regarding what it can and cannot reveal and to what extent its findings can inform understanding and policy considerations. Every researcher should clearly state the study’s purpose and objectives; lay out a review of related literature; present a theoretical framework and explain methods; and, finally, present the findings and analysis. A researcher should also note the limitations of his study and provide a discussion of conclusions and interpretations. The conclusions should follow from and be coherent with the study’s methods and limitations.
The study is sound research if the researcher takes all these steps. If things are clearly stated so that they are conceivably reproducible and the conclusions that the scholar states reasonably follow from the methods, findings, and limitations, then it is sound, beneficial, or “good” research. It is a bogus approach for any observer, evaluator, or critic to say that a research study is “bad” or “flawed” because it (a) does not meet all of the objectives that the critic wished it had included or met, (b) cannot lead to causal statements, or (c) led the investigator to suggest policies or ideas that the critic does not like. If a researcher is clear about the study’s theoretical framework, methods, conclusions, and limitations, and her conclusions and recommendations generally fit the findings and overall framework, then it is a decent study and people can learn from it. It can inform the field of inquiry.
Big Picture of Some Homeschool Research
Overall, research has found many positive things connected to homeschool students and families. There are many ways to do a review of research and, depending on the reviewer’s worldview and what message the reviewer wants to convey to the reader, the reviewer can easily slant things and affect the reader’s understanding of what the body of research says. There are several reviews of research on homeschooling, and a careful reading of each of them might reveal the author’s biases, whether slight or strong.
Reviews of research can be very helpful because they help us see the big picture. They can show us trends. They can indicate overall findings regarding a particular phenomenon. They can help us not get stuck on looking at an individual tree or two and miss the significance and beauty of the forest.
One unique review is the one that I did. It has been published in a peer-reviewed journal. Before I get to that, however, I would like to take a short side trail about the meaning of peer review.
Peer Review
In academic publishing, the goal of peer review is to assess the quality of articles submitted for publication in a scholarly journal. Before an article is deemed appropriate to be published in a peer-reviewed journal, it must undergo the following process:
- The author of the article must submit it to the journal editor, who forwards the article to experts in the field. Because the reviewers specialize in the same scholarly area as the author, they are considered the author’s peers (hence “peer review”).
- Typically, the reviewers do not know the identity of the author and the author does not know the identity of the reviewers.
- These impartial reviewers are charged with carefully evaluating the quality of the submitted manuscript.
- The peer reviewers check the manuscript for accuracy and assess the validity of the research methodology and procedures.
- If appropriate, they suggest revisions. If they find the article lacking in scholarly validity and rigor, they reject it.
Because a peer-reviewed journal will not publish articles that fail to meet the standards established for a given discipline, peer-reviewed articles that are accepted for publication generally exemplify the best research practices in a field….
My review in the peer-reviewed journal is of only and all peer-reviewed homeschool studies on the learner outcomes of academic achievement, social development, and success in adulthood. The article also gives the demographic characteristics of the US homeschooling population, reasons that parents choose to homeschool, and proposals for future research on parent-led home-based education. Here it is, open access:
Ray, B. (2017b). A systematic review of the empirical research on selected aspects of homeschooling as a school choice. Journal of School Choice: International Research and Reform, 11(4), 604-621. https://doi.org/10.1080/15582159.2017.1395638
This review includes the peer-reviewed published research of roughly 45 scholars spanning about 30 years. Four of the studies were based on representative national samples in the United States. At least one involved a careful matched-pair design. Other studies control for various background variables while examining learner outcomes. The researchers live all over the United States and in other countries. Clearly, there is a breadth and depth of scholarly work and scholars investigating homeschooling.
What was uncovered in this peer-reviewed review of peer-reviewed research? In 11 of the 14 peer-reviewed studies (78%) on academic achievement, homeschool students performed significantly higher than institutional school students.
Out of 15 studies on social and emotional development, 13 of them (87%) showed clearly significant positive outcomes for the homeschooled students compared to those in conventional schools.
And 11 of the 16 studies (69%) on success into adulthood and college showed positive outcomes for the homeschooled compared to those in conventional schools.
Overall, 35 of the 45 peer-reviewed studies (78%) found that the homeschooled students or graduates of parent-led home-based education performed significantly better than their conventional or institutional school peers in terms of academic achievement, social and emotional development, and success into adulthood (including at college or university).
That is, 30 years of research by about 45 scholars in peer-reviewed journals has revealed significantly positive things related to homeschooling. There are very few pieces of empirical evidence from studies indicating that homeschool students or graduates of homeschooling are not doing as well as others; there is no body of research showing that they are being harmed at any higher rate than others. A helpful review of research for those who want to go deeper and that covers much more than only peer-reviewed publications is one that I published in a peer-reviewed journal, also in 2017.
Advocates and Advocacy
Some scholars will write or make claims about “advocacy research” or “advocates” of this or that. The first thing to keep in mind is that being an advocate of concepts, initiatives, or values is a core part of the human condition. Everyone is an advocate of something. One scholar might be an advocate of reducing child abuse. Another might be an advocate of Roman Catholic schooling. A third could be an advocate of seeing and understanding the world through the lens of critical theory, or of neo-Marxism. And a fourth might be an advocate of conventional government-run schools as the best educational format for US society.
People have asked me whether I am an advocate of homeschooling. My response is that because of my worldview and experience, I am an advocate of what helps children learn and enjoy learning so that they become adults who are literate and numerate, kind to their neighbors, and able and inclined to work hard and support themselves and their families and honor their Creator. I am a strong advocate of education in the classic sense. I have been a state-certified teacher, have taught in private and public schools, and have taught teachers as a former professor of education. Because of all this, I am an advocate of much dialogue and interchange between adults and children. I advocate for customizing pedagogical approaches and curriculum for each child, according to their strengths, weaknesses, dreams, and desires. Mentoring and one-on-one instruction are great for children (and adults). Much opportunity for positive social interactions, cooperation, and age integration are good for children and adolescents. More parental involvement, rather than less, is good. Children having time to know and enjoy peers, but being adult-oriented rather than peer-dependent is good.
To the extent that any of the aforementioned conditions can or are likely to be met in homeschooling, government schooling, or private institutional schooling, I am an advocate for those types of schooling….
Concluding Thoughts
A sound or respectable social science study and its report is one in which the investigator clearly states the study’s purpose, methods, and findings, and the interpretations and conclusions that the scholar makes reasonably follow from all that preceded them. A sound evaluation of a study considers all of this. Everyone who reads a study, or an evaluation of a study, should remember that every researcher and reviewer has a personal worldview that affects, biases, or slants what he or she does, writes, and says.
To date, the clear majority of research done on homeschooling finds positive things associated with the method of education. Consistently negative criticisms of various individual studies about homeschooling and homeschool parents and students or of a particular scholar who studies homeschooling are, in essence, a negative criticism of the large majority of homeschool researchers and their studies. The negative critics generally ignore or fail to recognize and explain what homeschool studies have legitimately contributed to the knowledge base and understanding of an important body of inquiry regarding a millennia-old and effective form of teaching and learning—homeschooling—in the United States and around the world.
Source: What is “Good” Research? A Homeschool Researcher Responds to Harvard Professor’s Criticism: Research Methods 101. Excerpt posted by permission of the author.