Indigenous Research: A Paradigm Shift from Statistically Insignificant to Statistically Underpowered

In 2007 I began graduate school, and in my first research course I had to find a qualitative and quantitative peer-reviewed article within my area of interest. My research interests resided within Native American higher education. After searching the institution’s journal catalog, I found very few articles that expounded beyond basic descriptive statistics. Perplexingly flummoxed, I expanded my search to book chapters, reports, and dissertations, and the results were nil.

I did come across Shelly Lowe’s (2005) article where she applied the term ‘not statistically significant’ to the lack of research on Native Americans. Specifically, as “footnotes indicating that findings on Native Americans are not statistically significant and so are omitted from the research are too often the only reference to Native Americans in much of the literature in higher education” (p. 39). In other words, Lowe was appropriating the term ‘not statistically significant’ or ‘statistically insignificant’ to refer to the problem of invisibility of Native American students in higher education research.

At that time, in 2007, I was only a first-year graduate student and was not well acquainted with statistical terminology. Now, thirteen years later, as a methodologist, I take issue with the misuse of the term for the simple reason that Lowe is implying that researchers conducted analyses on Native American college students. Yet, these researchers decided not to publish the results because the p-value was higher than 0.05 and indicated strong evidence for the null hypothesis, which was not the case.

You may be asking yourself what is statistical significance? Statistical significance is the result of a culmination of statistical concepts: (1) hypothesis testing, (2) normal distribution, and (3) p-values. These concepts are taken together to quantify whether a result is due to chance or some factor of interest. Nevertheless, the sample size can influence statistical significance, but it is not synonymous with small sample sizes.

The term was misused again in 2010 when referring to Native American high school students who had inadequate sample sizes (Faircloth & Tippeconnic, 2010) by citing Butterfield’s 2003 summary report* on Native American students as an example of ‘statistically insignificant.’ Later, in the widely disseminated and popular book Beyond the Asterisk: Understanding Native Students in Higher Education (Shotton, Lowe, & Waterman, 2013) the term ‘not statistically significant’ was misrepresented again as:

often excluded from institutional data and reporting, omitted from the curriculum, absent

from the research and the literature, and virtually written out of the higher education

story. In particular, Native American students are generally not reported or discussed in

quantitative research findings or are noted as not statistically significant (p. 2).

As a result, statistically insignificant has been misapplied in manuscripts (Willmott, Sands, Raucci, & Waterman, 2015), books (Warrior, 2013), and reports (Scarborough, Kares, Arenas, & Lewis, 2019) and are three of the examples I found doing a quick google search; I have even misused the term myself in my 2019 dissertation. In a 2016 blog post titled Why Fund ‘Insignificant’ Populations Kevin Jennings, who is an executive director of the Arcus Foundation, was invited to speak on a panel concerning the needs of lesbian, gay, bisexual, and transgender (LGBT) and Two-Spirit Native peoples. During the Q & A portion, an attendee asked, "given that LGBT people are a small minority and Native Americans are an even smaller one, isn't the population of LGBT Native Americans statistically insignificant?" Furthermore, the attendee followed up with, "why would you say to a foundation that they should fund statistically insignificant populations when they want their funding to have a big impact?" Mr. Jennings answered, "on a strictly mathematical basis, the questioner is right." However, he remarked that the question missed the mark and questions “what is significant anyways?” Unfortunately, the statistically insignificant term has trickled its way down from academia and has taken a pejorative connotation as ‘not worthy of attention.’

In my opinion, Indigenous scholars need to move away from the ‘not statistical significant/statistically insignificant’ terms when referring to the invisibility of Indigenous students for two reasons. Reason one: the term statistical non-significance is an actual outcome of statistical analysis and is a little misleading and implies that statistical analyses were conducted. This is akin to a quantitative researcher using the term saturation when a robust sample size has been achieved. Reason number two: the term has amalgamated into a concept where statistically insignificant is attributed to as inconsequential, unimportant, and irrelevant as evidenced behind the logic of Mr. Jennings and the attendee. Unfortunately, this logic could give researchers and funders credence to overlook and ignore Native American students and Native Americans as a whole.

Moving forward, I implore that Indigenous scholars use the correct terminology when describing the invisibility of Native American educational research. In my own research and in my upcoming manuscript, I began using statistically underpowered and ignored, “In other words, when it comes to empirical higher education research Native Americans are statistically underpowered, ignored, and relegated to footnotes or an asterisk,” (Marroquín, 2020, p. 73). Garland (2007; 2010) and Shotton, Lowe, and Waterman (2013) did get it correct when referring to the lack of quantitative research with an asterisk and I do applaud Ms. Lowe for garnering attention by attaching this problem to a catchy buzz phrase; however, instead of statistically insignificant, a better term or phrase is statistically underpowered. Thus, statistical power is associated with obtaining the ideal sample size, which allows researchers to detect a true statistical significant result or effect (Dorey, 2011).

*Faircloth’s & Tippeconnic’s 2010 report cites Robin Butterfield’s summary report Council of Chief State School Officers. Strengthening partnerships for Native American students. Denver, CO. April 9-11, 2003. However, the report is unavailable.

References

Dorey, F. (2011). Statistics in brief: Statistical power: what is it and when should it be used? Clinical Orthopaedics and Related Research., 469(2), 619–620. https://doi.org/10.1007/s11999-010-1435-0

Faircloth, S.C., & Tippeconnic, III, J.W. (2010). The dropout/graduation rate crisis among American Indian and Alaska Native students: Failure to respond places the future of Native peoples at risk. Los Angeles, CA: The Civil Rights Project/Proyecto Derechos Civiles at UCLA. Retrieved from https://www.civilrightsproject.ucla.edu/research/k-12-education/school-dropouts/the-dropout-graduation-crisis-among-american-indian-and-alaska-native-students-failure-to-respond-places-the-future-of-native-peoples-at-risk/faircloth-tippeconnic-native-american-dropouts.pdf
 

Garland, J.L. (2007). Review of the book Serving Native American Students: New Directions for Student Services. Journal of College Student Development, 48, 612-614.
 

Garland, J.L. (2010). Removing the college involvement “research asterisk”: Identifying and rethinking predictors of American Indian college student involvement. Retrieved from Dissertations & Theses: Full Text. (Publication No. AAT 3426253).

Jennings, K. (2016, April, 16). Why Fund 'Insignificant' Populations? PhilanTopic. https://pndblog.typepad.com/pndblog/2016/04/why-fund-insignificant-populations.html

Lowe, S. (2005). This is who I am: Experiences of Native American students. New Directions for Student Services, (109), 33–40. https://doi.org/10.1002/ss.151

 
Marroquín, C. (in press). The Validation of the North American Indigenous College Students Inventory (NAICSI). Journal of American Indian Education, 59(1) 73-103.

Scarborough, W., Kares, F.R., Arenas, I., & Lewis, A.E. (2019). Adversity and resiliency for Chicago’s first: The state of racial justice for American Indian Chicagoans. Institute for Research on Race and Public Policy. https://irrpp.uic.edu/pdf/publications/IRRPP_TheStateOfRacialJusticeForAmericanIndianChicagoansReport.pdf

Shotton, H., Lowe, S., & Waterman, S. (2013). Beyond the asterisk: Understanding Native students in higher education (First edition.). Sterling, Virginia: Stylus.

Warrior, R. (2014). Double-crossed (or stabbed twice), again and again (and again): Reflections on our data-driven academic economy. In Rosemary G. Feal (ed.) Professions 2013. Modern Language Association of America.

 
Willmott, K. E., Sands, T. L., Raucci, M., & Waterman, S. J. (2015). Native American college students: A group forgotten. Journal of Critical Scholarship on Higher Education and Student Affairs. 2(1) 78-104. https://ecommons.luc.edu/cgi/viewcontent.cgi?article=1023&context=jcshesa