Develop models and measures to predict retention

Included here for your review is a paper I submitted as part of my final project in FCS 6010 – Basic Research Methods, a research methodology course I completed while working on my masters in Educational Technology.

The paper proposed development of a early warning system to predict and address student retention.

Please note that while the proposed budget for the project is based on real numbers, the proposal was drafated and planned based on an imaginary budget funded by a $25,000 grant. The overall costs for completing a project of this type would require University staff time (which could be easily allocated) and incentives for students to participate in the study. Otherwise said, the cost to our institution would be relatively low while the potential gains would be very high.

Predicting Retention at a Midwestern University
Michael VanPutten
Western Michigan University

Purpose of the Study

The purpose of this study is to measure and predict factors involved with student retention at Western Michigan University in Kalamazoo, Michigan. The results may indicate directions for development of tools and strategies for improving student graduation rates at four-year institutions with 20,000 or more students.

Importance and Justification

The average cost of attendance for a full-time student at a four-year institution (including tuition, fees, housing, books, and materials) in 2003-04 was $15,100 at public institutions and $29,500 at private institutions (Rooney et al., 2006). A student’s decision to invest time and money in a college education is influenced by expected future income, available job opportunities, encouragement from parents or peers and academic ability (Perna, 2000). A consistent upward trend in earnings potential for college graduates has been observed during the previous two decades: Dr. Gary W. Phillips, Acting Commissioner of the National Center for Education Statistics, indicated in the release of the Condition of Education: 2000, that the earnings of males who held a college degree increased by 56% in 1998 compared to 19% in 1980 while women’s earnings increased by 100% in 1998 compared to 52% in 1980 (Phillips, 2000). A recent study, using a weighted sample of 11,933 from a population of 2.45 million, indicated that increased future income for college graduates is enjoyed by multiple ethnic groups, with increases to their gross income averaging $13,779 for African Americans, $11,117 for Caucasians, and $9,711 for Hispanics (Perna, 2000).

A spring 2006 enrollment survey conducted by the National Center for Education Statistics for the U.S. Department of Education indicated that 11 million first-time, full-time students pursuing a degree or certificate at four-year institutions had a graduation rate of 56% (Knapp, Kelly-Reid, & Whitmore, 2007). “Graduation rates” refer to the percentage of students who complete degree or certificate requirements within 150 percent of the normal program time (Knapp et al., 2007). First-time students are those who have not attended a secondary education institution (Knapp et al., 2007). These graduation rates are consistent with a 1995-96 national survey where 12,000 students who were identified as being first-time students from a “nationally representative sample” of 44,500 undergraduates (representing a total of 16.7 million undergraduates) were found to have a graduation rate of 51% from four-year institutions (Berkner, He, Cataldi, & Knepper, 2002).

A national survey conducted in 1995-96 of 12,000 post secondary students reported the following reasons for early departure: 26% needed to work, 16% indicated other financial reasons, 10% were done taking desired classes, 10% had conflicts at home or personal problems, 8% had a chance in family status, 7% were taking time off, 6% were not satisfied, 6% had job or military service conflicts, and 4% had academic problems (Bradburn & Carroll, 2002). Note that the report indicated that students in the sample were allowed to provide up to three reasons for early departure, 61% of the respondents indicated only one of the reported reasons, and 24% did not indicate any of the reported early departure reasons (Bradburn & Carroll, 2002). Bradburn and Carroll identified these 12,000 respondents by selecting students from the 1995-96 National Postsecondary Student Aid Study (NPSAS:96) who were identified as being first-time postsecondary students (Bradburn & Carroll, 2002). The NPSAS:96 study featured a “nationally representative sample” of 44,500 students from a population of 16.7 million undergraduates (Bradburn & Carroll, 2002). Note that NPSAS:96 is the same study utilized by Berkner, He, Cataldi, & Knepper to analyze and report graduation rates.

Based on the graduation rates reported in the Knapp et al. survey conducted in 2006, there are 4.84 million students who do not complete their college degree. If an individual were to consider the average 2003-04 costs of tuition reported by Rooney, one could estimate that if these 4.84 millions students were to attend only one year of college at a public institution and then choose not to continue, the students would collectively incur a minimum of $18.3 billion in debts. If these students were to complete their college education they would earn a higher salary that would enable them to pay off their college debt in just a few years.
Tinto identifies five conditions for promotion of student persistence: expectations, support, feedback, involvement, and learning (Tinto, 2003). In detail, these institutional conditions provide an environment where students are expected to succeed, provides sociological/psychological and academic support, provides early feedback regarding performance using early warning systems, involves students as members, and creates environments that foster learning. Tinto’s model has been supported by work done by Zea et al. who utilized a student’s ability to cope with college, individual self-esteem, academic integration, identification with the university, and experience of disrespect because of race, ethnicity, or religion as measures to predict a student’s likelihood to persist and complete a college degree (Zea, Reisen, Beil, & Caplan, 1997). Their study found that perception of disrespect had a negative effect on a student’s desire to continue their education. In addition, Zea et al. found that how students identify themselves with a University has an affect on the students’ decision to continue their education at that institution. A study by Cartney and Rouse reported that student retention could be improved when an institution utilizes small groups to socialize individual students experiencing feelings of disconnection from their classmates and/or the institution (Cartney & Rouse, 2006).

Early warning systems designed to identify students who are unlikely to complete their college education have also demonstrated effectiveness in retention efforts (Caison, 2007; Moller-Wong, Shelly, & Ebbers, 1999). Caison, using multivariate logistic regression to compare the effectiveness of early warning systems, found that including institutional student information system data (e.g. high school GPA, SAT scores, and total credit hours) paired with information regarding parental education background, declaration of a major and intention to work could be utilized with greater effectiveness in an early warning system to predict one-year retention rates (Caison, 2007). Early warning systems can also be designed to classify individual student retention rates into low, medium and high-risk categories of not completing a degree by utilizing academic performance indicators, individual student course enrollment statistics, and student demographic data (Moller-Wong et al. 1999). The methods utilized to identify the measures for classifying students into levels of risk included cluster analysis and a tree diagram (which identified eight clusters of risk) paired with logistical regression to identify which factors were the most significant in determining high and low levels of risk (Moller-Wong et al. 1999). Moller-Wong was also able to utilize logistic regression to determine which factors weighed more in early semesters versus later semesters over a fourteen-semester period (Moller-Wong et al. 1999). The Caison and Moller-Wong early warning systems tracking systems did not take into account the effects of student participation in academic advising or extra curricular groups and activities, frequency of change of major, if students were participating in courses that counted toward their declared major, total number of jobs, or the total hours allocated for study, employment and commuting to campus. One other weakness of the Moller-Wong system is the requirement to change what is being measured on a semester-by-semester basis in order to reproduce the prediction results of student persistence – which makes generalizing the result of her team’s system to other campuses difficult.

Theoretical Framework

Tinto’s model suggests that an institutional retention program should include goals for making students part of the university, a commitment to and focus on the education of students and their involvement in learning, as well as nurturing of both academic and social campus communities (Tinto 1987). Students are more likely to continue their education in an environment where they feel at one with the community, supported in their educational goals, and involved in the educational process (Tinto 1987). Tinto also recommends consideration of expectations, feedback and support as necessary conditions for the promotion of student persistence (Tinto, 2003). These aspects were discussed earlier in this proposal. In review, these institutional conditions provide an environment where students are expected to succeed, the conditions provide early feedback regarding performance using early warning systems, involves students as members, and environments that foster learning.

Previous research has demonstrated the potential effectiveness of early warning systems to predict students’ decision to complete an undergraduate degree. A revised tool that utilizes characteristics and measures of existing systems as well as additional measures may demonstrate greater effectiveness for predicting the likelihood that students will complete their education. The purpose of the proposed study is to determine if student utilization of academic advising, declaration of a major, student employment, and/or participation in extra curricular activities or groups have a positive effect on at-risk students’ decision to complete their college degree more so than their not-at-risk peers. The null hypothesis for this study states that academic advising, declaration of a major, student employment, and/or extra curricular activities do not affect an at risk student’s decision to complete their degree more than their not at risk peers.

Methodology

About the Research Design

The proposed study will utilize a quantitative survey and qualitative interview case study. The survey portion of the study will enable the primary investigator (PI) to measure Tinto’s five areas for improvement of student persistence: expectations, support, feedback, involvement, and learning. The interview portion of the study will enable the PI to gain a more in-depth understanding of the experiences and issues that affect individual student persistence. The resulting data collected will be analyzed to determine if there are significant differences between at-risk and not-at-risk student groups in terms of the affect of academic support, student employment, and extra curricular activities on their decision to complete their chosen degree.

The dependent variable will be the student’s persistence in continuing enrollment towards completion of a degree in their declared academic major. The independent variables will include student utilization of advising services and academic support, student employment, and participation in extra curricular activities or groups.

Sampling

Initiation of sampling will commence upon completion of review and approval of the proposed study by the Human Subjects Institutional Review Board (HSIRB). The PI will load name and address data obtained from the student information into a spreadsheet and generate a random sample to select potential participants. A unique numeric identifier will be assigned to each participant in order to facilitate confidentiality. The PI will issue invitations to study participants. Two major groups, at-risk and not-at-risk, will be utilized for the survey portion of this study. Each major group will have four age-based sub-groups with ages ranging from: under 18, 18-24, 25-39, and 40. These age ranges are comparable to those utilized in Knapp et al.’s 2006 survey where participant age ranges were reported as being: 1.74% under 18, 61.54% 18-24, 26.7% 25-39, 9.37% 40+, and 0.64% unknown (Knapp et al., 2007).

The PI will continue to utilize a random sample for selection of participants until a sample of 400 newly admitted undergraduate students from the Fall 2007 semester is obtained. The goal is to have 200 students in each of the major groups and at least 20 students in each of the sub-groups.

Eight students from the 400 survey participants will be randomly selected for a case study interview. A case study will be taken for one student from each of the eight age-based sub-groups.

Data Collection

Online surveys will be utilized for collection of data from the 400 survey participants. Data from the surveys will be logged to a comma-delimited format suitable for analysis with a statistical software package. The sample will be surveyed once every four weeks during the Fall 2007 and Spring 2008 semesters. Surveys will commence the first week of classes. Participants will be asked to complete a total of eight surveys during the two semesters. Each student will be issued a unique survey id and password to enable the PI to evaluate changes in student activities and perceptions during the course of the semester. The students in the sample will be provided incentive for participation in the survey. After each survey a drawing will be held for a $250 gift certificate to the campus bookstore. Students will be notified by e-mail at their University e-mail address when new surveys are accessible. Reminders will be delivered to students who do not complete a survey within 10 days of the survey being released.

The eight student interviews will be conducted through an online open-ended questionnaire. Students will be offered a $150 gift certificate to the campus bookstore as incentive for their participation in the interview.

Additional data will be collected from the student formation system. The PI will include data regarding total enrollment credits, GPA, ACT scores, declared major.

Description of Instruments Used in the Study

The online surveys will consist of questions designed to measure each students’ utilization of academic advising, if a major has been declared or changed, if the student is taking classes that count towards their major, if and where the student is employed, and if the student is participating in any extra curricular groups or activities. In addition the students will be asked to rate their satisfaction in the quality of advising received, their chosen major, the quality of the courses they are enrolled in, overall satisfaction in their current employment, and satisfaction in any extra curricular groups or activities they participate in. A five point Likert scale will be utilized for survey questions that measure student satisfaction. A sample of the proposed survey is attached to this report in Appendix A.

The student interviews will primarily feature open-ended questions. It is the intention of the PI to make students feel as comfortable as possible during the online in order to gain the most incite into the opinions and experiences of each individual student. A sample interview schedule is attached to this report and is presented in Appendix B.

Data Analysis Procedures

The PI will utilize a data analysis team comprised of institutional staff members with expertise in statistical analysis. Survey responses will be analyzed using a statistical software package. T-tests will be utilized to determine if there is any difference in the responses provided by the two groups. A summative report will be presented of response totals for each question/category. The data analysis team will determine the effect of each independent variable on the dependent variable using partial and multiple correlation. In addition, the data analysis team will utilize multivariate regression to determine if the existence of one or more independent variables causes a change to the dependent variable. Finally, a path analysis will be conducted to determine if a series of occurrences of independent variables influence the dependent variable.

Biographical Statement of the Primary Investigator

Michael VanPutten is the Web Developer for Enrollment Management at Western Michigan University in Kalamazoo, Michigan. Michael serves as president of the campus web users group where he provides leadership and assists to facilitate collaboration and development of web communication policies and standards with the campus community. Michael is also the president of VanPutten Interactive, a company through which he has endeavored to provide creative consulting, design and development solutions to Kalamazoo, Michigan and the greater West Michigan region. VanPutten Interactive has assisted individuals, commercial and non-profit clients to establish professional online identities, build online communities, reduce costs of operation, deliver services online, and collect payment for services via the Web.

Michael received his BA in Philosophy from Western Michigan University in 2000 and is completing his master’s degree in Educational Technology. Michael has over ten years of experience in web and multimedia development, which he has utilized to develop cutting edge instructional multimedia for higher education. Michael has led professional development workshops demonstrating best practices in implementation of projects using digital photography, digital audio/video applications, WebCT, Macromedia Dreamweaver and Flash, Adobe Photoshop, and Microsoft PowerPoint.

Timetable for completion of the study

Implementation of Online Survey Tools: Jul – Aug 2007
Conduct of Online Surveys: Sep 2007 – Apr 2008
Send Survey Reminders: Sep 2007 – Apr 2008
Analysis of Online Survey Data: Sep 2007 – Apr 2008
Conduct Interviews: Apr 2008
Code Interviews: Apr – May 2008
Analysis of Interview Data: Apr – May 2008
Write Report: May – Jun 2008

Budget

Personnel Costs
Principal investigator $10,400.00
(10 hours x 52 weeks x $20/hr)

Data analysis team $6,000.00
(10 hours x 36 weeks x $20/hr)

Data Collection & Analysis Costs
Vovici EFM Feedback Professional – 1 year license $2,495.00
(This tool provides an environment for development, implementation, and administration of online surveys)

QSR Nvivo7 $495.00
(Qualitative interview coding/analysis software)

Participant Incentives
Survey incentives $2,000.00
(8 x $250)

Questionnaire incentives $1,200.00
(8 X $150)

Total $22,590.00

Appendix A – Survey Questions

1. Have you received academic advising during the past four weeks?
( ) yes ( ) no
a. If yes, who did you received academic advising from:
( ) college advising ( ) faculty advisor ( ) department staff ( ) other: ____
b. If yes, how did you receiving advising:
( ) in person ( ) over the phone ( ) by e-mail ( ) other: _________
c. If yes, how would rate rate the overall value of your academic advising?
( ) very un-helpful ( ) not helpful ( ) neutral ( ) helpful ( ) very helpful
d. Please rate your level of satisfaction with the academic advising you received.
( ) very dissatisfied ( ) dissatisfied ( ) neutral ( ) satisfied ( ) very satisfied
2. Have you declared your academic major?
( ) yes ( ) no
a. If no, do you plan to declare an academic major in the next four weeks?
( ) yes ( ) no
b. Please indicate your current major: ______________
c. How did you select your current major? _______________
d. Have you changed your academic major in the past four weeks?
( ) yes ( ) no
e. Please rate your level of satisfaction with your current major:
( ) very dissatisfied ( ) dissatisfied ( ) neutral ( ) satisfied ( ) very satisfied
3. Were you employed during the last four weeks?
( ) yes ( ) no
a. If no, do you plan to seek employment?
( ) yes ( ) no
b. At how many places were you employed during the last four weeks?
c. How many hours did you work at your place(s) of employment during the last four weeks?
d. How would you rate your level of satisfaction with your current employment:
( ) very dissatisfied ( ) dissatisfied ( ) neutral ( ) satisfied ( ) very satisfied
4. Do you participate in any extra curricular or group activities?
(checkboxes of common available activities will be listed) Other: __________
5. How many hours do you usually spend in class?
6. How many hours do you usually spend studying?
7. How many hours do you spend commuting to attend your classes?
8. Which of the following support services did you utilize in the past four weeks?
( ) writing center ( ) library ( ) librarians ( ) career and student employment
( ) academic skills center ( ) intellectual skills development program ( ) trio student success program ( ) alpha mentoring program ( ) university curriculum
a. Please rate your level of satisfaction with the support services you utilized
( ) very dissatisfied ( ) dissatisfied ( ) neutral ( ) satisfied ( ) very satisfied
9. Please rate your level of satisfaction with Western Michigan University
( ) very dissatisfied ( ) dissatisfied ( ) neutral ( ) satisfied ( ) very satisfied

Appendix: B – Interview Schedule

The purpose of this online questionnaire is to enable Western Michigan University to gain understanding about your individual experience during your first two semesters. Please answer each question as completely as possible:
1. Why did you choose to pursue a degree at Western Michigan University (WMU)?
2. Which other universities did you consider attending prior to selecting WMU?
3. How many of your high school classmates selected WMU?
4. Please describe the WMU campus community.
5. What has your experience been while attending classes?
6. What did you like most about your first two semesters at WMU?
7. What did you like the least about your first two semesters at WMU?
8. What do you hope to gain by having a college degree?
9. What were the

most challenging aspects of being a student during the past two semesters?
10. Please tell us about a favorite course you took at WMU.
11. How would you describe your teachers at WMU?
12. How would you describe the staff at WMU?
13. How did you spend your time during the past two semesters?
14. How did you prioritize and allocate time effectively?
15. How do you think you have changed over the past two semesters?
16. Please describe your favorite activities of the past two semesters.
17. What are you looking forward to?
18. What do you plan to do after graduation?

References

Berkner, L., He, S., Cataldi, E.F., & Knepper, P. (2002). Descriptive summary of 1995-96 beginning postsecondary students: Six years later. Retrieved May 11, 2007 from http://nces.ed.gov/das/epubs/2003151/postsec4c.asp

Bradburn, E.M., & Carroll, C.D. (2002) Short-term enrollment in postsecondary education: Institutional differences in reasons for early departure, 1996-98. Retrieved June 12, 2007 from http://nces.ed.gov/das/epubs/2003153/

Caison, A.L. (2007). Analysis of institutionally specific retention research: A Comparison between survey and institutional database methods. Research in Higher Education, 48 (4). 435-451.

Cartney, P., & Rouse, A. (2006). The emotional impact of learning in small groups: Highlighting the impact on student progression and retention. Teaching in Higher Education, 11(1). 79-91.

Knapp, L.G., Kelly-Reid, J.E., & Whitmore, R.W. (2007). Enrollment in postsecondary institutions, fall 2005: Graduation Rates, 1999 and 2002 Cohorts; and financial statistics, Fiscal year 2005. Retrieved May 11, 2007 from http://nces.ed.gov/pubs2007/2007154.pdf

Moller-Wong, C., Shelly, M.C., & Ebbers, L.H. (1999). Policy goals for educational administration and undergraduate retention: Toward a cohort model for policy and planning. Policy Studies Review, 16 (3/4), 243-277.

Perna, L.W. (2000). Differences in the decision to attend college among african americans, hispanics, and whites. The Journal of Higher Education. 71(2), 117-141.

Phillips, G.W. (2000). The release of the condition of education 2000. Retrieved June 16, 2007 from http://nces.ed.gov/whatsnew/commissioner/remarks2000/6_01_2000.asp

Rooney, P., Hussar, W., Planty, M., Choy, S., Hampden-Thomson, G., Provasnik, S., & Fox, M.A. (2006). The condition of education. Retrieved June 16, 2007 from http://nces.ed.gov/pubs2006/2006071.pdf

Tinto, V. (2003). International student retention conference: Promoting student retention through classroom practice. Enhancing student retention: Using international research to improve policy and practice. November 5-7, 2003. Staffordshire University, Amsterdam. Retrieved June 16, 2007 from http:// www.staffs.ac.uk/institutes/access/docs/Amster-paperVT(1).pdf

Tinto, V. (1987). Leaving college: Rethinking the causes and cures of student attrition. Chicago: University of Chicago Press.

Zea, M. C., Reisen, C. A., Beil, C. , & Caplan, R. D. (1997). Predicting intention to remain in college among ethnic minority and nonminority students. The Journal of Social Psychology, 137, 149-160.

Leave a Reply

Your email address will not be published. Required fields are marked *

thirteen − twelve =