5. Assessment and Research

Graduates should be able to demonstrate their understanding of and ability to employ good assessment and research practices. In meeting this competency, students should demonstrate their experience with/ability to. . .

• Design and implement thorough assessment and research efforts including the identification of significant key issues and questions;

Notes: Assessment course – New Zealand study abroad assessment project. Other assessment? Survey of Graduating Seniors – How is the data assessed in order to identify areas for improvement at the departmental level?

Our educational programs must be assessed in order to measure many fundamental elements, most notably, the effectiveness of our efforts to support and assist students. Conducting research by means of analyzing data, conducting surveys, or interviewing students provides us with critical feedback we can use to improve our programs. In the Academic Programs Office, we engage in recruitment and retention activities for the College of Agricultural Sciences. One of the main challenges that we face is measuring the effectiveness of our efforts. I have learned that many forms of data need to be acquired in order to assess those measurements and turn the results into an appropriate course of action. We need to track the matriculation rates of students contacted during recruiting events; where students are coming to us from, both regionally, and based upon their academic interests in high school; the movement of students into and out of our college, and the reason for each change of major; the satisfaction, or lack thereof, of students with their academic advising and courses within their major; and placement of graduated students in the various Agriculture-related professions or progression toward graduate studies. However, this list only provides a few examples of the many factors we must consider in order to maximize our potential.

As administrator for the College of Agricultural Sciences study abroad program with Lincoln University in New Zealand for the last three years, I have seen a dramatic decrease in the number of participating students. I wanted to find out the reasons for this decline. The documents below describe the process that I engaged in to gather and analyze extant data and survey results.

Document: CAS Study Abroad Research Project

So what did I learn from this process? I learned that while it may be simple to determine a problem, it is very challenging to gather the right information and look at it in the right way to find a solution. I thought that if I just looked over some data on historical enrollment in the study abroad program and asked some questions regarding the intent of past participants a pattern of some sort would surface that I would be able to draw conclusions from. I thought wrong. Although it was interesting to examine the data that I acquired, it was not so easy to determine reasons for the decline in participation. However, as a first attempt at a full-fledged assessment project, the process taught me the importance of determining a very definable issue and asking the right questions in order to obtain relevant answers.

I spent one summer working with the International Programs Office and our department of Animal Sciences (ANS) to develop an international opportunities flier for ANS students. My main task over the summer was to research the catalogs of institutions overseas to find the best Animal Sciences programs available to our students who may wish to study abroad. However, what I really learned from this project, was the concept of curriculum integration. The research I was conducting, as well as the relationships being built between our college department and International Programs, was all part of this greater curriculum integration concept. The idea is that by having members of academic departments develop relationships with international programs staff where those two entities work together to examine course offerings at overseas institutions, the department is able to claim a higher level of ownership of the study abroad offerings while international programs is introduced to the needs of the students in that department through the greater process. The resulting international opportunities flier was, therefore, only the final product of the process of curriculum integration. Following is a quick look at the notes I kept while conducting my research:

Document: Curriculum Integration Project Research

And here is the final document that was created by International Programs staff after all of the content was compiled:

Document: ANS International Opportunities Flier

• Create instruments and/or protocols for assessing and researching these important issues and questions; and

Notes: What assessment tools have I used to gather and present data? Extant, surveys, interviews. Since we don’t learn to analyze data sets in this program, how can I address this section of the competency?

Creating the survey for the study abroad assessment project was actually really fun. But, entertainment value aside, it is much more difficult than I thought to design a set of inquiries that will yield useful results. Gathering the extant data from the datawarehouse and from International Programs was fairly straightforward. However, analyzing and attempting to draw conclusions from the data just created more questions rather than offering any answers. It would be very helpful to take some introductory statistics courses in order to learn techniques for effectively analyzing data sets.

Here is a handout that helped me to understand the process of analyzing extant data:

After gathering all of the data, I created the following spreadsheet by compiling all of the information therein. As stated before, I learned that looking over the extant data does not necessarily provide any obvious results. I was unable to draw any substantial conclusions from the data alone. However, this was all part of the learning process for me. I needed to find out on my own the limitations of relying too much on statistical data.

Document: Compiled Data for Analysis

After I became aware of the need for additional information in order to have something to actually assess, I decided to create a survey that would be directed to students who had studied abroad in the past. The first thing I learned was that it was necessary to have some of my peers take the survey as a test drive before going live with it. I was able to make many important changes after gathering the feedback from my test group. Next, I learned that even though I think I am asking a question that will give me exactly the answers I am looking for, the reality is that people are complex, and their answers are reflections of this complexity. Finally, the most important thing I learned conducting the survey is that I need to have a plan for how I will collect and analyze the survey responses even before I have written the questions. Simply sitting down and writing questions based upon the answers that you hope to receive is not the way to go. Following is a link to the actual survey as well as the compiled responses.

Link: Study Abroad Survey
Document: Survey Responses

• Credibly convey key findings and recommendations to stakeholders and constituents.

Notes: Presentation of New Zealand project findings. Dissemination of survey of graduating seniors data to department heads.

Once the research has been conducted, data has been analyzed, and findings have been deduced, it is time to present the information to an audience. For my final presentation in the assessment course, I chose to present by first identifying the problem, then describing participants, rationale behind the analysis, methods used to gather data, form of analysis, limitations, major findings, and recommendations. I learned that understanding the audience should inform the type of presentation to be created. Also, the findings and recommendations presented should be relevant to the audience. So, for some assessment projects, it may be necessary to create multiple presentations designed to be most informative to each different audience it will be presented to. Another key discovery for me was that although I may understand the results of my project, conveying that understanding to an audience is a whole different challenge. I found that at some points my listeners understood my findings, but couldn’t understand how I came to such a conclusion; while other times, my process of analyzing data made sense, but my results did not match expectations. This is an area that I will have to work on throughout my career in order to effectively communicate the results of future assessment projects.

Document: CAS Study Abroad Research Project Presentation