EPAC December 2008 online chat
Evaluating the "Foundations and Skills for Lifelong Learning" Metarubric Draft from AAC&U's VALUE Project
Friday, December 5 at 10 a.m. PT/12 p.m. CT/1 p.m. ET
On Friday, December 5th, 14 members of the EPAC community joined to evaluate the "Foundations and Skills for Lifelong Learning" Metarubric developed by a team of faculty from around the country as part of the Association of American Colleges & Universities' (AAC&U) VALUE (Valid Assessment of Learning in Undergraduate Education) project. The VALUE project is seeking feedback on their draft rubrics and we thought it would be interesting to center this month's discussion on evaluating the "Foundations and Skills for Lifelong Learning" Metarubric draft. We were fortunate enough to have Terry Rhodes, AAC&U Vice President, Office of Quality, Curriculum and Assessment, and VALUE Project lead join us for the conversation.
GUIDING QUESTIONS FOR THE METARUBRIC EVALUATION
1. Please comment on the criteria of the rubric:
If you get stuck on what to comment on, you could consider these questions:
- Are there any criteria that are missing?
- Given your understanding of this subject, are the criteria listed in this rubric appropriate?
- Were the criteria understandable?
2. Please comment on the levels of the rubric:
If you get stuck on what to comment on, you could consider these questions:
- Did the levels (1, 2, 3, 4) of the rubric cover the range of quality of the student work you reviewed?
- Did you review work that was far beyond 4 or far below 1?
- Were there any levels of typical student competency that were missed by the rubric—for instance, would you have prefer one level between, say, levels three and four?
- Do the four performance levels provide adequately for the type of eportfolios you use on your campus? Does Level 1 reflect expectations for freshmen performance? Does Level 4 reflect expectations for seniors?
3. Please comment on the performance descriptors of the rubric:
If you get stuck on what to comment on, you could consider these questions:
- Is the language of the performance descriptors understandable and useful?
- Do the performance descriptors for each level reflect consistent, progressively more complex and challenging performance expectations?
- From this experience in scoring, can you suggest better language for any of the performance descriptors?
RESOURCES AND ISSUES MENTIONED DURING THE DISCUSSION
CHAT PARTICIPANTS
- Trent Batson, Web2ePortfolio Initiative
- Gary Brown, VALUE Initiative (Critical Thinking) and Web2ePortfolio Initiative
- Ruth Cox, Academic Technology Division, San Francisco State University
- Ginny Curley, Coordinator of college-wide portfolio, Nebraska Methodist College
- Betsy DeGeorge, University of Tennessee College of Social Work, Office of Research and Public Service
- Kate Drezek, Office of Academic Assessment, Virginia Tech
- Glenn Johnson, Penn State
- Judy Patton, Portland State University and VALUE initiative (Integrative Learning)
- Catherine Paul, University of British Columbia's Office of Learning Technology - Coordinator of the ePortfolio community of practice
- Terry Rhodes, AAC&U and VALUE Initiative
- Teggin Summers, ePortfolio Initiatives office, Virginia Tech
- Marc Zaldivar, ePortfolio Initiatives office, Virginia Tech
- John Ittelson, EPAC co-facilitator
- Helen Chen, EPAC co-facilitator and Research Scientist, Stanford Center for Innovations in Learning
CHAT TRANSCRIPT
---------------------
Ginny: Hi, This is Ginny Curley at Nebraska Methodist College in Omaha, Nebraska. I coordinate our college-wide portfolio. We have just begun using the portfolios as an outcomes assessment measure and, therefore, this rubric comes just in time!
---------------------
Virginia Tech: Hi all. This is Marc Zaldivar, Teggin Summers, and Kate Drezek from Virginia Tech. Marc and Teggin are in the ePortfolio Intiiatives office; Kate is in the Office of Academic Assessment.
---------------------
Betsy: Betsy DeGeorge: Publications Manager, University of Tennessee College of Sociqal Work, Office of Research and Public Service
---------------------
Glenn: Hello Everyone! Glenn Johnson here from Penn State. Program evaluation/assessment has been a hot topic here of late.
---------------------
Trent: Hi, Trent Batson from the Web2ePortfolio Initiative
---------------------
ruth: Hi, Ruth Cox here from SF State, Academic Technology Div.--our campus is involved in the VALUE project, with a Liberal Studies undergrad. pilot (160 students using ePortfolios). We are getting lots of requests for Program Evaluation assistance. Good to "see" familiar faces here!
---------------------
Moderator (Helen): Welcome Betsy, Glenn, Virginia Tech Team! I'd like to also mention that Marc will be presenting on "Institutionalizing the E-Portfolio: Addressing Assessment, Pedagogy, and Professional Development Issues for Widespread Adoption (Learning Technology)" at the January ELI meeting. Glenn's campus Penn State will also be presenting on "Social Blogging Platforms as E-Portfolios." Great to have you here!
---------------------
Gary: Gary Brown, also with Value Initiative (Critical Thinking) and Web2ePort
---------------------
Judy: Hi, Judy Patton here from Portland State U. I am on the integrative learning rubric group for the Values project.
---------------------
catherine: Hi! I'm Catherine Paul from UBC's Office of Learning Technology - Coordinator of the ePortfolio community of practice here
---------------------
Terry: Hi. Terry Rhodes from AAC&U - overseeing whole VALUE project. Welcoming feedback and comments.
---------------------
Trent: Could you say more about the rubrics you're developing, Terry?
---------------------
Glenn: Thanx for the mention of the Penn State presentation at ELI. If you are there I encourage you to hear Cole, Brad and Carla talk - GREAT stuff!
---------------------
Terry: We have a total of fourteen rubric teams. Our goal is to shift conversation away from reliance on snap shot scores as proxies for learning toward richer, more nuanced and complex statements of learning expectations on a broad range of learning outcomes, i.e. rubrics that can be used nationally to talk about what is shared by faculty across the country.
---------------------
Trent: that's really good,
---------------------
Trent: these are rubric structures?
---------------------
Terry: Trent, I should invite Judy and Gary to talk since they are on rubric development teams.
---------------------
Judy: Yes, they are rubric structures that span a range of learning outcomes. Those identified in the Liberal Learning doc from AACU.
---------------------
Betsy: Could anyone state the initial goals behind the AAC&U's Rubric?
---------------------
Judy: The work is actually in a second phase. Institutions across the country used the first draft rubrics to score student work last spring. The rubrics are now being revised using the feedback from those activities.
---------------------
Gary: Certainly our team's discussions are "nuanced, complex" and sometimes downright difficult. Central is Trent's concern, expressed elsewhere, that we have to balance assessment with instruction. We know too well that testing is not, by itself, teaching.
---------------------
ruth: Could someone speak to the choice of the over-arching qualities of: "Curiosity, Motivation, Independence, Transfer & Reflection" as core elements of Life-long learning? They resonate with my "Maslow-ian" background on the qualities of self-actualizing people.
---------------------
Moderator (Helen): As Terry mentioned this rubric is one of many being developed by teams of faculty from around the country to help institutions might assess student ePortfolio work based on these learning outcomes. Since these rubrics are still works in progress, AAC&U is actively seeking feedback. To guide our conversation, here are the metarubric evaluation questions.
---------------------
Moderator (Helen): Great question to get us started Ruth since those elements on the lefthand side of the rubric are the criteria.
---------------------
Terry: All, the goal was to see if we could extract shared or common criteria or elements that had been developed or identified by faculty and others on campuses around the country. If we could demonstrate a level of shared criteria for outcomes, then we could talk about a shared conversation and general expectations of what learning looked like when we saw it from students regardless of what type of institution someone was attending.
---------------------
Judy: Terry, do you know anything about the conversations in the LLL group? I found the rubric to be quite lovely in their simplicity. However, I wonder how this particular rubric will be used. I can see graduates taking it to use in their life postgraduation or giving it to entering students to let them self evaluate and set goals...
---------------------
Trent: This is great work, Terry -- very much needed as a leg up on different campuses
---------------------
Betsy: What were the main developments that took place between phase 1 and phase 2?
---------------------
Trent: Judy: yes, students setting goals, planning their study -- excellent
---------------------
Ginny: In response to Judy, I can see using this as a way to introduce the goals of a liberal education to entering students (especially in our healthcare-focused institution) and then as the measurement tool for their portfolios when they graduate. If faculty use the same language throughout their education, they will hear the message more clearly.
---------------------
Terry: Judy, you pick up on one of the key hopes we have for rubrics, i.e. that students can use these to develop their own abilities to reflect on and evaluate their own learning and to be able to see what their own view looks like compared to what faculty might think.
---------------------
Judy: Betsy, in my experience, a rubric was developed, several institutions beta-tested it and sent feedback using the guiding questions back. Groups just completed revisions of the rubrics or are completing them now for a second go. I don't know. Terry?
---------------------
Trent: As Steve Ehrmann said in his post from Australia, campus reps ask what technology to choose and think that's the first step; if this work through AAC&U can help people see the rubric-definitions as the first step instead, the work will address this other big problem
---------------------
Moderator (Helen): Ginny, you raise an important point - the audience for these rubrics. For now, it looks like AAC&U is interested in getting feedback from the perspective of faculty/instructors/evaluators of student ePortfolio work and are these metarubrics practical and useful. Perhaps later on student feedback could be solicited.
---------------------
Ginny: Terry's comment sparked a question we discuss often: how to engage students in the metacognitive process
---------------------
Terry: Betsy, the difference between phase 1 and phase 2 is simply having had campuses use the rubrics with student portfolios or sample work and then revising the rubrics. We will do another revision in the spring (phase three)after campuses use the rubrics after fall term with their students' work and then "publish" drafts early summer so everyone can have access to them to use and continue to develop.
---------------------
Judy: Trent, oh, I so, so agree. Institutions need to begin with what they want to find out about their student learning and find or create a system that allows them to do that.
---------------------
Trent: Terry: did you say that 14 campuses are now part of VALUE?
---------------------
Ginny: Question: Do most institutions share their rubrics with students prior to the grading process?
---------------------
ruth: I agree with Ginny that reinforcing the ideas throughout (using the same language) is important. Also feel that setting some context for students would be important. "What are the qualities of a lifelong learner in Higher Ed., The Workplace, Adult/Senior Life?
---------------------
Glenn: I agree, communicating high expectations in a consistent fashion is important for avoiding a 'mystery learning' approach. Being clear and intentional is an important first step, but embedding these into the curriculum ... what can we do to make sure that this next step is ensured? It has to be more than the technology right?
---------------------
Terry: All, we have a couple campuses that are involving students in looking at and using the rubrics to begin to see if they make sense to them.
---------------------
Trent: A huge challenge is deciding what evidence, how broad a range of evidence, can be used to meet the rubrics.
---------------------
Betsy: The elements look good to me. As I look at the rubric, though, the feature that I see as missing has something to do with the ability to sort and exclude. In a world of extreme information overload, don't you think that a student's ability to decide what not to pursue is almost as valuable as their ability to decide what to pursue?
---------------------
Trent: good point, Betsy
---------------------
Ginny: To Terry's comment: Great idea to share with students in terms of do they understand them? I love the language and believe it captures our goals...but I often think students are stumped and I need to work to adjust my language for different audiences.
---------------------
Gary: Ruth's point suggests the importance of bringing in friendly employers and community leaders into the review process....
---------------------
ruth: Besty--great point. Knowing how to "Sort" effectively is going to be a big life-long skill we all will need.
---------------------
Moderator (Helen): Interesting point, Betsy. There's so much talk about 21st century communication skills for this "digital" generation/net gen, etc. Is this a unique lifelong learning skill necessary for our current generation of students?
---------------------
Trent: If these rubrics result in just another way of grading, of course, shucks, not so good
---------------------
Betsy: I think that the digital natives do have some sorting skills available to them that I don't think I have.
---------------------
Trent: I think that after these rubrics become understood and part of the national conversation, the next step is addressing how to broaden the definition of student work and how they can bring evidence into the rubrics
---------------------
Ginny: The rubric speaks to the challenge of today ---to teach students to engage at a deep level with content. I don't know if it is "new" but I do believe there are so many avenues for information today that students need to think before they gather...
---------------------
Trent: students create the content
---------------------
Terry: We do not see these as grading rubrics. Of course, they can be deconstructed or translated into more specific rubrics for a particular program or set of assignments - more granular. The goals would still be that the core or shared criteria are also reflected in the granular versions - Washington State has good examples of this for critical thinking.
---------------------
Ginny: Terry - can you explain more about them not being grading rubrics? Are you saying they are to be used for outcomes assessment, but not for student evaluation, per se?
---------------------
Betsy: Trent, I don't think I agree with you on that, philosophically. I'm not that constructivist.
---------------------
Terry: Trent, We have been encouraging rubric teams to think in multimedia terms as they define criteria and performance descriptors to encompass many modes of demonstrating learning both through the curriculum and the co-curriculum.
---------------------
Trent: no problem, Betsy
---------------------
Trent: I like co-curriculum
---------------------
ruth: Following up on Trent's comment--in many disciplines it makes a lot of sense to pull employers into the discussion of "what evidence" they feel is important to their discipline. In many depts. this is pretty clear & in fact has helped us to guide the "ingredients" we include in student portfolios. When we are talking about qualities of Life long learning, workforce development comes up for me.The element of "teamwork" or being able to communicate within a group seems important to lifelong learning.
---------------------
Gary: One challenge is to understand grading and assessment are or ought to be synonymous with feedback
---------------------
Betsy: I can groove with co-curriculum.
---------------------
Ginny: I agree with allowing students to create their ePortfolios. Don't we learn about their skills from what they select in addition to how they discuss it? As I encourage students to draw from co-curricular and life experience, current events, etc. I can see if they are applying their knowledge and skills beyond constructed assignments.
---------------------
Moderator (Helen): Building on Gary's comment about assessment and feedback, taking a closer look at the levels and thinking about the student work on your own campus, do you see a mapping with the performance levels 1 to 4? Are 4 levels enough, too many?
---------------------
Terry: Ginny, we are approaching the rubrics from the standpoint of the most accomplished levels as approximating what we would hope for students as they leave a four year institution. The other end represents entering or beginning levels. So, clearly, they can be used for grading a single assignment or set of assignments, but they are general and therefore most useful as written for more general assessment of student learning.
---------------------
Judy: Ginny, I would go a step further and say if students aren't allowed to create their own eports, they become a rather bureaucratic assessment formula.
---------------------
Gary: John's point is also key--defining rubric levels in ways that include quality and not just quantity and that illuminate quality distinctions.
---------------------
Ginny: Judy - I agree completely. And when this happens, the learning is often lost.
---------------------
Glenn: If we rubrics to score we are using them to articulate differences, right? Used as an instructional strategy, i.e., the design and development of these are a learning opportunity in itself. In a way it is a shame that these would be a 'given'.
---------------------
Trent: I'm thinking of experiential learning, internships, service learning, semester abroad, field work, gap year, etc etc -- the people running these programs have good experience in how to gather evidence of learning from out-of-the-classroom work.
---------------------
Betsy: I am wondering how, this rubric for assessment is different from the kind of rubric you might set up for a paper portfolio?
---------------------
Gary: To Trent's point, we've found internship supervisors are often more receptive to rubric based feedback than many of the faculty we've worked with.
---------------------
Terry: Trent, and we have teams working on rubrics related specifically to civic engagement, etc. The obvious issue is that none of these outcomes is neat and tidy; there is overlap, etc. - we're trying to suggest, though, that if we think something is important for our students to learn, we should articulate what that learning looks like.
---------------------
ruth: Thanks for clarifying, Terry. One quality that I am uncomfortable with is "Independence" as it's written the rubric...What we hope for as students leave the institution is in many ways the opposite to "presents insights without guidance from others". With a valuing of teamwork, cooperation, listening-skills etc as social/community elements that will be essential to life long engagement.
---------------------
Glenn: What is more valuable to support in the long run, developing the capacity to apply 'given' rubrics, or developing the capacity to create quality rubrics?
---------------------
Ginny: Terry - Agreed. I am putting this in our context of having students develop their portfolios over four years and then grading them at the end...to encourage engagement. The grades must be holistic to reflect development over the course of those years. Certainly, the early entries would not be expected to be at a "Level 4"---but by the end we'd hope to see that (or at least a 3).
---------------------
Trent: Yeah, Terry, that is the key to the issue: learning is messy, not neat.
---------------------
Moderator (Helen): would it be different from a paper portfolio? The focus is on the learning outcome -- foundations and skills for lifelong learning.
---------------------
Betsy: Yes, I think it would be different
---------------------
Gary: Glenn, great question not to be glossed over but pondered...
---------------------
Betsy: I think that the technological skills are new and should therefore be represented within the rubric
---------------------
Terry: We see the rubrics as useful and applicable for any type of portfolio or collection of student work - we do strongly believe that student reflection on their learning should be a part of any form and that the rubrics, I think, assume student reflection as part of the evidence.
---------------------
Trent: In the field of writing, we use a writing rubric to "train" teachers how to grade papers so we have a reasonable norm (rater-reliability). But, it's in the training where evidence can be defined and broadened so we don't just have a scoring system.
---------------------
ruth: Yes, very different--esp. with hyperlinking to co-curriculur, multimedia, community based work.
---------------------
Ginny: Betsy- I don't see a difference between paper assessment and ePortfolio assessment as it this rubric is written.
---------------------
Betsy: If it is not different, then don't we have an old rubric we can use?
---------------------
Virginia Tech: Ruth, I agree with you: how would you assess this quality of independence on a portfolio anyway? Is the "4" level, really the polar other end of the "1" level?
---------------------
Trent: Like, in writing, do we include email as writing, or chat, or writing for the Web, or Facebook "what are you doing now" responses
---------------------
Gary: V Tech's question may pertain to the whole rubric--attributes of the individual and his/her character are hard to discern from the evidence presented in the ePortfolio
---------------------
Betsy: I think that we are in a new world, with new skills. ePortfolios serve those skills in ways that paper portfolios do not.
---------------------
Virginia Tech: Gary -- right: and they should not be confused either. The rubric should be for the artifact(s) and their reflection on learning, not the individual character.
---------------------
Ginny: Betsy - good question! I suppose my use of the rubric is to assess the cognitive skills and values...not so much the communication of those in terms of method. Because students will be capturing learning in a variety of ways, it is not so important (to me) to capture if that was communicated via paper or electronic forms,
---------------------
Betsy: Trent, exactly, Email is writing. Chats are writing. Social networking is new and can serve education in new ways. I don't think we should miss this!
---------------------
ruth: I think you need to take one "signature", comprehensive project where students are actually exploring their learning styles (like the "Intellectual Autobiography" paper our Lib. Studies write). Take a range of students (with same assign.), and try to apply the rubric.
---------------------
Betsy: The experience we are having here in this session is incredibly different from any kind of education that could be experienced a few years ago. I don't think we could apply a 1960's rubric to this.
---------------------
Trent: I think we have a number of layers of judgement and reflection between student work, work that is "handed in," faculty interaction with the work and the student, and then the final layer of entering a value in a cell in the assessment management system. Those are both technological and human layers. Technology allows more layers and tracking, so the more we define parts of these layers, the more free we are in others.
---------------------
Judy: Part of what we see if eports is that it gives students a way to use the developing digital communication strategies so we can keep somewhat up with how learning is happening for the current cohort. We found that digital portfolios allowed students to connect their learning in ways not available to paper portfolios and also the very design is a critical/creative thinking activity.
---------------------
Gary: OK Betsy, let's start a critical engagement rubric
---------------------
ruth: I like this overview of 21st Century Skills. http://www.21stcenturyskills.org/index.php?Itemid=120&id=254&option=com_content&task=view
---------------------
Glenn: Learning is messy, absolutely. But I think this project is trying to answer how we best make sense of it? Applying a national rubric/lens, or analyzing this using local rubrics/lens and then finding ways to make these local rubric criteria interoperable across rubric instances? Which approach provides a better return on investment to learning for us?
---------------------
Moderator (Helen): Speaking of writing taking new forms other than just papers, etc., check out the longitudinal Stanford Study of Writing: http://ssw.stanford.edu/
---------------------
Terry: Ruth, various campuses do rely upon capstones of one sort or another as the primary place to obtain artifacts for rubric application
---------------------
Betsy: The ISTE National Ed Tech Standards struggle with this same question, I think. I like that overview Ruth.
---------------------
Betsy: Gary, that sounds good.
---------------------
Trent: And, Betsy and Gary, we'll expect a report at the next chat
---------------------
Moderator (Helen): Would anyone like to comment on the performance descriptors? Could you see yourself (and your faculty) understanding and using this rubric to evaluate student work?
---------------------
Ginny: We used some similar categories (not nearly as well-written, however) just this semester to examine portfolios. I appreciate the holistic approach the descriptors take.
---------------------
Trent: I'm not prepared, teacher
---------------------
Terry: All, I have to tell you that it is definitely rewarding to hear from people who understand rubrics, their purpose and relationship to student learning...
---------------------
Gary: Does the progression from rarely to always provide adequate handles for promoting/assessing quality?
---------------------
Ginny: The earlier comment about Independence is worthy to raise again. I am thinking about how I would have assessed that based on a submitted portfolio
---------------------
Terry: Ginny, good point. We will make sure all of this feedback gets to our rubric team members for next round of revisions.
---------------------
Betsy: Yes, Ginny's point was a good one. . . the necessary balance between dependence and independence.
---------------------
ruth: Rarely, Sometimes, Usually, Actively/consistently work for me, but descriptors could be simplified further.
---------------------
Ginny: All - I am sorry to have to leave this discussion - It is time to administer a course evaluation for a colleague! I am eager to see the final product and look forward to future discussions!
---------------------
Judy: I agree with Betsy. Lifelong learning now includes a "works well with others" section.
---------------------
Virginia Tech: Or, not only independence, but also curiosity and motivation. These personality characteristics would have to be "read through" whatever evidence is being presented in the portfolio. Compare that to "synthesis" which is a bit more direct. I can look at portfolio evidence and see if it is synthesized well, or reflected upon well. But can I really tell if the person was motivated to put it there? Or if they were curious about what they were building? Maybe...but probably not consistently.
---------------------
Trent: The learning goals seem to be engagement (first three), synthesis, and reflection. And the various levels of performance seem to measure how well a student has mastered the skills of inquiry in different fields.
---------------------
Glenn: Self-authorship is something that has been referred to here in one program. (Kegan and Baxter Magolda come to mind as a reference for this.)
---------------------
Moderator (Helen): Hi everyone -- I know we're getting close to the end of our time. If you haven't done so already, can you please enter your email address/contact information so I can follow up with you once the transcript is posted and summary of resources.
---------------------
Trent: On some campuses, an ethical goal is set, or international understanding -- are the rubrics looking only at core cognitive development? And are there examples of assignments that might go with the different rubrics?
---------------------
Moderator (Helen): Thanks for the reference Glenn -- Baxter-Magolda is actually on the advisory board for the VALUE project.
---------------------
Trent: this has been a great conversation, everyone
---------------------
Judy: I second that. Thank you!
---------------------
Terry: Thank you all - very helpful feedback.
---------------------
Gary: yes!
---------------------
ruth: Thanks Helen & John & all--good to see you. Want to extend an invitation to you all to attend our "Day of Dialogue on ePortfolios" here in San Francisco, Feb. 25. http://conference.csuprojects.org/eportfolios
---------------------
Trent: bye to everyone
---------------------
Moderator (Helen): Also, as Terry mentioned, there are more rubric drafts ready to be evaluated and the VALUE project is seeking more partner campuses to help pilot these drafts. If you are interested, please email VALUE Initiative Manager, wendemm AT gmail DOT com.
---------------------
Betsy: Thanks.
---------------------
Moderator (Helen): Also, please let me know if there would be interest in doing another EPAC chat like this one on another rubric. Thanks everyone & happy holidays!
Comments (0)
You don't have permission to comment on this page.