Demonstrating Improvement: The Final Part of the Cycle

Outcomes assessment has a rhythm to it: name the outcome, measure it, make improvement, measure again to demonstrate effectiveness of the improvement. Start all over again.

Demonstrating improvement can be a long time coming especially if initial measures include time to establish benchmarks or norm instruments. A narrative on a web page of University of Nebraska - Lincoln offers Exemplars: How is the Outcomes Assessment Process Contributing to Improvements? Most of the exemplars reflect the ongoing nature of the inquiry, not yet claiming success. The page is published by UNL's Office of Undergraduate Studies.

The pertinent question appears to be how much time does it take to actually demonstrate improvement (or no improvement, of course). A colleague hard at work on an accreditation report is struggling with how to highlight the good faith effort of academic programs as they repeat assessments to test for improvement. The answer lies in multi-year assessment cycles with provision for annual measures.

Postscript: A few days ago, Pat Williams (U of Houston) addressed this issue (with the same example) on the Assess this! blog, although I spotted that blog entry just yesterday. Clearly, we were influenced by the same online conversation in a listserv—and the same persistent question.

© 2009 Mary Bold, PhD, CFLE. Email contact: bold[AT]marybold.com. The content of this blog or related web sites created by Mary Bold (www.marybold.com, www.boldproductions.com, College Intern Blog) is not under any circumstances to be regarded as legal or professional advice. Bold is the co-author of Reflections: Preparing for your Practicum or Internship, geared to college interns in the child, education, and family fields. She is a consultant and speaker on assessment, distance learning, and technology.

Tech Note: Locating Assessment Blogs

Ephraim Schechter has been collecting URLs of assessment blogs, a result of a recent exchange on the ASSESS listserv. To locate Schechter's list, go to the Internet Resources for Higher Education Outcomes Assessment web site currently hosted by North Carolina State University and do a Control-F (or "Find") for "blogs." Plural case makes a difference!

Of course, Schechter's entire web site deserves periodic review as resources are continually added and updated. For efficiency, Schechter offers a page listing just the New Links of the past couple of months. (So, at present, the list of blogs also appears under March 2009 on the New Links page.)

© 2009 Mary Bold, PhD, CFLE. Email contact: bold[AT]marybold.com. The content of this blog or related web sites created by Mary Bold (www.marybold.com, www.boldproductions.com, College Intern Blog) is not under any circumstances to be regarded as legal or professional advice. Bold is the co-author of Reflections: Preparing for your Practicum or Internship, geared to college interns in the child, education, and family fields. She is a consultant and speaker on assessment, distance learning, and technology.

Professionalism in Test Administration

At the SAM International Business Conference in Las Vegas this week, faculty members from the Business Department of Texas A&M University - Corpus Christie described the successful launch of a project aimed to support students in taking a major field test (MFT) by publisher ETS. The paper presented at conference will be published in proceedings in the coming months.

Briefly, the faculty devised several strategies for the weeks leading up to the MFT administration, alert to "the small things" that can make a difference in performance on a standardized test. Students had access to online quizzes on sample questions that they could take as many times as they liked. Students were prepared for the formality of a secure test through a letter explaining procedures and even how to locate a seat in the testing room. Faculty included an anonymous "customer satisfaction" survey after the administration in order to learn how students experienced the testing.

The A&M faculty team refer to their project as professionalism in test administration. They devised means to evaluate virtually all of their strategies, including how students used the alloted scratch paper that the Department systematized in distributing and collecting. Not the least of their evaluation is looking for correlations between strategies and test results. They are satisfied that improvement can be seen already.

(Reference to proceedings will be provided here after the publication date.)

© 2009 Mary Bold, PhD, CFLE. Email contact: bold[AT]marybold.com. The content of this blog or related web sites created by Mary Bold (www.marybold.com, www.boldproductions.com, College Intern Blog) is not under any circumstances to be regarded as legal or professional advice. Bold is the co-author of Reflections: Preparing for your Practicum or Internship, geared to college interns in the child, education, and family fields. She is a consultant and speaker on assessment, distance learning, and technology.

Faculty/Student Supports: Moving to ePortfolios

At this week's 2009 International Business Conference sponsored by the Society for Advancement of Management (SAM), a conversation arose in a session about use of ePortfolios in online education. A Texas institution's representatives gave a presentation about an ePortfolio project in its first year and an audience member from a North Carolina school offered another view from the perspective of the planning year of an ePortfolio adoption.

Both institutions described the need to engage faculty in a new ePortfolio initiative and both acknowledged that resistance is fairly common. At the Texas school, faculty are supported in the ePortfolio work through the addition of staff: ePortfolio coaches. The coaches are Master's prepared professionals who have had personal experience in creating an ePortfolio in their own degree program and who can assist students directly (and virtually) in the two-year process of portfolio creation. The coaches offer general support, primarily technical but extended to screening for the need for proof-reading, without being responsible for curriculum content (or grading). This type of work falls into the category of "unbundling," by which non-curriculum tasks of faculty are handed off to other workers in online environments.

At the North Carolina school, the support needed for students working on their ePortfolios is expected to be supplied by faculty members. The program plans to offer faculty the choice of either maintaining an advising role with students or trading it in for a coaching role specific to the ePortfolio.

Both approaches indicate the importance of assigning oversight to students in their ePortfolio work. A helpdesk can provide front line assistance but additional guidance is needed, particularly if the ePortfolio spans more than one course or more than one semester.

(Disclosure: I was one of the presenters as a consultant to the Texas school.)

© 2009 Mary Bold, PhD, CFLE. Email contact: bold[AT]marybold.com. The content of this blog or related web sites created by Mary Bold (www.marybold.com, www.boldproductions.com, College Intern Blog) is not under any circumstances to be regarded as legal or professional advice. Bold is the co-author of Reflections: Preparing for your Practicum or Internship, geared to college interns in the child, education, and family fields. She is a consultant and speaker on assessment, distance learning, and technology.

Batson's short article on the lack of "there" in ePortfolio

Campus Technology's Trent Batson wrote about ePortfolios earlier this month in the article, Eportfolio: There's No 'There' There.

The article presents basic descriptions of the ePortfolio (a place to put your stuff, a technology for assembling evidence of your learning) alongside the potential for student-centered learning (students owning their own work and tying it to their curriculum). And Batson presents larger issues, too: open education, longitudinal learning.

It's always a pleasure to learn of Batson's thinking (even in a short article) but the real learning of this piece comes from the reader comments. The mix of responses supports one of Batson's points (made in his first paragraph), that "ePortfolios mean differing things to different people."

You won't find brand names in Batson's piece or in his readers' comments. In fact, one of the points made is that a technology need not be called an ePortfolio to be used for that purpose. In my experience, I think the most user-friendly and powerful non-ePortfolio technology is the PDF (best manipulated in Adobe Acrobat). It has a learning curve and it's a creation tool, not an assessment tool. Among the technologies that call themselves ePortfolio, I like TaskStream's LAT or Learning Achievement Tool. It's user-friendly for student-authors and also provides assessment functions for instructors.

(Blogger disclosure: I have consulted for TaskStream but I always like to point out that I was a customer first.)

© 2009 Mary Bold, PhD, CFLE. Email contact: bold[AT]marybold.com. The content of this blog or related web sites created by Mary Bold (www.marybold.com, www.boldproductions.com, College Intern Blog) is not under any circumstances to be regarded as legal or professional advice. Bold is the co-author of Reflections: Preparing for your Practicum or Internship, geared to college interns in the child, education, and family fields. She is a consultant and speaker on assessment, distance learning, and technology.

LSU: Student Voices on Campus Budget Cuts

Students on the LSU-Baton Rouge campus, the system's largest, have responded to the state's education budget crisis with a student campaign called S.O.S. - Save Our Schools. The group meets face to face as well as on Facebook, and hopes to meet with state legislators, too. The S.O.S. Blog links to their press coverage.

More student voice is heard through editorials in the Baton Rouge campus Daily Reveille. This week the paper's editorial called for performance-based budget cuts, the "fairest way to weather this ugly financial situation." The LSU Board of Regents will be responsible for making large cuts for the 2009-10 fiscal year and the Board is being lobbied by everyone from students to the governor to consider basing the cuts on performance record (not just size of budget). The online Daily Reveille even has a menu tab for Budget Cuts (set between Sports and Entertainment), reflecting the intense interest in higher ed funding for the campus and for the state.

Cuts in Louisiana began this academic year, with cuts in the millions for most of the the state's institutions. The Baton Rouge campus cut more than $10 million, about a third of the cut for the entire LSU system. Mid-year strategies across the system included reduction of adjuncts and student workers. The next cuts will necessarily be broader in scope. For example, LSU-Shreveport is discussing a trimming of administration, reducing its colleges from four to two. Degree programs would not be cut according to this proposal, just its number of deans.

Louisiana will seek federal stimulus funding but the higher ed cuts for next fall will still be about $219 million. Campus administrators in the LSU system are preparing their own figures for review by the state's House Appropriations Committee this spring. Just as in many other states, the conversation underway in Louisiana does not assume that federal help or a recovered economy will make things right any time soon.

© 2009 Mary Bold, PhD, CFLE. Email contact: bold[AT]marybold.com. The content of this blog or related web sites created by Mary Bold (www.marybold.com, www.boldproductions.com, College Intern Blog) is not under any circumstances to be regarded as legal or professional advice. Bold is the co-author of Reflections: Preparing for your Practicum or Internship, geared to college interns in the child, education, and family fields. She is a consultant and speaker on assessment, distance learning, and technology.

Digital Assessment: Adobe's Guide

Adobe provides a rich guide to Digital Assessment intended for use with their product, Adobe Acrobat. This style of assessment is introduced as appropriate for multimedia and other electronic student works. The primary means of assessment using Acrobat software is simple markup, the process by which you can add a "layer" of notes and graphics on top of a student page. The system requires that students submit their work as PDF files (or the instructor must convert Word documents and other file types to PDF).

Two powerful techniques (powerful in terms of impact on students) are placing audio comments in the PDF and adding comments to individual frames of video. Audio comments are embedded in the PDF and can then be heard by students opening the file with Adobe Reader. Instructors must consider whether a transcript of the audio clip is also appropriate; providing alternative access to comments is a best practice in educational technology. The audio clip can be created within Adobe Acrobat without use of any other software (newer versions of Acrobat required).

Commenting on video (frame by frame) is an appropriate assessment technique when students submit video embedded in a PDF. Because the instructor feedback can be tied directly to the work, assessment is stored in the most convenient location even if the student's video submission is additionally made through other channels.

For assessment archives, PDF files provide a compact method of storing student works. Video files, especially, are handled conveniently.

© 2009 Mary Bold, PhD, CFLE. Email contact: bold[AT]marybold.com. The content of this blog or related web sites created by Mary Bold (www.marybold.com, www.boldproductions.com, College Intern Blog) is not under any circumstances to be regarded as legal or professional advice. Bold is the co-author of Reflections: Preparing for your Practicum or Internship, geared to college interns in the child, education, and family fields. She is a consultant and speaker on assessment, distance learning, and technology.

Tech Note: Podcasts and Mobile Devices

An eCampus News report on podcast lectures at SUNY-Fredonia concludes that student learning (measured by course exam) was greater for students who listened to a podcast lecture than for students who attended a live lecture. Broadcast via iTunes University, the podcast afforded students the advantages of pause and rewind, which are admittedly rare options in the live lecture hall.

The report includes mention that 80% of students accessed the podcast via laptop. "Only" 20% used a mobile device. But the 2009 Horizon Report, the annual prediction of coming trends in learning technology, assures us that mobile is coming. The SUNY-Fredonia research may be the first signal that student users of mobile computing (for school purposes) are increasing. The minority of mobile users in this sample may be the early adopters, hinting at what will impact higher ed in the next year. (The Horizon Report presents technologies according to a timeline; mobile computing is projected for "one year or less.")

© 2009 Mary Bold, PhD, CFLE. Email contact: bold[AT]marybold.com. The content of this blog or related web sites created by Mary Bold (www.marybold.com, www.boldproductions.com, College Intern Blog) is not under any circumstances to be regarded as legal or professional advice. Bold is the co-author of Reflections: Preparing for your Practicum or Internship, geared to college interns in the child, education, and family fields. She is a consultant and speaker on assessment, distance learning, and technology.

CHEA introduces the Federal Update

When the Higher Education Act (HEA) bill was being developed, CHEA followed the progress with the HEA Update. This month, a new online publication launched on the CHEA web site: Federal Update.

In familiar format, CHEA mixes news and reflection in this Update. First topic: the negotiated rulemaking whereby regulations for the new legislation are determined for the accrediting agencies (and, by extension, for the agencies' member institutions).

(The HEA is now commonly referred to as HEOA, for Higher Education Opportunity Act of 2008.)

Distance education administrators will take note that the first Federal Update includes questions about how HEOA requirements for assuring student identity will be addressed. For a short and clear explanation of this issue, along with the other major changes in the new law, see CHEA's comparison chart of previous law and the HEOA.

© 2009 Mary Bold, PhD, CFLE. Email contact: bold[AT]marybold.com. The content of this blog or related web sites created by Mary Bold (www.marybold.com, www.boldproductions.com, College Intern Blog) is not under any circumstances to be regarded as legal or professional advice. Bold is the co-author of Reflections: Preparing for your Practicum or Internship, geared to college interns in the child, education, and family fields. She is a consultant and speaker on assessment, distance learning, and technology.

EDUCAUSE: Economy's Impact on IT

EDUCAUSE surveyed member institutions in January 2009 about the impact of the economic downturn on campus IT.

The survey results obviously focus on computing issues but also point to trends that exist beyond the IT department. (N of 577 from 409 EDUCAUSE-member institutions reflected a response rate of 8.8% and yielded margin of error +/-4%.)

Some results are expected (budget reductions are already underway, hiring and travel are on hold). Others are not so predictable (disparity of results between CIOs and faculty members). Some results are clear: a call for more online and regional conferences, a commitment to sustainability on campus.

Key Findings:
Numbered pages 3 - 6 [PDF pages 5-8]

This report is short enough to warrant some time spent in the details. Recommended:

Detailed results with expert graphics begin on numbered page 7 [PDF page 9].

Comments (scan for italic text) begin on numbered page 12 [PDF page 14].

© 2009 Mary Bold, PhD, CFLE. Email contact: bold[AT]marybold.com. The content of this blog or related web sites created by Mary Bold (www.marybold.com, www.boldproductions.com, College Intern Blog) is not under any circumstances to be regarded as legal or professional advice. Bold is the co-author of Reflections: Preparing for your Practicum or Internship, geared to college interns in the child, education, and family fields. She is a consultant and speaker on assessment, distance learning, and technology.

Edentity: Evolving Meanings

At the CSU ePortfolio Day of Dialogue in San Francisco last week, John Ittelson referenced his use of the term "edentity" in 2001 to introduce the concept of the electronic portfolio that could consolidate a student's academic record and perhaps even a person's lifelong record. Today, the term is used commercially by a handful of web companies to describe various services, most of them tied to web site marketing.

The urbandictionary has defined edentity as one's identity on the Internet and that comes close to the Acxiom company's service of authenticating student identity—relevant to Ittelson's use of the term but much more pertinent to the security question before administrators of online education: how do you know who your student is?

In February, Acxiom partnered with Campus Technology to produce a free webinar on the subject of identity authentication. (An aside by one of the presenters revealed that more than 300 people attended virtually.) The archived webinar can now be downloaded by registering with Campus Technology. Acxiom has also partnered with WCET members to test identification methods, some of which are documented in an Acxiom White Paper. The affordable means of authentication is to pose security questions to students as they work online and this is the thrust of Acxiom's service.

© 2009 Mary Bold, PhD, CFLE. Email contact: bold[AT]marybold.com. The content of this blog or related web sites created by Mary Bold (www.marybold.com, www.boldproductions.com, College Intern Blog) is not under any circumstances to be regarded as legal or professional advice. Bold is the co-author of Reflections: Preparing for your Practicum or Internship, geared to college interns in the child, education, and family fields. She is a consultant and speaker on assessment, distance learning, and technology.

Economy: Impact on Higher Ed

The Lawlor Group, a marketing firm for the education community, opened 2009 with a report on Trends and Tips following the company's research into recruitment and enrollment at independent collleges and universities. The Lawlor report is a synthesis (just 4 pages) but includes some of the statistics gathered to draw seven conclusions about how private institutions might respond to trends in the marketplace. Some of the trends are tied directly to the current economic downturn: students' and families' inability to pay, pragmatism in seeking "better value" schools even when ability to pay is not a problem, and swirling (patchworking credits from several institutions). The report offers suggested solutions for these and other trends.

A colleague at a public institution shared an observation about a new trend in recruitment that she attributes to the economy, also. In her competitive admissions program, prospective students have always been assertive in tracking their applications. In recent weeks, record numbers of applicants have moved beyond assertive to demanding, especially in seeking refunds for application fees after being rejected. The staff's interpretation is that the economy is driving the increase in applications and perhaps the same economy is making the application fee worth recovering.

© 2009 Mary Bold, PhD, CFLE. Email contact: bold[AT]marybold.com. The content of this blog or related web sites created by Mary Bold (www.marybold.com, www.boldproductions.com, College Intern Blog) is not under any circumstances to be regarded as legal or professional advice. Bold is the co-author of Reflections: Preparing for your Practicum or Internship, geared to college interns in the child, education, and family fields. She is a consultant and speaker on assessment, distance learning, and technology.

Day of Dialogue: Accessing WASC Rubrics

At the CSU ePortfolios - Day of Dialogue last week in San Francisco, an interactive session focused on institutional assessment. Several participants commented on the high value of WASC's rubrics for making that assessment in "Educational Effectiveness Review."

The rubrics can be accessed through the WASC web site (click on menu item Document Library). Topics of these Educational Effectiveness rubrics include the Program Learning Outcomes, General Education Assessment, Capstone, Program Review, and Portfolio.

Link to go straight to the Portfolios Rubric:
Rubric for assessing the use of portfolios for assessing program learning outcomes

Another approach to the WASC rubrics is to see how an institution organizes them within an assessment website. CalPoly's web page does it well.

© 2009 Mary Bold, PhD, CFLE. Email contact: bold[AT]marybold.com. The content of this blog or related web sites created by Mary Bold (www.marybold.com, www.boldproductions.com, College Intern Blog) is not under any circumstances to be regarded as legal or professional advice. Bold is the co-author of Reflections: Preparing for your Practicum or Internship, geared to college interns in the child, education, and family fields. She is a consultant and speaker on assessment, distance learning, and technology.