1	The Development Process of an On-Line Portfolio System to Integrate Academia and Workplace Competencies Mark Erickson
2	Today’s presentation Brief history of assessment activities CCIS learning objectives Steps to create on-line E-portfolio Tools and process Discoveries, pitfalls and solutions Mired in the muck Rubrics before and after pilot Next steps Discussion
3	About us Northeastern University, Boston, MA, USA 18,000 undergraduates (full and part-time) 4,000 graduate students (full and part-time) College of Computer and Information Science (CCIS) 700 undergraduate students in the 5 year undergraduate co-op program Alternating co-op program. Required for undergraduates, optional for master’s students Students complete 3, six- month co-op periods 200 students on co-op per term CCIS Masters and PHD programs
4	Our goal: increase integration CCIS implemented new curriculum which includes academics and co-op Integrated Learning Model Extensive assessment research, focusing on what is learned in classroom and co-op How do classroom and co-op learning reinforce each other? URL for research details Prior research based on student self-reports and employer evaluations Accuracy and consistency issues for employers and students Students all think they are proficient Need for more objective measures E-portfolio with rubrics to provide measurable criteria against which to rate student achievement.
5	E- portfolio E-portfolio next phase of integration efforts Provide rubrics and more objective criteria Help students better understand, monitor, and direct their own learning on co-op Inform academic faculty of what students learn while on co-op Inform employers about student skill sets, leading to better co-op positions and a better “fit”
6	Development grant Received three year National Science Foundation assessment grant to develop on-line electronic portfolio Starting the second year of the grant Completed initial student pilot and moving forward with Design of rubrics Re-evaluation of learning goals Creation of database and structure Build structures and multiple views of data
7	E-portfolios On-line portfolios are HOT, everyone seems to be building a system Way to document and share various types of skills or achievements Ability to upload “artifacts” or samples of work that demonstrate abilities Many are limited in scope I.e. program certification goals Showcase work Moderate objectives
8	CCIS portfolio Students (design input) Enhanced ability to observe own career development Employers (design input) View selected student portfolios with permission Co-op faculty (design input) Monitor activity for co-op grades Assess student learning Academic faculty (design input) View portfolios of students in their classes Other parties as defined by students
9	Software: buy or build? Complex set of learning goals with depth Variability in co-ops Different needs of different constituencies Easy to navigate for all users Multiple links and connections Nature of CCIS documentation Decision: Need to build own software
10	OSP 1.0 Open Source Portfolio Initiative Decision to use OSP for a one year pilot while designing own software Temporary solution while developing rubrics Pilot 10 students over one year How will they describe their learning? What will students document? Is material same as survey data? What did students think was important? http://www.theospi.org/
11	Beginning webpage
12
13	OSP 1.0 Example
14	Shared co-op experiences
15	Pilot results Students better understood learning goals and contribution of co-op Students hated the OSP 1.0 portfolio software Difficult to navigate Confusing Lack of flexibility and ability to customize Link one-to-one Led to reframing of College’s learning objectives No easy way for administrators to view material Helped solve the “Rubrics Problem.” (more later)
16	Simultaneous development of rubrics A rubric is a set of categories that define and describe the important components of the work being completed, critiqued, or assessed Each category contains a gradation of levels of completion or competence with a score assigned to each level and a clear description of what criteria need to be met to attain the score at each level
17	Five skill levels Novice Co-op 1 Advanced beginner Co-op 1 or 2 Competent Co-op 2 or 3 Proficient Co-op 2 or 3 or FT Expert Co-op 3 or FT
18	Skill level descriptions
19	Description of CCIS curriculum Programming skills Technical knowledge Theoretical foundations Technical judgment Complexity, design, and abstraction Effective work and problem solving Communications and learning Learning skills Communication skills Creative thinking skills
20	Rubrics: communication skills
21	Rubric complexity Working from academic goals, started developing rubrics for each of 120 separate learning goals Hundreds of examples were needed Employers in focus groups want simplicity We became bogged down in endless discussions Stuck Looking for a new approach How to move forward?
22	Back to the pilot Reviewed all documentation in pilot Reviewed all “data” to develop a picture of skills at each level Data from student and employer evaluations Data from student e-portfolio descriptions Developed new set of macro-learning objectives that are Less granular and less academic More focused on career and personal development
23	Learning: Pre-pilot
24	Learning rubric, post-pilot
25	Journal/Blog E-portfolio
26	Build scaffolding
27	Where we are now OSPI 1.5 fall and waiting for 2.0 in spring Much more flexibility Covers most of our “wish list” Revising some of CCIS goals Reframing at a higher level Starting to build framework, input and output views, mock up views Fall pilot on hold pending software upgrade Additional employer focus groups
28	More activities Set up on-line journal Capture on-line journal results and pipe to portfolio Help students remember and articulate learning Redesign on-line employer evaluations to better match categories in e-portfolio Pipe results directly to portfolio Possibly add individualized learning goals at start of student assignment Timing and tracking issues
29	Discussion