1	The Development of an On-line Portfolio System to Integrate Academia and Workplace Competencies Melvin Simms and Mark Erickson
2	About us Northeastern University, Boston, MA 18,000 undergraduates (full and part-time) 4,000 graduate students (full and part-time) 700 Undergraduate students in the College of Computer and Information Science (CCIS) CCIS Masters and PHD programs 5 year undergraduate co-op program Alternating co-op program. Required for undergraduates, optional for master’s students 200 students on co-op per term
3	Our goal: increase integration Implemented new curriculum which includes academics and co-op Help students better understand, monitor, and direct their own learning on co-op Inform academic faculty of what students learn on co-op Inform employers about student skill sets, leading to better co-op positions and a better “fit”
4	Integration: current phase Received three year NSF assessment grant to develop on-line electronic portfolio Starting the second year of the grant Completed initial student pilot and moving forward
5	The context Continuing to assess knowledge, skills and abilities required to successfully compete in the computing field. Identification of measurable criteria for the identified skills through surveys and focus groups Continuing to focus on creating quality assignments E-portfolio provides documentation to reflect the skill level based on measurable criteria
6	Today’s presentation Brief history of assessment activities Steps to create on-line portfolio Tools and process Discoveries, pitfalls and solutions CCIS learning objectives Mired in the muck Rubrics then and now Next steps Discussion
7	Integrated curriculum: questions Assessment What is learned in each setting, co-op classroom? How to maximize knowledge integration? How do classroom and co-op learning reinforce each other? Portfolio How do we measure and document learning? How do we store documentation and communicate?
8	Assessment related co-op research Assessment overview Over last 5 years, have conducted extensive research of employers, faculty, and students to find out what is learned in co-op and in the classroom. Emphasis on co-op learning Students and employers assess skill level versus job skill level Current research agenda Need for more objective measures Solve issues of accuracy & consistency Assess quality of assignment E-portfolio using specific rubrics
9	E-portfolios On-line portfolios are HOT, everyone seems to be building one Way to document and share various types of skills or achievements Ability to upload “artifacts” or samples of work that demonstrate abilities Many are limited in scope i.e. program certification goals Showcase work Moderate objectives
10	CCIS portfolio Students- Enhanced ability to observe own career development Employers View selected student portfolios with permission. Co-op faculty Monitor activity for co-op grades Assess student learning Academic faculty View portfolios of students in their classes Other parties as defined by students
11	E-port design participants Students Focus groups Portfolio pilot project Required evaluations Faculty Development of learning goals Design of portfolio Employers Required evaluations Focus group
12	Portfolio software: buy or build? Complex set of learning goals with depth Variability in co-ops Different needs of different constituencies Co-op faculty, academic faculty, students, employers, others as selected by students. Easy to navigate for all users Multiple links and connections Nature of CCIS documentation Decision: Need to build own software
13	OSPI 1.0 Open Source Portfolio Initiative Decision to use OSPI for a one year pilot while designing own software Temporary solution while developing rubrics Pilot 10 students over one year How will they describe their learning? What will students document? Is material same as survey data? What did students think was important? http://www.theospi.org/
14	OSP 1.0 Example
15	Pilot results Students better understood learning goals and contribution of co-ops Students hated the OSPI 1.0 portfolio software Difficult to navigate Confusing Lack of flexibility and ability to customize Lead to reframing of College’s learning objectives No easy way for administrators to view material Helped solve the Rubrics Problem. (more later)
16	Simultaneous development of rubrics A rubric is a set of categories that define and describe the important components of the work being completed, critiqued, or assessed Each category contains a gradation of levels of completion or competence with a score assigned to each level and a clear description of what criteria need to be met to attain the score at each level
17	Five skill levels Novice Co-op 1 Advanced beginner Co-op 1 or 2 Competent Co-op 2 or 3 Proficient Co-op 2 or 3 or FT Expert Co-op 3 or FT (2 – 3 years Exp.)
18	Skill level descriptions
19	Description of CCIS curriculum Programming skills Technical knowledge Theoretical foundations Technical judgment Complexity, design, and abstraction Effective work and problem solving Communications and learning Learning skills Communication skills Creative thinking skills
20	Rubrics: communication skills
21	Rubric complexity Working from academic goals, started developing rubrics for each of 120 separate learning goals. Hundreds of examples were needed. Employers in focus groups want simplicity We became bogged down in endless discussions Stuck Looking for a new approach How to move forward?
22	Help from the CCIS design recipe Problem analysis and data definitions Examine input and output data and nature of relationship. Develop technical contract and purpose (in English) and header Look at examples with expected results Template, all the data the function can use Body of the function, the meat Test using samples from step 3
23	Back to the pilot Reviewed all documentation in pilot Reviewed all “data” to develop a picture of skills at each level Data from student and employer evaluations Data from student e-portfolio descriptions Developed new set of macro-learning objectives that are less granular
24	Learning: Pre-pilot
25	Learning rubric, post-pilot
26	Build scaffolding
27	Where we are now OSPI 1.5 fall and 2.0 winter Much more flexibility Covers most of our “wish list” Revising some of CCIS goals Reframing at a higher level Starting to build framework, input and output views Continuing with small pilot in fall and larger in spring Additional employer focus groups
28	More activities Set up on-line journal Capture on-line journal results and pipe to portfolio Help students remember and articulate learning Redesign on-line student and employer evaluations to better match categories in e-portfolio Pipe results directly to portfolio Possibly add individualized learning goals at start of student assignment Timing and tracking issues
29	Discussion