Research ReportETS RR–13-22Synthesizing Frameworks of HigherEducation Student Learning OutcomesRoss MarkleMeghan BrennemanTeresa JacksonJeremy BurrusSteven RobbinsNovember 2013

ETS Research Report SeriesEIGNOR EXECUTIVE EDITORJames CarlsonPrincipal PsychometricianASSOCIATE EDITORSBeata Beigman KlebanovResearch ScientistGary OckeyResearch ScientistHeather BuzickResearch ScientistDonald PowersManaging Principal Research ScientistBrent BridgemanDistinguished Presidential AppointeeGautam PuhanSenior PsychometricianKeelan EvaniniManaging Research ScientistJohn SabatiniManaging Principal Research ScientistMarna Golub-SmithPrincipal PsychometricianMatthias von DavierDirector, ResearchShelby HabermanDistinguished Presidential AppointeeRebecca ZwickDistinguished Presidential AppointeePRODUCTION EDITORSKim FryerManager, Editing ServicesRuth GreenwoodEditorSince its 1947 founding, ETS has conducted and disseminated scientific research to support its products andservices, and to advance the measurement and education fields. In keeping with these goals, ETS is committed tomaking its research freely available to the professional community and to the general public. Published accounts ofETS research, including papers in the ETS Research Report series, undergo a formal peer-review process by ETSstaff to ensure that they meet established scientific and professional standards. All such ETS-conducted peer reviewsare in addition to any reviews that outside organizations may provide as part of their own publication processes. Peerreview notwithstanding, the positions expressed in the ETS Research Report series and other published accounts ofETS research are those of the authors and not necessarily those of the Officers and Trustees of Educational TestingService.The Daniel Eignor Editorship is named in honor of Dr. Daniel R. Eignor, who from 2001 until 2011 served theResearch and Development division as Editor for the ETS Research Report series. The Eignor Editorship has beencreated to recognize the pivotal leadership role that Dr. Eignor played in the research publication process at ETS.

Synthesizing Frameworks of Higher Education Student Learning OutcomesRoss Markle, Meghan Brenneman, Teresa Jackson, Jeremy Burrus, and Steven RobbinsEducational Testing Service, Princeton, New JerseyNovember 2013

Find other ETS-published reports by searching the ETS ReSEARCHERdatabase at obtain a copy of an ETS research report, please n Editor: John SabatiniReviewers: Lydia Ou Liu and Brent BridgemanCopyright 2013 by Educational Testing Service. All rights reserved.CRITERION, ETS, the ETS logo, GRADUATE RECORD EXAMINATIONS,GRE, LISTENING. LEARNING. LEADING., TOEFL, and TOEIC areregistered trademarks of Educational Testing Service (ETS).ISKILLS is a trademark of ETS.

AbstractThe public, education, and workforce sectors all have expressed interest regarding the keyknowledge, skills, and abilities that enable individuals to be productive members of society.Although past efforts have attempted to create frameworks of student learning outcomes, theresults have varied due to different perspectives and goals. Thus, the purpose of this paper was togather and review relevant higher education frameworks, determine their commonalities andcreate domains, and identify the assessments that Educational Testing Service (ETS) hasdeveloped in each of the domains. After a thorough review of the relevant frameworks, sevenkey domains were identified: creativity, critical thinking, teamwork, communication, digital andinformation literacy, citizenship, and life skills. Also discussed were the issues of educationversus work contextualization and the assumption of foundational quantitative reasoning andliteracy skills informing these seven domains.Key words: student learning outcomes, higher education standards, higher education, creativity,critical thinking, teamwork, communication, digital and information literacy, citizenship, lifeskillsi

Table of ContentsPagePurpose. 3Method . 3Selection Criteria . 3Frameworks Reviewed . 5Results: The Critical Domains . 12Creativity . 12Critical Thinking. 14Teamwork . 15Effective Communication . 16Digital and Information Literacy . 17Citizenship . 18Life Skills. 20A Note About Quantitative Skills . 21Existing Educational Testing Service (ETS) Assessments . 22The ETS Personal Potential Index (PPI) . 25Comparable Efforts . 26Future Directions . 28Frameworks From Education and Work Perspectives: Portfolio Versus Stacked, BadgesVersus Certificates? . 29Conclusion . 31References . 32Appendix . 37ii

List of TablesPageTable 1. Frameworks of Learning Outcomes . 6Table 2. A Summary of the Seven Critical Domains and Use in Educational Testing Service(ETS) Products and Assessments . 13Table 3. Alignment Between Burrus et al. (2013) O*NET Domains and Critical Domains . 26Table 4. Comparison of National Research Council (NRC) Domains and Critical Domains . 28iii

In 2006, the Commission on the Future of Higher Education, under the guidance of thenSecretary of Education Margaret Spellings, released A Test of Leadership: Charting the Futureof U.S. Higher Education (U.S. Department of Education, 2006). The report painted the pictureof a changing landscape where colleges and universities increasingly face internationalcompetition; low rates of access and degree attainment; and questions about the knowledge,skills, and ability of their graduates. Among other recommendations, this report called forinstitutions to demonstrate better what students have learned and can do as a result of theircollege experience.Another source of pressure comes from within higher education. In 2006, the NationalAssociation of State Universities and Land-Grant Colleges and the American Association ofState Colleges and Universities published the Toward a Voluntary System of AccountabilityProgram (VSA) for Public Universities and Colleges (McPherson & Shulenburger, 2006). TheVoluntary System of Accountability (VSA) was reportedly designed to increase the quality ofhigher education, both to be more transparent for its constituents, but also as a responsibilityaccordant with its vital role in American societal and economic success. The VSA provided aframework of methods and assessments to gather data across institutions in three coreeducational outcomes: critical thinking, analytical reasoning, and written communication skills.Although some have argued that the VSA was merely a response to calls from theSpellings Report (even though the project was underway before the commission’s; Miller, 2008),other efforts to define and measure key areas of student learning have come from withinacademe. For example, the Association of American Colleges and Universities (AAC&U) alsobegan the Liberal Education and America’s Promise (LEAP) initiative in 2005. Seeking to“recalibrate college learning to the needs of the new global century” (AAC&U, 2007, p. vii),LEAP focuses on defining key student learning outcomes, disseminating best practices ininstruction, and encouraging quality assessment among its members. For institutions of highereducation and the organizations that represent them, measuring student learning serves thefunction of ensuring quality, both internally and to their constituents.Finally, calls for better evidence of student learning have come from employers. Severalrecent reports have stated that, despite high rates of unemployment, employers claim they cannotfind sufficiently skilled workers (Deloitte & The Manufacturing Institute, 2011;ManpowerGroup, 2012). This skills gap is seemingly a result of employer dissatisfaction with1

the products of higher education. A study by the Conference Board, Corporate Voices forWorking Families, Partnership for 21st Century Skills, and the Society for Human ResourceManagement surveyed more than 400 employers across the United States, asking theirperceptions of those currently entering the workforce (Casner-Lotto & Barrington, 2006). Thesurvey did not focus exclusively on college graduates but referred questions to those entering theworkforce with high school diplomas, 2-year degrees, and 4-year degrees. Respondentsemphasized the need for more applied skills, such as oral and written communication, teamwork,and professionalism, over basic academic knowledge in areas such as English, math, and science.Additionally, over 25% of respondents indicated that college graduates were deficient in severalkey skills, such as leadership and written communication.A study conducted for the AAC&U by Hart Research Associates (2010) sampled over300 executives from a wide array of organizations to determine their perceptions of the needs ofcollege graduates. Of those surveyed, 88% said that graduates need higher levels of learning andskills than they had in the past. Only 28% of respondents agreed that 4-year colleges anduniversities are doing a good job of preparing students for the modern workforce. Thus, theworkforce sector is vested in defining and measuring student learning so that it can ensure asufficiently skilled supply of new employees.These three sectors—public, education, and workforce—all pose different questions tohigher education as they weigh the high cost of a degree. Are students acquiring the keyknowledge, skills, and abilities to function as productive members of society? Are collegegraduates ready to enter and succeed in the workforce? Though differing in origin, each of thesequestions has led members of these respective sectors to discuss the focus of higher education,resulting in a litany of proposed frameworks for student learning.Literature that seeks to define student learning in higher education comes from diverseperspectives and targets diverse goals. Organizations in education, government, public policy,and industry all have a vested interest in the direction of higher education. These interests varyfrom societal good to human capital development to self-determination and autonomy. Theseframeworks take many forms. Some, such as the VSA (McPherson & Shulenburger, 2006),simply mention key areas, such as broad general education outcomes, on which higher educationshould focus. Others, such as the Conference Board study (Casner-Lotto & Barrington, 2006),conduct surveys of key constituents to determine what factors are important to them.2

As a result of varying motives and methods, research in this area has come to a range ofconclusions. Should higher education focus on the traditional liberal arts education or geartoward work-relevant competencies? Should the framework for higher education be broad,leaving room for local determination, or specific as to ensure continuity across graduates? Theseare just some of the questions that arise. Ultimately, this multitude of efforts to create commonunderstanding across higher education has led to some confusion.PurposeWe began this project with three goals:1.To gather and review outcomes frameworks relevant to higher education,considering the social, educational, and occupational perspectives. This focus onhigher education is a key consideration of this paper. Though preparing students forthe workforce is and always will be an important role of many educational systems, aframework of outcomes designed to define what a student should learn and haveachieved in college is different than one that seeks to certify the competenciesnecessary for entering the workforce.2. Determine commonalities among these frameworks. Our team thoroughly reviewedthe relevant literature to identify and define common areas of learning. Our reviewyielded seven key domains that largely represent these established frameworks.3. Identify assessments that Educational Testing Service (ETS) has developed in each ofthese domains and the extent to which the assessments align with the definitionspresented here. By consulting with representatives from across ETS, we haveidentified both operational assessments and research tools that have been designed toassess many, but not all, of these domains. This alignment will be discussed in thefinal section of the report.MethodSelection CriteriaUsing several relevant frameworks, the research team assembled them in two ways. First,we looked at the domains—broad, high-level skills—within each framework. By comparing andcontrasting these across all domains and determining those factors that were included in amajority (at least four of seven) of the frameworks reviewed, we were able to identify the seven3

critical domains presented below. Next, we examined the operational definitions—the learningoutcomes used to define each domain. By analyzing these, we were able to identify theknowledge, skills, and abilities that define the critical domains.Given the three goals stated above, we used three criteria to select relevant frameworks:population criterion, scope criterion, and specificity criterion. For the purpose of this paper, thepopulation criterion referred specifically to frameworks targeted at higher education. For thescope criterion, we focused on frameworks that span across disciplines and are appropriate for allinstitutions of higher education. Finally, the specificity criterion included only those frameworksthat provided operational definitions of the knowledge, skills, and abilities discussed in theframework. A more thorough discussion of each criterion is below.Frameworks targeted toward higher education (population criterion). Manyprevious efforts to identify key skills have focused on either basic work requisite skills or generaleducation outcomes. For example, the National Work Readiness Certificate was developed usingwork-contextualized cognitive test items representing the ability to use applied mathematics,applied reading, and information location as part of workplace task performance. It is widelyused within the community college and career technical training sectors as it is most appropriateand specific to work training readiness and workplace success. From another perspective, severaleducationally focused frameworks have been created to understand higher education readinessand success by either focusing on primary and secondary education (e.g., the Common CoreState Standards) or spanning all levels of education (e.g., National Research Council [NRC],2008, 2010, 2011, 2012).Much of the debate about the value of postsecondary education, as well as questions ofdegree quality, has focused on 4-year (i.e., bachelor’s) degrees. Thus, we only consideredframeworks that explicitly targeted a bachelor’s level education or, in the case of workforceexamples, the expected skills of those holding a 4-year degree.Multidisciplinary and trans-institutional (scope criterion). Given the emphasis onassessment and accountability over the past several decades, many individual institutions ofhigher education have developed frameworks of learning outcomes for their students. Similarly,professional organizations and fields vary in the requirements and expectations of their members.Thus, we considered only educational frameworks that were intended to go beyond any oneinstitution or field of expertise. For example, a recent study by Shultz and Zedeck (2012)4

outlined characteristics that were indicative of effectiveness in the law profession. Similarly, theU.S. Department of Labor, Employment and Training Administration (USDOL-ETA, n.d.)published a competency model specifically for advanced manufacturing. Although theseframeworks may provide valuable insight for some key skills, we chose to focus on those thatwere identified as vital across fields.Specific in the learning outcomes or competencies expected (specificity criterion).Many efforts that outline key skills for learners and workers stop short of providing operationaldefinitions of these skills. In reviewing the literature, we saw this embodied in several ways.Efforts that commonly use the term 21st century skills (e.g., NRC, 2008, 2010, 2011, 2012) havethoroughly reviewed the scientific literature and often refer to skills such as critical thinking,communication, and teamwork, among others that we discuss here. Similarly, surveys ofemployers (e.g., AAC&U, 2010; Casner-Lotto & Barrington, 2006; Hart Research Associates,2010; Manyika et al., 2011) broadly refer to these same skills. In both cases, however,operational definitions that include supporting learning outcomes or competencies—statementsabout specific knowledge, abilities, and dispositions inherent to these domains—were lacking.For the purposes of this report, such specificity was necessary to generate a commonunderstanding among all groups, as well as to inform potential future work in curriculum andassessment development.Frameworks ReviewedGiven these criteria, we identified seven frameworks for consideration (see Table 1). Wehad considered organizing these frameworks according to the sector of the publishingorganization (i.e., policy/government, education, workforce), but the work itself is rarely asdiscrete as such segmentation would suggest. Though many of the published reports around theseefforts come from a single entity, the frameworks themselves are often vetted, if not developedin collaboration with, members of other sectors. The AAC&U’s LEAP framework wasdeveloped with significant input from employers. The Assessment & Teaching of 21st CenturySkills (ATC21S) framework was initiated by a consortium of corporate partners—Cisco, Intel,and Microsoft—but conducted in collaboration with the University of Melbourne and otherinstitutions of higher education.5

Table 1Frameworks of Learning OutcomesFrameworkFramework for Higher EducationQualificationsEuropean Higher Education AreaCompetenciesLiberal Education and America’sPromiseFrameworks for Learning andDevelopment OutcomesThe Degree QualificationsProfileThe Assessment & Teaching of21st Century SkillsAbbreviated titleQAA-FHEQETA Competency ModelClearinghouse’s GeneralCompetency Model petusQuality Assurance Agency forHigher EducationEuropean Commission: EuropeanHigher Education AreaAssociation of American Collegesand UniversitiesThe Council for the Advancement ofStandards in Higher EducationThe Lumina FoundationCollaboration among Cisco, IntelMicrosoft, the University ofMelbourne, and othersU.S. Department of Labor,Employment and TrainingAdministrationWith that said, many of the frameworks are written to meet the goal of the sponsoringorganization. For example, the framework put forth by USDOL-ETA, though informative forhigher education, is designed to represent the skills of employees and not the knowledge, skills,abilities, and mindset of a well-rounded citizen (a term frequently used by Lumina’s DegreeQualifications Profile, LEAP, and others). Similarly, the Degree Qualifications Profile (DQP)outlines the abilities of degree holders, not necessarily prospective employees. However, as weultimately seek to synthesize these models, these different perspectives will strengthen the finalproduct, as the domains that are common across all three sectors will have been determined withbroader endorsement.Quality Assurance Agency for Higher Education—Framework for HigherEducation Qualifications (QAA-FHEQ). In 1997, the National Committee of Inquiry intoHigher Education released their Dearing Report (see Dearing, 1997), which indicated a greatneed to create a national framework for higher education qualifications (Quality AssuranceAgency [QAA], 2008). In response, the Framework for Higher Education Qualifications (FHEQ)in England, Wales, and Northern Ireland was developed by the QAA (2008). In 2001, the FHEQpublished the first edition of the framework and completed an updated second edition in 2008.This framework has been designed to meet the standards of the Bologna Declaration on the6

European Space for Higher Education as well as align with the Framework for Qualifications ofthe European Higher Education Area (QAA, 2008).Briefly, the purposes of the FHEQ, as described in The Framework for Higher EducationQualifications in England, Wales and Northern Ireland: August 2008 (QAA, 2008), are thefollowing: to provide important points of reference for setting and assessing academic standardsto higher education providers and their external examiners, to assist in the identification of potential progression routes, particularly in thecontext of lifelong learning, and to promote a shared and common understanding of the expectations associated withtypical qualifications by facilitating a consistent use of qualifications titles across thehigher education sector (QAA, 2008, p. 6)The FHEQ is a “qualifications framework” (QAA, 2008, p. 7) with five levels. Theselevels are numbered 4–8 (following Levels 1–3 as indicated by the National QualificationsFramework and the Qualifications and Credit Framework), with Level 4 referencing highereducation certificates, Level 5 indicating foundation degrees and/or diplomas of highereducation, Level 6 including bachelor’s degrees, Level 7 referring to master’s degrees, and thehighest FHEQ level, Level 8, including doctoral degrees (QAA, 2008). Each level has specificqualifications and skills associated with it. A qualification descriptor provides a statement ofspecific outcomes an individual should demonstrate as well as a global description of the abilitiesone should possess at that level (QAA, 2008).European Higher Education Area Competencies—The Bologna Declaration and theTuning Project. In 1999, the Bologna Declaration was signed by higher education officials from29 European countries in an effort to create a greater level of trust and transparency throughoutthe European education systems (Education, Audiovisual, and Culture Executive Agency[EACEA], 2012). Thus began multiple proceedings, both formally and informally, to assist withthese reforms, the most notable of which was the Tuning Project. Funded by the EuropeanCommission (the executive body of the European Union), the Tuning Project was developedthrough the efforts of faculty members (Adelman, 2009). The term tuning refers to thedevelopment of a framework that ascertains specific learning expectations for students in each7

area of study. In turn, clear expectations are set for each subject area while academic autonomyis retained (Marshall, Kalina, & Dane, 2010).While the initial Bologna Declaration sought to create a system based on two cycles—undergraduate and graduate—and to specify requirements to transition from the first to thesecond (EACEA, 2012), the Tuning Project assisted with the design, development,implementation, and evaluation of each cycle (Tuning Educational Structures in Europe, n.d.).Thus, while similar, the Bologna Declaration and Tuning Project had different objectives. TheBologna Declaration led to the development of the National Qualifications Framework, whichattempts to articulate the differences between qualifications in each cycle (EACEA, 2012). In theTuning Project, the goal is to provide qualifications on the institutional and pan-European fieldlevels. The Tuning Project provides faculty members and institutions with the information toadequately describe “cycle degree programs at the level of subject areas” (e.g., biology,sociology; Adelman, 2009, p. 48).To do this, faculty members are assigned the task of developing reference points andstatements of learning outcomes, levels of learning, and desired competencies in particulardisciplines using a methodology that ensures all statements are transparent and comparable(Adelman, 2009). The Tuning Project asserts that uniformity is not ideal in degree programs, butrather the ability to find points of convergence, reference, and common understanding betweenthem. Tuning articulates what a specific curriculum/program intends to do, but does not attemptto map one specific pathway (Adelman, 2009; González & Wagenaar, 2003).Association of American Colleges and Universities—Liberal Education andAmerica’s Promise (LEAP). In 2005, LEAP was established and has since served as anadvocate for liberal education. Working with hundreds of higher education institutions at alllevels (i.e., community colleges, colleges, and universities), LEAP provides information about“key outcomes of a quality education” (AAC&U, 2011, p. 1) and indicates how well institutionsare assisting students in reaching the desired outcome through research and reports (AAC&U,2011). In addition to this research, LEAP has established several partnerships with colleges anduniversities to create the Campus Action Network (e.g., individuals from campuses across thecountry work together to discuss best practices in the field) and has formed the NationalLeadership Council (composed of business persons, academics, and policy leaders who advocatefor all students to have access to opportunities in key education outcomes; AAC&U, 2011).8

Central to the LEAP initiative are the views set forth in College Learning for the NewGlobal Century issued by the LEAP National Leadership Council in 2007 (AAC&U, 2011). Thereport argued the importance of focusing on key education and learning outcomes and offeredrecommendations for achieving these results (AAC&U, 2011). Using the report and research as aguide, LEAP has denoted four domains that are considered “the essential learning outcomes”:knowledge of human cultures and the physical and natural world, intellectual and practical skills,personal and social responsibility, and integrative and adaptive learning (AAC&U, 2011, p. 7).In addition to the Essential Learning Outcomes framework, the LEAP NationalLeadership Council has proposed a new framework for excellence that includes seven principles:1. Aim high—and make excellence inclusive2. Give students a compass3. Teach the arts of inquiry and innovation4. Engage the big questions5. Connect knowledge with choices and action6. Foster civic, intercultural, and ethical learning7. Assess students’ ability to apply learning to complex problems (AAC&U, 2011)LEAP has emphasized the importance of developing a partnership between educators, students,and society to reach this level of excellence (AAC&U, 2011).The Council for the Advancement of Standards in Higher Education (CAS). In 1979,the Council for the Advancement of Standards in Higher Education (CAS) began with thepurpose of creating, developing, and disseminating standards in higher education to educatorsand their institutions (Strayhorn, 2006). Specifically, CAS aims to support and enhance thequality of programs and services in the field of learning and development (CAS, n.d.). In aneffort to meet these goals, delegates from 36 professional higher education associations,represented by a range of organizations in the United States and Canada—including both forand not-for-profit organizations—serve as members of the CAS Board of Directors (CAS, n.d.).CAS has developed standards of professional practice for a range of areas in the realm ofeducation and has created 16 learning and development domains, including intellectual growth,effective communication, realistic self-appraisal, enhanced self-esteem, clarified values, careerchoices, leadership development, healthy behaviors, meaningful interpersonal relationships,independence, collaboration, social responsibility, satisfying and productive life styles,9

appreciation of diversity, spiri

population criterion, scope criterion, and specificity criterion. For the purpose of this paper, the population criterion referred specifically to frameworks targeted at higher education. For the scope criterion, we focused on frameworks that span across disciplines and are appropriate for all institutions of higher education.