Use and problems in the language of discipline-based qualification statements:
learning from Tuning and its analogues

Clifford Adelman

Abstract: This essay is an empirical account of English language use, across three continents, in 40 Tuning and analogous discipline-based statements of desired demonstrated competences and learning outcomes in higher education. It is primarily concerned with lexical and semantic matters, takes the perspective of the student as the primary reader and beneficiary of these statements, and is as much proscriptive as it is analytical. It provides frequencies of verbs used in such statements, flags commonly but unacceptable verbs and syntax, offers a different grouping of competence-oriented verbs from that inherited from Bloom et al’s Taxonomy, and suggests what we should do in revisiting statements of learning outcomes that have taken root in the literature.

Keywords: language, verbs, syntax, voice, and diction level; competence; meanings and contradictions; reference points; benchmarking versus Tuning.

I. Introduction: What this essay will do and why

This essay follows the spirit of the first evaluation of Tuning competence statements^[1] in that it is grounded in prose actually written and promulgated by discipline-based committees. While it endorses most of the notion of “competence” set forth in the follow-up recommended template for writing Degree Profiles,^[2] it departs from the perspective of that document in the categories of its analysis, and explicitly rejects the use of some key terms (e.g. “ability”) in statements of both competences (generic and subject-specific) and learning outcomes. The language of competence and learning outcome statements is its principal topic, and it finds many current statements wanting on lexical and semantic grounds.

We begin with the following questions:

— What, empirically, are the lexical and syntactic qualities of current competency and learning outcome statements found in discipline-related guidances across national settings?

— Do these statements provide clear direction to current and prospective students as to what they are expected to demonstrate to qualify for degree awards?

— Which lexical features of those statements are blockages to student understanding and disciplinary clarity?

— Which lexical features in these statements are most likely to fulfill the ideals of universal clarity of learning outcomes to all stakeholders in higher education?

— How frequently are these features used in extant statements of desired competences and learning outcomes, and in what ways can they be grouped to facilitate future expansions and refinements?

— Along the way, what are the difficulties with the concept of “competence” and how might we come to live with the term? What statements do we encounter in declarations of qualifications for students in higher education that are not “competences” and in what ways? And what do we do about them?

Exploratory answers to those questions are offered on the basis of close-reading^[3] of high-sounding and high-impact documents that are granted an easy pass in the world of higher education, particularly those documents dealing with what they claim to be student learning outcomes, student qualifications, and student competences. They come from many countries and in many languages. Some, in fact, are national system policy documents; some are association goal statements; some are intended as rallying points for improvement by academic organizations; some are guidances for accrediting and quality assurance bodies; some are voluntary cooperative undertakings by groups of like-minded faculty and administrators. We give them a pass because their intentions are noble.

Under close reading, however, the student is sometimes either buried or outrightly forgotten in these documents, and language has turned fields of dreams into fields of mud. However noble in intent, the diction and semantic discourse of too many of these statements have become what George Orwell (in his seminal essay, “Politics and the English Language”) called “blah,” words that fall like snow, obliterating distinctive features of the landscape. We all lose our way, and I am sure that writers in languages other than English have made similar observations. Noble intentions deserve better. At the same time, however, close reading reveals the extent to which these statements succeed in providing students a clear map to the types of actions that will carry them on the way to qualifying for degree awards, and the extent to which the faculty groups responsible for constructing discipline-based reference targets for student learning have moved toward that end. Close reading unfortunately turns up both the exemplary and the mumbled in the same sentences, e.g. “the qualifying student will synthesize information and recognize relevance,”^[4] and part of our job is to muffle the mumbling.

This exploration is not about the learning process, nor is it intended as a guide to curriculum design and curricular delivery by academic units. It does not address all the features of a degree program profile, nor the theoretical nature of cognitive competences versus psychomotor competences versus affective competences versus what Sadler^[5] calls “graduate attributes.” On the other hand, though it agrees with Weinert that “competence has become a fashionable term with a vague meaning not only in public use, but also in many social sciences” and is loaded with “considerable surplus meanings,”^[6] this article uneasily retains the word out of methodological consistency: just as verbs used by the sources of our analysis are counted, so are the sources’ use of pivotal terms. This, admittedly, is an imperfect choice, particularly in light of critiques the first version of the U.S. Degree Qualifications Profile^[7] received from the field, claiming that the word “competence” implied minimal acceptable performance, hence did not meet standards of qualification statements. It is no surprise that the second version of the DQP^[8] shifted from “competence” to “proficiency.”

It is also worth noting at this point that while this article proposes groupings of words in the writing of competency statements for higher education, it does not pretend to offer a definitive theoretically-based taxonomy. It is an a posteriori account and analysis, and is informed far more by canons of language style and basic linguistic units than psychology or educational theory. It invites the reader to reflect, dispute, resolve, and re-write.

1. Why is language important in statements of competence and learning outcomes in higher education?

The writing of competence and learning outcome statements in education is not a new phenomenon in the English language environments of this paper. Since the publication of The Taxonomy of Educational Objectives by Benjamin Bloom and colleagues in 1956,^[9] there has been a stream of modifications and adaptations and accompanying lists of appropriate words to use in such statements,^[10] but no systematic examination of how those taxonomies and words have actually been used by generations of education developers. That said, why is examination of formal learning outcome statements in higher education important?

— Both syntactic and lexical features of these statements signal statement-type (competence, metacompetence, discrete outcome) and status (possession, development, goal).

— Voice (declarative, imperative, subjunctive) signals the student-reader (and others) as to whether fulfilling or exceeding the performance described is assumed, required, or simply desired, i.e. the intent of the statement.

— Diction level (on a continuum from concrete/specific to abstract/generalized) provides focus, a camera lens setting on the breadth, tractability, and accessibility of the statement’s stated subject.

I am looking at the outputs of different groups of faculty in different disciplines in different countries, outputs that pretend to describe desired points of cognitive development and application in the careers of higher education students, outcomes that these various writing groups understand to represent something called “competence.” Whether these writing groups acknowledged it or not, anything we call a “competence” represents a convergence of declarative knowledge, disposition, and cognitive activity. For example, a student who “identifies” X has to know something about X, is inclined (either autonomously or responding to a direction from an external source) to isolate X from a stream of phenomena, to classify and label it, and (though unstated) merge this activity with others. This is obviously a complex configuration, multifaceted, and sometimes indeterminable.^[11] It is a “dynamic combination”, though the contributions of “metacognitive skills, interpersonal… skills, and ethical values”^[12] will differ by task and environment, and may not be present at all. As will be demonstrated, we read hundreds of Tuning and Tuning-analogous statements of what it takes for students to qualify for a degree in a specific field that unfortunately reflect a far less thoughtful and sensitive a notion of “competence.”

II. Sources, Inclusions, and Exclusions

My sources come from countries in which English is the native language (even when other languages hold legal status or are spoken by notable segments of the population): the U.S., U.K., and Australia. Confining the universe to native English language sources produces a modicum of commonality, minimizes linguistic noise, and at least clamps borders on usage. The one noted exception in this universe lies in my inclusion of the default European English used in the presentation of official Tuning documents from the places of origin of the Tuning enterprise in the Netherlands and Spain.

Some 40 discrete Tuning, benchmarking, and discipline profile statements were examined: eight (8) from the Australian Learning and Teaching Council (ALTC),^[13] ten (10) from the UK’s Quality Assurance Agency (QAA),^[14] ten (10) from the European Tuning collection, and 12 from Tuning USA.^[15] These 40 documents (see Appendix A for a full list) contain 1177 statements of expected student competencies or learning outcomes at the bachelor’s/1st degree cycle level (though, in many cases, they also include such statements at the associate’s/short cycle, master’s, and/or doctoral levels). My analysis focuses only on stated bachelor’s degree/1st cycle competences^[16] and learning outcomes with the exception of Australian cases in which the Master’s is the de facto qualifying degree (architecture, education). Where lists of competences/learning outcomes are presented at both “threshold” and “typical” levels, as they frequently are in QAA documents, the statements under “typical” are chosen for inclusion. The ALTC refers to all its outcome statements as “threshold,” in part to distinguish their intent and standards control from the European Tuning’s “expected and intended” outcomes.^[17]

Having acknowledged these sources, it is important to note that they do not share the same purposes or forms, hence their language tapestries will differ in notable ways. A benchmarking statement, such as those of the QAA, is different from a Tuning production. The unit of analysis in a benchmark is the institutional program or department, and any statements of expected learning outcomes are dependent on the offerings and distributional modes of that program or department. By one interpretation, the QAA is not defining degrees, rather the factors that programs and departments consider and weight in designing or revising a curricular program. Thus, a QAA statement can introduce its template of reference points with the virtual command: “We expect all programmes to address the following areas… so that graduating students” will know, understand, apply, etc. “those aspects of the discipline,”^[18] but not to indicate comparative weighting of the curriculum or its various modes of delivery. These are not requirements for the award of degrees. The student is not the party addressed. All statements of knowledge about the field’s core concepts and principles of application are expressed as consensus for academic departments, e.g., for Geography:

— reciprocal relationships

— spatial variation

— patterns, processes, interactions and change

— significant of spatial and temporal scale on physical processes, human processes, and their interactions

— patterns of change

— diverse manners of representation of the human and physical worlds

— range of analytical and observational strategies^[19]

Benchmarking statements are very persuasive on this discipline profile territory. When they turn the page to student behaviors, on the other hand, their language becomes uncomfortable. Verbal phrases that otherwise would be considered as required learning outcomes become noun phrases governed by fuzzy conditionals, e.g. “students should develop competence in.” Such phrasing does not in any way diminish general masteries that we would acknowledge the field to promote, but too often we get a noun-phrase, such as “decision-making” that walks us into a dead-end wall. What is “decision-making”? Without a context and a challenge, without what Natural Language Processing calls frame-agent-theme-destination,^[20] it lacks grounding. If we said, instead, “selecting sources and choosing, describing and defending a path of investigation of an unscripted problem,” we are describing what the student does under the otherwise formless banner of “decision-making.” The language landscape of QAA documents is worth including as much to illustrate such alternatives as for its declaratives.

Another feature of benchmarking or Tuning documents in professional fields involves the inclusion and (in many cases) the dominance of statements referring not to what the student does or acquires or learns during the course of education for which universities are responsible, rather to professional activities that take place in work environments after the student leaves higher education. The difference between academic and professional competences is well marked in the literature, and its starkest example in our universe of documents is the QAA benchmark statement for Social Work. Technically, there are 61 competence/learning outcome promulgations, but of these, 37 apply only in future work contexts, e.g. “support service users to take decisions and access services, with the social worker as navigator, advocate and supporter” or “manage uncertainty, change, and stress in work situations.”[21] While the rhetoric of presentation may assume (without explicitly saying so) that the first cycle program in social work prepares a graduate to evidence these behaviors, they are not presented as program level learning outcomes, and I chose not to include them in the analysis of the language of competences that an institution of higher education could assess in the course of a student’s degree program. The European Tuning document for Nursing is similar: of 47 subject-specific competences, twelve (12) describe post-degree professional activities that could not be demonstrated even in the clinical portion of the nursing curriculum (for which five competences open with the clinical context stage direction, “Using nursing skills, medical devices, and interventions/ activities to provide optimum care, demonstrates the ability to… ”.[22]). Those 12 statements are not included in the language analysis that is the subject of this article.

This last observation raises the methodological question of which statements made under the label of “competences” or “learning outcomes” were excluded from the universe of analysis and why. For a notable example, in the European Tuning Reference Points for Business, one reads the agreed on competences, generic and subject specific, only in appendices reporting on the results of surveys concerning their importance. The 31 generic competence statements include 14 describing desired aspects of personal development that, as expressed, could never be directly observed or derived from observation, e.g. “Ability to plan and manage time,” and two statements of desired “appreciation” or “awareness” that are not competences.^[23] These 16 statements are not included in the analysis, nor are three similar statements in the subject-specific list, e.g. “work assignment abroad” is not a competence or outcome: it is part of a desired business curriculum.

Then, too, the reader should be aware of the range in the number of competence statements included in these documents. The Australian ALTC discipline groups were instructed to be parsimonious, and followed instructions: the maximum number of outcome statements in any of the Australian reports utilized is seven (7). On the other hand, the Tuning USA project in Marketing included discrete “sub-competences,” all 167 of which are worth marking (and more so than the 21 categories in which they are organized).^[24] The number of learning outcomes included in a discipline’s summary seems to be largely a function of the level of abstraction determined by the writing committee. This is a camera lens decision: the close-up results in the detailed list of the Marketing group. The reader knows everything the student does on the road to the degree. A wide-angle photo inevitably results in a higher degree of abstraction, and the likelihood of fewer competences included.

III. Bias and Control

There is a bias in this analysis of which the reader should be fully cognizant at the outset: I am one of four authors of the U.S. Degree Qualifications Profile (DQP), a document analogous in purpose, though in neither form nor official standing, as the Qualifications Framework for the European Higher Education Area (QFEHEA), i.e. while is it also focused on generic student qualifications for degrees at three levels (short-cycle, bachelor’s, and master’s), its competence statements are far more detailed than the “wide-angle diction” of the QFEHEA.^[25] The DQP adopted strict language rules, developed in part by reading analogous products from other countries, and in languages other than English (particularly German and French), partly inspired by the principles of learning outcome statement forms advocated by Benjamin Bloom and his colleagues, and presents a striking contrast to these other outings.

The various projects under the umbrella of “Tuning USA” do not share this set of rules, though statements in harmony with those rules occasionally turn up in both interim and final reports of these projects. The difference is explained by control: the DQP is an iterative creation under the sponsorship of a non-governmental organization, the Lumina Foundation for Education, which has edited and published the Beta document and funded a process of feedback and planned subsequent editions featuring the impact of both feedback and further thoughts of the original four authors. No governmental authority is involved in this work.

The “Tuning USA” projects, while also funded by the Lumina Foundation, are neither controlled nor endorsed by Lumina. State higher education authorities in Indiana, Kentucky, Minnesota, Texas, Utah, and, recently, Montana, have gone their own ways under their general understanding of what Tuning is designed to accomplish; but in none of these cases has the state authority intervened. These efforts have now been joined both nation-wide by the American Historical Association (AHA) and the American Communications Association, and regionally by the Midwest Higher Education Compact (MHEC). To the extent to which state higher education authorities function in any way beyond their roles as organizers of faculty groups, grant applicants, distributors of funds, and process managers in these projects, it certainly is not with a set of language rules, review, or approval. The same can be said for voluntary national scholarly associations such as AHA or voluntary informal regional associations such as MHEC. Across all of this is a fragmentation that works against convergence of Tuning forms among disciplines, and leaves mechanisms of control out in the cold. While language is a central engine of the DQP, it is, at best, an afterthought in the work of the various Tuning USA projects, though it is obvious from their competency statements that a majority of these undertakings absorbed productive language principles from interaction with the DQP process.

The situation is very different in analogous undertakings elsewhere. Both the Australian and English documents were produced and/or sponsored by government agencies. And while participation in Tuning Educational Structures in Europe project is voluntary, the undertaking has the official backing of the European Commission and was folded into the Bologna Process in 2005. None of this means control-by-ministry, but it does carry government endorsement. Each of these three sources operates with the same assumptions and processes, and issues its conclusions either in exactly the same forms (Australia’s ALTC) or in highly analogous forms (QAA and Tuning Educational Structures in Europe).

IV. Language Rules

In many ways, the DQP is faithful to Benjamin Bloom’s notions of competence, mastery, and evaluation–reflected in the dominant part of speech employed and its logical extension. As previously noted, a “competence”^[26] is a complexity exhibited or demonstrated by students (or anyone else, for that matter), therefore it is something the student does, therefore it is driven by operational verbs. More pointedly expressed by DQP co-author, Peter Ewell, “competency statements in the DQP are deliberately and relentlessly couched in ‘action verbs’ that describe what students at particular [degree] levels should be expected to do.”^[27] Seen from the engine of the verb, a statement of competence is incomplete without examples of assessments/assignments that would elicit the student behaviors that allow the degree of competence to be judged. The original DQP made that point a core principle of any statement of student learning outcomes, though it did not include examples of assignment prods; Ewell’s monograph and the DQP second edition, in 2014, include those examples. While discipline profiles (the Australian Learning and Teaching Council) and subject benchmarking documents (the UK’s Quality Assurance Agency) often include brief sections describing the processes of assessment that should be considered, these are neither logically nor organically connected to the core markings of desired subject content and student learning, and are usually placed in the position of a coda to the principal document. For the DQP, the line from verb to assignment is a maxim—and not just any verb. Only operational, that is, “measurable, action-oriented” verbs are admitted. This rules eliminates from the text of its degree qualifications statements a heavy quintet^[28] of English nouns and noun phrases used by the Australians, the British, other Americans, Scottish, the Irish and in a plurality of European Tuning statements. In the language world-view of the DQP authors, too many of these nominals, particularly when placed in the opening clause of a competence statement, are either deceiving, wrong, irrelevant, distracting, or downright meaningless. The DQP is very explicit as to the terms it rejects “because these do not describe discrete activities that lead directly to assessments.”^[29]

V. Setting Aside Unproductive Language

With that principle in mind, and in search of forms of competence and outcome statements that are clear and unambiguous, it is incumbent on us to set aside assertions that are not competences, putative competences that are neither taught nor improved in higher education, and the most common distracting phrasings. These involve sentences governed by nouns, and that open with noun clauses. My principal targets for elimination are “ability,” “capacity,” “understanding,” “awareness,” and “appreciation,” and their verb variations, and this essay is not the first to object to the use of such barren terms in student learning outcome statements.^[30] Some critics call them “white noise.” I obviously do not think we can afford white noise in competency statements or discrete learning outcomes, nor did the early evaluation of European Tuning competence statements by the Netherlands Organization for International Cooperation in Higher Education,^[31] whose successor project, CoRe2 noted on its Website, “European higher education institutions were experiencing difficulties in expressing the learning outcomes of their programs in clear and concise competence descriptions”^[32] The examples cited below will demonstrate the prescience of this understated judgment, and not merely for European sources.

1. The distracting “white noise” of “ability” and “capacity”

The most guilty opening in the 40 documents examined is the English noun, “ability,” its verb form in “able to,” and its principal analogue, “capacity.” German comes closer to the problem with the word in English: die Fähigkeit means “aptitude.” At least in U.S. education discourse, “aptitude” is what we would call a “red flag” word. It labels something that is putatively inherent in an individual, and its equivalent in discourse is “ability.” However ensconced “ability” is in the literature on competence, we would do well to remember that “quantifying differences in individual mental ability” led to IQ testing and its consequent strengthening of “pervasive forms of discrimination”.^[33] Its history carries too much of this unwanted baggage.

But there are other reasons for avoiding the term. “Ability” is not something an individual does. One doesn’t know a student has an “ability” to do anything, or a “capacity” for anything until the student actually does it, and the competence is reflected in what the student does. As Richard Shavelson notes, a competence is impossible to infer without an “observable performance”,^[34] and one does not get an observable performance from “ability.” We do not judge the unseen. That is, one cannot assess an “ability,” whereas one can write prompts that extend the description of a competence demanding that a student “identify,” “categorize,” “differentiate,” “design,” “disaggregate,” “reformulate,” “evaluate,” etc.

“Ability” also gets tangled up in the intent of a qualification document. Are the stated outcomes for students subjunctive/conditional or indicative/descriptive? That is, are they something we hope would happen or might happen or something that actually happens? If our learning outcome statements are introduced with a conditional, as in “the student should have developed,” the next word or phrase will be nominal, and the writers are sorely tempted to bridge the conditional state with content by the use of “the ability to” or “the capacity to.” I take the Australian perspective on this issue: a qualifications statement is not a subjunctive or conditional in any language; it is not a wish list; it is not a set of hopes.

Why one should tell students that they will “develop” conditional “abilities” that never might be realized instead of telling them directly what they will do in the course of a program is a diversion of the first order. In its first draft of a Tuning statement, the Indiana History Group of the Tuning USA project took the “ability” language governance in directions that, when read slowly, also lead into brick walls, e.g.

Ability to acquire understanding…

Ability to become familiar with…

Ability to gain experience…^[35]

And the Minnesota Tuning USA “general competence” statements for both biology and graphic arts added the fog-filled, “Ability to show awareness…,” hardly a qualifying statement of anything.

Consider that “competence” of “ability to acquire understanding.” What is it that a student does to demonstrate “ability” here? The student does not demonstrate “ability”; the student demonstrates actuality! That said, one must ask, “What does a student do to ‘acquire’? What does a student do to ‘understand’? to ‘acquire understanding’?” “Acquire” is an interim process, a generalized activity that consists of locating, organizing, cataloguing, etc. Can anyone locate, organize, catalogue etc. “understanding?” And as for “understanding,” the verbs that describe what students do in “understanding” render the notion of “acquire” moot. That is, if one “understands X” one has already “acquired” X.

Some of our colleagues are cognizant of this current/future status dissonance in outcome statements, a dissonance in which “ability” lives. The ALTC’s standards statement for the Bachelor of Law degree, for example, asserts that “ability” (in this case, “to respond to ethical issues and exercise professional judgment”) “is not fully formed [at the point of graduation]… but should have commenced development”.^[36] Even then, one does not “develop” abilities or capacities, rather the concrete skills, cognitive operations, and behaviors that demonstrate incorporation. If that is so—and it is–then why use the phrase, “develop ability,”^[37] as an outcome for a degree qualification?

If only these qualifications and benchmark statements eschewed “able” or “ability,” and proceeded directly to the verbs that follow, the red flag would disappear. Another route around the knock-off “ability” while saving the idea is offered, with my slight emendations, by the QAA benchmarking statement in Physics. One could follow this model with “graduates have learned

— how to formulate… identify… use… present

— how to… model… approximate

— how to plan, execute, and report… analyse… evaluate… relate”^[38]

and save the day of clear communication with students, faculty, examiners, and employers.

Reflective faculty Tuning groups are not always ignorant of this problem. For example, in its first draft of bachelor’s degree competency statements, the Indiana Chemistry Group of the Tuning USA project offered 68 competences/learning outcome goals, all of which began with “ability to,” followed by the verb which indicated the true substance of the competency statement;^[39] in its final version, Indiana Chemistry contracted to 36 competency statements, none of which included “ability to” anywhere.^[40] All competences were thus illuminated. Similar reconstructions are evident in the revised Indiana Tunings of history and elementary education where, in both cases, the bulk of “ability” statements were supplanted by the verbal, “demonstrates.” These reconstructions illustrate what should happen during the re-writing tasks marked at the end of this essay.

2. The binds of “knowledge” and “understanding”

In the Bloom et al.’s Taxonomy,^[41] the concepts referenced by “knowledge” and “understanding” are linked under the cognitive domain, with knowledge describing recall, and “understanding,” supplanted by “comprehension.” “Comprehension”^[42] is conceived not as a single act, rather a sequence of translation, interpretation, and extrapolation, and at the same time that Bloom explicitly excludes “a single (unanlyzed) term such as ‘understanding’.”^[43] Yet “knowledge” and “understanding,” dominating the lead clauses of competency and learning outcome statements, are found in national qualification frameworks, discipline Tuning, and subject benchmarking statements. The terms are often presented without antecedents or objects, as if whatever they mean is intuitively known, and needs no details. Knowledge is presented as a matter of student possession, as in the German besitzen that introduces knowledge sections of the National Qualifications Framework,^[44] or the Australian Learning and Teaching Council’s simple assertion that “a student will have depth of knowledge in a particular disciplinary area.”^[45]

“Knowledge” and “understanding,” often used as synonyms in these documents, often appear with an introductory adjective indicating a general type of knowledge or understanding, e.g. broad, specialized, advanced, integrative, comprehensive, theoretical, practical. Again, it is assumed that everyone knows on their pulses what the resulting phrases, e.g. “advanced understanding,” mean. But these are opaque labels. One is tempted to ask, “So, what might ‘not so advanced’ mean?” Ellipsis has no place in competence and learning outcome statements.

Furthermore, properly speaking, and in English, “understanding” is not a synonym for “knowledge,” however much the two are related. “Understanding” is a cognitive process, one that brings into play enough operations (in addition to those specified in the Bloom et al taxonomy’s division between knowledge and comprehension) such as description, inference, testing, and visualization so as to add depth to the individual’s “knowledge” of facts, relationships, formulas, etc. Why not describe to students what they are doing in their heads instead of pushing it under the rug with “understand”?

While “understanding” is centered in individuals, hence students (in our case), and can be presented in verbal form, when we come to Tuning and subject benchmarking, “knowledge” is an irrevocable noun, and one which takes objects. The reference point templates of Tuning specify “knowledge” of a disciplinary “what.” That is where the lists begin: legal regulations, error analysis, transport phenomena, major wars, poetic forms, auditing principles, and on and on. This is where the nouns and adjectives and their clauses take over, with both allowance and expectation for considerable detail. There is nothing objectionable about this in competence or learning outcome statements—provided that (a) the governing verb goes beyond the fact of possession to one specifying presentation of some kind (e.g. “demonstrate,” “display,” “perform”) and (b) the disciplinary contents are specific enough as to leave no doubt that context is nursing, for example, and not economics. If nothing else will suffice, the ideal form of a knowledge competence/learning outcome would read, “The student acquires and demonstrates knowledge of X, Q, and M,” whether X, Q, and M are facts, ideas and theories, terminology, methods, processes, genres—or some combination of these. Otherwise, as Bloom and his colleagues pointed out, “knowledge” invokes simple remembering and recall, acts that are assumed in virtually all other cognitive dynamics, and, for students already enrolled in bachelor’s degree programs, hardly a stand-alone performance criterion that qualifies one for a degree.

Let us think about “knowledge” with some of the distinctions the economist Fritz Machlup put on the table for consideration. Is “knowledge” first, something one possesses or an action of the mind? In Machlup’s question, is it the “known” or “the knowing”?^[46] In our terms, is it a noun or a verb? Or, as the German higher education qualifications framework was configured in 2005, is it “knowledge” or Wissenserschließungen (“ways of demonstrating knowledge” in my loose translation)? This is an epistemology question. Because in order to answer, one must determine “the ways of knowing” or “ways of getting to know” (all the verbs that should be in qualification frameworks or Tuning learning outcome statements), as well at the types of what is possessed. One of the problems with our use of the word “knowledge” as something possessed, Machlup points out, is that we often wind up with logical tautologies and dead ends, such as “I have possession of what I posses” (not his example, but that is the form of the statement that results; a classic instance being the QAA program profile statement for accounting, “basic knowledge and understanding is characterised by knowledge of a topic…”^[47]).

What the simple use of verbs in competency statements does not resolve, if we accept Machlup’s analysis, is the necessity of communication. That is, as he says, knowledge is not really knowledge if it is contained solipsistically—within the knower. To be sure, that is part of Machlup’s economic analysis, since inventions and patents and art and, indeed, any new “knowledge” depend on more than one possessor, hence on various forms of communication, reaching out, implanting elsewhere. Particularly in higher education, even under the desired goal of autonomous behavior explicit in many qualifications frameworks, communication is essential. You don’t know anything unless it moves from one place to another, or unless some direction of communication is embedded in the ways you know. Put another way, no one knows what a student knows unless the competence statement includes a vehicle of expression.

3. The conundrum of “awareness”

Strictly speaking, “awareness” and its verbal form means consciousness, and, the typical respondent might say, “we do not teach university students how to be conscious.” True, and Bloom et al point out that, in the sense of consciousness, “awareness” is part of the affective domain, not the cognitive domain, where the bulk of university learning and instruction takes place. But when we bring the language of awareness/consciousness into the cognitive domain, as 10 of our 40 documents did, it should be in the sense of students’ awareness of their own behavior, hence is not really a competence. That is not the way “awareness” was invoked by the writers of our documents, though: instead, they created tautologies of “awareness.” How so? To say that students are identifying or analyzing or explicating or even just citing a subject, topic, theory, method, etc., it is a given that those students are already “conscious” of the substance they are addressing. “Awareness”—read ‘basic consciousness’—is not a separate competence, rather involved in all competences. On the other hand, as a meta-competence, this substrate of all competences becomes “after some time and repetition, automatic,” hence exists “at a low level.”^[48] If so, there is no reason to include it—noun or verb—in a qualifying competence or learning outcome statement. To put it in the gentle manner of the 2007 NUFFIC evaluation of existing Tuning language, aware/awareness amounts to “stating the obvious.”^[49]

4. “Communication”: putting the verb back in

Both degree qualifications frameworks and Tuning-type maps usually present communication in very generalized terms. Exceptions cite specific communication media (from oral to PowerPoint to “wikis, blogs, and podcasts”^[50]), or genres such as technical reports, laboratory notebooks, exhibit catalogues, program notes in music, architectural models, documentary videos, and extended discursive essays. Yes, these are all noun phrases, and ideally should be used in both qualifications framework statements and discipline-specific Tuning or benchmarking as examples of what students can expect in a program on the way to earning credentials. But below the surface of these examples lie two related principles through which we can puts the verb back in its desired controlling position for the core competency activities of communication: intentionality and locus or origin.

Fritz Machlup wrote long before digital technology, its devices, and apps opened up a universe of communication lines in five or six dimensions. Yet, as he pointed out, the difference between ordinary yapping and “communication” lies in “talking [and writing, and texting, and e-mailing and blogging] for a purpose” [italics mine], and for which every language presents a range of verbal options. In English, to pick a sample of Machlup’s sample: “reporting… warning… requesting… advising, persuading, directing… convincing, permitting, teaching… edifying, confirming, affirming, denying, misleading,… ”^[51] are all purposes. The point is not to use all of these in competency statements, rather, instead of a blank “communication,” tell us what kind and for what purpose. And, as Machlup suggests, think about what role the student is playing as the originator of communication: “transporter… transformer… processor… interpreter… analyzer… original creator,”^[52] to which we would add in our time to follow the bazaar model of knowledge creation, “contributor… editor…”^[53] Then put the noun in verbal form, e.g. “disseminate,” “debate,” “respond,” “negotiate,” and many other forms of communication, and we will have given shape to an otherwise vague command. In Levin’s Natural Language Processing taxonomy in the context of higher education, these are verbs of “transfer,” not verbs of different purposes or behaviors such as complaints or advising.^[54]

As for locus, the nature of communication actions (and their verbs) varies by setting. Group work settings (to which the mushy slogan of “teamwork” is applied) require the student to negotiate, feedback, interact, argue, and contribute. The QAA benchmark statement for Social Work offers a set of such communication activities that can be elicited and judged in the process of both field work and future professional life: “consult actively… liaising and negotiating across differences… challeng[ing] others when necessary”.^[55] These are different settings than group projects within university courses. But within the brackets of their study in universities, students display, disseminate, write, and speak in a variety of forms and media, with specifications in a learning outcome statement carried by nouns and noun phrases in apposition. The broad default term of “communicate” does not do justice to all these variations of place, its conditions and expectations. Students deserve more precise and concrete parameters for academic behavior.

4.1. Interim conclusion

Why spend time on these issues before an empirical account of the uses of verbs in statements of competence and learning outcomes? Because out of 1177 competency statements in our basic universe, 322 are either not governed by verbs at all (including those introduced by “ability”) or lead with such non-operational verbs. Put another way, and assisted by the insights of Natural Language Processing, there are verbs that describe “states” and verbs that describe “events,”^[56] to which subsequent research added a class of “indeterminate”^[57] (for example, from our universe of verbs, “refer,” “apply,” “report,” “describe,” and “indicate”). At the very least, the verbs that drive competence and learning outcome statements refer to events, not states. Thus, when one considers the verbs actually used by the writers of the 40 documents that are the sources for this analysis are considered in Section VI, most of these 322 could justifiably be removed from the universe under consideration on the grounds that they do not represent observable events.

VI. Gross Data

At its first cut, the 1177 competence/learning outcome statements in the universe under consideration included 1637 provisionally qualifying English verb cases. Which verbs do we observe most frequently? Table 1 provides the account. What can we say about this collection of English verbs used in statements of competence or learning outcomes by discipline-based working groups?

1. Overall, and with the exception of the problem verbs asterisked, it is a very credible collection. If students are the principal audience, they know what they are expected to do in their various educational activities in order to qualify for a bachelor’s/1st cycle degree in the field addressed. The fact that writers of competency and learning outcome statements in European Tuning projects, the UK’s Quality Assurance Agency, the Australian ALTC, and Tuning USA converged naturally on this set of verbs lends support to our theory of inter-subjective judgment with which this article concludes. As the German philosopher, Ludwig Wittgenstein, offered, Eine Sprache vorstellen heißt sich eine Lebensform vorstellen, “to imagine a language is to imagine a way of life,”^[58] and when the English language is applied in an environment of higher education competencies by writers spread more than half way around the world, we have a natural demonstration of what “a way of life” means in the world of higher education.

Table 1

Verbs used by all four source blocks at least a total of 10 times

	Number of uses
Demonstrate	155
Evaluate	75
Identify	74
Analyze	72
Apply	61
Understand*	60
Know/have knowledge*	54
Use/utilize	53
Explain	50
Recognize*	49
Communicate*	36
Describe	36
Present	34
Develop	29
Design	24
Interpret	24
Reflect*	24
Create	21
Manage	21
Synthesize	21
Solve	19
Formulate	19
Select	16
Research	15
Plan	15
Work*	15
Interpret	14
Assess	14
Collect	11

* After one removes the asterisked verbs (discussed below) plus those used only once by only one of the 40 sources, the original 1637 verb cases shrinks to 1414.

2. Some of the frequencies with which we find these verbs are the products of specific disciplines in each of the four collections. For example, there were six (6) engineering reports in our portfolio, a factor that accounts for the over-weighting of the verbs “solve” and “design.” Another—and exemplary—case in the related vein is that of the Australian Learning and Teaching Council’s Science Standards, a document that presented a competence map that could be applied to experimental sciences (the field used was chemistry) and the non-experimental fields (also known as “formal science”) encompassed by mathematics. While nearly all experimental sciences use and integrate mathematics, there is a distinct difference in the fields themselves, so the division is a perceptive one to begin with. The very prominent verbs in this presentation reinforce these two paths:

— Experimental science students: observe, measure, classify, test hypotheses, revise, modify, gather, synthesize, design, select, evaluate, record, and interpret.^[59] A very impressive range of verbs that, in fact, describe what chemistry or biopsychology or physics or geology students do.

— Formal science students: prove, model, formulate, extract, translate (from non-mathematical language to mathematical representations), reason through algorithms, infer, and calculate^[60] (well, these operations are common to experimental science, too, but in a supportive role).

The point, again, is that discipline drives language,^[61] and carefully constructed as it is, this Australian collection stands out from others, including those from Australia. With a much fuller list of discipline profile statements from all language sources, one could also determine the effects of comparative licensure (mandatory as in nursing, voluntary as in engineering, partial as in clinical psychology within psychology, and semi-professional in which licensure is not an issue, as in business administration) on the range and frequencies of outcome-driving verbs.

3. “Demonstrate” and “present,” together, account for a significant portion of verb use, pointing generically to what students do that allows faculty to judge whether they have attained competence in the specific factor at issue. They are often used, however, to cover either non-competences, e.g. “Demonstrates an ability to …,” or to walk around the idea of possession, e.g. “Demonstrates knowledge of …” The first of these cases remains a problem; the second does not, provided that the nature of the knowledge demonstrated is specific, e.g. “Demonstrates knowledge of major turning points in European history since 1700,” as opposed to generalized, e.g. “Demonstrates knowledge of European history.”

1. Putting a Microscope to the Data: 6 Problem Verb Types in Competence/Outcome Statements

All of the verbs indicated below were used in competence or learning outcome statements in the 40 documents examined for this study. Most should be avoided in such statements on the grounds that they represent states, not events of learning, or, at best, may inhabit an indeterminate space between those categories, but some can be used with appropriate additions. I urge only that writers of discipline-specific statements reflect on what these verbs refer to and represent, and whether the writers feel wholly comfortable that the verbs provide students with operational paths to higher education attainment.^[62] Writers in languages other than English should reflect in the same way, though with other syntactic and lexical contexts. As Levin pointedly notes, “ties between verb behavior [the way we use those words in relation to other parts of our written or spoken sentences and clauses] and verb meaning is not particular to English.”^[63]

1. Verbs that describe routine student activities in education or learning tasks in specific disciplines.^[64] These are neither competences nor outcomes in either subject-specific or generic contexts:

ask, conceptualize, consider, parse, practice, question, read, reason, think

2. Verbs used in the course of assignments or instructions given by faculty, and/or serve as behavioral commands. They are neither outcomes nor competences:

comply, consult, discuss, list, promote, propose, weigh

3. Verbs that refer to states of generalized consciousness, not actions that demonstrate competence in anything:

ascertain, aware, become familiar, recognize, realize

4. Verbs that lead to outcomes observable (if at all) only in uncertain futures or in the future action of others:

anticipate, coach, foresee, mobilize, motivate

5. Verbs that are basically statements of fact, not competence:

has/have, possess

6. General behaviors that are not connected to any cognitive action, let alone outcome:

act, work

Reviewing competence and outcome statements previously written and containing these verbs in lead roles, setting them aside, and rephrasing the statements with active, concrete (that is, “operational”) verbs that lead directly and logically to faculty-generated assignments and other assessments would produce a body of powerful and focused discipline-based parameters that would naturally increase the portability of credentials.

VII. Putting Common Language Together: How can our Working Verbs be Grouped?

Some 127 “working verbs,” i.e. those that describe what students do in the course of demonstrating their competences and learning were deployed across the 40 documents examined. Counting repetitions, there were 1291 cases of these 127 working verbs, but not all are worthy of our consideration (indeed, 21 of the 127 verbs were used only once by the writers of the documents under consideration). All of them are “operational,” that is, they logically lead to concrete assignments eliciting student behaviors that allow faculty to judge whether outcome specified had been attained—and to what degree it had been attained. That said, there are different strengths of “operational” verbs, depending on the burden they place on learning objects (nouns and noun phrases that specify the subjects of action), a feature of linguistic valency. For example, the default command, “demonstrate,” as the lead verb in competence and outcome statements is polyvalent, and using it allows a combination of generalized nouns to take on the burden to describing what the student does, e.g. “demonstrates specialized knowledge of…” followed by case nouns from the discipline’s core.

These verbs can be grouped in terms of 17 types of activities as set forth below, each of which constitutes a configuration of what Weinert calls “performance-specific concepts of cognitive competencies”.^[65] Keep in mind that there are many other constructive, operational verbs that could be included in these categories, but they were not used at all by the writers of the 40 documents at issue. These groups are not intended as a taxonomy, do not follow Benjamin Bloom’s taxonomic division of six (6) knowledge-based goals, and, I propose, are what Sadler would call “intersubjective.”^[66] That is—and I invite readers to respond (and hence prove the case)—I created these categories inductively, labeling them with gerunds that are not repeated in the set of infinitives under each category, by sorting and re-sorting the verbs used by Tuning and benchmarking writers into sets grounded in the notion of analogous activities while keeping in mind Levin’s observation (granted, based on a far more rigorous quantitative linguistic analysis) that “verbs in English and other languages fall into classes on the basis of shared components of meaning.”^[67] And I am convinced that other native speakers of English, poring over those 127 verbs with the question, “What, roughly, are these verbs asking students to do in the course of their learning?” would come up with similar configurations, similarly labeled, and without recourse to elaborate psychometric assessment.^[68]

There are no perfect matches here—or anywhere else in the world of competency statements. There is no implied rigid isolation; there is no implied hierarchy. There is no one given sequence cross-cutting all disciplines.^[69] Many of these verbs could fall in more than one cognitive action category. For examples: “identify” could be a case of delineating, “extract” might be considered an act of preparation, and aspects of what I call “formatting” are involved in analysis. Table 2 matches the 17 categories of verbs emerging from 40 Tuning and Tuning-type documents against over 200 categories Levin induced from a massive analysis of 3,024 verbs in ordinary language use, indicating the labels Levin used in her classification of the meanings and behaviors of these verbs.^[70] Again, there are no perfect matches and, depending on the behavior of these verbs, Levin invokes some of them in more than one category. That only five (5) of the 17 Working Verb Groups I propose do not find analogues in Levin’s account provides one leg of a triangular backing for the “intersubjective.”

All of these can be adopted, in different weightings, by the diversity of disciplines, hence fit neatly into degree profile construction, even as national codes, regulations and standards will affect the status of verbs in different fields. Accountants and chemists have different notions of “appraising”; nurses and economists hold very different objects of “manipulating,” and these contextualized verbs differ in parameters by national law (accounting) and national regulations (nursing). It is also obvious, for example, that “processing” verbs will not be common in profiles of the visual and performing arts, that “operating” verbs will be prominent in business profiles, that “deliberating” verbs will sprout in law programs, “inquiring” verbs in laboratory science, “making” verbs in engineering, “converging” verbs in human service fields ranging from social work to nursing, and that all fields will demand certifying, re-thinking, communicating, valuating, explicating, examining, combining, etc. verbs to different degrees. And these lists certainly can be opened up with verbs little used in contemporary competence discourse.

Table 2

Matching 17 Working Verb Groups to Levin’s Verb Classes

Working Verb Groups Empirical Content	Levin’s Analogous Verb Class (Page References)
A) Preparing (artifacts, materials, tools, texts) Access, acquire, collect, extract, gather, locate, obtain, retrieve, seek	Obtaining (142)
B) Delineating Categorize, characterize, classify, define, describe, determine, frame, identify, prioritize, specify	Characterizing (181)
C) Explicating Articulate, clarify, explain, illustrate, interpret, outline, translate	No comparable grouping found
D) Examining Analyze, compare, contrast, differentiate, distinguish, extract, formulate, map	Separating (165)
E) Inquiring Experiment, explore, hypothesize, investigate, research	Investigating (198)
F) Formatting Arrange, assemble, collate, organize, sort	Build Verbs (172)[71]
G) Combining Assimilate, consolidate, connect, integrate, link, synthesize, summarize	Amalgamating (160)[72]
H) Making Build, compose, construct, craft, create, design, develop, generate, model, shape, simulate	Create Verbs (175)
I) Utilizing Apply, carry out, conduct, demonstrate, employ, implement, perform, produce, show, use	Performance verbs (178)[73]
J) Operating (executive functions) Administer, control, coordinate, engage, lead, maintain, manage, navigate, optimize, plan, undertake	No comparable grouping found
K) Deliberating Argue, challenge, debate, defend, justify, resolve	No comparable grouping found.
L) Valuating Audit, appraise, assess, evaluate, judge	Verbs of Assessment (196)
M) Communicating Convey, display, disseminate, express, respond	No comparable grouping found
N) Converging (for group academic work) Collaborate, contribute, interact, negotiate, participate	“Correspond Verbs” (200)
O) Re-thinking Accommodate, adapt, adjust, improve, modify, refine, reflect, review	Change of state (244-245)
P) Certifying Cite, document, observe, record, reference, source (v)	No comparable grouping found.
Q) Processing Calculate, determine, estimate, manipulate, measure, solve, test	Measure verbs (272)

 

How and where does this collection of verb groupings depart yet intersect with the six dimensions of the Bloom taxonomy (knowledge, comprehension, application, analysis, synthesis, and evaluation)? I avoided the task since it would involve crossing gerunds with nouns, mixing a purposeful hierarchy with a purposefully random ordering, and confronting a rational theory structure with empirical chaos. This exploration relied on what discipline committees actually wrote, thus revealed warts and bumps of which the writers were evidently not conscious, and hence is valuable in its own right as a marker of current practice. The work of Levin and her successors on the field of natural language processing (NLP) is more concerned with other features of language dynamics,^[74] but is obviously very helpful in providing authoritative reference groups of verbs.

VIII. What do we do at the end of the day?

This exploration has strayed outside its language boundaries, but that is inevitable. Talk about words and sentence forms in any language leads to concepts.

Obviously, a considerable amount of work, thought, and negotiation went into the production of the 40 documents in the universe examined. Their forms, tones, and expression were conditioned by national traditions, formal instructions, guidelines, and the often unstated assumptions of writers about the nature of subject-specific qualification statements in higher education. Some of them are exemplary; some of them treat us to exemplary moments; some have demonstrated how reflection and revision have brought them closer to communicating accessible substance; some still need to follow those reconstructive examples.

Thus, consider all these documents and their relatives as Beta. They are not set in cement. A majority of their creators have a great deal of revisiting and re-writing yet to do, though the task should not be difficult. Reassemble the teams that wrote these statements of competencies and learning outcomes; add new members from disciplinary faculties (which will help address a long-standing critical mass problem in Tuning projects); put two editors from outside the academy at each disciplinary table, and command them all to reconstruct Tuning-type templates to reflect, simultaneously, the stuff of the disciplines and the operational learning actions students must take to master that stuff. Then run structured surveys of faculty and student concentrators in participating departments/faculties to validate the revised templates.

We are not writing discipline profiles and competence statements for yesterday’s or today’s students, but for tomorrow’s, and a tomorrow in which digital translating systems will likely be far more accurate and effective communication environments than currently. New languages will enter the world of higher education competence, benchmarking, and learning outcome statements, and will likely provide more alternatives than our current misused English lists. The old categories, the sloppy constructions, the default phrases of Orwell’s “blah” will not do for an environment that is changing the conditions and volume of information and information-tasks at warping speed. To leave students, employers, and societies writ large in the hammock of tired language formulations is simply not fair in light of this trajectory. We need more discipline in our language constructions, more reflection, more questioning of what our words mean, and more centrality for a new and genuine range of verbs that address the future, not the past. Many of the nouns and noun phrases that form a discipline’s reference points will change, as knowledges and processes do not stand still. No competence or learning outcome statement is forever. To put all this in a Kantian construction, we cannot leave these evolving intuitions in limbo without a vibrant and convincing library of forms. The 40 documents examined for this essay provide the beginnings of that library, but it should be obvious that we have more work to do. It is not that we are not involved with setting benchmarks or defining learning outcomes or marking acceptable threshold levels of performance, rather we need to rethink the way we are involved—and we can start with our verbs.

But the story does not end with the verb. Particularly in the disciplines, verbs lead to nominal reference points, the objects of what historians, nurses, chemists, civil engineers, and economists identify, analyze, create, extract, calculate, and combine. We have said it above with the authority of Kant in support: a verb-dominated statement of competence is incomplete, so we sit at our team tables and practice writing full sentences, beginning with forms (operational verbs leading logically to assignments and assessment prods) and the intuitions that reference components of the discipline profile. Yes, this all means retreading old ground, but we owe such trekking to our future students.

 

 

Appendix A

Documents Used for the Core of this Study

1. Australian Learning and Teaching Council, Academic Standards Statements for:

Accounting (2010), Architecture (2011), Creative and Performing Arts (2010), Education (2011), Engineering and ICT (2010), History (2010), Law (2010), Science (2011)

2. Tuning USA, final reports to the Lumina Foundation from:

Indiana Commission for Higher Education (Chemistry, Elementary Education, History, 2010); Midwest Higher Education Consortium (Marketing, 2013); Minnesota State Colleges (Biology and Graphic Arts, 2010); Texas Higher Education Coordinating Board (Chemical Engineering, Electrical Engineering, Industrial Engineering, Mechanical Engineering, all 2011); Utah Board of Regents (History and Physics, 2010)

3. Quality Assurance Agency (UK), Subject Benchmark Statements for:

Accounting (2007), Archaeology (2007), Biosciences (2007), Business (2007), Economics (2007), Engineering (2010), Geography (2007), Physics (2008), Psychology (2007), Social Work (2008)

4. Tuning Project/Tuning Subject Area Group, Reference Points for the Design and Delivery of Degree programs in:

Business (2009), Chemistry (2008), Earth Sciences (2009), Education (n.d.), European Studies (2008), Mathematics (n.d.), Nursing (2011), Occupational Therapy (2008), Physics (2008), Theology and Religious Studies (2012)

Bibliography

Adelman, Clifford, Peter T. Ewell, Paul L.Gaston, and Carol Geary Schneider. The Degree Qualifications Profile. Indianapolis: Lumina Foundation for Education, 2011.

     .“Competence, Technology, and Their Discontents: an Essay.” Inside Higher Ed (on-line), June 6, 2013.

     . The Bologna Process for U.S. Eyes: Re-learning Higher Education in the Age of Convergence. Washington, DC: Institute for Higher Education Policy, 2009.

Australian Learning and Teaching Council. Learning and Teaching Academic Standards Statement: Science. Chippendale, NSW, Australia, 2011.

     . Learning and Teaching Academic Standards Statement: Bachelor of Laws. Chippendale, NSW, Australia, 2010.

Biggs, John, and Kevin Collis. The Structure of Observed Learning Outcomes (SOLO) Taxonomy. New York: Academic Press, 1982.

Bloom, Benjamin S. et al. Taxonomy of Educational Objectives Book I: Cognitive Domain. New York: Longman, 1982.

Brent, Michael R. “Automatic Semantic Classification of Verbs from Their Syntactic Contexts: an Implemented Classifier for Stativity.” Paper presented to the Annual Meeting of the European Chapter of the Association for Computational Linguistics, 1991. acl.ldc.upenn.edu/E/E91/E91-1039.pdf.

Brigham Young University. “What Are Learning Outcomes?” https://deanofstudents.byu.edu/content/what-are-learning-outcomes. Received 1/10/ 2014.

Carson, John C. “Defining and Selecting Competencies: Historical RefIections on the Case of IQ.” In Defining and Selecting Key Competencies: Theoretical and Conceptual Foundations, edited by Dominique Simone Rychen and Laura H. Salganik, 33-44. Seattle, WA: Hogrefe and Huber, 2001.

De Bruin, Lucie et al. Competences in Education and Cross-Border Recognition: Evaluation of the Usefulness of Learning Outcomes and Competences for International Recognition. The Hague: NUFFIC, 2007.

Engineering Council [UK]. UK Standard for Professional Engineering Competence. London, UK, 2013.

Ewell, P.T. The Lumina Degree Qualifications Profile (DQP): Implications for Assessment. Washington, DC: Association of American Colleges and Universities, 2013.

Hochschulrektorenkonferenz, Kultusministerkonferenz und Bindesministerium für Bildung und Forschung 2005. Qualifkationsrahmen für Deutsche Hochschulabschlüsse. www.hrk.de/de/download/datien/QRfinal2005.pdf. Received 4/7/2007.

Indiana Commission for Higher Education. Tuning USA Final Report: the 2009 Indiana Pilot. Indianapolis, IN, 2010.

Kipper, Karin, Anna Kornonen, Neville Ryant, and Martha Palmer. “A Large-scale Classification of English Verbs.” Language Resources and Evaluation 42 (2008): 21-40.

Lakoff, George. Irregularity in Syntax. New York: Holt, Rinehard, and Winston, 1970.

Levin, Beth. English Verb Classes and Alternations: a Preliminary Investigation. Chicago: University of Chicago Press, 1993.

Lokhoff, Jenneke et al. A Tuning Guide to Formulating Degree Programme Profiles. CoRe2 Project. Bilbao, Groningen and The Hague: University of Deusto Press, 2010.

Lorin W. Anderson and David R. Kratwohl, eds. A Taxonomy for Learning, Teaching, and Assessing. New York: Longman, 2001.

Machlup, Fritz. The Production and Distribution of Knowledge in the United States. Princeton: Princeton University Press, 1962.

Midwest Higher Education Compact Cross-State Tuning Initiative. “Marketing & Psychology: Competencies + Silos.” Minneapolis, MN, 2013 Supplement.

Minnesota Office of Higher Education. “Final Report. Grant number 6142 Tuning Project.” St. Paul, MN, 2010.

Moon, Jennifer. The Modular and Programme Development Handbook. London: Kogan Page, 2002.

Quality Assurance Agency. Subject Benchmark Statement: Accounting. Gloucester, UK, 2007.

     .Subject Benchmark Statement: Engineering. Gloucester, UK, 2010.

     . Subject Benchmark Statement: Geography. Gloucester, UK, 2007.

     . Subject Benchmark Statement: Physics, Astronomy and Astrophysics. Gloucester, UK, 2008.

     . Subject Benchmark Statement: Social Work. Gloucester, UK, 2008.

Raymond, Eric S. The Cathedral and the Bazaar: Musings on Linux and Open Source by an Accidental Revolutionary. Sepastopol, CA: O’Reilly Media, 1999.

Rich Pell. “FrameNet, PropBank, VerbNet.” Power Point presentation. faculty.ist.unomaha.edu/ylierler. Received January 2, 2014.

Rychen, Dominique Simone, and Laura H. Salganik. “A Holistic Model of Competence.” In Key Competencies for a Successful Life and a Well-Functioning Society, edited by Dominique Simone Rychen and Laura Hersh Salganik, 41-62. Gottingen: Hogrefe & Huber, 2003.

Sadler, D. Royce. “Making Competent Judgments of Competence.” In Modeling and Measuring Competencies in Higher Education: Tasks and Challenges, edited by Sigrid Blömeke et al., 13-28. Rotterdam: Sense Publishers, 2013.

Shavelson, Richard J. “An Approach to Testing and Measuring Competence.” In Modeling and Measuring Competencies in Higher Education: Tasks and Challenges, edited by Sigrid Blömeke et al., 29-43. Rotterdam: Sense Publishers, 2013.

Tuning Educational Structures Experts’ Group. “Tuning Australia Pilot Project 2010-2011.” Groningen, March 2011.

Tuning Project. Reference Points for the Design and Delivery of Degree Programs in Occupational Therapy. Bilbao: University of Deusto Press, 2008.

Tuning Subject Area Group for Business and Management Education. Reference Points for the Design and Delivery of Degree Programs in Business. Bilbao: University of Deusto Press, 2008.

Tuning Subject Area Group for Nursing. Reference Points for the Design and Delivery of Degree Programmes in Nursing. Bilbao: University of Deusto Press, 2008.

University of Illinois. “Tips on Writing Learning Outcomes.” www.library.illnois.edu/infolit/learningoutcomes.html. Received 1/10/2014.

Weinert, Franz E. “Concept of Competence: a Conceptual Clarification.” In Defining and Selecting Key Competencies: Theoretical and Conceptual Foundations, edited by D. S. Rychen and L. H. Salganik, 45-65. Seattle: Hogrefe and Huber, 2001.

Wittgenstein, L. Philosophische Untersuchungen/Philosophical Investigations. New York: Macmillan, 1953.