External Quality Assurance

How to use Constructive Alignment (CoAl) for accreditation and external evaluation purposes?

Since all UM programmes need to be accredited by the NVAO (the Accreditation Organisation of the Netherlands and Flanders), a constructively aligned programme is crucial to guarantee and show the quality of your programme to internal and external stakeholders. In short, accreditation concerns the trajectory in which an accreditation authority awards (nationally) recognised degrees. These programmes are included in the official register of the relevant country (e.g., the CROHO in the Netherlands or the Higher Education Register in Flanders).

Accreditation relates to the assessment of the quality of the programme and focuses on learning outcomes. Currently, the NVAO assessment takes place via a “limited accreditation” or “extended accreditation” (see assessment frameworks). If the institution has applied for the institutional audit, the programmes are assessed through “limited accreditation”.

Constructive Alignment and Accreditation

In the four standards of the NVAO Dutch accreditation framework there are three central questions:

  • What does the programme aim for? (cf. Standard 1);
  • How is this realized by this programme? (cf. Standard 2);
  • Are the objectives achieved? (cf. Standard 3 and 4).

Looking at these central questions and looking at CoAl, similarities are recognizable within documents provided by external parties such as the Dutch Inspectorate of Education and on a more international level within e.g. the Organisation for Economic Co-operation and Development (OECD).

 

 

The Dutch Inspectorate of Education is responsible for the inspection and review of schools and educational institutions in the Netherlands. This Inspectorate has completed a report in which they want to encourage universities and universities of applied sciences (Hogescholen) to improve the quality of assessment. The inspection is of the opinion that institutions for higher education should give extra attention to the consistency in assessment, to the professional development of teachers and educational leaders, and to the organisational integration of assessment. Within this report, they mention the use and importance of CoAl.

The concept ‘alignment’ is also explicitly included in the flow chart of quality of assessment in higher education within the possible areas for improvement.

 

What does the programme aim for? (intended learning outcomes)

Standard 1.: The intended learning outcomes of the programme have been concretised with regard to content, level, and orientation; they meet international requirements.

As for level and orientation (bachelor’s or master’s), the ILOs fit into the Dutch qualifications framework. In addition, they have to tie in with the international perspective of the requirements currently set by the professional field and the discipline with regard to the contents of the programme. Insofar as is applicable, the ILOs are in accordance with relevant legislation and regulations.

How is this realized by this programme? (Teaching-learning environment)

Standard 2.:
 The curriculum, staff and programme-specific services and facilities enable the incoming students to achieve the ILOs.

Explanation: The contents and structure of the curriculum enable the students admitted to achieve the intended learning outcomes. The quality of the staff and of the programme-specific services and facilities is essential to that end.

 

Are the objectives achieved? (Assessment)

Standard 3.: The programme has an adequate assessment system in place. The tests and assessments are valid, reliable and transparent to the students. The programme’s examining board safeguards the quality of the interim and final tests administered.

Within different assessment types it should be indicated what they are supposed to assess and why they do so. In concrete terms, it means that it should be expressed how the assessment relates to the objectives of the course and to other assessment types which take place within the same course. This can be proved through e.g. an assessment scheme and/or an assessment matrix (see Example: Assessment matrix).

Are the intended learning outcomes achieved?

Standard 4.: The programme demonstrates that the intended learning outcomes are achieved.

The level achieved is demonstrated by interim and final tests, final projects and the performance of graduates in actual practice or in post-graduate programmes.

Following the principles of CoAl in the process of design of the educational programmes automatically prepares the programme for its (re-)accreditation. These principles can be visualised for accreditation and external evaluation purposes in different ways. However, this visualisation entails more than an overview. It reflects specific programme goals, measureable student learning outcomes, and a well-articulated plan for timely implementation, strategic data collection, and analysis. Findings should then be used to inform, confirm, and support programme level change and facilitate continuous programme level improvement. And then of course, we keep on learning. It is important that all these elements will be revised and reflected upon every year.

 

Consideration whenwriting self-evaluation reports

Following the principles of CoAl in the process of design of the educational programmes automatically prepares the programme for its (re-)accreditation. These principles can be e.g. visualised for accreditation and external evaluation purposes in different ways. Below you can find some examples of how the different Faculties at Maastricht University tackle this. But first there are some considerations that can be taken into account.
Every programme writes a critical (self-)reflection (Kritische Reflectie, KR), according to the NVAO-framework (for limited programme evaluation). The Review Committee (Evaluatiebureau) can provide guidelines, reviews, and writes the report. Based on this, the Board requests an accreditation at the NVAO. NVAO then does (not) accreditate or gives time for recovery and improvement.

  • Considerations related to Standard 1.
    ILOs per education unit/course must be finalized and the chosen method of assessment per education unit/course must be integrated into the learning objectives. This can be proved through e.g. an assessment scheme and/or an assessment matrix (see next tab).
  • Considerations related to Standard 2.
    How is this realized by this programme? (Teaching-learning environment)
    Standard 2: The curriculum, the teaching-learning environment and the quality of the teaching staff enable the incoming students to achieve the ILOs. Explanation: The contents and structure of the curriculum enable the students admitted to achieve the intended learning outcomes. The quality of the staff and of the programme-specific services and facilities is essential to that end.
  • Considerations related to Standard 3. and Standard 4.
    Within different assessment types it should be indicated what they are supposed to assess and why they do so. In concrete terms, it means that it should be expressed how the assessment relates to the objectives of the course and to other assessment types which take place within the same course. Furthermore, the function of assessment is of interest: is it intended to select, to classify, to place or to certify? Within summative assessment, at the end of a course, generally the certifying function will come into view in relation to the course objectives and, indirectly, also to the final qualifications of the programme. From the point of view of level monitoring, especially these kind of summative certifying assessments are of interest.
    It is also important that it can be proven that there is a clear relationship between learning outcomes and learning objectives (possibly also learning trajectories) and that all learning outcomes are adequately assessed (e.g. in terms of level of difficulty).

UM Examples

All UM faculties have inspirational practices related to how they (try to) assure CoAl and how they make CoAl e.g. visible for accreditation and external evaluation purposes. Examples from the Faculty of Law, FASoS and SBE can be found below.

 

Example: Faculty of Law

Faculty of Law
This is an example of an overview that is made per programme, linking the Dublin Descriptors to the final qualifications/end terms. The overview also shows which objectives occur and/or are assessed on a course level (only related to compulsory courses).

This is an example of an overview of assessment methods per programme and per year provided as appendix to the accreditation.

Example: Faculty of Arts and Social Sciences

Faculty of Arts and Social Sciences

Good practice external quality assurance: External Advisory Boards

One possible instrument for the periodic renewal of the curriculum and the updating of the final qualifications could be the establishment of the so-called ‘External Advisory Boards’ (EABs). These Boards are convened by the programme coordinator or the Faculty Board and are composed of alumni and of professionals employed or active in fields closely related to the aims of the educational programme. The idea behind these Boards is to create a network of ‘friends of the programme’ who could provide valuable input as to the future development of the programme, the mix of intended learning outcomes, and the positioning of the educational programme vis-à-vis its external environment. In this context, the EABs become part of the quality assurance circle, and provide input for the annual evaluation and revision of the curriculum, which informs the yearly revision of the Educational and Exam Regulations (the OER-cycle).


FASoS Case study of the External Advisory Board:

Composition and appointment procedure
At FASoS the members of the EABs are appointed for five years. The programme coordinators are asked to provide a list with a balanced (international) mix of both men and women, recent and older graduates, people from the public as well as the private sector and professionals from big companies as well as small ‘start-ups’. Shortlists are drawn up with approximately 14 members per Board. Upon approval by the programme coordinators and the Faculty Board, the members are invited to become an EAB member for a period of five years, during which they meet once a year in Maastricht. Apart from the EAB members the annual meeting is attended also by the Associate Dean of Education (Faculty Board member), the Chair of the respective Programme Committee, as well as the programme coordinator.

Content of the annual meetings and added value in terms of quality assurance
During the annual meetings the EAB members discuss the latest developments in the sector and the larger external environment of the programme, provide feedback about the final qualifications and the curriculum. During re-accreditation processes they provide input also for the self-evaluation reports.

In order to make sure that the EAB’s suggestions for improvement/change will actually be taken into account, minutes are taken during the meeting. These minutes are subsequently sent to the Programme Committee (PC) responsible for the educational programme. The PC discusses the recommendations from the EAB, and drafts a plan of action regarding how to implement (some of) the EAB suggestions. This action-plan is submitted to the programme coordinator who further implements the plan and reports back to the PC. The PC’s annual report reflects on and evaluates the improvements introduced by the programme coordinator, and charts out potential further lines of improvement. This evaluation report is placed on the agenda of the next annual EAB meeting, and the session starts with this overview, before proceeding to the latest developments. In this way the cycle is complete and feeds into the next yearly cycle leading to continuous attention to updating and improving the intended learning outcomes of the programme.

 

Example: School of Business and Economics

Click here for higher respolution of the table that is depicted above.

Assessment matrix

An assessment matrix counts two dimensions: the topics learned, and the levels of learning goals. If a course uses several assessment formats (written exam, paper, presentation, etc.), one can in principle design an assessment matrix for each component of the assessment portfolio. In the School of Business and Economics (SBE), where the final written exam is in most cases the major piece of assessment, we work with a form that splits the final exam into 8-12 topics and includes other assessments at a global level.

The other dimension of the matrix is the level of learning. That is, again in SBE, based on the four Dublin indicators:
1. cognitive, 2. academic attitude, 3. global citizenship, 4. interpersonal competences.
The first category, being the most important one for most written exams, is further subdivided into 1a. knowledge and understanding, 1b. application of knowledge and understanding, 1c. ability to make judgements, 1d. ability to communicate, 1e. and learning skills (note that this resembles closely Bloom’s hierarchy of proficiency levels).

Adding the two dimensions together results in the assessment matrix. The cells in the matrix are filled with questions in the exam or other assessment formats. The example provided below is from the first year SBE course Quantitative Methods. Since this is the first-course students encounter, it won’t surprise that many of the higher order learning goals are not covered.

From this matrix we read that beyond the final exam, assessment consists of three quizzes and a student project. Quizzes and exam are directed at the first two cognitive levels, knowledge and understanding and the application of it, whereas the student project aims to assess student’s verbal ability to communicate the outcomes of a statistical analysis. In general, there are additional formats of assessments, such as papers, presentations, and contributions to PBL sessions, that are better suited to assess these higher order proficiency levels, where the written exam typically focuses on the cognitive level.

Example of an assessment matrix