Program development and evaluation

The SWOT is a unique opportunity to bring in students’ perceptions, as we want to make sure that we don't lose the emphasis on students and their learning experience. Besides students (our main stakeholders), feedback from other stakeholders will fully inform the revision of the PLO’s

• The SWOT not being a satisfaction survey, it is best to use a questionnaire with open-ended questions. These types of questions allow for respondents to express their opinions using their own interpretation of the questions, therefore enriching the information that is collected.
• Likert-types questions may in general be too limiting as they must be expressed in such a way as to clarify/zoom on a very particular aspect asked upon, with the risk of missing the opportunity to gather as much the opinions of the stakeholders. I other words, it may be a “hit or miss”.
• If you still wish to use Likert-types of questions, please let us know and we will be pleased to look into your request.

At most once: it depends on how the chosen theme course is designed.

There are two possible scenarios:

• Scenario #1: This course is in fact used as an instrument to test a new course, before assigning o it an official code. This may be the case for example when a new course is requested by a group of students or a professor, based on the idea that this course could attract students and offer added value to the program. So before assigning it an official code, the academic unit could use this instrument (i.e. course with selected topic) to test the hypothesis (that this course will have sufficient audience). In this case, it will not be necessary to include it in the curriculum analysis because it is in fact a testing instrument.

• Scenario #2: This course targets specific learning outcomes, e.g. communication or ethics, and the use of different topics serves to keep the content up to date as it would be more relevant to the expertise of the professor, the discipline of the students or current affairs. In this case, this course should be considered in the curriculum analysis because the academic unit aims at well-defined program learning outcomes, regardless of the topics covered in the course in question. As at each instance a different professor may be in charge, it would be recommended that the program leader completes the questionnaire for this course. Another equally valid option would be for a group of professors to convene and agree on the answers to the questionnaire.

• Here, the definition of the "Taught" dimension is considered in a more inclusive way. For example, during discussions between the thesis supervisor and the student, the supervisor provides explanations to guide the student in the research process. This can be considered an element of teaching.
• The feedback offered by the supervisor is used to guide the student, who then uses this guidance to refine his/her research work, hence another teaching element. Although this teaching activity is not offered as a traditional lecture in a large hall (for example), the objective remains the same.
• The "Practice" dimension is also covered, as the student submits interim reports which are reviewed and corrected by the supervisor. This exercise would be intended to give the student the opportunity to practice his or her research skills (e.g. academic ethics) before making a formal defense in front of a jury (which brings us to the "Evaluated" dimension).
• In general, all three components, namely 'Taught', 'Practice' and 'Evaluated' are covered by this type of learning activity. As at each instance a different professor may be in charge, the programme leader or a group of professors convening together could fill-in the form for this course to ensure more consistency.

• Our surveying tool is very generic in that sense that it has been designed for four-year program with 120 credits. For programs that are offered over one year (Master’s for example) with a reduced number of PLO’s, it is quite natural to expect each course to cover the lower bound (three PLO’s only).

• Please note that a course is defined by its (course) learning outcomes the indication of the level (I, R and A) required for students to reach for each of the indicated program learning outcomes. If the course learning outcomes are not the same - because the levels are different (in autumn and winter) - one can conclude that these courses are in fact different. More precisely: between the two course instances, one would expect that there may be some different material, students may do different exercises, different activities, which should naturally lead to different assessments. Our recommendation here would be to define a new course code for the second course to be able to distinguish it from the first.
• If it is not possible to define a new course code, we suggest to enter this course twice using the "section" part to differentiate well these two courses. In this way, the coverage of the programme learning outcomes by both instances is clearly defined.

• The guiding idea is to always align the course learning outcomes with the indicated PLOs (3 to 6 for a four-year program).
• Let us take an example: PLO #1 says that students should be able to "use linguistic conventions to interact confidently in the target language with ease and accuracy". This is reflected in the course learning outcomes. Noe, the professor has complete freedom to guide students using pedagogical methods that are appropriate but quite different from another professor. Although the pedagogical methods used by one professor may be different from another professor, the coverage of the target PLO’s and the level (I, R and A) at which they need to be covered by these courses cannot be different. Please remember that the course learning outcomes are the same for both courses!
• If the coverage of the PLO’s changes from one professor to another, we may lose consistency, i.e. you have students who have mastered (for example) PLO#1 but have never heard of PLO#4 (assuming that the course in question was supposed to cover PLO#1 and PLO#4), and this may need to be addressed by the academic unit to ensure consistency. Again, this is why the course learning outcomes should be the same for both courses!

• If we are talking about a programme with 15 PLO’s, the answer is negative. It is not natural to expect a course to address so many PLOs, for the simple reason that the PLO’s will be very "diluted".
• If we are talking about a programme with only 5 or 6 PLOs, it is always a question of balancing depth with breadth, which must be decided at the level of the academic unit. Since a course is offered in a limited time, covering more PLO’s (for example 6 PLO’s versus 4) would imply more dilution as the limited time is spread over a larger number of program learning outcomes.
• As a general rule, we expect each PLO to be covered at least once at each I, R and A levels, and preferably twice.

• First, let’s clarify that just because it is a Master’s or PhD programme, the level of coverage will not necessarily receive an (A) - advanced level. The 'I', 'R', and 'A' levels refer to the degrees conferred, so they are relative to the degree itself.
• In the mentioned example of a master's level course, there is a need to upgrade students who are coming from the undergraduate level. This will be the 'I' level. Then (remember the famous spiral from I to A to R), there is a need progress toward 'A' level, passing by an ‘R’ level. The ‘A’ level would correspond to the level of development required of graduates in the program (whether undergraduate or graduate).
• Specifically, for someone coming out of a master's degree, the 'A' level for a given PLO will be referring to the PLO expression. This highlights the role of action verbs and the context that needs to be expressed through the PLO’s (these topics are covered in the series of online courses on Brightspace), and it is expected that these PLO’s are different for each degree. For example, when we talk about PLO’s in Master’s and PhD, we would more use action verbs such as analyse, formulate, create, develop... On the other hand, for an undergraduate degree we may use more verbs like enumerate, demonstrate, explain, discuss, …. It is important to remember that the level of a PLO is defined by its action verb, its context and final purpose.
• This being said, if in a particular course more than one level is covered, it is the highest that needs to be mentioned.

• Indeed, given the need to use resources as efficiently as we possibly can, we propose that for a given programme, the following courses (please note that by “courses”, all instructional activities are meant) be included:

  • All compulsory courses.
  • A "representative" selection of optional courses.
    • This selection can be based on the following criteria: expectations/requirements of the programme (expressed by the PLOs), courses that are directly offered by the academic unit, availability of faculty for this activity, popularity of the courses among students, etc...
    • The number of courses would be the number of electives that students must take. For example, if students are required to take 3 courses out of 10 courses, the 3 courses that would best fit the above criteria would be chosen.
  • For elective courses, please follow the same procedure as for optional courses.

There are two cases:

• If the covered PLO is not concerned by the language of learning (for example, a PLO stated as “developing an algorithm for an industrial robot” does not seem to be affected by the language in which the student is learning to develop the algorithm), completing the questionnaire for only one of the two languages would be sufficient. This is the case indeed where the PLO is language agnostic.
• If the language of learning plays a role in the PLO itself, then the questionnaire should be taken for both courses (French and English courses). This is indeed the case where the PLO is language dependent.