Peer-Instruction Unveiled: Measuring Self-Assessment Skills and Learning Gains in a Large Flipped Learning Environment

This is new research that I will present at the SRHE New Researcher Conference 2014.
The students in the pictures are from my 1st Year Introductory Economics class. Thanks to all of them for helping making this poster so nice. The merit is all theirs.

HEA Enhancement Event – York Nov 2014

I attended this HEA event while being in full teaching mode. I had to run from a late afternoon lecture to the station, jump on a train, get a wee sleepover in York, and a late return journey home. Well well worth the effort! HEA might be a little more money constrained, but it can still count of enthusiastic promoters and discipline leads!

Besides, they have adopted me as part of their enlarged family you know? After being awarded my Teaching Development Grant last year I started to get more and more involved with their events. I presented at 3 of their conferences in one year, tweeted, spammed everybody with my fliers, learned, promoted, disseminated, developed…and made some great connections and friends!

Well, well, without further rambling, let’s talk about this great event.

  • First of all we had a fantastic keynote talk by Mike Neary (@mikeneary) on the Lincoln “Student as Producer” model, as well as on many others great Cooperative Education initiatives. The Students as Partners theme as at the centre of the agenda, but many other issues were raised in the Q&A session, including how can we align the UKPSF framework with the principles of good student and partners practice.
  • The first workshop I attended was led by Alex Buckley (@ajbtwit) and Camille Kandiko. (@cbkandiko). They presented the first results of the UKES 2014 (UK Engagement Survey) and discussed with us about issues connected with cognitive testing and development of the UKES questionnaire. I will definitely have to read the full report, and there is an interesting blog post as well. We also had some interesting conversations about how the data process flows within each individual institution, and it was fascinating to hear about different experiences across the sector. I need to think much more about what discussed in this workshop. More posts will follow.
  • The second workshop was led by the mighty Kathy Wright (@HEAEducation). Kathy tackled issues related to student feedback and NSS from a refreshing perspective, leading us to reflect on what can be done at an individual level (as module convenors, or leaders of specific units) to prevent bad surprises from occurring. She encouraged us to think about our teaching journey, sailing off with our crew, chasing for a treasure island full of great learning objectives. We explored what could hamper our journey, how we could avoid shipwrecks, and what could help us sailing safe and sound to destination. Again, more posts will follow, as I am planning to illustrate my own journey as I thought of it during the workshop.

This was a great event, which I would recommend to everybody who is passionate and committed to good practice in learning and teaching in HE. I tweeted away for the whole duration of the day and I posted a Storify report of tweets (#HEAenhancement).

Photo 12-11-2014 15 48 06 Photo 12-11-2014 15 47 53

If They Don’t Talk, Post-it!

This year I am in charge of delivering Support Sessions for the module I teach and convey (Introductory Economics). Support Sessions are non-compulsory ‘open’ office hours targeting students who feel that they are falling behind with the material. The sessions are timetabled and scheduled to take place in teaching venues. Everybody is welcome to attend and ask for clarifications on either the theory taught in Lectures, or the problems assigned in Seminars and Workshops. The set-up is clear, attendance to these sessions is quite good, but…

But, as usual, students are very reluctant to ask questions. At the beginning of every session I would switch the projector on, bring with me notes about everything covered in recent theoretical or applied teaching moments, and ask the fatidic question: “What would you like me to cover today?” Punctually, I would be met by silence and perplexity, if not embarrassment.

Argh, argh, Argh! What to do? Should I start picking on material that I think is difficult? Deliver a mini-lecture that covers core points? Dismiss the class in a huff telling students to come back next week with questions?

Nah, none of the above. I just have to accept the fact that my students do not feel comfortable enough to speak up in public and admit, in front of everybody, that they are experiencing difficulties with any part of the material taught. After all, would I be prepared to do the same? Would I walk in a class and tell the teacher, with all my peers listening to me, “I did not understand this”?

Well, I found the solutions to all my problems and I am a very happy lecturer now. I attended a very powerful workshop by Prof Phil Race at the Anglia Ruskin Assessment Fiesta a few weeks ago, and I got an excellent idea: Post-it!

Yes, that is right, Post-it is the way!
At the beginning of each session I distribute a sticky yellow Post-It slip to each student in the venue, and I ask them to write down what they would like me to cover in the Support Session, then I circulate 1 or 2 A4 sized blank sheets. Students stick all the Post-It slips on the A4 sheets. I like that they can see what other students have written: very often the same questions repeat over and over. Once the sheets are back with me, I quickly sort them on my desk. This allows me to address student queries in order, and build the session in a clear and organised way. Once the plan of action is in front of me, I start covering the issues raised, one by one, until completion. The additional benefit of this approach is that, once students see that I am addressing their specific concern, they gain confidence to raise their hands and ask for further clarifications on the points they are interested in.

At the beginning of each session I am flooded with Post-It notes now. At the end of the session students have gathered the information they needed, and we all live happy thereafter. Well…until the next Support Session!

TurningTekky! TurningTechnologies Conference

I attended the TurningTechnology Users Conference held in Manchester on 22 September 2014. My first time at this kind of events.

I have been tweeting a lot (as usual):

These resources were made available from TurningTechnologies:

I enjoyed the event overall and I would definitely recommend it. However, I think this conference would be even more beneficial to new comers (rather than “users”) to the world of Student Response Systems.

The keynote by Eric Mazur was excellent. (I would have not expected any less!).

He also suggested to link to resources on Peer Instruction techniques: the peer instruction network.

TurningTechnologies have other interesting online resources for Student Response Systems users, available from the following links:

Finally, of course there is my own resource and contact page, linked to the HEA Project that I am developing on fostering Academic Self-Efficacy with the aid of Student Response Systems.

Anglia Ruskin “Assessment Fiesta” – 17 Sep 2014

I recently delivered a workshop on methods to integrate Student Response Systems and VLEs for formative assessment and teaching evaluation at the Anglia Ruskin University “Assessment Fiesta”. It was a great opportunity to meet colleagues, listen to interesting presentations, and showcase my project on Academic Self-Efficacy and learning technologies (When Student Confidence Clicks). So here follows my account for the day and what I am most certainly taking home from this experience.

A link to the storify tweets collection for the day is here.

First of all, I listened to Kate Little (NUS) (@katelittle)
in conversation with Prof Sally Brown (@ProfSallyBrown) on issues related to assessment and feedback, its role in the NSS and I found out about the NUS Assessment Benchmarking Tool, which I want to use to check that my assessment strategy matches with NUS objectives.

  • The plan of NUS is taking up Student Unions to the same level of analysis of NSS scores adopted by HEIs.
  • Consistent and ‘authentic’ assessment are paramount values, whereas anonymity does not seem to be a big concern.
  • Setting up the scene on plagiarism by saying during inductions ‘not to do it’ does not seem the best approach.
  • Electronic feedback is ok, but students want to be able to become personal with the marker if need be.
  • Start early with administering assessment, embed task with your feedback to elicit further action and response to feedback.

I then went to a workshop by Phil Race (@RacePhil). My first time: very powerful. There was so much to take in! What I learnt…

  • One hasn’t learnt something unless she hasn’t spoken about it. Students need to verbalise. (And this is why teachers learn even more than students while they teach).
  • I learnt how to use post-its to generate quick and useful feedback within a group of people. (Apparently Phil is famous for this, but to me it was an epiphany).
  • An important question to ask (aside from what, when, where, and how) is ELSE?
    That creates more inquiry and stimulates deepening the analysis.
  • “Show students what quality is” says Royce Sadler.

There was a showcase of good practice made of short 7-minutes presentations:

  • I loved Debbie Holley’s (@debbieholley1) presentation on short videocasts she posts on her VLE, where she gets personal and re-assuring to the students. Really nice! She has a good blog with ideas as well.
  • I also liked Matthew East’s (@mdleast) presentation on group wikis developed and delivered via VLE from students to students. Excellent work and idea.

I delivered my own workshop, which was well received (chuffed!)

The final remarks (with the help of Sally Brown):

  • Students are better learners if they understand how learning occurs.
    (Sally Brown always cites education literature to students).
  • “Show them what quality is” says Royce Sadler.
  • Read Royce Sadler, Dylan William, and David Nicols (that is suggested by Debbie Holley) on assessment and feedback.

Ex-post entry:
Anglia Ruskin (Cambridge) had a presentation delivered by Prof Mantz Yorke, which pretty much relates to issues of student self-efficacy. I want to find out more about this. An introduction about his talk is here. (This was on 1 Oct 2014).

Technical Degrees? Deja vu!

Ed Milliband re-launches the plea for vocational training, newly re-branded as “technical degrees”. In principle I am all in favour. The British vocational training system is appalling and something needs to be done about it. A few concerns though:

  • To which extent this idea will be a true innovation, rather than re-heating the same old soup? I am slightly perplexed. The revision of the vocational system promoted through the 1990s did not lead to any good. NVQs deliver qualifications not recognised by the labour market and the return on investment of such qualifications is short-termed and very low. Continous revisions to them system cannot but do worse as the labour market becomes suspicious about degrees that are not known and are not tested long enough to provide any sort of information on the skills delivered.
  • The common denominator between Italy (my home country) and the UK is poor vocational training. Yet, the two education systems could not be more different. So, to which extent can the two countries say: “We want to import the German model”? The dual-system that works so well in Germany might not have an easy life in either Italy and the UK, if not adapted to their specific environments (and that is where the problems start!). Besides, the vocational system in Germany works very well because it is linked to a sound apprenticeship scheme, where all the stakeholders have a say. Is there fertile ground to do the same in the UK? Social dialogue in Italy and the UK is very different from social dialogue in Germany; the corporatist approach promoted in the German way to negotiate across social stakeholders is unknown territory in the Italian and Anglosaxon cultures.
  • The reform introduced to the HE and FE system in the UK from the 1990s transformed good polytechnics in bad universities. Now it seems that we want to reverse back praising the value of proper vocational education. To which extent will this be a return to polytechnics? In other words: who is going to produce these technical degrees?
  • What will be the implications for Universities? We (sadly) spent the past few years scrapping the idea of ‘knowledge’ to substitute that with the mission of delivering ‘skills’. So now we will have ‘skills’ delivered by university institutions and “technical degrees”. What will be the difference? Will this change generate more competition or less competition in the tertiary education sector?
  • Finally, my thoughts go to the poor employability officers scattered around HE institutions in the UK. How will they assess, design, and promote the ‘new’ skills generated by technical degrees? How will they market them and differentiate them from the regular degrees? (Good luck guys!)

There is no doubt that the British education system needs to address issues related to vocational training and unskilled unemployment. Perhaps, rather than adding new degrees to a newly re-reformed (and already distressed) HE system, we should find the courage to go back to the old: polytechnics that worked very well and produced skills relevant for the labour market.

Experiential Learning: I played the violin!

I recently attended the HEA Annual Conference in Birmigham. A great event overall, and I will begin my account of it talking about a great session I attended there on “Experiential Learning” delivered by the mighty Laura Ritchie.

When we entered the session venue, the room was already set with violins, cellos, violas and scores: intimidating to say the least. Very few of us had ever touched an instrument. (It took me 40 years to ever go near a violin!) So off we went, the session started with Laura asking us to lift our violin and hold it like the torch of the Statue of Liberty. This allowed us to turn it onto our shoulders and have it just in the correct position, without even knowing it. (Pedagogy Principle 1: when you lay out your teaching material you have to set it so that students will not even need to think about what is the “right” way of approaching it; it will be all orchestrated (pardon the pun) for them so that they can slide into the right practice seamlessly!) We started to get familiar with our instruments, plinging it, finding notes between giggles and hesitations. (Pedagogy Principle 2: learning should be fun and thrilling after all). After this first induction, we were ready to pick up our bows; here we went with another easy demonstration on how to hold the bow right as we were supposed to. After a little of practice, Laura asked us to take turns and observe each other to correct our techniques, helping each other out to fix problems. (Pedagogy Principle 3: peer-instruction anyone?) More was added to our set of skills (jumping across weeks of what constitutes  the syllabus for professional players). We were playing our first concerto, we had conductors, and we were asking the conductors what we needed out of them: tempo, prompting …or ‘breathe’ as they call it! (Pedagogy Principle 4: student feedback to their instructors, what do student need out of us? Are they able to articulate it?) We were asked to invent a short tune, and then look at somebody else in the room. The person we picked was supposed to replicate exactly what we did, and play it on her instrument. (Pedagogy Principle 5: peer-instruction and collaboration. Observe each other, replicate each other’s practice). Lots of emotions were going through my mind while I was struggling to catch the tune, mastering the technique, finding notes, holding the bow…and trying to make my violin producing a sound (for how atrocious it was). Laura remarked that we were struggling, but that we did not give up. By the end of the session we were able to play “Old MacDonald’s Farm”…all by ourselves! (Pedagogy Principle 6: It does not matter how small, you need to give your students an objective to achieve. You need to make sure that they constantly see a purpose to the effort they are putting in their learning. You need to allow them to see the light at the end of the tunnel, a little reward for the investment they make, a sense of reward that brings satisfaction ‘right here and right now’, not just at the end of the module, and not just at the end of their degree).

Laura’s session was a truly inspirational and humbling experience: I think I am a good teacher, and I used to think that I knew exactly how it feels being a student learning new material, drawing on my experience and my memories as a student. Yes, there is some truth in all this, and relying on our emotional intelligence is important (quite ironic that Alan Mortyboys, pivotal figure in the emotional intelligence literature, was sitting just next to me with his violin). The truth is that many moons have passed since I was a student, and I had to acknowledge that my emotional intelligence alone does not make up for the fact that I am no longer a spring-chicken in my teaching profession. I needed to be reminded how it feels to be a student struggling to grasp new material, something never learnt before, with peers all around me, who could judge me and make fun of me, but who could also help me and support my own learning. Thank you Laura for bringing me back to a more humble-self, you certainly left a mark.

 

Gender bias and fairness of treatment: a tale of demand and supply?

I recently attended a seminar on Unconscious Bias. The principle is simple yet always problematic: the evaluation of others is affected by our own stereotypes and attitudes, which we cannot see ourselves. So far so good…

The conversation kicked off on career progression, and continued with career progression in academia, and then again on gender bias. Ok, I started to get a little itch right then. Gender bias is indeed a huge problem in academia, and I strongly believe it should be addressed. But it is not the only source of inequality and unfair treatment. I would say that ethnicity, sexual orientation, or nationality, just to mention other three, also play an important (negative) role…yet they are widely ignored. Think, for instance, how much the UK has benefited from the brain drain from other countries (like mine, Italy). A large number of extraordinary researchers and teachers, yet: how many academics not born and bred in the United Kingdom are in leadership positions in our university system?

Ok, back to gender bias –do not get me wrong- I fully embrace the cause, even though I have some concerns to share about the debate around it. The seminar continued illustrating methods and techniques to monitor on career progression procedures. With a few interjections and some (quite angry) comments about the current situation from the audience (90% composed of female colleagues). What I struggle to understand is: where is the novelty? Every modern institution has Human Resources and Equality and Diversity policies to tackle gender discrimination. It is quite difficult to evaluate their impact, but they are in place. So the next question is: what is missing?

I will use the language that I am familiar with (as economist) to explain my point of view. Imagine that we are in the market for “fairness in the work environment”. There is a supply of fairness delivered by HR and E&D policy and procedures. This is complemented and informed by data, statistics, and all sort of interviews on experience of applicants to career progression. A lot of time and effort is spent on the supply side, but what about the demand side? Yeah, right: what about demand for fairness? Females (or other groups subject to discrimination) are not well represented in leadership position, but are they applying for these positions? (We can observe the number of those who applied, but not the number of those who got discouraged by the system and did not. This is what we call sample selection bias). The problem is that particular groups (e.g. females) have experienced discrimination for decades… centuries really! So, even if institutions are taking care of the supply side of fairness, the demand might be lagging behind because it takes time for discriminated groups to raise their own awareness and aspirations to come to a position of even demanding for fairness. Assessing this empirically it is not an easy task, but at least we should acknowledge it!

So, the problem is there: there is still under-representation for particular social groups. But can we clearly pinpoint whether that is a supply or a demand issue? Somebody might say: it does not matter, the problem is still there! The answer is, yes, the problem is there, but the policy you need to put in place to address it is completely different! I think we should never neglect the supply side; we should continue to monitor it and to improve it. But on the demand side we are truly lagging behind. I would like to see more training, mentoring and networking to raise awareness within social groups affected by discrimination. (What I often see, instead, is a lot of entrenchment and anger). It takes time and effort, but it is essential. Now, there are two reasons why institutions push on the supply side: (1) it is easier to tackle, (2) it prevents legal disputes and protects institutions from being sued. This is perfectly understandable. But how about people (e.g. females)? How can we incentivise pressure groups to abandon entrenched, angry positions to take real good care of the demand side? Who should do it? Hard to tell, but that is the direction to go now.

Is it assessment or anatomic dissection?

Despite being a ‘false beginner’ in teaching practice I had to re-train to gain my PGCert credits to obtain my HEA Fellow status over the past two years. (Not yet required in Scotland, where I worked prior to joining UEA). I attended several modules of our MA in Higher Education Practice here at UEA, and shared the journey with a number of colleagues from different schools and discipline. One of the most verbal and enthusiastic contributors teaches anatomy in the School of Medicine and spends a lot of time dissecting bodies to teach our future doctors how the human body system works. He is really a fun guy; I guess it takes a little sense of humour if you embrace such a challenging profession. (I did not use to be squeamish when younger, but I am now. I could not do it, I think).
Well, we have been talking a lot about assessment in our sessions, and we are just out from an intense examination marking period, so the association comes spontaneous: is assessment (summative assessment especially) an anatomic dissection exercise?
Many colleagues believe that good practice consists in indicating explicitly the number of marks awarded for each part of the questions asked to our students in every paper we give out to them. Marking guidelines become even more complex: “give three points if students write this…drop two points if students do not say that”. There are great advantages in doing so: (1) if you have a large number of colleagues involved in marking the same set of scripts you can guarantee fairness and uniformity, (2) you can help less experienced markers with very firm criteria, (3) you can point students to important passages of their examination papers.
Nice heart here, it works fine and pumps the blood beautifully. But lungs are not working so well and oxygen does not transfer to the blood. The gallbladder is inflamed, it should better be removed, we can do without it! So the heart is worth 5 points, lungs 4 points, gallbladder just 1 point. Ok…the patient would live, even with a few respiratory problems. Dissection session terminated, put down your knives clean up and move on to the next.
But the heart is not just a muscle, and the brain not just a set of synapses. Can we really ascertain what is more important? And what happens when we look at a human person all together, at her thoughts, emotions, interactions with other human beings? Along with anatomy our medical students learn about compassion, communication with the patient, personalised treatment and information. Are we giving all this to our students when we are marking?
Anonymous marking places a nice clean sheet over the body to dissect, covers modesty and contributes to guarantee that all bodies are seen as the same. But –my colleague confirms- not all human beings are the same. True anatomy wisdom can teach us about similarities as well as differences, and good doctors will have to deal with both: choosing treatments and interventions that are tailored to each individual case. On top of that, patients hate being anonymous to their doctors; they want to be recognised, and they want to be informed about their health status clearly, honestly, and personally.
I will stop now with the Medicine metaphor (my knowledge in the field really ends here). But I am wondering whether marking procedures could do better than being imprinted to anatomic dissection exercises. Assessing our students’ performance and abilities, as showcased by their scripts, should consist of much more than adding up scores gained across answers and sub-answers. How does the script look like in its entirety? How is the writing style? Is there internal coherence? What is our gut feeling as experienced markers? (And how can we help our junior colleagues to develop their own gut feeling?) When teaching Macroeconomics I tell students that the whole is more than the sum of its parts. (Microeconomists will excuse me here, I hope). I wish I could do the same when I mark scripts, and in fact, I do…whenever I can!

Pimp my Barrow…Pimp my Teaching

Last Saturday I came to campus (as usual) to catch up with some work and I found myself immersed in our students’ “Pimp my Barrow” end-of-exam-period celebration. As new comer, I was yet to get acquainted with this event, but I discovered it even has a Wikipedia entry! Groups of students decide on a theme, design and create their fancy dresses as well as make additions to their barrow around their theme. They race from Norwich city centre to UEA, where a big party explodes. After a few hours of banter, they storm back in town to collect money for charity and to continue on with the celebrations.

While standing on the balcony overlooking the campus square going insane, I was amazed by the ingenuity of our students, the originality of their costumes, and the wit of some of their ideas. At some point I caught myself shaking my head and thinking (perhaps with a little envy for the fun they were having) that if ever our students applied the same craft, originality, independence and thinking in their studies our job would be so much easier and fun. I must admit, I felt immediately ashamed of that thought…because what came to my mind next is that very often we are the ones to blame. If our students can do extraordinary things with a carton box, a little face paint, and some old charity-shop old clothes, can you imagine what they could do with the right learning tools, academic knowledge, and thinking freedom? if they are so great at pimping their barrows, we should be equally great at pimping our teaching. Image