Five-second tests

In the lead up to a recent software release, I had the chance to do some formal user experience testing with a small group of people who work on Digital projects all the time. So I was a bit nervous that they would find it a bit too obvious. But they liked the approach, and it proved to be a very productive morning in terms of collecting their feedback on the system. Here are two approaches that I found worked nicely together: Five-second tests followed by findability tests.

One of the testing techniques is called “Five-second tests” and the idea is that after seeing a screen for 5 seconds, we want to find out if users remember what they saw, understood the purpose, how they felt about it, and what they would expect to do next.

So I worked out a sequence of about a dozen screens in our test environment that reflected the main stages of the user journey. I gave them each a blank table to fill in and explained it with this example

Example

 
 

Page number	Elements Please write down all of the main elements you remember from the page you saw	Purpose What do you think this page was about? What benefit does this page give me?	Impressions What is your first impression? What grabbed your attention? What went through your head?	Next What might you click on next?
99	Purple boxes labelled F, E, D, C, B, A Title something about career levels Another title something about a career plan template	Should tell me what’s expected of someone at each career level	I tried to remember which career level I was. It would be nice if the system knew that.	I’d click on the box with my career level on it or the career plan template or both

 

We did it in a conference room, as the first section of a facilitated user experience testing workshop. Most of the users had never seen the screens at all before, and I gave virtually no explanation of what the system was for apart from it was “about careers”. The instructions I gave were: 

• Each screen is shown for 5 seconds, or in sections of 5 seconds if it scrolls.
• Each user then completes the columns in their impressions table on OneNote without discussion (in order to avoid groupthink).
• Jot down other ideas as they come to you, even if not answering the question.
• When the majority of the participants have finished writing, we’ll move on to the next screen.

 

After the workshop, I combined all the rows about the same screen into one table for analysis. Here are some of the things I noticed:

• Different people had completely different memories and understood the purpose differently on each screen. Some of that reflected their own interests (what they wanted to get out of it), and some of the comments revealed that their first behaviour instinct was to “play” with it by clicking on things that looked clickable and interesting to get an impression of what it could do before formulating a purpose for using it.
• One or two screen designs had a “marmite” effect – most users loved them or found them unremarkable, but a few hated the visual metaphor. I guess that’s always going to happen, but we’d always aim to have the unlikers in the minority. In fact this “marmite effect” delayed the earlier stages of the design because we simply couldn’t find a visual design that the handful of sponsoring stakeholders all liked, and could not proceed without them all basically “liking” it at some level.
• Another comment revealed that one user had his own internal schema of the way to think about careers which centred first on a personal skills claim, and since this user journey didn’t start from where he wanted to start, it wasn’t something he wanted to explore. This reinforced a design principle to allow users to explore from the perspective that makes sense to them. Of course we rarely guess all the perspectives people will come up with first time, and certain ways of exploring are impractical to implement.

I know there are some tools out there that try to automate 5-second tests. I wonder if any of my readers have tried any of those? Please do comment if you have.

Findability tests

The clever thing about starting with 5-sec tests is that the users work out for themselves the purpose of the site, and it leads very nicely to the next Findability tests – how would you go about finding what you’ve just seen, in the corporate environment? In a way the 5-sec tests created a half-remembered experience like you might get after watching an all-hands conference call, or seeing a corporate news mail that mentioned it. We encouraged them to use their laptops to attempt to find the pages from whatever starting points came to their mind, and asked them to record where they started and what they actually found (our real system wasn’t live at that point).

Findability tests were extremely helpful in creating an action plan for us to get links to the new tool from all the places that users might go looking for it. We also unearthed quite a few other sites about the same subject that we weren’t aware of, and it led to an action plan to bring those together.

Dumbledore’s pensieve – is there any other way to get knowledge out of an expert?

Dumbledore: “I use the Pensieve. One simply siphons the excess thoughts from one’s mind, pours them into the basin, and examines them at one’s leisure. It becomes easier to spot patterns and links, you understand, when they are in this form.” And of course, Harry Potter is then also able to see what Dumbledore knows.

There’s a wealth of hidden knowledge in the people we know and work with. But it’s trapped inside their skulls, and the responsibility for sharing it isn’t theirs alone. So short of invasive surgery, how do we get it out?

There are lots of good reasons for doing this – the problem is most apparent when people leave the business. It’s at this point that we suddenly realise that they knew a lot of stuff nobody else seems to know. So we either live with a sudden outbreak of ignorance or some poor soul has to scrabble around to pick up as much valuable knowledge as they can from the rubble.

But this is not to say that the best time to seek the knowledge in an expert’s head is just before they leave. It’s something we can do at any time, and doing it saves us from relying exclusively on the one person who knows how system X works, or how we deal with customer Y.

One name for this process is knowledge elicitation. Others call it knowledge acquisition. Most obviously, ask them to tell you everything they know. But it isn’t always as easy as that.

• It is not always easy to persuade an expert to take the initiative to share their knowledge by producing materials themselves – they are often very busy.
• Sometimes experts have forgotten the reasons for their judgements – they have become rules of thumb (heuristics, tacit knowledge).  It takes a questioning interaction for them to reflect and uncover the core of their expertise.
• Sometimes experts don’t realise how much they know (another aspect of tacit knowledge) so may assume too much of their audience.
• The topic may be broad and it may not be clear where to start, what questions to ask, or how to structure the knowledge.

There are many ways to go about elicitation, and to overcome the pitfalls, the crucial thing is to include an active process of getting the knowledge out of them. Here is a summary of the best suggestions I’ve gathered:

• Just ask them to tell you
• Interview them
• Facilitate a workshop
• Draw out their knowledge informally
• Observe them at work

Just ask them to tell you

Get the expert to either give you a presentation, or to write something about how they do what they do.

Simply asking about a particular subject is sometimes the easiest way to draw out knowledge. Ideally the requestor chooses a specific subject, shows personal interest in the subject, expresses confidence in the expert, agrees a timeline or deadline and chases it up. The requestor reads or listens, gives feedback and asks questions which prompt further clarification.  The request can come from

• someone senior (line manager, team leader or project manager)
• a peer (for example in the community of practice)
• a learner (e.g. someone new to the project)

The method the expert uses could be:

• An informal presentation/explanation/knowledge call from an expert (often a virtual meeting, though face-to-face works better)
• Masterclass – ask an expert to tell the story to a small group of a situation, what they did and why.  Participants ask questions.  The session is recorded and stored in an open-access repository. 
  • Short masterclasses can be incorporated into regular team meetings
• White papers/blogs – ask an expert to write short documents on aspects of their area of expertise. Some will use Word documents, others may use blogs. A few may wish to share via video. Some will prefer this to the masterclass format.  These can be
  • “How to” and “Why” information for reference
  • Creative ideas for discussion

Interview them

Interviews
can be as formal or as informal as you see fit. But make sure you know what you’re asking about and how you’re doing this asking. The phrasing of your questions is crucial.

Interviewing the expert puts more emphasis on the interviewer to ask the questions that draw out the knowledge.

• “Can I pick your brains” is a common opener for an informal interview.
• It is also common to open a question session via phone, instant message or email.  That may lead to a scheduled 1:1 meeting.
• Mentoring meetings are an opportunity for expert interviews
• Regular reviews with a line manager are an opportunity for an informal expert interview
• When a person is leaving the company, there is a formal exit interview, but don’t wait for that.  As soon as the departure is announced, it is important to prioritise the time of the expert to absorb their knowledge.

In one-to-one settings it can be useful to ask the following types of probes

• Why would you do that? (converts a statement into a rule). Use the five whys.  Keep asking until you get to the deepest reason why.
• How would you do that? (generates lower-order rules)
• When would you do that? Is it always the case? What could vary in the situation that would change the rule? (reveals how general the rule is and may draw out other rules)
• What alternatives are there?
• Can you tell me more about …?
• Who would you ask about…?  Knowing the network of people the expert refers to is an important part of the knowledge.
• Encourage them to draw diagrams (process flow, connected things, categories)

It is important for the interviewer not to constrain the conversation too much by forcing a line of questioning too quickly.  The process of reflection on the questions needs time.

• Start by ‘settling down’:
• Be QUIET! Be a GREAT listener. Don’t interrupt.
• Ask only open questions one at a time
• Listen/watch carefully for tacit actions and remarks and help the subject explore the logic behind these actions or comments and rediscover what led them to that selection, choice or decision, by asking appropriate questions.

Prepare the interview “lightly”. The interview should be semi-structured, not tightly planned. Try thinking about the core topic of your interview as a tree trunk, there may be many branches for you to explore, but you’ll always need to return to the trunk to continue your climb or you’ll end up out on a limb!

Be aware that an expert may not know why, and may seek to justify their work, and may give spurious justifications for perfectly valid decisions/rules.

The interviewer is responsible for writing up (or drawing up) the knowledge.  It is very helpful if the output is then reviewed by the expert, for correction and to add additional depth.   Ideally the knowledge is stored in a shared area so that others could find it via search.

Facilitate a workshop

It can be good to gather together one or more experts to create a knowledge output in a workshop format.  Most workshop techniques work best face to face.  A few can work in a virtual meeting session with careful adaptation and extra facilitation skill.

Where the topic is specific, and the question(s) are well understood, consider using

• After Action Reviews, or Lessons Learned workshops. 
• Fishbone diagrams.  Mindtools guide.  Allows analysis of cause and effect.
• Brainstorming
  MindTools guide. Participants contribute ideas freely without initial critique.  Research shows that more and better ideas are generated if the session includes time for individual brainstorming.
• Learning histories for complex events (MIT guide). Several participants share their perspectives against a consistent framework (the timeline).  A facilitator then brings these perspectives together.  The resultant report is then discussed by all participants in a subsequent call or workshop, adding new observations.
• Knowledge cafes. Wikipedia explanation. David Gurteen on benefits. Allocate each topic to a table with a host.  The host facilitates discussion and ideas are recorded on a paper tablecloth or clipboard.  After 15 mins, the group moves to the next table.
• Gallery walk. Each topic has a poster or stand around the room, with a host.  The host gives a brief presentation and then there is debate.  When the whistle blows, the groups circulate to the next stand. 

Sometimes it is not clear where to start to gather expertise on a broad topic.  In this case, consider starting with

• Top 10 things. Ask each group to list the top 10 considerations for repairing a fault, planning a project or whatever.
• Brown-paper exercise (creating and repeatedly sorting/categorising post-it notes) also known as Card Sorting (research guide)

The facilitator is responsible either to write up the output or to appoint a scribe before the workshop begins.  The scribe, often not an expert, usually helps create the charts and visuals in the workshop itself, then finishes them off afterwards. It can be helpful to ask the expert to be the scribe, especially for diagrams, but make sure they have time to tidy up the output afterwards.  A good discipline is for workshop output to be circulated for review within 2 days of the workshop.

Draw out their knowledge informally

Time spent informally with experts can lead to storytelling, debate and comparing notes.  Opportunities may occur at the watercooler or coffee machine, while eating lunch together, when staying away from home with colleagues on business, at the pub or in celebrations or other teambuilding events.  Any of these may be described as a “fireside chat”.

The knowledge shared in informal settings is often hard to document at the time, but be creative: use napkins, paper tablecloths, smartphone audio recordings (with consent) etc.  Good practice is for a learner to make notes shortly afterwards, and to ask the expert(s) to review their output.

Observe them at work

We can learn from experts by observing their actual work. This is sometimes called Protocol Analysis, and can be on-line or off-line.  It can be self-reported or (better) reported by someone else who is shadowing them.  It is important to observe more than one example. Beware that talking while working can interfere with the work itself.

What makes virtual teams tick?

Nancy Dixon has a very sensible blog post this week about the challenges of setting up and running a virtual team​. Nothing earth-shattering, but a compilation of pragmatic wisdom

• have a face to face kick-off
• keep teams small
• build a rhythm of regular meetings, agile style
• set up peer coaching meetings
• establish norms for communication 
• do face-to-face at least once a year
• take advantage of travel to add an extra day to work in the same place as another team member

She has some good links also to the research evidence

What would I add?

• Having a shared space like OneNote or Microsoft Teams or Slack reduces the burden of email
• 1:1 catch-up calls between members of the team with no particular agenda, recognising that it is normal and healthy to chat without a task

Most of you work in virtual teams, so what would you add to Nancy’s list? What are the ‘norms for communication’ that you think work?

Subscribe by email

Search all my blog posts

Why NASA encourages people to sleep at work

In an amazing breakthrough, scientists have discovered a revolutionary treatment that makes you live longer. It enhances your memory and makes you more creative. It makes you look more attractive. It keeps you slip and lowers food cravings. It protects you from cancer and dementia. It wards off colds and flu. It lowers your risk of heart attacks and stroke, not to mention diabetes. You’ll even feel happier, less depressed and less anxious. Are you interested?

Nothing about this fictitious advert is inaccurate. It isn’t describing a new wonder-drug, but rather the proven benefits of a full night of sleep. The evidence has been documented in 17000 scientific reports. And the treatment is free.

In the 1990s NASA refined the benefits of sleep for astronauts. They discovered that naps as short as 26 mins deliver huge benefits – 34% performance improvement and 50% increase in overall alertness. This research evidence is what led to the NASA nap culture at work right across the world – not just in space. Procter and Gamble and Goldman Sachs offer free sleep hygiene courses. Nike and Google have “nap pods” available in their buildings.

I’ve been reading Matthew Walker’s 2018 bestseller “Why we sleep” and it goes further than I’d ever heard before in explaining the brain science of sleep. Also here are a load of videos of the same guy talking about different aspects of sleep. I wanted to jot down some notes about just one of those benefits: the way sleep contributes to learning, and what that says about how people learn in relation to their work.

I’m interested to know if you have any personal stories of the benefits of changing your sleep patterns (especially if you have dared to sleep during work time), or if you have found other articles or videos that give further insights or get the message across in compelling ways.

How memories work

Maybe you’ve watched the Disney movie “Inside out” in which short-term memory orbs are moved while the child sleeps.

The memory orbs of Inside Out are sent through vacuum tubes down to ‘Long Term’, a library of endless shelves that hold Riley’s memories. From above, Long Term looks like the cerebral cortex, folded outer layers that make a mammal’s brain to resemble a walnut. Jellybean-like characters known as ‘Mind Workers’ pick memories off a shelf and thrown them into the ‘Memory Dump’, a deep chasm where the unwanted orbs go dull and the information they carry – such as old phone numbers and piano lessons – is soon forgotten. Turns out the movie-makers did their homework and this is a good illustration of what happens. (Read more in Forbes magazine)

The “Why we sleep” book explains much more about memory, particularly in Chapter 6, which I found quite compelling.

You’ll probably have heard the basics of circadian rhythms, REM and non-REM sleep before. If not, the book explains that in earlier chapters.

Sleep refreshes and creates capacity

I noticed “fried brains” twice recently at work. Once it was my brain trying to take in a load of new stuff, the other time was after I spent an hour teaching a colleague a complicated technique. Both times it was late afternoon. Both times someone got grumpy to the point of “can’t take any more”.

These are examples of the limited short term memory in the hippocampus getting full. Cramming more in risks losing information or getting memories muddled up. In controlled trials, having a nap relieved the saturation effect – just having a break did not. Looking deeper, it was non-REM sleep and in particular the electrical pattern called “sleep spindles” that seem to cause the restorative benefit.

So given that, why would we even consider intensive all-day lecture-style training? Shorter segments distributed with sleep in between are going to create capacity to take more in, and be much more effective.

Have I ever been to sleep at work? Not often, but I had that awful flu-like illness in January, and decided to keep things going by working from home when I could. Several times I was exhausted and had a nap during the day. I noticed how much more capacity I seemed to have in the late afternoon when I got back to work than normal for that time of day.

Sleep presses “Save”

The simplest effect compares memory after a number of hours. If that time includes non-REM sleep, 20-40% more is remembered.

Unexpectedly the material is remembered better after sleep than before it – it’s as if the brain repairs the memories. And the neuroscientists can show that the memory is being retrieved from the long term store in the cortex rather than the short term store in the hippocampus.

This also applies to motor skills.

Tantalisingly he describes some new research about enhancing memory by electrically stimulating the brain during sleep, by rocking the body, and by playing associated sounds. Who knows what those sleep pods in our workplaces might include in future!

Sleep to forget what’s not relevant

It was fascinating to read some new research in which people were told some information to remember and other information to forget. Before sleep both were remembered much the same. After sleep, the memories were much less strong for stuff they needed to forget. Sleep is doing a cleanse and selects what to remember. The “Memory dump” in the film isn’t too far from the truth.

In the workplace we are all flooded with an information overload. Attention to detail is absolutely necessary in order to do a good job – or for an IT system or network to work for people – which is our business after all. But sometimes after diving deeply into the detail it’s hard to see the big picture again. There’s something beneficial about breaking off from the detail and taking another look at how everything fits together after a night’s sleep.

Sleep for creative problem solving

I guess I knew that sleep helps with creative design work and technical problem solving. I actively use this effect by priming my mind with the landscape of the problem towards the end of the day and then just leaving it, before picking it up first thing in the morning (often before breakfast). Just after sleep is a very creative time, indeed I sometimes have to scribble the ideas on paper because I couldn’t type them fast enough on the computer and waiting for computer updates totally spoils the flow. As expected, the book confirms that this is REM sleep at work.

What’s your experience of sleep and work? How can we use these insights when going about our work, particularly when it involves learning something new?

Leadersmithing: get better at leadership by building your muscle-memory

Is leadership something you are born with, or something you can learn?

Eve Poole says we can all get better at leadership and her new book Leadersmithing takes a very robustly-researched view of the learning process by which we become better leaders.

In fact the first third of the book is about this theory of how we learn: Insights from neuroscience, muscle memory, self-regulation, reflective judgement and learning to learn all play a part in making us better leaders.

She says that leadership is about what leaders do when faced with a set of “critical incidents” – challenging situations that are everyone faces. And crucially, it is possible to practice our responses to those critical incidents by seizing the opportunities in everyday life on small things, so we are prepared for bigger things – not just by having theory in our heads, but by having patterns imprinted in our “fast brain”, the amydala. Which she calls “muscle-memory”.

70% of learning is from doing

In my company we often talk about the 70:20:10 model in which 70% of our actual learning is through direct personal experience of wrestling with something new. Learning to drive is the classic example to illustrate this idea: 70% of your ability to drive comes by doing some driving, 20% from the driving instructor and 10% from reading the theory books.

Poole has added several new dimensions to this way of learning for me:

• The idea that it is possible to practise tricky interpersonal leadership situations by deliberately setting them up. You “do not want to sit still for the next decade, waiting to get mugged by experience in order to become more job-ready” Instead “like a vaccine – the right kind of experience in the right dose to equip you to prevail when ‘experience’ comes knocking. You can curate this kind of learning for yourself.”
• Recognising the role of the “fast brain” gut reaction and the emotional content of pressured situations, and practising that too. “From what we know about neurobiology, the more emotionally charged the situation is in which these skills are acquired, the deeper the resulting memory and its retrievability under pressure in the future.”
• Adding household and family situations as a context for leadership learning – as well as job shadowing, acting up, volunteering and the like.

Eve’s TEDx talk

If you want to get a preview I recommend her excellent TEDx talk (embedded below). In 18 mins she covers just a couple of very personal stories from the book – but the book is so rich with ideas you’ll want to read it as well.

The 17 critical incidents for leadership learning

Eve Poole supports her ideas from years of observing how leaders actually learn at Ashridge business school “the vast majority cited specific critical on-the-job incidents that had taught them what they needed to know”. Here is her distillation.

1. Stepping up
2. Taking key decisions
3. Coping with increasing change
4. Managing ambiguity
5. Taking a risk
6. Accepting when you get it wrong
7. Key board/stakeholder meeting
8. Doing the maths
9. Joining the dots
10. Motivating and influencing others
11. Flexing style
12. Delegating to and empowering staff
13. Dealing with poor performance
14. Listening to staff
15. Knowing when to seek help and advice
16. Giving and taking feedback
17. Work–life balance

And then an example of the practicalities she describes for No1: “How do you practise stepping up beforehand? By finding small spheres in which you reign supreme, so you can test your mettle when the buck stops, even on something that seems minor. Stick your neck out organizing your team away day or the family holiday. Chair a committee or a task group. Deliberately draw fire, or take the blame, or stand by your decisions. All of these flex the muscles you need to build your core strength for stepping up when the day comes.”

I’m quite comfortable with some of the 17. Others not so. Now I’m becoming more aware of when these things crop up, and observe myself and how I respond to those situations.

I find myself remembering lots from this book and recommending it to people. Hope you find it useful too.

When they ask for help – don’t give it! (An enjoyable Barefoot experience)

I was using the new Barefoot presentation recently in a primary school near where I live in Leeds, and I really like it. A big improvement.

Near the end, when the teachers started exploring the resources on the barefootcas.org.uk site they decided to find stuff that was directly relevant to their class and their curriculum.

Jane, an older teacher, found the Scratch water cycle activity and said it was totally relevant to her geography curriculum – “that’s exactly what we teach”. She had never used Scratch before, but tried one of the debugging activities. It was really interesting to sit with her and watch her work out what to do. I didn’t help, just encouraged her for clicking things to see what would happen. Eventually she found the studio. Brow creased very hard trying to understand the loop construct, but she got it and was extremely pleased with herself for debugging her first programme. She said that the only help I gave her was to encourage her to persevere.

John, a younger teacher, called me over to help him with a solar system activity he had found. I suggested he just try clicking something, and so he called over their IT coordinator to tell him what to do. I stopped the IT coordinator from helping him, and suggested again that he just tried clicking things on screen, reassuring him that he couldn’t break anything. He was able to work out how to make some things happen for himself. And of course the penny dropped that “tinkering” and “debugging” are the whole point. It’s not about following instructions step by step, you have to explore.

So I really enjoyed that experience, and they did too. It was nice that the Barefoot team sent me some quotes from the feedback sheets

“I found this workshop very helpful and not as overwhelming as I anticipated. It was delivered in a very helpful way – thank you very much.”

“Really useful session. Great resources and really well explained/demonstrated. Thank you.”

“A superb session – lots of new ideas/resources. Excellent deliver of training.”

“Thank you for a great session!”

So if you aren’t already a Barefoot volunteer, I recommend it. Find out more at the barefootcas.org.uk site.

Of course the “When they ask for help – don’t give it” title is a provocative thing. Normally I would want everyone to give help freely and promptly, as I try to do. But sometimes it is better to take more of a coaching approach than a mentoring approach, especially when you are able to be supportive in other ways (encouraging, being there to help if they still can’t do it). The judgement call is when to help directly and when to coach, I guess.

The Hammock: People remember the first and last bit, and not much else

I had an enjoyable time chatting to Tim Riesterer live on air during a recent internal Academy masterclass on persuasive storytelling.

One of Tim’s main points was about the hammock effect, which he illustrated with this diagram. If you are trying to get a message across, people will usually pay attention at the beginning and the end, but much less in the middle.

“They will manage a social smile, make sure to nod and pretend they listen to you. But really, they won’t.”

In most presentations the meat is in the middle. That’s good for a hamburger, but not for a presentation.

And Tim’s solution is “start hot, end hot and get spiky in the middle”.

But since hearing his talk I’ve been wondering: what does that mean in practice? How can I actually use this insight when I’m trying to get my ideas across?

So I did a bit more research online, and here are some of the best bits I found. I’d be interested for other people to add their own reflections in the comments about how to make this practical.

Primacy and recency

First I wanted to check that the hammock is real and not fake news. It turns out there’s some well-known brain research behind Tim’s hammock diagram. The serial position effect is a cognitive bias with two main parts:

• Primacy bias: the things you present first, or the first impressions you give will be remembered best.
• Recency bias: the things you say near the end will also be remembered well.

Wojciech Mendyka puts it very well. Paying attention is literally that: it has a cost. Our brains are lazy.

This rapidbi webpage has a good summary of primacy and recency, and an excellent list of further reading on the effect.

Start hot

It’s easy to start presentations with comments about the technology not working properly, or a planned piece of corporate introductory material, or other irrelevant clutter. If people remember the first things you say best, then making the most of that opportunity is vital. Start hot.

Tim suggested covering first something that will be new to your audience (especially if that will destabilise their previous perceptions).

Here are the best of 12 other ideas I found

• Be contrarian
• A series of rhetorical questions
• A compelling soundbite
• A startling fact
• Connect with something in current affairs
• Use the word Imagine
• Arouse curiosity

And my own favourite is to tell a story with a point, using Shaun Callahan’s model, which I blogged about before: Storytelling and self-disclosure in a business blog.

End hot

If the last things people hear are the most remembered, then finishing well is important. End hot.

Tim suggested not ending a presentation with Q&A, which is quite radical. He quite rightly says that Q&A often goes off the main topic, so it is a great idea to leave people with the main thing you are talking about, rather than the random detail someone asked about. For the masterclass I persuaded him to try out that idea. We did some Q&A at several points during the talk, then very near the end, and then the intention was to leave him a few minutes to wrap up. Of course the Q&A overran a bit, which made it hard for Tim to squeeze everything in, but I think it was a better thing to finish on.

Another tip is to tell them something they will want to re-tell to someone else.

Get spiky in the middle

The idea of being “spiky in the middle” is to re-grab people’s attention frequently. But how can we do that in practice? What are the “grabbers”?

Tim suggested progressive reveal: don’t show the audience everything you are about to say, because then there is no element of surprise or anticipation, which is boring. So he suggested powerpoint animations to reveal one point at a time. On the other hand, I can remember listening to presenters who bored the socks off me by using progressive reveal on slide after slide of bullet points. These same people often refused to give their slides till after the event, even on asking, which made them seem “precious” and made it harder for me to take notes. One big reason this fails is that you need to grasp an overview of the subject area as an overview, not piecemeal, to get a mental picture of the subject landscape and then understand how the presenter’s solution fits in.

Rapidbi suggest using practice or other activities as a down-time segment rather than one continuous talk

Mendyka recommends making it unexpected. Ask a provocative question to challenge the status quo. Ditch powerpoint sometimes.

Facts and figures wake some people up.

In our Academy masterclasses we interrupt the main speaker with comments and questions to make it more of a conversation, knowing that when the voice changes, people’s attention is reset.

Your ideas

What are your ideas for this? How can we “start hot, end hot and get spiky in the middle”? All comments welcome below.

Storytelling and self-disclosure in a business blog

A while ago I did a video blog about my standing desk which attracted lots of interest and 28 comments on the internal version of this blog, including some nice humour and a real bit of two-way conversation as people asked questions and I answered. Today I’m reflecting on why it was popular, and what seems to engage people in general when I write these blog posts. Partly because I want to keep my own blogs interesting, popular and relevant, and partly because I want to see lots of other interesting and useful blogs getting written. I think there were two things that hit the sweet spot:

• Storytelling
• Self-disclosure

I’ve been influenced by the Anecdote blog so I’ll take two quotes from there and think out loud about how that particular blog of mine fit those criteria.

Storytelling: can you spot a business story?

In order to get better at telling stories, it is fun to think about what the elements of a story are, and then try to spot those elements. So before I analyse the standing desk blog, here’s an infographic from anecdote.com and then a little test for you to try.

Now for the little test: I enjoyed reading ten paragraph-long “stories” on The Story Test and trying to decide whether they really were stories or not, based on those criteria. As the authors say, you can only get better at telling stories if you can recognise how stories work.

And what about my standing desk blog? Does it fit these criteria?

• Time and place marker. Yes. My blog post starts “It’s more or less a year since…”
• Seeing and feeling something happening. Yes. I actually change my desk during the video. You literally see it on video. I hope you felt the effort as I pushed the heavy box up the stairs.
• People and dialogue. Maybe. It is a personal story, so you know it is about me, so I think that counts as a specific person. There’s no dialogue in the video, but it certainly starts in the Newsfeed afterwards.
• Something unanticipated happens. Yes. Standing desks are still sufficiently unusual to be unanticipated. Many people commented on the cardboard boxes, which is quirky and somewhat unexpected.
• Has a business point. Yes. It’s all about wellbeing in the workplace.

If you are still reading, I’d like to encourage you to do some storytelling in your own blog.

Self-disclosure

Here’s another quote from anecdote.com about the value of disclosing personal details in your stories.

In August last year I was running a session helping people find examples to illustrate the business benefit of their products. I was listening to one participant speaking:

“I’ve got an example… I’m going on holidays soon, and have been doing the planning using XYZ Software. It’s very useful in identifying all the tasks that need to be done and keeping organised…” She continued talking for several minutes, maintaining the same vague, ambiguous speech pattern.

When she finished, I asked if she remembered the four things that indicate if something is a story or not (time marker, specific events, characters and something unanticipated). She nodded.

I explained that by adding in a few details she could turn what she’d said into a story. For example, I asked, “where are you going on your holiday and when?” She instantly turned to me and said “that’s none of your business”.

And she was right. It was none of my business.

But her desire to keep things completely impersonal was a barrier to moving from assertion to example; from something vague to something her audience could picture and comprehend.

The desire to keep things impersonal is a significant barrier to telling stories and business.

Personally, I would think nothing of saying something like “I’m going on a camping holiday with my family to the south coast in March. We’re going for three weeks so we’ll need to take lots of gear; camping, fishing, boating… Especially in March, when the weather is less predictable. So, to keep track of everything, last week I’ve started using XYZ software. It’s been great helping me to…”

So one of the reasons I think my standing desk blog was so popular was because it included self-disclosure and those kind of personal details. And that carries a risk – we instinctively don’t want to give out personal information in a public setting, we fear judgement and criticism. Looking back at my standing desk posts

• I showed you the inside of my house and my workroom. And yes, I did tidy up before taking the video. I might even have brushed my hair. Interestingly I can only think of a tiny handful of other BT people who I’ve seen in their homeworking setting – and that was when using webcams in a Lync conference.
• I know that there is a BT culture of criticising photos of the workplace by pointing out health and safety risks. I took the risk of showing my non-standard, non-approved set-up, and the possible criticism it might attract.
• I’m also aware that standing desks are a target for mockery by my favourite comedians. You can see that I pre-empted that by including a couple of humorous photos in the blog posts.

Isn’t it interesting that I was relatively defensive in the way I did the self-disclosure, trying to avoid showing weakness? But actually, as I think about it, some of the self-disclosure that has been most memorable for me in my company has been disclosure of apparent weakness. A few have blogged publicly about their mental health challenges, which I find profound and admirable. A senior leader blogged publicly about his bad back, and that really struck a chord of human connection with me even though I have never met the guy. TED talks are full of self-disclosing stories. That’s food for thought.

So the learning here is that self-disclosure in storytelling, whether in blogs, public speaking or one-to-one, is an important way to build emotionally resonant connections with other people, and makes the message stick too.

So what do you think? Is storytelling as important as I am suggesting, or should we stick to the bullet points and statistics? Is self-disclosure worth it?

Thinking fast and slow about big purchasing decisions

Thinking fast and slow about ITTs

A colleague responded to a blog post I did the other day about Daniel Kahneman’s excellent psychology book “Thinking fast and slow”, and we got to talking about a very practical application of Kahneman’s theories in the way we make big purchasing decisions in our company. But actually this approach to decision-making applies to lots of other areas in work and in life: my son choosing a University, which product to buy on Amazon, where to go on holiday.

ITTs

Have you ever been involved in an Invitation To Tender (ITT) process? It could have been responding to an ITT (where your customer is considering your company as their supplier) or issuing an ITT (where you are doing a big purchase from another company). Either way, I wonder what your feelings were about the process?

I’ve responded to some pretty big ITTs in my time as a solutions architect, and my experience is that the process was cumbersome and it was rare that we could say we had done a really good job in responding. My brain could barely cope with the hundreds or thousands of questions asked. There was never enough time to do the job properly.

I’ve also been involved in evaluating ITT responses. Each one usually runs to hundreds of pages, and I was expected to digest all of these, scoring them on hundreds of points – sometimes for every question. It was daunting, caused major brain overload, and I wonder if good decisions came out of it.

To make it easier, the questions sometimes were in a spreadsheet with a “Full compliance, partial compliance, or non-compliance” score claimed by the supplier. Of course when responding, the goal was to interpret each question so you were seen to be as compliant as possible. The result was that each supplier claimed compliance with almost everything. With so much information it became hard to see the difference, and to see the reality behind the apparently compliant responses. How do you judge between them?

My colleague emailed me his reflections on a particular ITT process he went through, and how we could improve things by “Thinking fast and slow” about ITT responses. It’s very pertinent to me at the moment, because we’re just at the beginning of a lengthy ITT process.

Like me, he is a prolific blogger in BT, usually with somewhat more of a technology focus than me. With his permission, I’ve pasted some of what he said below (obscuring the supplier names and adding some pictures). A semi guest post. I think you’ll find it interesting. He says:

“The Current Problem

Making decisions about the outcome of an ITT is a vital process. Experience, at least in the IT space, is that the process is a cumbersome one, and fails to use the best evidence and practice about how people actually make decisions.

If we take once recent ITT – a well-run and orderly affair, with a good outcome – we can see a number of aspects:

1. The scoring mechanism has taken many working weeks of effort to complete, with each number revised and re-revised, weightings modified, and overall totals tallied;
2. It is likely that only two individuals have fully reviewed all the numbers in the scores;
3. The scoring has in fact been secondary to our judgements on the overall effect of the proposals to our ability to deliver improved service over the long term at reduced cost;
4. Scoring has not been able to properly reflect identified showstoppers;
5. Scoring has not been highlighted the real market and technical position of some of the suppliers.

Let’s look at Supplier A, for example. They offered us something that sits very well against what we asked for in terms of being a software-based hardware-agnostic platform. However it doesn’t work for us: it doesn’t support provision of what we need to lots of our type of server – a non-starter.

In contrast, Supplier B offered us a device that is nothing like what we asked for but highly competent and in practice could be deployed, and they were still in the development process with that device. Another couple of suppliers did well because of their capabilities in one technical area but the ITT in fact never asked for that.

It’s clear that we make decisions not using the scoring process itself but through other mechanisms and use the scoring process only as a cumbersome vehicle to express those decisions. We need to look at how we really do work.

How People Make Decisions

I’m going to lean very heavily on Professor Daniel Kahneman’s work. Kahneman is a Nobel laureate whose lifetime has been spent in studying how people actually make decisions and the mechanisms that can be used to help do this. “Thinking, Fast and Slow” (ISBN-10: 0141033576) contains the material I’ll be using.

There are many ideas in Kahneman’s work, with many aspects of thought covered in the text. I will argue here that there are two important ideas. The first is that thought and decision-making has two different systems (his System 1 and System 2, which correlate exactly to his Fast Thought and Slow Thought of the title), both of which have validity and both of which need to be used in good decision-making. Sometimes it’s impossible to use both, but knowing they both exist and something about how to make them work is a start.

Fundamentally people have two systems by which they think. System 1 works very rapidly and is unaware of many aspects of a decision – it is biased by what is front of mind (the “information availability” factor), does not understand statistics, can be tricked by expectation management and “anchoring”, but is very good at combining a complex set of factors and getting to a decision.

System 2 in contrast takes its time. It is able to weigh factors and (if it is aware of these aspects of thought) can make allowances for tricks, can apply statistical analysis, can go back over information and remember past experience and new factors; but is much less decisive than System 1.

I propose that we use System 1 and System 2 by marking ITT responses twice: once on first reading, and once after proper analysis. However the way we do the marking is critical, and depends on the second aspect of Kahneman’s work.

He shows, building on others’ work, that complex approaches to trying to classify an important decision simply doesn’t work as effectively as people think it does, and that there is another way to do things.

There is a great deal of content in the book and I would fail at trying to summarise it. But it boils down to this: trying to score every result for every question and apply a weighting to it is ineffective as a mechanism for decision-making.

In contrast, he puts forward an approach that has been widely used and proven. A good example here is the Apgar test, used for identifying health of a new-born baby. Doctor Virginia Apgar realised that there was no single simple way to judge whether a new-born was healthy or not. Some doctors didn’t really look at the baby at all, or only looked at one or two factors, while others applied complex tests that took time and made a simple urgent decision a complex and untimely process.

To counter this inconsistent and poor decision making, Apgar proposed that each child would be given a score on only five factors, which were latterly given the title “Activity, Pulse, Grimace, Appearance, and Respiration” (at once making her name immortal and her existence as a person largely forgotten.) This mechanism introduced a simple, consistent, timely and complete way to judge whether a new-born needed urgent intervention, observation, or is essentially healthy.

APGAR is used in an urgent scenario but Kahneman shows very similar characteristics in other much longer-running decision-making processes, such as investment fund management. Adding complexity slows us down, and makes our behaviour in making decisions more obscure.

This is all very interesting but what does it mean for an ITT process? We are certainly not going to boil down an ITT to five or eight requirements. We still need to get a full set of requirements defined. However my proposal is that we mark the responses based on a summary set of what is actually important to us – with the responses from an ITT being used (twice!) to give us the marks.

Many of these overall considerations will be consistent across ITTs:

• Can we Deploy it? [D]
• Is the solution Robust? [R]
• Can we Operate it? [O]
• Do we believe we can work with the suPplier? [P]
• Is the proposal Complete? [C]
• Will it Improve customer service? [I]
• Will it reduce our Costs? [C]
• Is the proposal a good Strategic fit? [S]

This is then the DROPCICS process. The dropkick is taken, judged, and then reviewed again in slow motion.

We can of course look at each of these as they are my initial take but my proposal is that we give each response a mark out of ten for each of these factors. However we need to consider each of the numbers in turn, not just the total: any solution that scores less than 5 on any mark is almost certainly unacceptable except perhaps as a niche player. The total may then be used to compare.

In our first review – the System 1 review – we simply read the response and assign the marks at the end. Those numbers are then fixed.

In our second, System 2, review we work through and attempt to formulate a response based on a more complete review. These numbers can be changed through the process. We would use the System 1 results as a focus for specific issues; the less-than-5 scores need to be validated as correct.

At the end of the process we primarily use the System 2 results, as this will have all the information learned throughout the process. However we will then go back to our System 1 results and look to identify the discrepancies. It may be that the differences are easily understood, but we need to be aware that our System 1 is proven to be a highly capable complex decision-making engine.

If we look at the recent ITT on this basis, many of the factors that were problems would come through very clearly. One of the solutions was undeployable (and inoperable, too, in fact). Another solution was incomplete and not a good strategic fit. So System 1 would have highlighted this quickly, given us a focus to check we’re right and let us move on to other questions. System 2 would then get us to a simple set of numbers based on real due diligence and judge the outcome openly without hiding the process in a thousand weighted numbers.

Summary Process

1. Identify the small number of factors we are going to mark upon.
2. Read each response and give a System 1 mark out of 10 for each of the factors.
3. Work through the process of understanding the detail, with clarifications, corrections and full detailed analysis, illuminated by System 1 marking, and come out with a System 2 mark.
4. Identify the reasons for discrepancies between the two sets of marks and agree a final mark for each, primarily based on System 2 but with System 1 factored in as appropriate.
5. Agree whether there is a reason why a response with a single low score should be left in the process.
6. Use the marks to agree a final outcome.”

So what do you think of the DROPCICS proposal? Is it workable? Have you ever done decision-making that way? Have you any other suggestions to improve how we deal with ITTs?

Cognitive bias catches out the guy who got the Nobel prize on cognitive bias

A couple of years ago I loved reading Daniel Kahneman’s bestselling “Thinking fast and slow“. He won a 2002 Nobel prize in Economics for his work in psychology, and the book is a really good survey of the evidence about our cognitive biases.

Or so I thought.

Recently a friend in academia posted on Facebook a link to an article in Retraction Watch in which Daniel admits “I placed too much faith in underpowered studies”. The issue is with his 12 citations of research in a chapter on “priming,” in which the memory of something is said to unconsciously influence a person’s behavior going forward.

The irony is heavy, that in Kahhneman’s own first published paper (in 1969) he warns about exactly this bias – the tendency of researchers to trust the results of small scale studies. In that chapter about priming, 11 of his 12 citations were underpowered. It doesn’t mean his theories are wrong, but it does mean they are less proven than it might appear from the confidence of his writing.

I admire him for publishing a retraction. Admitting you were wrong is hard to do.

So was the riddle of the surgeon unconscious bias or priming?

In another of my recent blogs the riddle of the surgeon most of us assumed that the surgeon was male. I wonder why that happened?

• It could have been an unconscious bias that being a surgeon is a man’s job. Where might that come from? 11.1% of consultant surgeons in England are female (up from 3% in 1991), so the majority of surgeons encountered in the memory of people doing the riddle will have been male.
• The wording of riddles, like puns, is designed to mislead us in subtle ways. The riddle is short but uses eight male-related words. Does that prime us to think in male terms?

“A father and son are involved in a serious car accident. The father dies instantly while the son is airlifted to hospital in a critical condition. When the son arrives at emergency, a surgeon walks in, looks at the patient, and says, ‘I can’t operate on this boy, he’s my son’. So, who is the surgeon?”

Two different explanations for why that riddle caught us out. How do we know which explanation is right?

The dilemma of experience that validates theories

So that gives me a dilemma about what to do with the theory of priming

• In attempting that riddle we have personally experienced something that seems to validate the theory of priming. It “feels” right (but there is at least one other plausible explanation).
• The examples, anecdotes and small-scale research evidence of priming in Kahneman’s book are compelling (but he now admits the studies are underpowered).
• In the popular media Derren Brown and hypnotists often talk about priming (but is that part of their misdirection?).

What do we make of this? In trying to understand the ways we deceive ourselves are we in fact sometimes deceiving ourselves with inadequate theories?

My conclusion at the moment is that the theory of priming is only an approximation, a simplified partial model of what is going on. A bit like the Bohr model of the atom (electrons like planets going round a nucleus). The Bohr model is a helpful simplification at GCSE level, but is actually disastrous because it predicts that all atoms are unstable, which is not true.

And there are other examples of false models. I’ve written before about ‘Learning styles’: a zombie theory that refuses to die? in academic and business worlds. Many books and articles have been written questioning the orthodox theories of economists – for example the rational actor, the “trickle-down effect” and austerity.

So the lesson I’ve taken is to hold models like this about human behaviour lightly. It is a possible way of explaining some things that go on in our minds, but it is not a complete answer, and might even be the wrong explanation some of the time.

Have you ever come across theories that seem attractive, plausible or useful, but in fact turn out to be half-truths?