Sophie Tidman and Carolyn pick up where the event left off, exploring why evaluation so often feels heavy, awkward or quietly avoided. And what changes when you stop treating it as a final judgement and start using it as a way of noticing what’s really going on as work unfolds.
They talk about how what gets measured quietly shapes behaviour and why the most important shifts are often the hardest to name. And how developing better language and sharper patterns of noticing helps people make sense of those shifts, rather than losing them because they don’t fit existing measures.
Along the way, they draw on Sharon Varney’s work on leadership in complexity and the limits of looking only in the rear-view mirror. Erin Manning’s idea of the minor gesture, the subtle movements where change actually starts. And Nora Bateson’s notion of warm data, the relational, context-rich information that helps organisations make sense of themselves without pretending things are tidy.
Apple Podcasts
Spotify
Buzzsprout
Amazon MusicTranscript
Sophie: [00:00:00] Hello and welcome to the Mayvin Podcast. The episode today comes out of a recent Mayvin alumni webinar called Measuring in the Messiness, where we explored evaluation in organizations. We were moving away from thinking about it as a technical exercise, but as something human, political, often misunderstood and really central to how we learn in organizations.
So in this episode, myself, Sophie Tidman and Carolyn Norgate are picking that conversation back up. We talk about why evaluation often shuts learning down, how attention itself becomes an intervention and why noticing shifts in how people are being, not just what they're doing matters deeply. Enjoy.
Sophie: So last week I had a really interesting conversation [00:01:00] with one of the participants of our event on measuring in the messiness on evaluation, and she said it got her so excited about evaluation and she was doing some really exciting stuff and she'd been allowed, like in her organization, she'd got the leeway to do some really interesting stuff and I was so amazed.
Because I really didn't think I could ever get excited about evaluation, but the fact that I managed to get excited about evaluation for that event and help somebody else get excited about was really lovely feedback. Great. How do you feel about evaluation?
Carolyn: Well, I think it's become a bit of a loaded word when you talk about impact.
We often talk about working in the messiness, so we talk about measuring in the messiness as well, I think. Once you get into it, really quite easy to get excited about it. But it has a weight to it as a word that it's attached over the years.
So, maybe we just need to help kind of give it, you know, reclaim evaluation was as something that can be exciting for 2026.[00:02:00]
Sophie: Yep. I talked in the webinar about it often being. It feels very much like capital E evaluation. Yeah. Rather than a process, part of the conversation, a moment in time. So I think the mindset with which people do evaluation, so is this, is this a tool for, accountability mm-hmm.
And making everything stacks up and having the last word on whether this program or intervention was successful or failure. First of all, that's generally impossible given how quickly things move and things are always in flux and changing and the difficulty of measuring, the things that are very important.
And also it stops the learning because of that.
Carolyn: I think the way it's being weighted. It's when it's seen as the end of a linear process, like the last bit of the problem solving, you know, if you were framing something in that way rather than as an adaptive ongoing.
[00:03:00] Interesting, fascinating, but often quite messy process of change or process of learning or process of development. Mm-hmm. So I think, yeah, I really like that kind of, let's not thingify it, let's see it as an ongoing dynamic part of the process. And as an opportunity for learning. It's one of the ways of bringing reflection and reflexivity into any intervention or change process.
Sophie: Yeah. And one of our roles, isn't it as OD practitioners, is to help channel people's attention, energy, where's most useful? We solve often with very good questions or framing. So I think kind of actually being very proactive and talking about evaluation quite early on and going, well, what would actually be different?
What would, if this was wildly successful, what would be different in your lives? And that provokes such a different conversation. Amongst clients and really is part of the shift [00:04:00] that's part of the intervention itself. I think we were talking a lot about recently clients who've had really big shifts, participants in our programs, but found it very difficult to articulate it.
Like, oh, well, nothing really changed. And then they'll talk about something about how they approached an issue about how they were, what was the example you gave recently from, wearing responsibility Lighter.
Carolyn: Yeah. We often use that phrase. The work is serious, and of course you take the work seriously, but how do you hold yourself more lightly in it, so that you can move through it, with some ease. So I think we've been having that conversation and a learning set and towards, and this was probably, maybe that was in the sort of first or second learning set and by the third or fourth, which was the end of the cycle.
And we were doing a bit of, light evaluation. I didn't call it that with them, but it was me doing a bit of a quality check in terms of has [00:05:00] this process made a difference and helping them notice the difference it might have made to them. This conversation started around.
I dunno, really? The network's been good. Uh oh. Yes, the network's been good. I mean, it's a bit like the Monte Python sketch, you know, what's, what's this program ever done for us? But then, you know, each of them started thinking gradually about things they'd done, or were in the process of doing and learning how to do or be different.
The difference in how you are being versus what you are doing, is a harder to notice. I mean, our whole practice based learning methodology is about encouraging people to notice their being and not just their doing.
How can I, have greater sense of presence, get my voice heard, those are all sort of being kind of things as opposed to doing kind of things. So encouraging people to notice their being, I think is, is really critical. It links to me to something I think I said in that, that webinar and that was, it was for our alumni, wasn't it?
So we thought [00:06:00] yes, because the conversation had been so fruitful and generative and got people excited. We thought we'd do the podcast as well. But I remember. Quite often when I was an internal, I don't think I notice this as much now, but when I worked internally, there was often a lot of anxiety about evaluation at the start to processes, and needing really clear plans for how we were gonna do it. I'd be sort of doing some of our evaluation as I was going along, having those kind of conversations with people. And at the point at which I wanted to go back and engage some of the commissioning clients around it, I couldn't get any traction, even though there'd been all this anxiety at the start.
And I think my hypothesis as to why that is, is actually people have noticed the difference. I can't quite quantify it, but they know something's changed as a result of that change process or learning process or both. And so the anxieties all [00:07:00] gone. And I think that's, there's sometimes something around an evaluation process is a bit of a container for anxiety and a place to put it.
And I still, you know, my learning from that was, okay, how can I have some conversations with them that help them notice that difference rather than just moving quickly onto the next thing. So some of the learning comes out of it. Yes. We notice the capacity we're building, we can, you know, take an appreciative approach essentially to, okay, what's now working?
You said the difference you wanted a year ago was, and how are you seeing that and what's unexpected come up? What surprised you? What haven't we got?
Sophie: There's this quote from Alan Watts, I really love in relation to a lot of this practice based work, which is hurrying and delaying alike are ways of avoiding the present.
And it often feels that people delay the start of programs. How, um, evaluation is. Is often an avoidance of starting, like wanting more data. So somebody said, we say we need more [00:08:00] data.
What we need is collective courage to start.
Carolyn: Mm. Lovely.
Speaker: That, it's that delay in kind of like, but I need to know, I need to know it's gonna be successful, how it's gonna be successful.
Carolyn: Yeah, I think the notion of baselining comes in there in terms of the data we have at the start, and to be able to know it's different, and it's like, well, what's good enough? And what are the different ways of knowing? So there might be some, you know, a lot of our clients have annual staff surveys and pulse surveys, and so you can have some useful proxy data from those. It's easily accessible.
But also, what are the stories? What's an emblematic story that you're hearing right now, which is why we're doing this work. And what might you like the kind of archetypal story to be at the end of this?
And so it's, it's a different kind of data.
Sophie: Yeah. And it's, what gets measured, gets done.
Carolyn: Mm-hmm.
Sophie: And so sometimes we're focused on the visible and what Aaron Manon calls the kind of major gestures of change, like the kind of. [00:09:00] Sort of tangible metrics, but all the things that shift them are the intangible, the less visible.
So, you know, how can we language them and, and have a way of talking about them. So it becomes normal to talk about the relationships, the identities, the kind of, the information flows, the things that are less talked about, less languaged, because words make worlds, as we often say.
And then on the evaluation. Yeah. When it loses interest and people just want to move on. So we've got one program at the moment, haven't we? That some of our internal clients are very frustrated 'cause the program was very successful and it's been scaled up and they want to build on it, but they're being dragged off. 'Cause, 'cause it's like, oh, well this idea, this new idea, people want something new because it, it assuages is a bit of an anxiety that they're doing something rather than the same old thing.
Carolyn: Mm.
Sophie: But sometimes actually just holding them. Do you remember what this is about?
Carolyn: Mm. Yeah. And that it can still respond to the context a year later. [00:10:00] The context may now be slightly different. But so what do we need to tweak around it? But it's not necessarily, something that needs to be thrown out. Yes. The sort of urgency for novelty, yeah.
I think another, well, it's a bit like you, you know, Alan Watts quote, it's partly, by doing something new, it can take a bit longer to get it going. And, you know, there's more acknowledgement in novelty than there is in something continuing to work really well. And as humans, we all seek acknowledgement and, want to know we're making a difference which brings us back round to actually, hang on, this is making a difference.
Sophie: Yes.
Carolyn: So how can we have a conversation about. What your concerns are around the difference it's making. And again, that comes back to the helping people notice the differences it's making.
Sophie: Mm-hmm.
Carolyn: I really liked the phrase you used in the, um, webinar around sort of cold data sort of metrics, isolated metrics often to warm data.
So stories we talked about, relationships, you know, what's changing [00:11:00] feelings, you know, how does it feel around you? Yeah.
Sophie: Yeah, I think that's Nora Batson's kind of, way of thinking about it, which I really love. You know, her warm data labs, which are basically lots of talking around stories and kind of admit the ephemeral nature of them.
She talks about not even really trying to measure or track it, but actually those conversations being the shift.
Carolyn: Someone who was there at the session had a lovely, analogy where we were talking about warm and cold data, and she was saying, you know, if you think about a relationship, the kind of peak moments that people notice might be the big anniversaries.
Mm-hmm. You know, the wedding day itself, but actually it's the cups of tea around the table where, critical conversations can happen or, day-to-day conversations, but that's the stuff that makes the relationship. It's not the big Yes. That's where the quality comes.
The quality comes in all those small moments. Back to Manning's minor gesture. Yes. The major gesture of the 25th anniversary party or whatever it is, is a moment.
Sophie: Yeah. But [00:12:00] the minor gestures are all the little patterns that braille bring mm-hmm. Build up in the everyday.
Carolyn: Talked about politics of evaluating as well, didn't we?
The phrase, that I remember James, our colleague, introduced me to years ago now around who's looking, which sort of, when we bring a sort of action research, quality into thinking about evaluation, that asking yourself, asking. Your colleagues, clients, whoever you're working with, you know, who's looking.
'Cause it might not just be the person who's commissioned the work or who's sponsoring the work. It might be that there's some critical stakeholders that aren't obvious. And so what's the politics of what's happening around this work and what do we need to be aware of as we go into it?
There's always some small p politics at play. But where might you need to pay attention?
Sophie: Yes. And it's always political, isn't it? What, what data you choose, what you [00:13:00] choose not to use. You choose to, to make more visible, more prioritized for who you are. Always making choices there. So rather than pretending that this can be objective. Actually being critically subjective about it, and open about it as far as you can.
I always feel a little bit uncomfortable doing leadership programs where, the senior sponsor gets to decide what good, what a good leader is in their organization and who's, who's behaving well as a leader and who's not.
Because actually a lot of the time when we're doing leadership programs, it's the countercultural moves that are most needed to interrupt things that aren't usually welcomed.
Carolyn: Mm-hmm.
Sophie: It's not, you know, we're not teaching heroic leadership where you get patted on the back. We're often kind of helping people to follow their instincts to maybe be a little bit more subversive and, and add their difference.
Carolyn: It's slightly different to the who's looking point, but it's about what's the situation calling for, and sometimes what [00:14:00] the situation or the clients, the, you know, depending on where you're working, your customers, citizens, patients. You know, what's the situation calling for from you as a leader?
And that might not be the conventional move. And systemic patterns can get very stuck in organizations. So to be able to experiment with a different pattern, a different response, it's quite courageous. But that, yeah. To your point about the senior leader, their expectation of what good leadership looks like.
As their journey was to become that senior leader might not be what the journey needs to be for the people five years, 10 years behind them and or what the situation calls for right now.
Sophie: Yes. And it's very empowering, isn't it, kind of saying to participants, well, what do you need? What would success look like for you?
This is your, this is your community. What do you need from each other? It starts a whole different conversation about leadership in the organization.
Carolyn: [00:15:00] Yeah. And takes evaluation into a very different space in terms of, everyone being responsible for their own learning and their own outcomes.
If everyone takes responsibility for that, then, the system itself is doing its evaluation as it goes, and all of those people noticing together, that's where you might get a pattern shift.
Sophie: Hmm. We did. Part of what. Initiated this event was doing some interesting evaluation processes with clients. So one thing we've been trying this year is fish bowls. So a group of four or five people sit in chairs in the middle of, the room. Have a conversation, for example, about what, what is this telling us about leadership here, what, what we need as a community of leaders next and outside that circle, it's everybody else, including sometimes senior sponsors, listening into that conversation, really observing, with the opportunity to come, to come in and out of the circle so that that inner circle,
has some safety, has some [00:16:00] containment, but it also has some fluidity and movement. It's a really nice way of kind of getting a little bit more traction and direction of the conversation when you have quite a large group of people.
We'd got to the point people had unearthed quite a lot of data and then this was the kind of So what now what part of the conversation? They were sort of telling stories, part of that they were lodging it in their context and what was happening more widely. So it was quite a rich, contextually rich discussion.
And at the beginning we were like, people gonna be, people are gonna be honest here. People gonna be really open. Do they feel safe enough? 'cause this could feel quite. Exposing. Yeah. Yeah. Exposing. And it was surprising how, how open they were actually. So we did it in the alumni session as well online.
That was a little bit of a risk. But as we say, evaluating the session for us, like if we just run the same kind of thing with the same kind of questions, the same kind of people. It gets very boring for us and probably for others as well. So it's kind of [00:17:00] adds a bit of edginess for us to have something a bit different and I think it worked.
Yeah, people seem to feel the energy from that and the spontaneity that was, getting different voices in the room.
Carolyn: People knew that it was their opportunity to speak and to get into conversation. Enabling different kind of voices in a different kind of container for the conversation.
Sophie: Yeah. The other thing we haven't mentioned yet is this idea about looking upstream when evaluating, looking at the glimmers of what's beginning when things are all in shift.
Mm-hmm. All in flux. Sharon Vanney talks about this, doesn't she? In terms of the rear view mirror, when we're looking at lessons learned and evaluating, we're often looking in the rear view mirror, right? It's gone. It's never coming back. How do you
Carolyn: drive? Only looking in a rear view mirror. Yeah. Yeah. Badly.
Yeah.
Sophie: Um, and so, this kind of being really in the present to look at what's starting, what's becoming is a real skill.
There was this lovely quote accompany the things as they are in the process of [00:18:00] becoming, and I wonder what that, you know, as a kind of way of closing a program and thinking about what next, to your point earlier about the search for novelty.
Finding those new archetypal or emblematic stories that are gonna kind of motivate us and we're gonna move towards.
Carolyn: Yeah, I sometimes use that seasonal metaphor, what's in spring for you? Where are the new buds as just a way of people helping people, because this all comes back to noticing and you being in the here and now in your own embodied experience.
And that's not language that is really easy for a lot of people. This is very OD language that we're using. But maybe just using that metaphor of like, what's just wintered, what's either died off or, or, hibernated for now and what's in the, what's in the spring, where are the new shoots that you can see that that sometimes really helps people, I think, get a sense of the glimmer of the new, of what's starting to come [00:19:00] through.
So I think that artful approach can be really helpful sometimes in helping people get into their day to day.
This is where it absolutely is exciting, that opportunity to see this as a way of helping a system learn and grow, helping people learn and grow.
Sophie: And if you think, the faster the world is changing externally to us, the more continuously we have to sense and act and adapt. You know, if you think about, catching a ball, for example, somebody kind of moving on a sports field, they're continually moving, feeling their environment and kind of changing and acting and responding to that and then getting feedback.
So really we're talking about organizations becoming sensing and adapting better, just becoming learning organizations in that way. And I think we didn't talk about it in the event, but it's been been very much in my mind the kind of the impact of AI on that. Both in terms of, there's a lot of very interesting evaluations of, of AI interventions and it's quite frustrating actually 'cause people aren't seeing the [00:20:00] returns.
And actually it's changing how they think about how they evaluate AI. I think is really important about what they're trying to do, being really clear about what they're trying to do. What they're trying to measure.
Carolyn: Yeah.
Sophie: So I think a lot of this could be applied to AI interventions, but also the fact that ai, where it's going is providing a lot of information, actionable information towards the edges of organizations and could.
Potentially in the future become a whole nervous system for an organization. So really kind of having that information always to hand continually and, in a way that our role as humans is about the judgment, discerning sense making of that data.
Carolyn: And the feeling side of it, that kind of messy mushy, which is data sometimes harder. Yeah. Come back to data. How do we bring our human experience, into evaluation and not just, let our AI agents find all the cooler data.
Sophie: AI as a mirror will maybe just amplify the cold [00:21:00] data that we tend to prioritize anyway in evaluation. But when we use it as a mirror to kind of challenge ourselves a bit, a bit more of a trickster technology, then we can use it to highlight the things. Love that. Yeah. Highlight the things that we don't pay enough attention to.
Carolyn: Yeah. Well, blind spots. That could be really interesting. Yes, yes. It's right. Okay. I feel another podcast coming on about how we've been using AI in evaluation. But watch this space maybe.
Sophie: I'm sure it's not gonna be the last conversation. Anyway. Thank you very much
Carolyn: Carolyn. Thank you Sophie. And yeah, I hope everyone else is equally as excited about evaluation now.
Sophie: So that's our attempt to make evaluation a little less terrifying and possibly even interesting, maybe even exciting. If you are noticing a few [00:22:00] different things since listening, I would love to hear about it and we will take that as a win. Thanks for listening. See you soon.