Evaluating AI tools for research analysis

To coincide with the publication of our new web pages dedicated to helping you use AI and GenAI responsibly in your learning and research, we asked Kate Jones, a third year PhD student, to tell us more about her own use of AI and her views on it:

I haven’t used AI to write this blog post, but I did think about it. The technology is moving so fast, it’s hard to keep up with the possibilities of GenAI tools. But as researchers we have the skills we need to experiment with them work out together how best to make use of them.

We also need to be mindful of the pitfalls of GenAI. We know chatbots hallucinate – they make things up and create nonsense outputs, tend to over-simplify and don’t understand our prompts. As a parent I am of course well used to handling entities that do this. Prompting an AI effectively has commonalities with trying to get my children to put out the recycling (“no, turn the bin around so the lid faces you and you can open it”). Patience, clarity and repetition all play a part.

We also need to be mindful of data privacy, ethical considerations, and the cost and environmental footprint of GenAI. I’ve learnt a lot about these thanks to the Centre for Postdoctoral Development in Infrastructure, Cities and Energy (C-DICE) and their training events on AI for researchers run by the brilliant Naomi Tyrell at Research Your Way. Just as we prompt AI chatbots, so Naomi’s training has prompted me to give them a go.

Recently I’ve been assessing whether using a GenAI to support me to conduct qualitative coding and thematic analysis. My data consists of nearly 20 unstructured interviews – around 100,000 words. There is guidance available, but it mostly considers the impact of GenAI on what we write – an essay, a thesis, a paper – so the guidance didn’t fit. And I am not an expert – most of us aren’t! – so I have tried to apply my researcher skills to critically analyse whether it is right to use it.

Choose your GenAI carefully

I chose a GenAI that provides comparatively high privacy levels: in this case, Claude.ai, and there are others. Claude.ai has privacy settings on by default, complies with GDPR, and on the whole, doesn’t use the data that’s uploaded to train its models – except in some circumstances.

My rule of thumb for Chatbots – stay away from the thumbs up. Providing feedback can result in your conversation being used to train the AI model, even if sharing is turned off.

Experiment to evaluate its capabilities

To evaluate the GenAI’s ability, I ran a small-scale experiment with the free version. I uploaded two interview excerpts I had already coded manually, and my coding framework. I prompted Claude.ai to create an ‘artifact’ showing me the coding visualised as highlighted text with comments in the margin, and then provided prompts to improve the process. I compared Claude’s coding to mine, then I asked it to provide insights about inter-coding agreement; which was moderate to good. It suggested how we could improve and provided insights about my coding style.

Make sure your data is ‘AI-ready’

For me, this is critical to comply with ethics and integrity standards, GDPR and for my own peace of mind. For my data, it involved not only removing names and other identifiers like job titles, but removing organisational names, locations, named projects or departments, the date and time of the interview, and participant numbers. I also removed references to sensitive, unpublished or proprietary data, projects, products and types of work. This is a manual process, requiring human judgement, and has also helped me familiarise myself with my data so I can better judge when GenAI outputs don’t seem quite right.

Use GenAI to enhance your research capabilities

I prompted Claude.ai to ask questions about my framework, to ensure understanding and improve coding processes. Claude.ai offered me several different analysis approaches, and with clear prompting and repetition of tasks, has been able to create narratives and summary charts on key themes. It hasn’t told me anything I didn’t already perceive, but it is capable of synthesising a large amount of data much quicker than me, and adds a verification step that I think could help reduce bias in research that relies on single-coder analysis.

I’ve now paid for a subscription that means Claude.ai can work harder for me. I’m still overwhelming it, at which point it sets a timer for five hours, when I can refresh and carry on. (This is still quicker than the children’s refresh rate. They seem to think recycling only needs doing once per school holiday.)

As researchers, we welcome trial and error, and we can thrive in uncertainty. I’m never going to be an expert in GenAI, but I will keep experimenting, asking questions, getting it wrong and taking small, careful steps forward.

Image and text copyright: Kate Jones. With thanks to Kate for writing this post.