Skip to main content
search

Using AZURE AI Foundry to develop a LLM model which identifies themes and labels data

Abstract

The project consists of 5 different phases in which we ultimately aim to develop an internal Large Language Model.
These 5 different phases consist of:
1. Setup & Scope - Define the classification criteria, validate keywords, select project summaries
2. Baseline - label projects to create a diverse gold standard
3. Pilot Runs in Azure AI Foundry that are run on selected summaries and validated against the gold standard
4. Evaluation & Iteration - Compare Foundry output vs. baseline
5. Scaling & Reporting - Apply to full dataset. If succesful, test generalizability

Aim

Test whether Large Language Models (LLMs) can:
- Extract relevant themes and subthemes from project summaries (NL/EN)
- Provide a scalable method to analyze hundreds of projects

Intended Impact of the Study

If the project is deemed succesful, the model will be scaled to more of our internal data. Better labeling of our internal data will be immensely useful for our colleagues, cost less time, be able to answer more questions and lead to an increase in job satisfaction.

Project Lead

Not Applicable

Keywords

AI; Data analysis; Keyword identification; Labeling; Large Language Model (LLM); Theme identification

Research Area(s)

Data collection processes; Digital tools

Study Design(s)

Feasibility or pilot study

Data Collection Method(s)

Quantitative

Participant(s)

Staff in funding organisations

Status

Active

Expected completion date: December 31, 2026

Stage

Data analysis; Data collection

Study Funding/Support

Not funded through a grant but conducted as part of a funding organisation continuous improvement activity

Dissemination Activity/Activities

Stakeholder engagement