Funder defends data security after ‘AI-based triage’ row - Research Professional News

Research network which rejected applicants for UKRI funds using AI says confidentiality was “entirely safeguarded”

A research network at the centre of a controversy over the use of “AI-based triage” to reject funding applications has said confidentiality was “entirely safeguarded” following concerns over data privacy.

Last week, several researchers applying to Crane—a network that supports efforts to strengthen the UK’s cybersecurity research ecosystem—reported on social media that half of all applicants had been rejected for funding after being assessed by an “AI-based review”.

Crane, based at the University of Oxford, is backed by a £6.5 million grant from the Engineering and Physical Sciences Research Council. As a network grant, Crane runs its own funding calls and launched a call worth £580,000 in June this year with the theme of “securing trust, autonomy, and resilience in future cyber ecosystems”.

Unsuccessful applicants to the call shared their rejection notice from Crane, which said “AI-based triage was used only to identify approximately 50 per cent of the strongest proposals to take forward to full human review”.

The notice said each proposal was “read independently by several different AI models against the same seven criteria used to shape this call” and then produced a score and a written justification with supporting quotations from the text. To “guard against any one model’s idiosyncrasies”, Crane ran a second assessment using different AI models, with the resulting scores combined.

Data privacy backlash

The use of AI to assess applications has sparked a backlash among researchers. One rejected applicant, Matthew Ivory, senior research associate at Lancaster University, told Research Professional News he had data privacy concerns, questioning whether Crane had “suitably ring-fenced any models” to ensure no data left their secure servers.

“How do I know that our ideas…are not now in the public domain and to be freely shared with others?” Ivory asked, saying he would not be “wholly comfortable” with AI companies having access to novel research proposals “that have been carefully written by academics [and] which may feed into large language model responses to other individuals”.

Asked about such concerns, Crane said the AI models they had used were secure.

Mark Ryan, Crane co-investigator and professor in computer security at the University of Birmingham, said: “I can reassure you that the confidentiality of the proposals was entirely safeguarded in keeping with our privacy policy and the announcement in the call.”

“The AI services were entirely ring-fenced, and their models were ‘read only’ with respect to the proposal data. The models were not given internet access, nor the ability to draw on information other than the specific elements of the research proposal that they were assisting in the evaluation of.

“That proposal data has not contributed and will not contribute to any model training. No public AI services were used in the evaluation, and the text of the proposals never left the confidential control of the review team.”

Transparency concern

Other researchers also raised concerns over the transparency of the process. John Stott, another rejected applicant and a senior lecturer in astrophysics at Lancaster University, told RPN he didn’t know AI would be used without human oversight.

“While I’ve heard there could potentially be AI use when allocating future funding, I hadn’t realised that it had already started so I was quite surprised,” Stott said.

“I’d been under the impression that if AI was used it would be to help human reviewers…but a human would still read the whole application.”

In its rejection notice, Crane said: “We recognise that AI-assisted assessment is a new and evolving part of the review process, and we are continuing to evaluate how well it aligns with the judgments of human reviewers on this call.”

Move towards more AI

Momentum has been growing to adopt AI in grant assessment. In a recent announcement of changes to its assessment processes, the national funder UK Research and Innovation said it would be “exploring the safe use of AI-assisted assessment, supported by appropriate controls and safeguards”, starting with a retrospective trial comparing AI with human reviews.

A study released as a preprint last week comparing AI with human reviews on UKRI grants said AI “may usefully play a role in contemporary funding decisions” including by helping to triage weaker proposals.

Asked about the use of AI by Crane to assess applications for UKRI funds, a spokesperson said: “UKRI policy does not allow the use of generative AI to conduct application assessment. We are following up with the Crane network to review this situation in detail and ensure compliance with our assessment conditions.”