Cultural consequences: How will REF2028 affect our research environment?

Cat Davies, Dean for Research Culture, explores the hidden consequences of the shift towards assessing research culture in REF2028.

Much has been said since last week’s publication of the Future Research Assessment Programme’s initial decision report, which outlines the revised assessment formula determining the allocation of £2 billion in annual research funding post-REF2028.

In my earshot alone, some colleagues are joyously anticipating a boost to the inclusive culture they’ve been working for. Others have been wryly cynical about tokenism and ever new ways that institutions will find to game the system. Hyperbolic responses to the ‘slashing’ of the value of outputs have tried to pull REF into the culture wars. A degree of healthy indifference has been palpable too.

A headline aim of REF2028 is to facilitate an “inclusive and collaborative research system that supports a diversity of people, ideas, institutions, methodologies, outputs, and activities”. While this is a worthy aspiration and necessary for sustaining quality research, it is not an end in itself. What are the intended consequences of this healthier research culture? What unintended impacts might follow? And how might our research culture play out from the very act of change-making itself?

Bringing more colleagues into the fold

Let’s start with the inclusion of a wider range of people. New for REF2028, institutions can choose to return contributions from staff without a contractual responsibility for research. This potentially widens the eligible pool to technical professionals, research managers and career developers, postgraduate researchers, practitioner and industry partners, teaching and scholarship colleagues - anyone that can demonstrate a substantive link to the institution (defined as 0.2 FTE for at least 6 months). This change should finally make visible the hidden strategic and operational work that enables those of us with conspicuous authorship.

But what’s beyond better recognition and morale-boosting? These submission-newbies naturally work in teams and also represent a wider variety of protected characteristics, so this change should straightforwardly foster more diverse research teams, leading to more innovative, impactful work. Earlier in the pipeline, it could drive more diverse recruitment as institutions will need to show that their workforce has a range of backgrounds and assets beyond a clutch of 4* outputs.

The formal recognition of practitioners, technicians, and a range of other job types might accelerate porosity of careers. If staff who move in and out of academia from other sectors become ‘REFable’ (RIP), their value and respect should rightly increase, and ‘alternative’ career paths become more attractive.

One of the most welcome outcomes of this change would be a dismantling of the ego-driven aspects of HEI environments that our systems have created, along with the toxic superiority that some traditional researchers show for research enablers.

So what are the downsides? Paradoxically, this drive for inclusion could lead to greater competition. Given the widening of the REF-returnable staff pool and the need to keep the size of the return manageable, there will be less space for staff to have their work included in REF2028. Despite academics’ resentment of the REF, they do want to be in the game. The removal of the minimum and maximum number of outputs attached to individuals risks reviving the REF2014 stigma of being in or out, incurring disengagement and damaged egos. With finite space in the submission, more researchers will be on the bench with a risk that some groups, e.g. ECRs will be disproportionately excluded. Ring-fencing for career stages or profiles might be an answer, but this might also bring negative consequences. For example, if outputs by postgraduate researchers become eligible for submission, quality pressures on our researchers-in-training will likely increase. Similarly, non-academic staff could feel unnerved to be part of the REF, where before they were largely protected from the anxieties around inclusion and performance.

There is a fear that greater inclusivity will compromise research quality. This is patently false. More diverse teams are more widely cited. Losing unrecognised, talented colleagues presents very real risks for research quality. More broadly, some warn that the shift towards assessing culture will diminish the importance of quality research. I’ve yet to hear that from smaller institutions of course, who stand to gain more from the revised allocations.

Diversifying what research looks like

REF2028 aims to support a wider range of outputs and activities (although it’s worth remembering the eligible range in previous exercises). This pledge is enacted in all three rebranded elements:

• The increased weighting placed on activities relating to People, Culture and Environment, alongside the reduced weighting on research outputs
• In Contribution to Knowledge and Understanding, at least 10% of the subscore is devoted to broader contributions to the advancement of a discipline
• Similarly, in Engagement and Impact, an evidence-based statement on wider contributions beyond the submitted case studies will be worth at least 20% of the subscore.

These changes should at last reward the important business of EDI work, mentoring, public engagement, advocacy, software development, and the many other ways that researchers spend their time. The explicit acknowledgement of diverse outputs should enable researchers to be as proud of their replication studies, preregistered protocols, datasets, review articles, policy reports, practitioner guides, as they are of journal articles. In turn, we should recognise colleagues as richly talented, not merely as producers of outputs, improving morale and subduing imposter syndrome. The removal of the requirement for a minimal rating of 2* for underpinning research will also legitimise valuable impacts based on nonstandard research.

For all these gains, there will be trade-offs attached to an enriched conception of research. Reliable internal and external peer review will require familiarity with a wider range of outputs and activities, presenting a heavier load for some colleagues, e.g. those brokering peer review. The process of submission will also be challenging – can a sound sculpture or oral CPD methodology be easily uploaded?

Acknowledging the gravity of the change itself

The formal shift towards a focus on research culture is unexpectedly radical to some, notwithstanding the recommendation for an equal weighting of the three elements recommended by the FRAP International Advisory Board. The very act of shaking up such a powerful system signals an openness and flexibility not traditionally associated with government directives. The REF2028 decisions are a welcome fruition, or at least budding, of the government’s people and culture strategy.

This win demonstrates the power of careful evidence-based lobbying by courageous, creative groups such as the Hidden REF (‘Celebrating all research outputs’) who should now take their moment of glory. Other activist groups such as those behind DORA, COARA, and The Metric Tide have achieved great influence through their determination and deep understanding that research assessment is a powerful tool to shape research culture for the better.

Judging by my inbox, last Thursday’s announcements have provoked stirrings in previously hard-to-reach corners of my institution. By this single personal metric, Research England and the FRAP have already achieved a foundational aim – to recognise the importance of a positive research culture. Ears are pricking up among members of the professoriate previously cynical about the research culture drive. Researchers buy into our work for various reasons, and the REF incentive will be a big one.

For genuine buy-in, the REF must keep its nose clean. Its proposed methods are in line with good culture. It’s experimental and consultative. It responds to one of our most pressing challenges. It will try to keep its burden ‘proportionate’, e.g. by continuing established good practices and by using a word-limited structured form for the People & Culture element rather than the sprawling freeform narratives of its predecessor. Its abolition of the census date for calculating the volume measure in favour of counting the workforce over a more sustained period should help to make research culture and quality business-as-usual, rather than just assessing the research we do when someone is watching.

I’m hopeful that the exercise will help wither problematic aspects of our research culture like exclusion, hypercompetition, false hierarchies, and intimidation. The weaving of culture through all three elements of the assessment and the requirement to evidence cultural improvement should safeguard against performativity, though this will depend on metrics (see my next blog..).

Outlook

As a dean for research culture, my response to the initial decisions is predictably upbeat. Of course, colleagues should question this. The worst case scenario beyond 2028 would be that the revisions actively damage our culture. With an estimated price tag of £471 million in 2021, the sector cannot afford to get it wrong.

Equally damaging would be an ineffective REF. I would even accept no change to the overall results (see David Kernohan’s tentative calculations of REF2021 data with 2028 weightings which reveal marginal differences) as long as REF2028 drives systemic and behavioural changes to improve our research culture and therefore quality.

My hope is that REF will be used to enable the excellent research culture practices that are already developing in our institutions, and to reward actual progress. HEIs should use the REF to free up their staff to get on with what they do best, with collaboration, integrity, and autonomy. They must desist from launching performative activity that will make us all even busier.

With thanks to Elizabeth Adams, Lizzie Garcha, and Nick Plant for their helpful comments on an earlier draft of this piece.

An edited version of this blogpost was published by the Times Higher Education on 30/06/23.