January 2012

Highlights from workshop on evaluation criteria on science-policy interfaces

The SPIRAL project (work package3) arranged a workshop in Brussels on the 15th December 2011 addressing evaluation criteria for science-policy interfaces (SPI). There were eighteen participants, including eleven from outside the SPIRAL project. The workshop consisted of two sessions. The first session focused on participants’ experiences of science-policy work, and in particular on connecting successes or failures of SPIs to influence behaviour to the issues addressed by a draft set of evaluation criteria. In the second session the evaluation criteria were revised in plenary discussions. Here we highlight some important issues raised in discussions during the workshop.

Science and policy were seen as distinct domains with their own cultures, both facing the challenge of a lack of incentives for taking part in SPIs. Scientists tend to focus on peer review of their research, and policy makers tend to see SPI work as low-priority. However, participants noted that there is no need to lure every scientist and policy maker to take part in SPIs. SPIs can be seen as translators between the two cultures, and it is enough that some people work in this domain. Thus, there is nothing ‘wrong’ in focusing solely on science or policy, but it is important that enough translators exist to build bridges over the gap between science and policy to ensure that the communication needs of both communities are met.

Capacity building and iterative processes in SPIs are seen as important in producing commitment in related actors as well as for producing targeted outcomes. Capacity building is essential for example regarding the science behind the messages (e.g. increasing understanding of what scenarios mean). Without common concepts discussions are much harder. And iterative processes create a sense of continuity, which produces motivations to learn and take part in SPI work. Continuity can also facilitate the development of adaptive governance mechanisms for biodiversity and ecosystem services.

Requirements for consensual decision making, common to many SPIs, were seen as a barrier to innovation and action: there are often certain actors using their veto to halt progress. Thus, it can be essential for SPIs to use a range of conflict resolution strategies, and there can be a need for strong leadership to get things moving. Transparency in processes and decision making was also seen as general principle according to which SPIs should work.

Communication of uncertainty was considered a key challenge for SPIs: essential for enhancing credibility, but involving risks of reducing clarity or usefulness. Uncertainty should be communicated throughout science-policy processes, not only in produced outputs. Interestingly, it was suggested that while pictures, maps and figures may be effective in conveying key messages, they often present issues in simplistic terms and often do not communicate uncertainties.

Regarding the outcomes of SPIs, it was noted that unexpected outcomes or spin-off effects may be produced by SPIs. In some cases these can be results of social learning triggered by the SPI.

Overall, it was thought that the modified set of criteria could be useful in analysing SPIs, and could be a useful reference list for designing SPIs or SPI processes. However, it was stressed that the criteria should not be used to judge SPIs that score low on one or more of the criteria. It is not necessary for every SPI to meet every criterion to be successful, and indeed some of the criteria can be contradictory – the right balance will depend on the context and purpose of the SPI.

Finally, two further challenges for criteria development work in SPIRAL were identified. Firstly, to address in more detail issues associated with policy maker behaviour (the criteria at present focus more on science side and internal processes of SPIs). Secondly, to incorporate issues which resonate with informal SPIs.

 Contact:

Jari Niemelä | Simo Sarkki | Rob Tinch

----------------------------

November 25, 2011

 In WP 3 we examine the effectiveness of science-policy interfaces (SPI) in developing and supporting mechanisms for encouraging social behaviour that reduces negative human impact on biodiversity and ecosystems. We see a chain of influence from evaluation criteria to behaviour. SPIs are informing, shaping and influencing policy-making. Policy is by definition about effecting on people’s behaviour. SPIs can have also direct impacts on behaviour of wider public, for example by raising awareness and willingness to act. Thus, WP 3 acknowledges this chain of influencing on behaviour and concentrates on the first part of the chain: changing policy-makers behaviour through supporting development of well working science-policy interfaces. We do not develop the evaluation criteria for judging SPIs, but to clarify plausible success factors, which can help when developing emerging or existing interfaces.

First workshop and interviews for developing evaluation criteria for effective science-policy interfaces

For developing the evaluation criteria for effective SPIs WP 3 organizes two workshops and a set of interviews. The first workshop and interviews have been conducted, and the second workshop will be held in December 2011. Workshop participants and interviewees are scientists and policy-makers with experience on SPIs in Europe and beyond.

The results of the first workshop included 44 draft criteria for assessing the effectiveness of SPIs. It was noted in the workshop that formal evaluation is quite rare, but can be very useful. Performance should not be assessed too early, as the outcomes take time to emerge, but process-oriented evaluation can be useful at all stages. Evaluation requires a clear framework of objectives, and indeed one of the benefits of evaluation can be the reflection entailed by the requirement to specify what the goals are.

The subsequent interviews tested the draft criteria and as a result of the interviews the number of criteria was reduced to around 20. In addition, the interviews mapped various barriers for effective SPIs as well as potentially good practices for connecting science and policy.

Success factors for effective science-policy interfaces

The highlights for success factors gained from the first workshop and interviews include conflict management strategies (e.g. third party facilitation); trust building (e.g. Chatham House rules; informal discussion settings); strong leadership to move things forward; inclusion of well known persons, which also contribute to overall credibility of the interface; well planned media relationships and outreach strategies; good balance between responding to policy demand and maintaining independence; inclusion of relevant expertise and stakeholders, which, however, might be challenging; and adaptive capacity and flexibility for example to allow taking advantage from windows of opportunity.

Forthcoming workshop and outputs

The second workshop will collect best practices and stress the relevance of the draft criteria for the cases, which are outlined by participants in a story-telling session. The criteria will be also assessed, developed and validated, and finally the relevance of the different criteria is discussed against different SPI types (see WP 1 science-policy interface mapping exercise). These criteria can then be used to design and support emerging and existing SPIs (e.g. in WP 4 test cases). As an ultimate objective WP 3 aims to assist SPIs– via evaluation criteria - to produce outcomes related to social learning regarding interfacing policy and biodiversity, enhancement of biodiversity policies, and finally also implementing measures leading to changing behaviour to safeguard biodiversity and ecosystem services.

The outputs of WP 3 will include deliverable including fact sheets about evaluation criteria, policy recommendations, and peer reviewed articles. These outputs will be created during the year 2012.

Contact:

Jari Niemelä | Simo Sarkki | Rob Tinch