L4D Learning for Democracy

" This report summarises the results of the research project ‘Exploratory Study on Methods used to measure the ICT-mediated Social Impact of grassroots organisations’. This study was commissioned by IPTS to feed into a forthcoming 2-year research project: Measuring the impact of eInclusion actors on Digital Literacy, Skills and Inclusion goals (MIREIA). The report maps and reviews evaluation and impact assessment methods that have been developed and used to understand the social and economic impact of eInclusion actors, with a focus on quantitative approaches and production of measures that relate to policy goals. The first step was the compilation and preliminary profiling of a list of 81 examples of impact assessment approaches, identified through a literature and practice review. Following a process of evaluation, eleven approaches were documented and analysed in more depth. After that, the three ‘most relevant’ approaches were then selected for further analysis with particular reference to their feasibility as inputs to developing the MIREIA framework, and with reference to illustrative case studies of their use by grass roots organisations. Finally this report includes additional case study analysis of examples of the use of impacts assessment methods in real-world ‘grass roots’ interventions based on the results of the validation workshop. The overall conclusion of the study is that the main purpose of the MIRIEA framework and indicators should be to support impacts assessment as a ‘process’ – the process of creating the conditions to enable the embedding and implementation of concepts, methods, tools and practices, and their subsequent adaptation and evolution through use. In other words, the main purpose of the framework and tools is to support ‘praxis’."

TABLE OF CONTENTS
Acknowledgements........................................................................................................................3
Preface ..........................................................................................................................................5
REPORT SUMMARY........................................................................................................................9
1. ...............................................................................................13 Background to this Report
1.1 .......................................................................13 The report in the context of the MMTSO study
1.1.1 ...................................................................13 Aims, objectives and definition of the study
1.1.2 ......................................................................................................14 Research background
1.1.3 ............................................................................................................16 Policy background
1.2 .............................................................................17 The study approach and research activities
1.3 ........................................................................................................19 Methodological approach
1.3.1 ..........................................................19 Initial selection of three ‘most relevant’ methods
1.3.2 .........................................22 Approach used to document and analyse selected methods
1.3.3 ..........................................................................28 Additional analysis and documentation
1.4 ............................................................................................................29 Structure of this report
2. 31 Documentation and Review of Impacts Assessment Scenario 1: Communities of Practice
2.1 ............................................................................................................31 Overview of Scenario 1
2.2 .......................................................................................................................35 The case studies
2.2.1 .............................................................................................................35 UK online centres
2.2.2 .........................................................................................................39 Spanish telecentres
2.3 .........................................................................................................42 Assessment of Scenario 1
3. ....................................45 Documentation and Review of Scenario 2: Cost‐based Evaluation
3.1 ............................................................................................................45 Overview of Scenario 2
3.2 ...............................................................48 Case Study: International Computer Driving Licence
3.3 .........................................................................................................52 Assessment of Scenario 2
4. ........................................................................................................................................53 Documentation and Review of Scenario 3: Outcomes Identification as a Collective Practice
4.1 ..................................................................................................................................53 Overview
4.2 .......................................................................................................................55 The case studies
4.2.1 .............................................................................................................................55 VET4e‐I
4.2.2 ...........................................................................................................58 Realising Ambition
4.3 .........................................................................................................62 Assessment of Scenario 3
5. ...........................................................................65 Additional Documentation and Analysis
5.1 .................................................................................................................65 Focus of this section
5.2 .........................................................................................65 Results of the Validation Workshop
5.3 .............................................................................................................66 Additional case studies
5.3.1 .............................................................................................66 Purposes and methodology
5.3.2 ............................................................................................66 Summary of the three cases
5.3.3 ................................68 Comparative analysis and synthesis of the additional case studies
5.3.4 .......................................................................73 Lessons for MIREIA from the case studies
6. .................................................................................75 Conclusions and Recommendations
6.1 ..............................................................................................................................75 Key findings
6.2 ........................................................77 Towards an Impacts Assessment Framework for MIREIA
6.2.1 ........................................................................................77 Impacts Assessment Principles
6.2.2 ...........................................................................................80 MIREIA evaluation paradigm
6.2.3 ................................................................82 A structural model for the MIREIA framework
6.2.4 .....................................................................................................86