Map Accuracy Assessment and Area Estimation: A Practical Guide

Introduction

Accurate and consistent information on forest area and forest area change is important given the reporting requirements for countries to access results based payments for REDD+1 . Forest area change
estimates usually provide data on the extent of human activity resulting in emissions (e.g. from deforestation) or removals (e.g. from afforestation), also called activity data (AD). A basic methodological approach to estimate greenhouse gas emissions and removals (IPCC, 2003), is to multiply AD with a coefficient that quantifies emissions per unit ‘activity’ (e.g. tCO2e per ha), also called an emission factor (EF)

Activity data as part of emission/removal estimates should follow the IPCC good practice principle of neither over- nor underestimating emissions/removals and reducing uncertainties as far as is practicable. Uncertainty (lack of knowledge of the true value) is related to two issues: accuracy and precision (see Figure 1). Accuracy is a relative measure of the exactness of an estimate and accounts for systematic errors also referred to as bias. Therefore, an accurate estimate does not systematically over- or underestimate the true value. 

Map accuracy can be quantified by creating an error matrix (also commonly called a confusion matrix), which compares the map classification with a reference classification. Precision is related to the random error (see Figure 1), which can be quantified by a confidence interval.
A confidence interval gives a range that encloses the true value of an unknown fixed quantity with a specified probability. A precise estimate would thus have a small confidence interval. Basic guidance on reducing uncertainty can be found in GFOI (2013).

The accuracy assessment of map data described in this document will provide: 

1. a quantification of the map accuracy through the creation of error matrices; 

2. area and area change estimates that are adjusted for the map bias, thus more accurate estimates; 

3. and a quantification of the precision of the area and area change estimates through the calculation of confidence intervals.

The underlying principle of the accuracy assessment is that it compares the mapped land classification to higher quality reference data, collected through a sample based approach. The higher quality reference data can be obtained through ground collected data, but as this is expensive and
labor intense it is more commonly obtained through satellite imagery or aerial photography with finer spatial resolution than the data that was used to create the map data. When relying on imagery for reference data and there is no high resolution imagery available higher quality data can be collected using a process considered more accurate, such as human interpretation of the reference data.

This document uses the publication Good practices for estimating area and assessing accuracy of land change, (Olofsson et al. 2014) as a framework to provide recommendations for designing and
implementing an accuracy assessment for land cover and land cover change maps, and for estimating area based on the results from the accuracy assessment. This guide covers the theoretical background and recommendations for the setup of the sampling design, reference data collection and the analysis of the results.

 The scope of this document 

This document provides the methodology and practical implementation for the procedure for estimating area and assessing accuracy of a land cover map from a single period in time or for change between two time periods. The document guides the user through the aspects of an accuracy assessment which can be used to quantify and reduce uncertainty of map data for transparent reporting. 

The step-by-step guidance on how to implement such an assessment complements the theoretical background provided by Olofsson et al. (2014). Following the guidance of this document and the supplementary materials the user can produce tables of accuracy estimates, confidence intervals for area estimation, and comparison of area estimation derived from map data, reference data, and adjusted area estimates using both map data and reference data.

This document is split into two sections, the first covering the theoretical background for constructing an accuracy assessment and the second guiding the user through a practical implementation of an accuracy assessment of land cover and change maps. Step-by-step instructions rely on several open source software applications including QGIS, R and Open Foris Collect Earth. 

R is a free software programming language and software environment for statistical computing and graphics. QGIS is an open source desktop GIS application that provides data viewing, editing, and analysis capabilities. Collect Earth is a tool that facilitates sample-based assessments and collection of reference data from medium, high and very high spatial resolution satellite imagery in conjunction with Google Earth, Bing Maps and Google Earth Engine. Previous
experience with these programs is recommended but not necessary. Experience with GIS, working with raster and point data, as well as in basic statistical analysis is required to complete the assessment.

Map Accuracy Assessment and Area Estimation: A Practical Guide