Calculate Classification Error in QGIS with GRASS


An updated tutorial is available here. We have seen how to perform a land cover classification using the Semi-Automatic Classification Plugin for QGIS. This post is about the calculation of classification error in QGIS; this is useful to assess the accuracy of land cover classifications, in order to identify and measure map errors. For more information about accuracy assessment see my previous post here.


In this tutorial, we are going to assess the land cover accuracy of the map, which you can download from here, using ROIs as reference. This map has 4 classes:
  1. Water;
  2. Built-up;
  3. Vegetation;
  4. Soil.
As ground truth data, we will use the ROIs (which you can download from here) created in my previous tutorial, already reclassified according to the above classes.
We will use the GRASS plugin for QGIS. Usually, this plugin is installed along with QGIS; for more information about GRASS integration in QGIS see here.
At the end of the post you can find the video of this tutorial.

  1. Activate the GRASS plugin (if not already activated)
    • Open Manage Plugins under the menu Plugins; select the GRASS plugin and click OK;

    • the GRASS toolbar will open.


  1. Create GRASS location and mapset
    GRASS requires that data is stored in particular directories called "location", and subdirectories called "mapset". For more information about GRASS data structure see here.
    The GRASS plugin offers an easy dialog for the creation of location and mapset.
    • Click the New mapset button (the icon with a star) of the GRASS toolbar; the dialog New Mapset will open;
    • Click the button Browse to select the Database directory where location and mapset will be saved; for instance we can create a GRASS directory inside the user directory; click Next;

    • Enter location name, for instance turorial, and click Next;

    • Now we need to set the projection of the location; the classification we are using is projected in WGS 84 / UTM zone 33N (it is in Italy), therefore we select this projection and click Next;

    • We need to define the GRASS Region extent (although we will refine it later); select Italy from the combo box and click the button Set; click Next;

    • Type the name of the mapset, for instance class_accuracy, and click Next;

    • Click Finish and a message will inform you about the process result.

  1. Import classification and ROI shapefile in GRASS
    We need to import the classification raster and ROI shapefile in the GRASS mapset.
    • Click the button Open GRASS tools (with a hammer icon) in the GRASS toolbar;
    • We are going to use the module "r.in.gdal" in order to import the classification raster; type r.in.gdal in the Module List filter and click r.in.gdal; 

    • click the button ... and select the classification_reclass.tif file you have already downloaded from here; type classification under Name for output raster map (this will be the name of the classification raster in our GRASS mapset) and click Run;

    • After a few seconds the process will be finished; click the button View output in order to load the raster in QGIS (the raster needs to be loaded in QGIS for the following steps);

    • Now we import the ROI shapefile using the module "v.in.ogr"; type v.in.ogr in the Module List filter and click v.in.ogr; click the button ... and select the ROI_reclass.shp file you have already downloaded from here;  type reference under Name for output raster map and click Run;

    • When the process is finished, click the button View output in order to load the vector file in QGIS.

  1. Convert reference vector to raster
    This process is required for the next step, in order to compare classification raster to a reference raster. Reference raster will have the same resolution of classification raster, and pixel values will be the same as the ROI's field "ID_class" (pixel outside ROIs will have null values).
    • We need to refine the GRASS Region extent according to our classification; in the tab Browser select the classification raster and click the button Set current region to selected map;




    • In the tab Module List select the module v.to.rast.attr;

    • select the reference as input vector map; type ID_Class under Attribute field; type reference_raster as output raster map; click Run;




    • At the end of the process, click the button View output in order to load the reference raster in QGIS (GRASS plugin requires the raster to be loaded in QGIS).
  1. Perform the accuracy assessment
    We compare land cover classification to reference data in order to assess the land cover accuracy.
    • In the tab Module List select the module r.kappa;

    • Select classification under Name of raster map containing classification result; select reference_raster as Name of raster map containing reference classes; type error_matrix.txt under Name of output file containing error matrix and kappa*; click Run and the error matrix is saved in a text file.
  




* Attention! It appears that there is a bug which does not allow you to select a custom directory where to save the error matrix (an error about illegal output file name as in this image).
For Linux user, the only way to save the error matrix is to type only the file name, which will be saved automatically in the user directory.
For Windows users it is possible to save the error matrix in any directory, by replacing the character / with the character \ .
For example, change the output path from
C:/Users/Username/Desktop/error_matrix.txt
to
C:\Users\Username\Desktop\error_matrix.txt

 Update 09/07/2013: this bug has been fixed in the upcoming GRASS GIS 6.4.3




We can open the file error_matrix.txt with any text editor. In the following example, I have highlighted in the text where is the error matrix (in yellow) and where are the error statistics (in green).


ACCURACY ASSESSMENT
LOCATION: tutorial Fri Jul  5 11:52:34 2013
MASK: none
MAPS: MAP1 = Labels (reference_raster@class_accuracy in class_accuracy)
      MAP2 = (untitled) (classification@class_accuracy in class_accuracy)

Error Matrix
Panel #1 of 1
 MAP1
     cat# 1 2 3 4
 M     1 285 13 18 0
 A     2 0 321 37 1
 P     3 0 9 2494 0
 2     4 0 10 54 96
Col Sum 285 353 2603 97

cat# Row Sum
c     1       285          13          18           0        316
l     2         0         321          37           1        991
a     3         0           9        2494           0       4169
s     4         0          10          54          96       7507
    12983

Cats % Commission % Ommission Estimated Kappa
1 9.810127 0.000000 0.892741
2 10.584958 9.065156 0.881633
3 0.359569 4.187476 0.983670
4 40.000000 1.030928 0.588028

Kappa Kappa Variance
0.892142 0.000077

Obs Correct Total Obs % Observed Correct
3196 3338 95.745956

MAP1 Category Description
1:  (no description)
2:  (no description)
3:  (no description)
4:  (no description)

MAP2 Category Description
1:  (no description)
2:  (no description)
3:  (no description)
4:  (no description)



We have calculated the overall accuracy of the classification that is 95.75% .
Also, from this report we can understand which are the classes that were not identified correctly, and possibly improve their accuracy. For instance, the class 4 (soil) has 40% of commission error that indicates confusion between soil and vegetation classes, as we can see from the error matrix. Therefore, we could collect some new ROIs for the soil class and perform a new classification.

Following the same steps, we could assess land cover accuracy using a reference shapefile resulting from field survey or from photo interpretation of high resolution images.
It is worth mentioning the QGIS plugin Accuracy Assessment that allows for the error matrix calculation, comparing every classification pixel to the corresponding pixel (i.e. at the same position) of a reference raster. It requires that these two rasters have the same pixel size and dimensions.

The following is the video of this tutorial.

Newer posts Older posts