{"id":1831,"date":"2016-05-21T17:51:09","date_gmt":"2016-05-22T00:51:09","guid":{"rendered":"http:\/\/blogs.oregonstate.edu\/geo599spatialstatistics\/?p=1831"},"modified":"2016-05-27T14:45:04","modified_gmt":"2016-05-27T21:45:04","slug":"cluster-analysis-r","status":"publish","type":"post","link":"https:\/\/dev.blogs.oregonstate.edu\/geo599spatialstatistics\/2016\/05\/21\/cluster-analysis-r\/","title":{"rendered":"K-means Clustering in R"},"content":{"rendered":"

Introduction<\/strong><\/p>\n

I am working with a dataset from a dynamic global vegetation model (DGVM) run across the Pacific Northwest (PNW) over the time period 1895-2100. This is a process-based model that includes a dynamic fire model. I am particularly interested in drivers of the fire regimes (the frequency and intensity of wildfire) produced by the model. The general question I am studying is \u201cwhat, if any, is the relationship between the climatic and elevation inputs of the model and the fire regimes produced by the model?\u201d A more specific pair of questions for this method is \u201cDo fire regimes cluster locally?\u201d and \u201cHow do fire regimes change over time?\u201d<\/p>\n

Method description<\/strong><\/p>\n

Overview<\/em><\/p>\n

I divided the data into two time periods: 1901-2000 and 2001-2100 to capture the historical and projected climates and fire regimes over those periods. To define a fire regime, I chose the two fire-related output variables \u2013 carbon consumed by fire and percent of model pixel burned \u2013 along with fire return interval (FRI) over the century time period. I used k-means cluster analysis in R to define four fire regimes.<\/p>\n

K-means clustering divides a dataset into a specified number of data point clusters and calculates centroids and cluster membership such that the Euclidean distance between each cluster\u2019s centroids its members is minimized. The relationships between scales of the input variables should be taken into consideration in this type of analysis as it affects the values of Euclidean distances calculated. The steps in the analysis are outlined below. An appendix of the actual commands used for the analysis are in Appendix A.<\/p>\n

Outline of steps<\/em><\/p>\n

For each of the two input files:<\/p>\n

    \n
  1. Open the NetCDF file<\/li>\n
  2. Reverse the order of the latitude values (change from highest to lowest<\/em> to lowest to highest<\/em>)<\/li>\n
  3. Filter out the na (no data) values<\/li>\n
  4. Copy data (carbon consumed by fire, fraction pixel burned, FRI) into a combined data set<\/li>\n<\/ol>\n

    For the combined dataset:<\/p>\n

      \n
    1. Normalize values for each variable to 0.0 to 1.0 using the minimum and maximum of each variable in the combined dataset.<\/li>\n
    2. Generate the four clusters<\/li>\n
    3. Output value distributions of the four clusters<\/li>\n
    4. Distribute resulting data into two output datasets, one for each of the two time periods\n
        \n
      1. Divide into two separate datasets<\/li>\n
      2. Distribute results into non-na data locations<\/li>\n<\/ol>\n<\/li>\n<\/ol>\n

        For each of the output datasets:<\/p>\n

          \n
        1. Create output NetCDF file<\/li>\n
        2. Write data to NetCDF file<\/li>\n
        3. Close NetCDF file<\/li>\n<\/ol>\n

           <\/p>\n

          Results<\/strong><\/p>\n

          Clusters<\/em><\/p>\n

          With my input data, the FRI values range from a minimum of 1 to a maximum of 100 years. The mean annual fraction area burned has a theoretical range from 0 to 1, and the mean annual carbon consumed ranges from 0 to 266.5 gm-2<\/sup>. Performing the cluster analysis using the original input values resulted in 3 of the 4 clusters driven primarily by the values of carbon consumed (Fig. 1a-c). Normalizing the values of each input variable using the variable\u2019s minimum and maximum, resulted in clusters driven by different variables (Fig. 2). For the remainder of the project I am using normalized variables.<\/p>\n

          \"SheehanCluster01\"<\/a><\/p>\n

          Fig. 1: Z-score distributions of input variables within clusters for clusters computed without normalizing input data. Clusters 1, 2, and 3 (A-C) exhibit strong overlaps in both fraction area burned, and time between fires, and strong shifts in carbon consumed between clusters. Cluster 4 (D) is the only cluster in which all three factors differ substantially with other clusters.<\/p>\n

          \"SheehanCluster02\"<\/a><\/p>\n

          Fig. 2: Distributions of normalized input variables within clusters. Each of the clusters was created by performing cluster analysis using normalized input data. All clusters have substantial shifts in the distributions of at least two factors compared to the other clusters.<\/p>\n

          Result maps<\/em><\/p>\n

          The clustering of fire regimes can easily be seen in maps of the fire regimes (Fig. 3). Areas without fire tend to be at higher elevations: in the Coast Range and Cascades of Oregon and Washington and the Rockies of western Montana and northwestern Idaho. High frequency fires (Low FRI) are common in the plains and plateaus of southern Idaho, south and central Oregon, and the Columbia Plateau in central Washington. The other two fire regimes, both of which have a Medium FRI are somewhat intermingled, but are present in mid elevations in the Rockies, in southwest Oregon, and in the Willamette Valley and Puget Trough regions of Oregon and Washington, respectively.<\/p>\n

          \"SheehanCluster03\"<\/a><\/p>\n

          Fig. 3: Fire regime maps for (A) the historical period (1901-2000) and (B) the future period (2001-2100). (C: carbon consumed; Med: medium; Frac: fraction of grid cell; No Fire: no fire occurred in the cell over the time period)<\/p>\n

          Fire regime change from the historical (Fig. 3A) to future (Fig. 3B) period include the appearance of the High Carbon Consumed, Medium Fraction Burned, Medium FRI fire regime around the edges of the plains and plateaus as well as into the Blue Mountains of northeastern Oregon as well as the spread of both Medium FRI fire regimes into the Coast, Cascade, and Rocky Mountain ranges.<\/p>\n

          Method critique<\/strong><\/p>\n

          I found this method to be very useful for my study. Fire and vegetation modeling results are commonly expressed using single variables over time or area and fail to take into consideration the relationships between multiple variables. This work indicates that breaking sets of multiple fire-related variables into clusters provides a way to better characterize regional characteristics as well as spatial changes through time.<\/p>\n

          I did find the data manipulation required for the method a bit tricky. Having to work around the na values within the data required a bit of labor. Converting matrix data to vector data for the methods was another minor inconvenience. All that said, however, I believe such manipulations will become second nature as my experience with R grows.<\/p>\n

          Appendix A<\/strong>: Annotated R commands used for the analysis of normalized input data.
          \n## Cluster analysis on normalized data.
          \n## The dataset is drawn from two files, one for the 20th
          \n## century and one for the 21st
          \n## Install the necessary libraries<\/p>\n

          library(“DescTools”) # for displaying data
          \nlibrary(“ncdf4″) # reading and writing netCDF files<\/p>\n

          ## Set the current working directory for data and results<\/p>\n

          setwd(‘\/Users\/timsheehan\/Projects\/MACA\/FireClustering\/Analysis’)<\/p>\n

          ## Open the NetCDF files for reading<\/p>\n

          histnc = nc_open(‘SummaryOver_1901-2000.nc’)
          \nfutnc = nc_open(‘SummaryOver_2001-2100.nc’)<\/p>\n

          ## Get the lat and lon variables<\/p>\n

          lon = ncvar_get(histnc,’lon’)
          \nlat = ncvar_get(histnc,’lat’)<\/p>\n

          ## The lat dimension is highest to lowest in the input
          \n## file, so its vector needs to be reversed.
          \n## Reverse lat into ascending order (lower case r rev)<\/p>\n

          lat = rev(lat)
          \n## Create the data frames for historical, future, and difference data<\/p>\n

          normHist = expand.grid(lon=lon, lat=lat)
          \nnormFut = expand.grid(lon=lon, lat=lat)
          \nnormDiff = expand.grid(lon=lon, lat=lat)<\/p>\n

          ## Get the fields from the input files
          \n## Note the lat dimension comes in from high to low, so
          \n## it needs to be reversed so that R can display it (upper case R Rev)<\/p>\n

          ## Consumed is the annual mean for carbon consumed
          \n## PBurn is the annual mean of fraction of area burned
          \n## FRI is the mean number of years between fires<\/p>\n

          normHist$Consumed = c(Rev(ncvar_get(histnc,’CONSUMED_mean’),margin = 2))
          \nnormHist$PBurn = c(Rev(ncvar_get(histnc,’PART_BURN_mean’),margin = 2))
          \nnormHist$FRI = c(Rev(ncvar_get(histnc,’PART_BURN_intvl’),margin = 2))<\/p>\n

          normFut$Consumed = c(Rev(ncvar_get(futnc,’CONSUMED_mean’),margin = 2))
          \nnormFut$PBurn = c(Rev(ncvar_get(futnc,’PART_BURN_mean’),margin = 2))
          \nnormFut$FRI = c(Rev(ncvar_get(futnc,’PART_BURN_intvl’),margin = 2))<\/p>\n

          ## Normalize the values prior to doing the analysis
          \n## Also get the z scores for distribution plotting later
          \n## Note that NA values from the data must be omitted before
          \n## taking the statistics<\/p>\n

          ## Loop over the three variables used in the clusters
          \n## foreach one filter out cells with no data using na.omit
          \n## Normalize the data based on the min and max value of each
          \n## variable.<\/p>\n

          for(nm in c(‘Consumed’,’PBurn’,’FRI’)) {
          \n## Temporary data for efficient processing
          \ntmpVect = c(na.omit(normHist[[nm]]),na.omit(normFut[[nm]]))
          \ntmpMin = min(tmpVect)
          \ntmpMax = max(tmpVect)
          \ntmpDiff = tmpMax – tmpMin
          \ntmpMean = mean(tmpVect)
          \ntmpSD = sd(tmpVect)<\/p>\n

          ## Normalize the data
          \nnormHist[[nm]] = (normHist[[nm]] – tmpMin) \/ tmpDiff
          \nnormFut[[nm]] = (normFut[[nm]] – tmpMin) \/ tmpDiff<\/p>\n

          ## Z scores
          \nnormHist[[paste(nm,’_Z’,sep=”)]] = (normHist[[nm]] – tmpMean) \/ tmpSD
          \nnormFut[[paste(nm,’_Z’,sep=”)]] = (normFut[[nm]] – tmpMean) \/ tmpSD
          \n}<\/p>\n

          ## Create the data frame for clustering
          \n## Again, we omit data with no value using na.omit<\/p>\n

          normWorkDF = data.frame(‘Consumed’ = c(na.omit(normHist$Consumed),na.omit(normFut$Consumed)))
          \nnormWorkDF$PBurn = c(na.omit(normHist$PBurn),na.omit(normFut$PBurn))
          \nnormWorkDF$FRI = c(na.omit(normHist$FRI),na.omit(normFut$FRI))<\/p>\n

          ## Perform the clustering analysis
          \n## This is the point of all the work being done here
          \n## The Lloyd algorithm runs efficiently on the large dataset
          \n## (~100k points)<\/p>\n

          normCluster = kmeans(
          \ndata.frame(normWorkDF$Consumed,normWorkDF$PBurn,normWorkDF$FRI),
          \n4,
          \nalgorithm=”Lloyd”,
          \niter.max=500
          \n)<\/p>\n

          ## Copy the clusters back to the work data frame<\/p>\n

          normWorkDF$Clust = normCluster$cluster<\/p>\n

          # Plot the cluster distributions<\/p>\n

          for(ndx in 1:4) {
          \nfNm = paste(
          \n‘\/Users\/timsheehan\/Projects\/MACA\/FireClustering\/Analysis\/Graphs\/NormClust_’,
          \nndx,
          \n‘.png’,
          \nsep=”
          \n)
          \npng(fNm)
          \ntmpDF = subset(normWorkDF,normWorkDF$Clust == ndx)
          \nplot(
          \ndensity(tmpDF$Consumed),
          \nxlim=c(0,1),
          \nylim=c(0.0,6.0),
          \ncol=’darkblue’,
          \nxlab=’Normalized Value’,
          \nmain=paste(‘Cluster’,ndx,’Distribution’)
          \n)
          \nlines(density(tmpDF$PBurn),xlim=c(0,1),col=’green’)
          \nlines(density(tmpDF$FRI),xlim=c(0,1),col=’red’)
          \nlegend(
          \n‘topright’,
          \nlegend=c(‘C Consumed’,’Fract. Area Burned’,’Time Betw. Fires’),
          \nlwd=1,
          \ncol=c(‘darkblue’,’green’,’red’)
          \n)
          \ndev.off()
          \n}<\/p>\n

          ## Add the cluster numbers into the original data frames
          \n## Note that NA needs to be taken into account using
          \n## [!is.na(hist$Consumed)] to get the indexes of those data
          \n## items that are not NA<\/p>\n

          ## Historical values are in first half of data frame, future in second
          \n## Calculate indexes for clarity<\/p>\n

          histClustStartNdx = 1
          \nhistClustEndNdx = length(normCluster$cluster)\/2
          \nfutClustStartNdx = length(normCluster$cluster)\/2 + 1
          \nfutClustEndNdx = length(normCluster$cluster)<\/p>\n

          normHist$Clust = NA
          \nnormHist$Clust[!is.na(normHist$Consumed)] = normWorkDF$Clust[histClustStartNdx:histClustEndNdx]<\/p>\n

          normFut$Clust = NA
          \nnormFut$Clust[!is.na(normFut$Consumed)] = normWorkDF$Clust[futClustStartNdx:futClustEndNdx]<\/p>\n

          ## Create matrices for results and display
          \n## png() tells R what file to store the results to<\/p>\n

          png(‘\/Users\/timsheehan\/Projects\/MACA\/FireClustering\/Analysis\/Graphs\/NormHistClustMap.png’)
          \nnormHistClustArr = matrix(normHist$Clust,nrow=331,ncol=169)
          \nimage(lon,lat,normHistClustArr)
          \ndev.off()<\/p>\n

          png(‘\/Users\/timsheehan\/Projects\/MACA\/FireClustering\/Analysis\/Graphs\/NormFutClustMap.png’)
          \nnormFutClustArr = matrix(normFut$Clust,nrow=331,ncol=169)
          \nimage(lon,lat,normFutClustArr)
          \ndev.off()<\/p>\n

          ## Calculate Euclidean distance between cluster centers<\/p>\n

          normClustCtrEucDist = data.frame(cbind(rep(0,4),rep(0,4),rep(0,4),rep(0,4)))<\/p>\n

          for(ndx_1 in 1:4) {
          \nfor(ndx_2 in 1:4) {
          \nnormClustCtrEucDist[ndx_1,ndx_2] =
          \nsqrt(
          \n(normCluster$centers[ndx_1,1] – normCluster$centers[ndx_2,1]) ^ 2 +
          \n(normCluster$centers[ndx_1,2] – normCluster$centers[ndx_2,2]) ^ 2 +
          \n(normCluster$centers[ndx_1,3] – normCluster$centers[ndx_2,3]) ^ 2
          \n)
          \n}
          \n}<\/p>\n

          ## Create Fire Regime Euclidean distance map between historical and future<\/p>\n

          normDiff$HistClust = normHist$Clust
          \nnormDiff$FutClust = normFut$Clust
          \nnormDiff$EucDist = NA<\/p>\n

          normDiff$EucDist = mapply(
          \nfunction(x,y) ifelse(is.na(y), NA, normClustCtrEucDist[x,y]),
          \nnormDiff$HistClust,
          \nnormDiff$FutClust
          \n)<\/p>\n

          normDiffEucDistArr = matrix(normDiff$EucDist,nrow=331,ncol=169)
          \nimage(lon,lat,normDiffEucDistArr)<\/p>\n

          ## Create NetCDF file and store results there. This yields a
          \n## NetCDF file that can be used independently of R<\/p>\n

          x = ncdim_def(‘lon’,’degrees’,lon)
          \n## Reverse the order of latitude values
          \ny = ncdim_def(‘lat’,’degrees’,rev(lat))<\/p>\n

          histFCNCVar = ncvar_def(‘HistFireCluster’,’classification’,list(x,y),-9999,prec=’short’)
          \nfutFCNCVar = ncvar_def(‘FutFireCluster’,’classification’,list(x,y),-9999,prec=’short’)
          \neucDistNCVar = ncvar_def(‘EucDistance’,’NormalizedDistance’,list(x,y),9.9692099683868690e+36,prec=’float’)<\/p>\n

          # Reverse the lat order of the values being output
          \nOutHistClust = c(Rev(matrix(normDiff$HistClust,ncol=length(lat),nrow=length(lon)),margin=2))
          \nOutFutClust = c(Rev(matrix(normDiff$FutClust,ncol=length(lat),nrow=length(lon)),margin=2))
          \nOutEucDist = c(Rev(matrix(normDiff$EucDist,ncol=length(lat),nrow=length(lon)),margin=2))<\/p>\n

          ## Output the variables and close the file<\/p>\n

          ncOut = nc_create(‘NormClusters.nc’,list(histFCNCVar,futFCNCVar,eucDistNCVar))<\/p>\n

          ncvar_put(ncOut,histFCNCVar,OutHistClust)
          \nncvar_put(ncOut,futFCNCVar,OutFutClust)
          \nncvar_put(ncOut,eucDistNCVar,OutEucDist)<\/p>\n

          nc_close(ncOut)<\/p>\n","protected":false},"excerpt":{"rendered":"

          Introduction I am working with a dataset from a dynamic global vegetation model (DGVM) run across the Pacific Northwest (PNW) over the time period 1895-2100. This is a process-based model that includes a dynamic fire model. I am particularly interested in drivers of the fire regimes (the frequency and intensity of wildfire) produced by the… Continue reading →<\/span><\/a><\/p>\n","protected":false},"author":7728,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[661527],"tags":[],"class_list":["post-1831","post","type-post","status-publish","format-standard","hentry","category-tutorials-2016"],"_links":{"self":[{"href":"https:\/\/dev.blogs.oregonstate.edu\/geo599spatialstatistics\/wp-json\/wp\/v2\/posts\/1831","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/dev.blogs.oregonstate.edu\/geo599spatialstatistics\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/dev.blogs.oregonstate.edu\/geo599spatialstatistics\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/dev.blogs.oregonstate.edu\/geo599spatialstatistics\/wp-json\/wp\/v2\/users\/7728"}],"replies":[{"embeddable":true,"href":"https:\/\/dev.blogs.oregonstate.edu\/geo599spatialstatistics\/wp-json\/wp\/v2\/comments?post=1831"}],"version-history":[{"count":5,"href":"https:\/\/dev.blogs.oregonstate.edu\/geo599spatialstatistics\/wp-json\/wp\/v2\/posts\/1831\/revisions"}],"predecessor-version":[{"id":1842,"href":"https:\/\/dev.blogs.oregonstate.edu\/geo599spatialstatistics\/wp-json\/wp\/v2\/posts\/1831\/revisions\/1842"}],"wp:attachment":[{"href":"https:\/\/dev.blogs.oregonstate.edu\/geo599spatialstatistics\/wp-json\/wp\/v2\/media?parent=1831"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/dev.blogs.oregonstate.edu\/geo599spatialstatistics\/wp-json\/wp\/v2\/categories?post=1831"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/dev.blogs.oregonstate.edu\/geo599spatialstatistics\/wp-json\/wp\/v2\/tags?post=1831"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}