Chapter 2 Data structures and main functions

2.1 Meteorological variables

Package meteoland assists in the estimation of the following daily variables over lanscapes (units in parentheses):

  • DOY: Day of the year ([1-366]).
  • MeanTemperature: Mean daily temperature (in degrees Celsius).
  • MinTemperature: Minimum daily temperature (in degrees Celsius).
  • MaxTemperature: Maximum daily temperature (in degrees Celsius).
  • Precipitation: Daily precipitation (in mm of water).
  • MeanRelativeHumidity: Mean daily relative humidity (in percent).
  • MinRelativeHumidity: Minimum daily relative humidity (in percent).
  • MaxRelativeHumidity: Maximum daily relative humidity (in percent).
  • Radiation: Incoming radiation (in MJ/m2).
  • WindSpeed: Wind speed (in m/s).
  • WindDirection: Wind direction (in degrees from North).
  • PET: Potential evapo-transpiration (in mm of water).

Since specific humidity is often specified instead of relative humidity, the package also allows reading and using this variable, although it will not be written in files:

  • SpecificHumidity: Specific humidity (in kg/kg).

2.2 Spatial classes

The package deals with three kinds of spatial structures: individual points, a set of pixels from a spatial grid and full (i.e. complete) grids. The package includes six S4 spatial classes, which are defined as children of classes in package sp. Spatial structures are used to represent both topographical and meteorological data. These are described in the following subsections.

2.2.1 Topography

Three classes are defined to represent the variation of topographic features (i.e., elevation, slope and aspect) over space:

  • Class SpatialPointsTopography extends SpatialPointsDataFrame and represents the topographic features of a set of points in a landscape.
## Class "SpatialPointsTopography" [package "meteoland"]
## 
## Slots:
##                                                                   
## Name:         data  coords.nrs      coords        bbox proj4string
## Class:  data.frame     numeric      matrix      matrix         CRS
## 
## Extends: 
## Class "SpatialPointsDataFrame", directly
## Class "SpatialPoints", by class "SpatialPointsDataFrame", distance 2
## Class "Spatial", by class "SpatialPointsDataFrame", distance 3
## Class "SpatialVector", by class "SpatialPointsDataFrame", distance 3
  • Class SpatialGridTopography extends SpatialGridDataFrame and represents the continuous variation of topographic features over a full spatial grid.
## Class "SpatialGridTopography" [package "meteoland"]
## 
## Slots:
##                                                           
## Name:          data         grid         bbox  proj4string
## Class:   data.frame GridTopology       matrix          CRS
## 
## Extends: 
## Class "SpatialGridDataFrame", directly
## Class "SpatialGrid", by class "SpatialGridDataFrame", distance 2
## Class "Spatial", by class "SpatialGridDataFrame", distance 3
  • Class SpatialPixelsTopography extends SpatialPixelsDataFrame and represents the continuous variation of topographic features over a set if cells in a grid.
## Class "SpatialPixelsTopography" [package "meteoland"]
## 
## Slots:
##                                                                                                  
## Name:          data   coords.nrs         grid   grid.index       coords         bbox  proj4string
## Class:   data.frame      numeric GridTopology      integer       matrix       matrix          CRS
## 
## Extends: 
## Class "SpatialPixelsDataFrame", directly
## Class "SpatialPixels", by class "SpatialPixelsDataFrame", distance 2
## Class "SpatialPointsDataFrame", by class "SpatialPixelsDataFrame", distance 2
## Class "SpatialPoints", by class "SpatialPixelsDataFrame", distance 3
## Class "Spatial", by class "SpatialPixelsDataFrame", distance 4
## Class "SpatialVector", by class "SpatialPixelsDataFrame", distance 4

Although the three classes have the same slots as their parent S4 classes, data frames in SpatialPointsTopography, SpatialGridTopography and SpatialPixelsTopography objects have only three variables: elevation (in meters), slope (in degrees from the horizontal plane) and aspect (in degrees from North).

2.2.2 Meteorology

Analogously to topography classes, three spatial classes are used to represent the variation of daily meteorology over space:

  • Class SpatialPointsMeteorology extends SpatialPoints and represents daily meteorology series for a set of points in a landscape.
## Class "SpatialPointsMeteorology" [package "meteoland"]
## 
## Slots:
##                                                                   
## Name:        dates        data      coords        bbox proj4string
## Class:        Date      vector      matrix      matrix         CRS
## 
## Extends: 
## Class "SpatialPoints", directly
## Class "Spatial", by class "SpatialPoints", distance 2
## Class "SpatialVector", by class "SpatialPoints", distance 2
  • Class SpatialGridMeteorology extends SpatialGrid and represents the continuous variation of daily meteorology across a grid of cells.
## Class "SpatialGridMeteorology" [package "meteoland"]
## 
## Slots:
##                                                                        
## Name:         dates         data         grid         bbox  proj4string
## Class:         Date       vector GridTopology       matrix          CRS
## 
## Extends: 
## Class "SpatialGrid", directly
## Class "Spatial", by class "SpatialGrid", distance 2
  • Class SpatialPixelsMeteorology extends SpatialPixels and represents the variation of daily meteorology for a set of pixels (cells) of a spatial grid.
## Class "SpatialPixelsMeteorology" [package "meteoland"]
## 
## Slots:
##                                                                                                  
## Name:         dates         data         grid   grid.index       coords         bbox  proj4string
## Class:         Date       vector GridTopology      integer       matrix       matrix          CRS
## 
## Extends: 
## Class "SpatialPixels", directly
## Class "SpatialPoints", by class "SpatialPixels", distance 2
## Class "Spatial", by class "SpatialPixels", distance 3
## Class "SpatialVector", by class "SpatialPixels", distance 3

In addition to their corresponding inherited slots, classes SpatialPointsMeteorology, SpatialGridMeteorology and SpatialPixelsMeteorology have two additional slots: dates (a vector of days specifying a time period), and data (a vector of data frames with the meteorological data). Although the three classes have a data slot containing data frames, meteorological data is in different form in each class. In objects of SpatialPointsMeteorology, there is one data frame for each point where variables are in columns and dates are in rows. In objects of SpatialGridMeteorology and SpatialPixelsMeteorology, each data frame describes the meteorology over a complete grid, or a subset of cells, for a single day. In these cases, the data frame has grid cells in rows and variables in columns.

2.3 Reading and writing meteorological data

2.3.1 Point meteorology (ascii/rds files)

Objects of class SpatialPointsMeteorology are stored in the disk using one data file for each of their spatial points. Files can be stored in ascii (i.e. text) format or rds (i.e. R data of a single object compressed) format. Package meteoland provides four input/output functions for point meteorology:

  • Function readmeteorologypoint() reads the meteorological data stored in one ascii/rds data file and returns a data frame.
  • Function writemeteorologypoint() writes the meteorological data of a single point as an ascii/rds file in the file system.
  • Function readmeteorologypointfiles() reads several ascii/rds files and returns an object of class SpatialPointsMeteorology.
  • Functions writemeteorologypointfiles() writes several ascii/rds files in the disk, one per spatial point. Metadata (i.e. the spatial coordinates of each point and the corresponding file path) is stored in an additional file.

2.3.2 Point/gridded meteorology (netCDF)

NetCDF is a set of libraries to write machine-independent data formats that support the creation and sharing of array-oriented scientific data. NetCDF is very useful useful to store spatio-temporal datasets, as it allows storing the grid topology, the time period being described and the spatial projection as well as the actual data all in the same file. Moreover, NetCDF can be read and written lazily, without having to load all the file in memory.

Package meteoland allows objects of classes SpatialPointsMeteorology, SpatialPixelsMeteorology and SpatialGridMeteorology to be stored in the disk as NetCDF thanks to the functions provided by ncdf4 R package, which provides an interface to Unidata’s netCDF library (version 4 or earlier). We try to conform to CF specifications in files written by meteoland, but improvements can always be made.

The following functions are available for input/output of point meteorology:

  • Functions writemeteorologypoints() writes meteorological point data on a NetCDF.
  • Function readmeteorologypoints() reads point meteorological data stored in one or several NetCDFs and returns an object of class SpatialPointsMeteorology. If several files are read, the function tries to merge the result.

And the following functions are available for input/output of meteorology on pixels or full grids:

  • Functions readmeteorologygrid() and readmeteorologypixels() read the meteorological data stored in one or several NetCDF file and return an object of class SpatialGridMeteorology or SpatialPixelsMeteorology, respectively. If several files are read, the function tries to merge the result. Rotated grids should not be read using this functions.
  • Function readmeteorologygridpoints() allows reading grid pixels as points. It returns an object of class SpatialPointsMeteorology. This feature becomes useful to read data from rotated grids.
  • Functions writemeteorologygrid() and writemeteorologypixels() write the meteorological data of the full grid or the subset of grid cells, for a NetCDF. The same functions allow adding and replacing data content in a previously existing NetCDF.
  • Function writemeteorologygridpixel() allows writing/replacing data for specific pixels in an existing NetCDF.

2.4 Visualizing input topography and meteorological data

Although very simple, the package provides two kinds of functions to visualize the temporal and spatial variation of meteorology:

  • Function spplot() has been redefined from package sp to draw maps of specific weather variables corresponding to specific dates. The function can be used on objects of class SpatialGridMeteorology and SpatialPixelsMeteorology.
  • Function meteoplot() allows the temporal series of specific variables on specific spatial points to be plotted. The function can read the data from the disk or from objects of class SpatialPointsMeteorology.

Similarly, function spplot() also accepts objects of classes SpatialGridTopography and SpatialPixelsTopography, so that topography can also be easily displayed.

2.5 Subsetting, merging and reshaping data

A number of functions are available to manipulate data structures and to make them available for other R packages.

2.5.1 Subsetting and reshaping topography data

Following the design of package sp, package meteoland provides several functions to manipulate topographic data structures (see fig. 2.1).

  1. It is possible to coerce objects between from topography classes to the corresponding class in package sp using functions as(). This includes coercing to spatial objects with or without data columns. Additionally, an object of SpatialPointsTopography can be coerced into a SpatialPixelsTopography using a call like as(x, "SpatialPixelsTopography").
  2. It is possible to coerce objects to the general-purpose class sf of package sf using as(..., "sf").
  3. Subsetting (i.e.[ operator) is allowed on all three spatial topographic classes. The valid arguments of this function call depend on the kind of structure.
Conversion between meteoland classes for topography and classes from other packages

Figure 2.1: Conversion between meteoland classes for topography and classes from other packages

2.5.2 Subsetting and reshaping meteorology data

Analogously, package meteoland provides several functions to manipulate spatio-temporal data structures (see fig. 2.2).

  1. It is possible to coerce objects between spatial classes using functions as().
  2. Subsetting (i.e.[ operator) is allowed on all six spatial classes (topographic or meteorological). The valid arguments of this function call depend on the kind of structure.
  3. The package provides functions to extract meteorological data (these allow reading data in memory or from the disk):
  • Functions extractgridindex() extracts the meteorology of a particular pixel index from a grid and returns a data frame with dates in rows and variables in columns.
  • Function extractgridpoints() extract the meteorology of point locations from a grid and returns an object of SpatialPointsMeteorology, similarly to the [ operator.
  • Functions extractdates() and extractvars() extract the meteorology of a set of dates (or variables, respectively) from a SpatialPointsMeteorology object, (or SpatialGridMeteorology or SpatialPixelsMeteorology) and returns a SpatialPointDataFrame (or SpatialGridDataFrame or SpatialPixelsDataFrame) for each date (resp. variable) in a named list. This kind of conversion is also useful to export meteoland outputs to package raster, which provides wrappers for the abovementioned sp classes.
  1. Finally, the package provides some functions to reshape meteorological data structures into more general structures of spatio-temporal data, so that they can be more easily processed, combined and stored using other R packages. In particular, objects of classes SpatialPointsMeteoroloy, SpatialGridMeteorology and SpatialPixelsMeteorology can be reshaped into objects of packages stars and spacetime by using calls like as(x, "stars") or as(x, "STFDF").
Conversion between meteoland classes for meteorology and classes from other packages

Figure 2.2: Conversion between meteoland classes for meteorology and classes from other packages

2.5.3 Merging meteorology data

Functions mergegrid() and mergepoints() take any number of objects of the same spatial class and return a merged object. The functions pool over dates and variables, but the objects to merge should have the same spatial structures (points, grids, reference systems, …). This is specially useful to merge data corresponding to different periods.

2.6 Summarizing meteorological data

2.6.1 Temporal summaries

The package provides functions to generate temporal summaries of meteorological data. These accept meteorology objects as input and return their corresponding spatial dataframe structure with the summary statistics in columns:

  • Function summarypoints() summarizes the meteorology of spatial points. It accepts objects of SpatialPointsMeteorology as input and returns an object of SpatialPointsDataFrame with point summaries for the requested variable. Temporal summaries can be calculated for different periods and using different summarizing functions (e.g. mean, sum, minimum, maximum, etc.).
  • Functions summarygrid() and summarypixels() summarize the meteorology of full grids or of subset of grid cells, respectively. They accept objects of SpatialGridMeteorology and SpatialPixelsMeteorology, respectively, as input and return an object of SpatialGridDataFrame and SpatialPixelsDataFrame, respectively, with temporal summaries for the requested variable over the range of dates indicated.
  • Function summaryinterpolationdata() works similarly to summarypoints(), but takes an object of class MeteorologyInterpolationData as input.

All temporal summary functions except summaryinterpolationdata() accept a data file, or a metadata file as input. This allows, producing summaries lazily, without loading complete data structures in memory. Of course, this mode of producing summaries may be slower than operating on objects already loaded in memory.

2.6.2 Spatial summaries

The package provides one function to produce spatial summaries. Function averagearea() averages the coordinates and meteorological values of any spatial meteorology object, returning an object of class SpatialPointMeteorology with a single point representing the average.

2.7 Meteorology estimation functions

2.7.1 Spatial interpolation

Package meteoland provides three functions for interpolating meteorological data (i.e., one for each data structure):

  • Function interpolationpoints() interpolates weather for a set of locations given in SpatialPointsTopography and returns an object of class SpatialPointsMeteorology.
  • Function interpolationpixels() interpolates weather for pixels in a grid specified in SpatialPixelsTopography and returns an object of class SpatialPixelsMeteorology.
  • Function interpolationgrid() interpolates weather for a whole grid specified in SpatialGridTopography and returns an object of class SpatialGridMeteorology.

Both functions require an object of class MeteorologyInterpolationData, which contains the X-Y coordinates, the meteorological data and topography of a set of weather stations as well as weather interpolation parameters.

## Class "MeteorologyInterpolationData" [package "meteoland"]
## 
## Slots:
##                                                                                                           
## Name:                    coords                elevation                    slope                   aspect
## Class:                   matrix                  numeric                  numeric                  numeric
##                                                                                                           
## Name:            MinTemperature           MaxTemperature    SmoothedPrecipitation            Precipitation
## Class:                   matrix                   matrix                   matrix                   matrix
##                                                                                                           
## Name:  SmoothedTemperatureRange         RelativeHumidity                Radiation                WindSpeed
## Class:                   matrix                   matrix                      ANY                      ANY
##                                                                                                           
## Name:             WindDirection               WindFields                  WFIndex                 WFFactor
## Class:                      ANY                      ANY                      ANY                      ANY
##                                                                                                           
## Name:                    params                    dates                     bbox              proj4string
## Class:                     list                     Date                   matrix                      CRS
## 
## Extends: 
## Class "MeteorologyProcedureData", directly
## Class "Spatial", by class "MeteorologyProcedureData", distance 2

When calling functions interpolationpoints(), interpolationpixels(), or interpolationgrid(), the user may require interpolation outputs to be written into the file system, instead of being stored in memory. For example, if interpolationpoints() is called with export = TRUE and a location directory, the function will write the data frame produced for each point into an ascii text file or a rds file. Metadata files will also be written, so that results can later be loaded in memory. Alternatively, interpolationpoints() can be forced to write results in a netCDF by specifying export = TRUE and exportFile = <filename>. Similarly, if interpolationpixels() or interpolationgrid() are called specifying exportFile = <filename>, the functions will create and add data to a netCDF. These options becomes important when dealing with very large spatial structures.

Functions interpolation.calibration() and interpolation.cv() are included in meteoland to calibrate interpolation parameters and evaluate predictive performance of interpolation routines before using them. Details of interpolation routines are described in chapter 3.

2.7.2 Statistical correction

One function is available for statistical correction of meteorological data series (i.e., one function for each data structure). Function correctionpoints() performs statistical correction of weather data series on a set of locations and it returns an object of class SpatialPointsMeteorology containing corrected weather predictions. Statistical correction requires an object of class MeteorologyUncorrectedData, which contains the X-Y coordinates and the coarse-scale meteorological data to be corrected, which includes a reference (historic) period and projected (e.g. future) period:

## Class "MeteorologyUncorrectedData" [package "meteoland"]
## 
## Slots:
##                                                                                                       
## Name:           coords  reference_data projection_data          params           dates            bbox
## Class:          matrix             ANY             ANY            list            Date          matrix
##                       
## Name:      proj4string
## Class:             CRS
## 
## Extends: 
## Class "MeteorologyProcedureData", directly
## Class "Spatial", by class "MeteorologyProcedureData", distance 2

The reference (historical) period is compared with observed meteorological data of the same period, and the routine uses this information to correct the projected (e.g. future) period. Therefore, apart from the MeteorologyUncorrectedData object, the correction function requires accurate meteorological data (for a set of spatial points or a grid). Normally, these data will be the result of spatial interpolation.

As before, when calling functions correctionpoints(), the user may require the outputs to be written into the file system, instead of being returned in memory. The options are the same as described for interpolation.

Function correctionpoints.errors() was included in the package to evaluate the errors of the less accurate and more accurate series. Comparisons can be made before and after applying statistical corrections. In the latter case, cross-validation is also available. Details of correction routines are described in chapter 5.

2.7.3 Weather generation

Function weathergeneration() can be used to generate synthetic weather series for a range of inputs, including a (non-spatial) data frame (corresponding to weather series of a single location) as well as objects of classes SpatialPointsDataFrame, SpatialPixelsDataFrame or SpatialGridDataFrame. The output of the function is of the same class as the input object. Weather generation algorithms are described in chapter 7.