1. Roberto De Almeida
  2. Pupynere
Issue #6 new

Appending to a NetCDF file

Nick Papior
created an issue

For reading and writing NetCDF files, it would be very nice to have a way to append to NetCDF files.

Thus adding mode 'a' would seem like a good addition for the code.

It could be made by simply reading in all contents and copy to a new object with write access, although this would read and write a lot, it would at least enable the feature of appending.

Comments (2)

  1. Roberto De Almeida repo owner

    Hi, Nick! This is something that I started to implement at least twice, but there're always some details that make me give up after a time. I agree it would be useful, specially if we're appending data to record variables (since it can be done without moving data around), but it's not on my priority list.

  2. rostislav_kouznetsov

    Hi, I have implemented the feature (not in the most elegant and pythonic way though..). It just copies the stuff into the new file. It is also writes into a temporal file before touching a target file.

    My main trouble is that the order of variables and attributes gets completely different from the original file when copying the stuff. Thus diff is not very usefrull tool to compare ncdumps anymore... It would be great to have some control on the order of variables written into a file.

    Hope, it is of some use.

    #
    # Wrapper for pupyrene to handle temporary files
    # "append" to existing netcdfs 
    # easy create variables and dimensions filled with data and attributes
    #
    #   Roux aka rostislav DOT kouznetsov AT gmail.com
    import os
    import sys
    import numpy as np
    import pupynere as netcdf
    class silamNCout(netcdf.netcdf_file): # Class for our netcdf files...
    
            # Create temporary file (not to destroy the prevous
            # version before writing succeeded) and copy there the
            # stuff from the previous file: a way to
            # append variables to existing file
            def __init__(self, filename, append=True, verbose=False, allowrecord=True):
                    (head,tail) = os.path.split(filename)
                    tmpfile=os.path.join(head,"0-%d-%s"%(os.getpid(),tail))
                    netcdf.netcdf_file.__init__(self, tmpfile, "w")
    
    
                    self.__dict__['_targetfile']=filename #Setattr writes also to self._attributes
                    self.__dict__['_verbose']=verbose     # We do not want it...
                    self.__dict__['_records']=allowrecord     # No record dimension
                    oldF=None
                   if append:
                          try:
                            oldF = netcdf.netcdf_file(filename,"r")
                          except IOError as e:
                             if verbose:
                                 print "Previous file can't be updated..."
    
                    # Copy from the oldfile
                    if oldF: # Copy all stuff from the old file
    
                            for k, v in oldF._attributes.items():
                                    if hasattr(self, k): #Overwriting targetfile is harmful
                                                         # it should not appear in files anyhow
                                            continue
                                    self.__setattr__(k,v)
                            for dim in oldF._dims:
                                    if (allowrecord):
                                            value = oldF.dimensions[dim]
                                    else:
                                            value = oldF.variables[dim].shape[0]
                                    if value>-1:
                                            self.createDimension(dim, int(value))
                                    else:
                                            self.createDimension(dim, None)
                            for key, var in oldF.variables.items():
                                     ovar = self.createVariable(key, var.typecode(), var.dimensions)
                                     for k, v in var._attributes.items():
                                             ovar.__setattr__(k,v)
                                     if len(var.data.shape):
                                            ovar[:]=var.data.copy()
                            oldF.close()
                    else:
                            self.history=""
    
            # 
            #
            def finalize(self):
                    if self._verbose:
                            print "Closing tmpfile", self.filename
                    self.close()
                    if self._targetfile:
                        if self._verbose:
                              print "Renaming %s to %s"%(self.filename,self._targetfile)
                        os.rename(self.filename,self._targetfile)
    
            # Checks if var exists and creates or verifies it
            # intended to be used with dimensions 
            # They have to be same and have same values
            # Time is unlimited dimension
            def createVar(self, varname, dtype, dims, vals, attrs={}, epsilon = 0):
                    if (varname,)==dims: # The variable is dimension itself
                            if (varname=='time') and self._records:
                                    dimlen = None
                            else:
                                    dimlen = len(vals)
                            if  varname in self.dimensions:
                                    if self.dimensions[varname] != dimlen:
                                            print "Dimension %s has size"%(varname,), self.dimensions[varname], "requested", dimelen
                                            self.close()
                                            exit(-1)
                            else:
                                    self.createDimension(varname, dimlen)
                    # values
                    if  varname in self.variables:
                            var=self.variables[varname]
                            if var.dimensions != dims:
                                    print "Wrong datatype for", varname, var.dimensions, ", requested", dims
                                    self.close()
                                    exit(-1)
                            if var.data.shape != vals.shape:
                                    print "Wrong shape of", varname, var.data.shape, ", requested", vals.shape
                                    self.close()
                                    exit(-1)
    
                            if dtype != 'c':
                                VallErr=np.any(abs(var.data - vals)>epsilon)
                            else:
                                VallErr= (var.data != np.array(vals))  #False  #FIXME Hack. Do not know how to handle char scalars
                                VallErr= False  #FIXME Hack. Do not know how to handle char scalars
    
                            if VallErr:
                                    print "Wrong data", varname,"in file",var.data, ", requested", vals
                                    self.close()
                                    exit(-1)
                            if ("units" in attrs) and (var._attributes['units'] != attrs['units']):
                                    print "Wrong units for ", varname, var.units, ", requested", attrs['units']
                                    self.close()
                                    exit(-1)
    
                    else:
                        var = self.createVariable(varname, dtype, dims)
                        if (vals != None):
                           # Trick to handle scalar variables...
                            if vals.shape:
                                    var[:] = vals
                            else:
                                    var.assignValue(np.array(vals))
    
                    # Set attributes
                    for k,v in attrs.items():
                                    var.__setattr__(k,v)
    
  3. Log in to comment