__finishInit ( self )

__finishInit is a private helper function that finishes initialization by combining attributes to form new ones.

Inputs: None

Returns: Void.

Affects: Initializes class member variables that are combinations of existing ones.

Exceptions: None.

__getExperiments (
        self,
        arg,
        dirname,
        names,
        )

__getExperiment is a private helper function called by os.path.walk in getExpList.

__getExperiments is called for each sub-directory in the experiments directory. Its purpose is to add any experiment directory paths from that directory that match the search criteria to the expList.

Inputs:

arg: a tuple containing all the filter arguments, plus the fileList to be appended to. The tuple elements are:

expName: a string defining what experiment names to accept (case insensitive). Use of unix file matching characters * and ? are allowed. If None, any experiment name allowed.

kinstList: a list of kinst (kind of instrument codes) integers that will be accepted. If None, all kinst values are accepted.

startDate: a python datetime in UTC. If None, do not reject any files.

endDate: a python datetime in UTC. If None, do not reject any files.

startDayOfYear: a Julian day number (1-366) after which to accept files from any given year. This can be used for example to only allow files from Spring, which has a start day of year of 80 (March 21). If None, do not reject any files.

endDayOfYear: a Julian day number (1-366) before which to accept files from any given year. This can be used for example to only allow files from Spring, which has a end day of year of 172 (June 21). If None, do not reject any files.

expList: the list of valid experiment paths to which is appended any newly-found valid experiment directories

showIgnoredExperiments: if False (default), ignore experiments where expTab.txt state code is set to ignore. If True, include those experiments.

enforcePathConvention: if True, only return experiments whose path is of the form convention 1999/mlh/20jan98d (YYYY/<three lower case letters>/DDmmmYY<optional single character>. If False (the default), only required path in the form 1999/mlh/*.

dirname: the directory name of the present directory

names: the list of filenames in the present directory

Returns: None.

Affects: Adds full file paths of any data files found that meet all filter criteria

Exceptions: None

__getFiles (
        self,
        arg,
        dirname,
        names,
        )

__getFiles is a private helper function called by os.path.walk in getFileList.

__getFiles is called for each sub-directory in the experiments directory. Its purpose is to add any file paths from that directory that match the search criteria to the fileList.

Inputs:

arg: a tuple containing all the filter arguments, plus the fileList to be appended to. The tuple elements are:

expName: a string defining what experiment names to accept (case insensitive). Use of unix file matching characters * and ? are allowed. If None, any experiment name allowed.

kinstList: a list of kinst (kind of instrument codes) integers that will be accepted. If None, all kinst values are accepted.

kindatList: a list of kindat (kind of data codes) integers that will be accepted. If None, all kindat values are accepted.

startDate: a python date/time in UTC (see time module - actually a tuple of nine integers) after which to accept files. If None, do not reject any files.

endDate: a python date/time in UTC (see time module - actually a tuple of nine integers) before which to accept files. If None, do not reject any files.

startDayOfYear: a Julian day number (1-366) after which to accept files from any given year. This can be used for example to only allow files from Spring, which has a start day of year of 80 (March 21). If None, do not reject any files.

endDayOfYear: a Julian day number (1-366) before which to accept files from any given year. This can be used for example to only allow files from Spring, which has a end day of year of 172 (June 21). If None, do not reject any files.

fileList: the list of valid file paths to which is appended any newly-found valid file paths

publicAccessOnly: if 1, only return files marked in fileTab.txt and expTab.txt as public. If 0, return all non-archive files. If 2, return public and public archive experiments with public files. If 3, return all files, including public, private, archived, and private archived.

enforcePathConvention: if 1, only return experiments whose path is of the form convention 1999/mlh/20jan98d (YYYY/<three lower case letters>/DDmmmYY<optional single character>. If 0 (the default), only require paths to be in the form 1999/mlh/*.

includeNonDefault: if 1, also include data files that are not default files in they match all other criteria. If 0 (the default), exclude non-default.

includeNonMadrigal: if 1, include all experiment files that are not in fileTab.txt. If 0, (the default) limit data files to those in fileTab.txt.

appendKinst: if 1, append kind of instrument integer to each file name, so that what is returned is a list of (file name, inst code) tuples. If 0 (the default), do not append instrument code; return a list of file names.

appendStartTime: if 1, append start time in seconds since 1950 to each file name, so that what is returned is a list of (file name, startTime) tuples. If 0 (the default), do not append startTimes.

includeRealtime: if 1, include realtime files even if includeNonDefault = 0. If 0 (default), do not include realtime if includeNonDefault = 0.

dirname: the directory name of the present directory

names: the list of filenames in the present directory

Returns: None.

Affects: Adds full file paths of any data files found that meet all filter criteria

Exceptions: None

__initFromHardCode ( self )

__initFromHardCode is a private helper function that reads information from the automatically editted lines.

Inputs: None

Returns: Void.

Affects: Initializes class member variables that are found in the constants at the top of this file. These constants were set during installation.

Exceptions: None.

__init__ ( self,  initFile=None )

__init__ initializes the MadrigalDB by reading from $MAD_ROOT/madrigal.cfg (or initString).

Inputs: String representing the full path to the configuration file. Default is None, in which case configuration file is $(__MAD_ROOT)/__confFileName (now madrigal.cfg).

Returns: void

Affects: Initializes all the class member variables.

Exceptions: MadrigalError thrown if error (see class description for specific conditions).

Notes:

Given the file $MAD_ROOT/madrigal.cfg (these names are set by constants above), or initFile, this constructor goes about loading some basic data about the database, including information such as:

-the database utility directory

-the www home base

-the cgi home base

Implemented using ConfigParser module. This reads ini type files. The only difference between ini files and madrigal.cfg is that ini files are broken into sections of the form:

[sectionTitle]

key1 = value1

key2 = value2

Since madrigal.cfg (or another initFile) has no section header, one section head called "madrigal" is appended to the beginning of the file.

__isValidEmail ( self,  emailStr )

__isValidEmail is a private helper function that does some checking to ensure a valid email address was found.

Inputs: emailStr - email address string to verify

Returns: 1 if no problems, 0 if not valid.

Affects: Nothing

Exceptions: None

__readConfFile ( self )

__readConfFile is a private helper function that reads information from the parsed config file.

Inputs: None

Returns: Void.

Affects: Initializes class member variables that are found in the config file.

Exceptions: MadrigalError thrown if any key not found.

__setFileAccess (
        self,
        arg,
        dirname,
        names,
        )

setFileAccess is a private helper function called by os.path.walk in setFileAccess.

Inputs:

arg: either 0 for public access, or 1 for private access.

dirname: the directory name of the present directory

names: the list of filenames in the present directory

Returns: None

Affects: sets all fileTab.txt files in dirname to be public or private, depending on arg.

Exceptions: None.

__str__ ( self )

__str__ simply calls toString

getBinDir ( self )

getBinDir returns the madrigal bin directory.

Inputs: None

Returns: The madrigal bin directory path. (eg /opt/madrigal/bin)

Affects: Nothing

Exceptions: None

getCGIHomeBase ( self )

getCGIHomeBase returns the full url to the cgi directory.

Inputs: None

Returns: String representing the full url to the cgi directory. (eg, http://haystack.mit.edu/cgi-bin/madrigal)

Affects: Nothing

Exceptions: None

getCgiDirPath ( self )

getCgiDirPath returns the directory path to the main madrigal cgi-bin directory.

Inputs: None

Returns: the directory path to the main madrigal cgi-bin directory. (eg, "/export/home/httpd/apache/cgi-bin/madrigal")

Affects: Nothing

Exceptions: None

getContactEmail ( self )

getContactEmail returns the email address of the site administrator.

Inputs: None

Returns: The email address of the site administrator.

Affects: Nothing

Exceptions: None

getContactLink ( self )

getContactLink returns contact email link tag (see getContactEmail for the email alone).

Inputs: None

Returns: The contact email link tag. (eg. <A HREF="MAILTO:brideout@haystack.mit.edu">madrigal@haystack</A><BR>)

Affects: Nothing

Exceptions: None

getDatabaseUtilityDirectory ( self )

getDatabaseUtilityDirectory returns the full path to the database utility directory.

Inputs: None

Returns: String representing the full path to the database utility directory. (eg, /opt/madrigal/bin)

Affects: Nothing

Exceptions: None

getDocDirPath ( self )

getDocDirPath returns the directory path to the main madrigal html directory.

Inputs: None

Returns: the directory path to the main madrigal html directory. (eg, "/export/home/httpd/apache/htdocs/madrigal")

Affects: Nothing

Exceptions: None

getExpList (
        self,
        expName=None,
        kinstList=None,
        startDate=None,
        endDate=None,
        startDayOfYear=None,
        endDayOfYear=None,
        showIgnoredExperiments=False,
        enforcePathConvention=False,
        )

getExpList returns a list of full experiment directory names that match the search arguments.

Inputs:

expName: a string defining what experiment names to accept (case insensitive). Use of unix file matching characters * and ? are allowed. If None (default), any experiment name allowed.

kinstList: a list of kinst (kind of instrument codes) integers that will be accepted. If None (default) or if list contains 0, all kinst values are accepted.

startDate: a datetime date. If None (default), do not reject any experiments.

endDate: a datetime date. If None (default), do not reject any experiments.

startDayOfYear: a Julian day number (1-366) after which to accept experiments from any given year. This can be used for example to only allow files from Spring, which has a start day of year of 80 (March 21). If None (default), do not reject any experiments.

endDayOfYear: a Julian day number (1-366) before which to accept experiments from any given year. This can be used for example to only allow files from Spring, which has a end day of year of 172 (June 21). If None (default), do not reject any experiments.

showIgnoredExperiments: if False (default), ignore experiments where expTab.txt state code is set to ignore. If True, include those experiments.

enforcePathConvention: if True, only return experiments whose path is of the form convention 1999/mlh/20jan98d (YYYY/<three lower case letters>/DDmmmYY<optional single character>. If False (the default), only required path in the form 1999/mlh/*.

Returns: a list of full experiment directory names that match the search arguments

Affects: Nothing

Exceptions: None

getExperimentDir ( self )

getExperimentDir returns the main experiment directory. No longer guarenteed to be unique,but will hold test files and geophysical experiments.

Inputs: None

Returns: The full experiment directory path. (eg. /opt/madrigal/experiments)

Affects: Nothing

Exceptions: None

getExperimentDirs ( self )

getExperimentDirs returns a list of full paths of all valid experiment directories.

Inputs: None

Returns: list of full paths of all valid experiment directories. Valid experiment directories are those of the form of the regular expression $/home/grail/brideout/madroot/experiments[0-9]*

Affects: Nothing

Exceptions: None

getFileList (
        self,
        expName=None,
        kinstList=None,
        kindatList=None,
        startDate=None,
        endDate=None,
        startDayOfYear=None,
        endDayOfYear=None,
        publicAccessOnly=3,
        enforcePathConvention=0,
        includeNonDefault=0,
        includeNonMadrigal=0,
        appendKinst=0,
        appendStartTime=0,
        includeRealtime=0,
        )

getFileList returns a list of full file names that match the search arguments.

Inputs:

expName: a string defining what experiment names to accept (case insensitive). Use of unix file matching characters * and ? are allowed. If None (default), any experiment name allowed.

kinstList: a list of kinst (kind of instrument codes) integers that will be accepted. If None (default) or if list contains 0, all kinst values are accepted.

kindatList: a list of kindat (kind of data codes) integers that will be accepted. If None (default) or if list contains 0, all kindat values are accepted.

startDate: a python date (see time module - actually a tuple of nine integers) after which to accept files. If None (default), do not reject any files.

endDate: a python date (see time module - actually a tuple of nine integers) before which to accept files. If None (default), do not reject any files.

startDayOfYear: a Julian day number (1-366) after which to accept files from any given year. This can be used for example to only allow files from Spring, which has a start day of year of 80 (March 21). If None (default), do not reject any files.

endDayOfYear: a Julian day number (1-366) before which to accept files from any given year. This can be used for example to only allow files from Spring, which has a end day of year of 172 (June 21). If None (default), do not reject any files.

publicAccessOnly: if 1, only return files marked in fileTab.txt and expTab.txt as public. If 0, return all non-archive files. If 2, return public and public archive experiments with public files. If 3, return all files, including public, private, archived, and private archived. (the default)

enforcePathConvention: if 1, only return experiments whose path is of the form convention 1999/mlh/20jan98d (YYYY/<three lower case letters>/DDmmmYY<optional single character>. If 0 (the default), only require paths to be in the form 1999/mlh/*.

includeNonDefault: if 1, also include data files that are not default files in they match all other criteria. If 0 (the default), exclude non-default.

includeNonMadrigal: if 1, include all experiment files that are not in fileTab.txt. If 0, (the default) limit data files to those in fileTab.txt.

appendKinst: if 1, append kind of instrument integer to each file name, so that what is returned is a list of (file name, inst code) tuples. If 0 (the default), do not append instrument code; return a list of file names.

appendStartTime: if 1, append start time in seconds since 1950 to each file name, so that what is returned is a list of (file name, startTime) tuples. If 0 (the default), do not append startTimes.

includeRealtime: if 1, include realtime files even if includeNonDefault = 0. If 0 (default), do not include realtime if includeNonDefault = 0.

Returns: a list of full path file names (strings) that match the search arguments

Affects: Nothing

Exceptions: None

getFileListFromMetadata (
        self,
        expName=None,
        kinstList=None,
        kindatList=None,
        startDate=None,
        endDate=None,
        startDayOfYear=None,
        endDayOfYear=None,
        publicAccessOnly=3,
        includeNonDefault=0,
        appendKinst=0,
        appendStartTime=0,
        includeRealtime=0,
        )

getFileListFromMetadata returns a list of full file names that match the search arguments using metadata only.

This method is very similar to getFileList, except that it assumes the metadata is correct, and doesn't search the actual experiment directories. It is therefore faster, but less robust.

Inputs:

expName: a string defining what experiment names to accept (case insensitive). Use of unix file matching characters * and ? are allowed. If None (default), any experiment name allowed.

kinstList: a list of kinst (kind of instrument codes) integers that will be accepted. If None (default) or if list contains 0, all kinst values are accepted.

kindatList: a list of kindat (kind of data codes) integers that will be accepted. If None (default) or if list contains 0, all kindat values are accepted.

startDate: a python date (see time module - actually a tuple of nine integers) after which to accept files. If None (default), do not reject any files.

endDate: a python date (see time module - actually a tuple of nine integers) before which to accept files. If None (default), do not reject any files.

startDayOfYear: a Julian day number (1-366) after which to accept files from any given year. This can be used for example to only allow files from Spring, which has a start day of year of 80 (March 21). If None (default), do not reject any files.

endDayOfYear: a Julian day number (1-366) before which to accept files from any given year. This can be used for example to only allow files from Spring, which has a end day of year of 172 (June 21). If None (default), do not reject any files.

publicAccessOnly: if 1, only return files marked in fileTab.txt and expTab.txt as public. If 0, return all non-archive files. If 2, return public and public archive experiments with public files. If 3, return all files, including public, private, archived, and private archived. (the default)

includeNonDefault: if 1, also include data files that are not default files in they match all other criteria. If 0 (the default), exclude non-default.

appendKinst: if 1, append kind of instrument integer to each file name, so that what is returned is a list of (file name, inst code) tuples. If 0 (the default), do not append instrument code; return a list of file names.

appendStartTime: if 1, append start time in seconds since 1950 to each file name, so that what is returned is a list of (file name, startTime) tuples. If 0 (the default), do not append startTimes.

includeRealtime: if 1, include realtime files even if includeNonDefault = 0. If 0 (default), do not include realtime if includeNonDefault = 0.

Returns: a list of full path file names (strings) that match the search arguments, and possibly kinst and starttime.

Affects: Nothing

Exceptions: None

getFullPathFromPartial ( self,  partialPath )

getFullPathFromPartial returns the full path to a file or directory based on a partial path.

Follows the rule that if partialPath begins with [/]experiments, then the full path is madroot + partialPath. Otherwise (pre madrigal 2.6) full path is madroot + experiments + partialPath

Input: parial path (eg, 2010/mlh/15jan10 or experiments10/2010/mlh/15jan10)

Returns: full path (eg /opt/madrigal/experiments10/2010/mlh/15jan10)

getHtmlStyle ( self )

getHtmlSyle returns the default html body tag for the site.

Inputs: None

Returns: The default html body tag for the site. (eg. <BODY BGCOLOR=#FFFF88 LINK=#008000 VLINK=#003366>)

Affects: Nothing

Exceptions: None

getIndexHead ( self )

getIndexHead returns the heading of the top level madrigal page.

Inputs: None

Returns: The heading of the top level madrigal page. (eg. Welcome to the Madrigal Database <BR> at Ishtar)

Affects: Nothing

Exceptions: None

getLocalRulesOfRoad ( self )

getLocalRulesOfRoad returns the local rules of the road.

Inputs: None

Returns: If the file madroot/local_rules_of_the_road.txt exists, returns the text of that. Else returns a default rules_of_road statement

Affects: Nothing

Exceptions: None

getMadServer ( self )

getMadServer returns the full name of the madrigal server (eg, haystack.mit.edu).

Inputs: None

Returns: String representing the url to the main database website.

Affects: Nothing

Exceptions: None

getMadroot ( self )

getMadroot returns the value of the environment variable __MAD_ROOT.

Inputs: None

Returns: The value of the environment variable __MAD_ROOT.

Affects: Nothing

Exceptions: None

getMadrootEnvVarName ( self )

getMadrootEnvVarName returns the name of the environment variable __MAD_ROOT (presently = MAD_ROOT).

Inputs: None

Returns: The name of the environment variable __MAD_ROOT.

Affects: Nothing

Exceptions: None

getMailserver ( self )

getMailserver returns the mailserver name.

Inputs: None

Returns: The mailserver name. If this heading is not found in madrigal.cfg, no error is thrown - simply defaults to localhost

Affects: Nothing

Exceptions: None

getMaxGlobalQueries ( self )

getMaxGlobalQueries returns the maximum number of global queries as an integer.

Inputs: None

Returns: The maximum number of global queries as an int. If this heading is not found in madrigal.cfg, no error is thrown - simply defaults to 2

Affects: Nothing

Exceptions: None

getMaxTempReports ( self )

getMaxTempReports returns the maximum size of the tempReports dir in GB as a float.

Inputs: None

Returns: The maximum nsize of the tempReports dir in GB as a float. If this heading is not found in madrigal.cfg, no error is thrown - simply defaults to 2

Affects: Nothing

Exceptions: None

getMetadataDir ( self )

getMetadataDir returns the metadata directory.

Inputs: None

Returns: The full metadata directory path. (eg. /opt/madrigal/metadata)

Affects: Nothing

Exceptions: None

getPythonExecutable ( self )

getPythonExecutable returns the full path to the python executable.

Inputs: None

Returns: the full path to the python executable. If this heading is not found in madrigal.cfg, no error is thrown - simply defaults to madroot/bin/python

Affects: Nothing

Exceptions: None

getRelativeCGI ( self )

getRelativeCGI returns the relative url to the cgi directory.

Inputs: None

Returns: String representing the relative url to the cgi directory. (eg, cgi-bin/madrigal)

Affects: Nothing

Exceptions: None

getRelativeTopLevel ( self )

getRelativeTopLevel returns the relative url of the top level directory in main database website.

Inputs: None

Returns: String representing the relative url to the top level directory in main database website. (eg, madrigal)

Affects: Nothing

Exceptions: None

getSiteID ( self )

getSiteID returns the site id number.

Inputs: None

Returns: The site id (integer) of the madrigal installation.

Affects: Nothing

Exceptions: If non-integer found

getTopLevelUrl ( self )

getTopLevelUrl returns the full url of the top level directory in main database website.

Inputs: None

Returns: String representing the full url to the top level directory in main database website. (eg, http://haystack.mit.edu/madrigal)

Affects: Nothing

Exceptions: None

getWWWHomeBase ( self )

getWWWHomeBase returns the url to the main database website(eg, http://haystack.mit.edu).

Inputs: None

Returns: String representing the url to the main database website.

Affects: Nothing

Exceptions: None

listFileTimes (
        self,
        expDir=None,
        relative=True,
        )

listFileTimes returns a list of (filename, datetime_ut) of all files in the experiment directory

Inputs:

expDir - the particular subdirectory of an experiment directory to list. If None (the default), will list all files for all experiment directories. May be an absolute path, or may start with experiments[0-9]*.

relative - if True (the default) give the path relative to the experiments[0-9]* directory. If False, give full path

Returns: a list of tuples, where each tuple has 1. the file path, and 2. a UT datetime object of the last file modification.

Exceptions: raised if expDir is not a valid experiments directory or subdirectory

setFileAccess (
        self,
        expDirectory,
        accessMode,
        )

setFileAccess sets all fileTab.txt files in all subdirectories of expDirectory to be public or private.

Inputs:

expDirectory: The full path to a directory in the experiment directory. That is, it may be madroot/experiments[0-9]* or any directory under it.

accessMode: either 0 for public access, or 1 for private access.

Returns: None

Affects: sets all fileTab.txt files in all subdirectories of expDirectory to be public or private.

Exceptions: If accessMode is not 1 or 0.

tarExperiments (
        self,
        tarFileName,
        startDate=None,
        endDate=None,
        excludePrivData=0,
        ignoreDirCon=1,
        includeNonDefData=0,
        onlyData=0,
        filetype=0,
        verbose=0,
        )

tarExperiments creates a tar file containing files from madroot/experiments[0-9]*.

Note: this method sometimes requires the modification of the fileTab.txt files found in the experiments directory. This is because some data files might be excluded, so that the fileTab.txt file will no longer be accurate. Because of this, all files to be tar'ed will be copied to /tmp/temp<random num>/experiments, where the fileTab.txt files will be modified. When done, this temp dir will be deleted.

Inputs:

tarFileName: The full path to a tar file to be created.

startDate: a python date (see time module - actually a tuple of nine integers) after which to accept files. If None (default), do not reject any files.

endDate: a python date (see time module - actually a tuple of nine integers) before which to accept files. If None (default), do not reject any files.

excludePrivData: if 1, allow data marked as private to be omitted (and the line from the fileTab.txt to be removed). If 0 (the default), all data, public and private, will be included.

ignoreDirCon: if 1, ignore convention that directory must be in form 1999/mlh/03sep99 (the default). If 0 , reject non-standard directories.

includeNonDefData: if 1, include all files listed in fileTab.txt. If 0 (the default), reject non-default files, and modify fileTab.txt to remove non-default listings.

onlyData: if 1, reject all files not listed in fileTab.txt. If 0 (the default), accept all files in a directory not mentioned in fileTab.txt.

filetype: format to save data files as. Default 0 is to leave present format unchanged. <type> is an integer as follows:

type = 0 Leave present format unchanged (default)

type = 1 Madrigal

type = 2 Blocked Binary

type = 3 Cbf

type = 4 Unblocked binary

type = 5 Ascii

verbose: if 1, print to std out the list of files included (relative path). If 0, (the default) print nothing.

Returns: None

Affects: created tar file tarFileName of selected files from madroot/experiments.

Exceptions: If unable to read any experiment file.

toString ( self )

toString returns a simple string representation of a MadrigalDB object.

Inputs: None

Returns: String describing a simple representation of a MadrigalDB object.

Affects: Nothing

Exceptions: None