Remote access using python

This page describes the remote Python API, and gives some examples of using this API. These examples have been tested on both Windows and Linux, and require only access to the internet and python 2.3 to run. It is available for download here.

The remote Python API is organized in the same way as the Madrigal data model, from Instrument at the highest level, down to the level of data values. Readers who are not familiar with the Madrigal data model should read the material in that section before proceeding with this tutorial.

The basic object in the remote Python API is the MadrigalData, found in the madrigalWeb module. To initialize MadrigalData requires only the url of the home page on any Madrigal 2.3 (or above) site as an argument. Calling the methods of this object will return all possible information from one Madrigal site. The other objects in madrigalWeb are simply there to hold returned information - for example, the MadrigalExperiment object holds information about one experiment.

MadrigalData has the following methods:

getAllInstruments - returns a list of all MadrigalInstrument objects at all Madrigal sites
getExperiments - returns a list of all MadrigalExperiment objects that meet the criteria you set at any Madrigal site
getExperimentFiles - returns a list of all default MadrigalExperimentFile objects for a given experiment id
getExperimentFileParameters - returns a list of all measured and derivable parameters in file
isprint - returns as a string the isprint output given file, parms, filters without headers or summary. This is the method that accesses the raw data in a Madrigal data file.
madCalculator - returns derived parameters for a range of latitudes, longitudes, and altitudes at a given time. Note that this method does not returned measured values from a file - use isprint for that. Instead, it is useful for accessing parameters available via the Madrigal derivation engine, such as magnetic field or MSIS parameters.
madTimeCalculator - is similar to madCalculator, except that it returns data from a range of times, but only returns parameters such as Kp that are independent of position.
radarToGeodetic - converts arrays of az, el, and ranges to geodetic locations.
geodeticToRadar - converts arrays of points in space to az, el, and range.

See the Madrigal Python API reference guide for complete documentation.

Two applications written with the remote Python API follow. The first is a simple regression test that is run to test web services when Madrigal is installed. The second is a script that downloads realtime data from any desired Madrigal site.

Simple regression test

This simple script calls the following MadrigalData methods:

getAllInstruments
getExperiments
getExperimentFiles
isprint
madCalculator

To use this regression test, cd to the examples directory in the installation directory, and type:

python testMadrigalWebServices.py http://www.haystack.mit.edu/madrigal

import madrigalWeb.madrigalWeb
import sys
import string
import difflib

if len(sys.argv) < 2:
    print 'usage: python testMadrigalWebServices.py <madrigal main url>'
    sys.exit(-1)

madrigalUrl = sys.argv[1]

outFile = open('testMadrigalWebServices.out', 'w')

# create the main object to get all needed info from Madrigal
testData = madrigalWeb.madrigalWeb.MadrigalData(madrigalUrl)

instList = testData.getAllInstruments()

# print out Millstone
for inst in instList:
    if inst.code == 30:
        outFile.write(str(inst) + '\n')
        

expList = testData.getExperiments(30, 1998,1,19,0,0,0,1998,1,22,0,0,0)

for exp in expList:
    # should be only one
    outFile.write(str(exp) + '\n')

fileList = testData.getExperimentFiles(expList[0].id)

for file in fileList:
    if string.find(file.name, 'mil980120g.002') != -1:
        outFile.write(str(file) + '\n')
        thisFilename = file.name
        break

outFile.write(testData.isprint(thisFilename,
                               'gdalt,ti',
                               'filter=gdalt,500,600 filter=ti,1900,2000'))
outFile.write('\n')

result = testData.madCalculator(1999,2,15,12,30,0,45,55,5,-170,-150,10,200,200,0,'bmag,bn')

for line in result:
    for value in line:
        outFile.write('%8.2e ' % (value))
    outFile.write('\n')

outFile.write('\n')

outFile.close()

Script to download realtime data from Madrigal

The following is a demonstration script that shows how real-time data can be imported from any Madrigal site that is updated on a real-time basis.

In this example, data is imported from http://www.haystack.mit.edu/madrigal from "Millstone Hill IS Radar". The following Madrigal parameters are retrieved:

year,month,day,hour,min,sec,gdlat,glon,gdalt,az,el,vo,dvo

for all records from the past 15 minutes.

Although the particular Madrigal site (http://www.haystack.mit.edu/madrigal), the instrument ("Millstone Hill IS Radar"), the parameters, and the times are hard-coded in this example, they could be easily be modified to be arguments.

To avoid missing data, we choose one parameter to be the filter parameter: vo. By filtering on this parameter, any "missing" values are filtered out.

To run this script requires the python Madrigal API be installed, which can be downloaded from http://www.haystack.edu/madrigal/madDownload.html.



import os,sys,os.path
import string
import time


import madrigalWeb.madrigalWeb


#constants
madrigalUrl = 'http://www.haystack.mit.edu/madrigal'
instrument = 'Millstone Hill IS Radar'

user_fullname = 'Put your name here!!!'
user_email = 'your@email.here'
user_affiliation = 'Put your affiliation here!!!'


# each line of data contains the following parameters
params = 'year,month,day,hour,min,sec,gdlat,glon,gdalt,azm,elm,vo,dvo'
filterParm = 'vo'
timeDelay = 15

# create the main object to get all needed info from Madrigal
madrigalObj = madrigalWeb.madrigalWeb.MadrigalData(madrigalUrl)


# these next few lines convert instrument name to code
code = None
instList = madrigalObj.getAllInstruments()
for inst in instList:
    if inst.name.lower() == instrument.lower():
        code = inst.code
        break

if code == None:
    raise ValueError, 'Unknown instrument %s' % (instrument)


# next, get a list of real time experiments in the last timeDelay minutes
startTime = time.gmtime(time.time() - timeDelay*60.0)
endTime = time.gmtime(time.time())


try:
    expList = madrigalObj.getExperiments(code, startTime[0],
                                     startTime[1],
                                     startTime[2],
                                     startTime[3],
                                     startTime[4],
                                     startTime[5],
                                     endTime[0],
                                     endTime[1],
                                     endTime[2],
                                     endTime[3],
                                     endTime[4],
                                     endTime[5])

except:
    raise ValueError, 'No realtime experiments found'


# assume there's only one realtime experiment, and get the file names
fileList = madrigalObj.getExperimentFiles(expList[0].id)

if len(fileList) == 0:
    raise ValueError, 'No realtime experiment files found'


# get data from each of the files
startDateStr = time.strftime('%m/%d/%Y', startTime)
startDateStr = ' date1=' + startDateStr
startTimeStr = time.strftime('%H:%M:%S', startTime)
startTimeStr = ' time1=' + startTimeStr
endDateStr = time.strftime('%m/%d/%Y', endTime)
endDateStr = ' date2=' + endDateStr
endTimeStr = time.strftime('%H:%M:%S', endTime)
endTimeStr = ' time2=' + endTimeStr

filterString = 'filter=%s,-1E30,1E30' % (filterParm) + startDateStr + startTimeStr + endDateStr + endTimeStr
for dataFile in fileList:
    result = madrigalObj.isprint(dataFile.name, params, filterString,
                                 user_fullname, user_email, user_affiliation)
    # make sure it succeeded
    if result.find('No records were selected') != -1:
        continue
    if result.find('****') != -1:
        continue
    print result