Talk:A two component pyre example

From DANSE

Discussion response

Even though this approach doesn't play as well with a component framework, because only 'Application' is used, here is an alternate approach...

There is a simple way I can think of... If your applications/components are structured similar to the following code snippets:

class Foobar(Application):
    class Inventory(Application.Inventory):
        import pyre.inventory
        command = pyre.inventory.str('command', default='')
        options = pyre.inventory.str('options', default='')
        ...

    def config(self, **kwds):
        for key,value in kwds.items():
            if key == 'command':
                self.inventory.command = value
            elif key == 'options':
                self.inventory.options = value
            ...

    def __init__(self, name, **kwds):
        Application.__init__(self, name)
        self.config(**kwds)
        ...

    ...

Then, another component can call your component (named "foo") with either:

foo_obj = foo.Foobar('example', command="mm", options="clean")

foo_obj = foo.Foobar('example2')

foo_obj.config(command="mm", options="clean")

Examples of this are currently found in:
gsl/infect/infecter.py
gsl/infect/loader.py
gsl/infect/builder.py

Another Example

This code uses fileIO.py (originally written for the Cobra client), which has functions for generic table or N-column file reading and writing. A driver "Table.py" uses "TableReader.py" and "TableWriter.py" to read a table from a file, possibly manipulate the data (exchanging rows & columns for example), then write the table to a file.

TableReader

#!/usr/bin/env python
#
# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# ReadTable.py
#
# 7/21/2004 version 0.0.1a
# mmckerns@caltech.edu
# (C) 2004 All Rights Reserved
#
# <LicenseText>
#
# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
#
__author__ = 'Mike McKerns'

from pyre.applications.Application import Application
import fileIO
import os

class TableReadError(Exception):
    '''Exception for improper TableReader usage'''
    pass

class TableReader(Application):
    '''table reader -- decends from cobra152/Local/Scripting/ReadTable.py'''
    class Inventory(Application.Inventory):
        '''Inventory declares and stores user modifiable variables'''
        import pyre.inventory    #for pythia0.6
        filename = pyre.inventory.str('filename', default='')
        intoRows = pyre.inventory.bool('intoRows', default=False)
#       return

    def config(self, **kwds):
        '''config modifies the variables used in the inventory by
accepting a dictionary of keywords and values; this dictionary can be
passed directly as a function call, or indirectly using the
commandline parser'''
        for key,value in kwds.items():
            if key == 'filename':
                self.inventory.filename = value
            elif key == 'intoRows':
                self.inventory.intoRows = value
        return

    def run(self):
        '''the main code block'''
        #pass inventory into non-global variables (function calls are tidy)
        filename = self.inventory.filename
        intoRows = self.inventory.intoRows
        #check for valid inputs
        if not os.path.isfile(filename):
            if filename:
                print "Error: filename '%s' not a regular file" % filename
            else:
                print "Error: No filename specified"
            #raise TableReadError
            return
        #convert bool to int
        if intoRows:  intoRows = 1
        else: intoRows = 0
        #put outputs
        self.header,self.data = fileIO.readTable(filename,intoRows=intoRows)
        if intoRows: type = 'row'
        else: type = 'column'
        print "data read by %s from '%s'" % (type,filename)
        return

    def __init__(self, name, **kwds):
        '''instantiate the application, and pass any keywords to config'''
        Application.__init__(self, name)
        self.config(**kwds)
        self.data = []
        self.header = []
        return

# main
if __name__ == '__main__':
    '''begin journaling services to log input/output/errors for
TableReader instance, and then run the main code block'''
    import journal
    rt = TableReader('test')  #instance of class TableReader (named 'test')
    journal.debug('test').activate()  #activate journal for 'test'
    rt.main()  #launch the main code block ('TableReader.run')
    print "header = %s" % rt.header
    print "data = %s" % rt.data

# version
__id__ = "$Id$"

# End of file

TableWriter

#!/usr/bin/env python
#
# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# WriteTable.py
#
# 7/21/2004 version 0.0.1a
# mmckerns@caltech.edu
# (C) 2004 All Rights Reserved
#
# <LicenseText>
#
# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
#
__author__ = 'Mike McKerns'

from pyre.applications.Application import Application
import fileIO
import os

class TableWriteError(Exception):
    '''Exception for improper TableWriter usage'''
    pass

class TableWriter(Application):
    '''table reader -- decends from cobra152/Local/Scripting/WriteTable.py'''
    class Inventory(Application.Inventory):
        '''Inventory declares and stores user modifiable variables'''
        import pyre.inventory    #for pythia0.6
        filename = pyre.inventory.str('filename', default='')
        data = pyre.inventory.list('data', default=[])
        header = pyre.inventory.list('header', default=[])
        isRows = pyre.inventory.bool('isRows', default=False)
#       return

    def config(self, **kwds):
        '''config modifies the variables used in the inventory by
accepting a dictionary of keywords and values; this dictionary can be
passed directly as a function call, or indirectly using the
commandline parser'''
        for key,value in kwds.items():
            if key == 'filename':
                self.inventory.filename = value
            elif key == 'data':
                self.inventory.data = value
            elif key == 'header':
                self.inventory.header = value
            elif key == 'isRows':
                self.inventory.isRows = value
        return

    def run(self):
        '''the main code block'''
        #pass inventory into non-global variables (function calls are tidy)
        filename = self.inventory.filename
        data = self.inventory.data
        header = self.inventory.header
        isRows = self.inventory.isRows
        #check for valid inputs
        if not filename:
            print "Error: filename not specified"
            #raise TableWriteError
            return
        #if os.path.exists(filename):
        #    print "Error: filename '%s' exists" % filename
        #    #raise TableWriteError
        #    return
        if not data:
            print "Error: no data"
            #raise TableWriteError
            return
        #convert bool to int
        if isRows:  isRows = 1
        else: isRows = 0
        #put outputs
        outfile = fileIO.writeTable(filename,data,header=header,isRows=isRows)
        if isRows: type = 'row'
        else: type = 'column'
        print "data written by %s to '%s'" % (type,outfile)
        return

    def __init__(self, name, **kwds):
        '''instantiate the application, and pass any keywords to config'''
        Application.__init__(self, name)
        self.config(**kwds)
        return

# main
if __name__ == '__main__':
    '''begin journaling services to log input/output/errors for
TableWriter instance, and then run the main code block'''
    import journal
    rt = TableWriter('test')  #instance of class TableWriter (named 'test')
    journal.debug('test').activate()  #activate journal for 'test'
    rt.main()  #launch the main code block ('TableWriter.run')

# version
__id__ = "$Id$"

# End of file

Table

#!/usr/bin/env python
#
# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# Table.py
#
# 7/21/2004 version 0.0.1a
# mmckerns@caltech.edu
# (C) 2004 All Rights Reserved
#
# <LicenseText>
#
# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
#
__author__ = 'Mike McKerns'

from pyre.applications.Application import Application
from ReadTable import TableReader
from WriteTable import TableWriter
import os

class TableError(Exception):
    '''Exception for improper Table usage'''
    pass

class Table(Application):
    '''table  -- driver for ReadTable and WriteTable'''
    class Inventory(Application.Inventory):
        '''Inventory declares and stores user modifiable variables'''
        import pyre.inventory    #for pythia0.6
        reader = pyre.inventory.facility('reader', default=TableReader("read"))
        writer = pyre.inventory.facility('writer', default=TableWriter("write"))
        read_commands = pyre.inventory.str('read_commands', default='')
        write_commands = pyre.inventory.str('write_commands', default='')
#       return

    def config(self, **kwds):
        '''config modifies the variables used in the inventory by
accepting a dictionary of keywords and values; this dictionary can be
passed directly as a function call, or indirectly using the
commandline parser'''
        for key,value in kwds.items():
            if key == 'reader':
                self.inventory.reader = value
            elif key == 'writer':
                self.inventory.writer = value
            elif key == 'read_commands':
                self.inventory.read_commands = value
            elif key == 'write_commands':
                self.inventory.write_commands = value
        return

    def run(self):
        '''the main code block'''
        #pass inventory into non-global variables (function calls are tidy)
        reader = self.inventory.reader
        writer = self.inventory.writer
        read_commands = self.inventory.read_commands
        write_commands = self.inventory.write_commands
        #reader/writer configuration
        if read_commands:
            exec 'reader.config('+read_commands+')'
        if write_commands:
            exec 'writer.config('+write_commands+')'
        #transfer data
        reader.main()
        writer.config(data=reader.data, header=reader.header)
        writer.main()
        return

    def __init__(self, name, **kwds):
        '''instantiate the application, and pass any keywords to config'''
        Application.__init__(self, name)
        self.config(**kwds)
        return

# main
if __name__ == '__main__':
    '''begin journaling services to log input/output/errors for Table
instance, and then run the main code block'''
    import journal
    t = Table('test')  #instance of class Table (named 'test')
    journal.debug('test').activate()  #activate journal for 'test'
    t.config(read_commands="filename='foo.py', intoRows=True")
    t.config(write_commands="filename='bar.py', isRows=False")
    t.main()  #launch the main code block ('Table.run')

# version
__id__ = "$Id$"

# End of file

MoreDiscussion

Let's start with a sample table found in "foo.py". I make a destinction between tables and N-column data in that tables are immediately readable by python (thus saved in a '.py' file).

# "foo.py": a sample table
header = ["foo", "bar", "baz"]
data = [
       [1,1,1],
       [2,2,2],
       [3,3,3],
       [4,4,4],
]

I can use "ReadTable.py" to read the table. From the command line, it would look like this:

dude@kittel>$ python ReadTable.py --filename=foo.py
 >> /home/mmckerns/dev/pythia-0.6/packages/pyre/pyre/inventory/Configurable.py:53:setCurator
 >> test(debug)
 -- test: setting curator to test(<pyre.inventory.odb.Curator.Curator object at 0x4062594c>)
data read by column from 'foo.py'
header = ['foo', 'bar', 'baz']
data = [[1, 2, 3, 4], [1, 2, 3, 4], [1, 2, 3, 4]]

I can also use "WriteTable.py" to write a new table. From the python interpreter, it would look like this:

dude@kittel>$ python
Python 2.3 (#9, Sep 26 2003, 18:07:03) 
[GCC 3.2 20020903 (Red Hat Linux 8.0 3.2-7)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import WriteTable
>>> wt = WriteTable.TableWriter('test', filename='baz.py')
>>> wt.config(data=[[1,2,3],[4,5,6],[7,8,9]], header=['one','two','three'])
>>> wt.main()
data written by column to 'baz.py'

Where "baz.py" will now contain:

header = ['one', 'two', 'three']
table = [
[1, 4, 7],
[2, 5, 8],
[3, 6, 9],
]

Table is a driver for both ReadTable and WriteTable, and we can run the simple test script (attached to the end of Table.py for simplicity, however usually you'd want it in a seperate file) to read data by rows from "foo.py" and write data by columns to "bar.py":

dude@kittel>$ python Table.py 
 >> /home/mmckerns/dev/pythia-0.6/packages/pyre/pyre/inventory/Configurable.py:53:setCurator
 >> test(debug)
 -- test: setting curator to test(<pyre.inventory.odb.Curator.Curator object at 0x4063278c>)
data read by row from 'foo.py'
data written by column to 'bar.py'

Where "bar.py" will now contain:

header = ['foo', 'bar', 'baz']
table = [
[1, 2, 3, 4],
[1, 2, 3, 4],
[1, 2, 3, 4],
]

Now, "Table" could also be expanded to similarly incorporate "TableMath" (a component that implements math operations on table elements) and "TableManipulation" (a component that manipulates the order of table elements).

Another small point of discussion is on error propagation -- the above examples catch errors by printing an error message and exiting the function with a return. This is not the best behavior to exhibit -- better is to raise an error, and let the top level application worry about the errors. So above, wherever there is a commented out "TableReaderError" or "TableWriterError", they really are best uncommented if the intent is to use them under "Table".

Also, similar components can be written for generic N-column readers/writers using the functions in 'fileIO.py'. The N-column reader/writer functions included in 'fileIO.py' allow spicification of a dialect -- so you can write 'acsii', 'comma-delimited', 'tab-delimited', or whatever you like. Further "fileIO.py" includes a function to auto-detect the dialect of a file it is trying to read. If you are interested, 'fileIO.py' and more recent versions of the above files can be found in the repository on arcs.cacr.caltech.edu in tutorials/reader-writer.

-- McKerns