Formula Construction and Evaluation
Formulas are textual expressions which specify computations that yield datasets e.g. images. Each frame currently contains a single formula which fully describes its image data. The methods used to display this data fall under the separate heading of "Display Parameters".
Every frame has a formula associated with it. This applies to blank frames also. To view the formula associated with a frame use the following equivalent methods:
Using the first two techniques will bring up the Formula Panel with the last formula that was being viewed or edited displayed in the text field. In order to see the formula associated with a particular frame requires changing the Frame number in the Connection Widget as described in Section 5. Formula evaluation mode and the arithmetic approach (operator library) to use are specified using pop-up menus as shown below.
6.2 Frame Types
A frame has the capacity to hold a number of datasets; the exact number depends on the frame's type, and the types currently supported are: "single-channel", and "three-channel". A single-channel frame supports one image, an overlay image, and a text entry. A three-channel frame supports three images (red, green and blue) instead of the one image. The one vs. three designation refers to the "base image" contents of the frame.
The operators in IISS formulas work at the level of image data; operands are typically frames, and it assumed in most cases that the base image data is to be used in the calculation. Operators will behave intuitively when arguments are of unequal types. For example, when adding a single-channel to a three-channel frame, the one dataset will be added to each of the three separately, to produce a three-channel result. The current operators are listed at the end of this overview. Operators can be nested.
Formula syntax provides for functions, arithmetic (infix) operators, frame references, numeric values, text strings and expression grouping. Functions typically have the form:
fcn_name[ arg1->val1, arg2->val2, ... argN->valN ]
The arrow symbol is constructed using two characters, ->. Alternatively, argument names can be omitted for simplicity in some cases:
fcn_name[ val1, val2, ... valN ]
A frame reference is required to have the form:
RowCol[[ Level ]]
where Row is an alphabetic row descriptor, Col is a numeric column descriptor, and Level is a numeric index into a frame-stack. The brackets [[ ]] define the data extraction operator.
6.5 Reading vs. Computation
Instead of having separate mechanisms for reading data and then computing over it, the IISS formulas accommodate both, to reduce unnecessary intermediate frames, and allow for compact expressions. The Read Panel is a convenient way for the user to construct one or more formulas for data input (i.e. Read*[ ] operators) and filling in frame data information. It is especially convenient when viewing image sequences with hundreds to thousands of images. The image sequence can usually be read into the spreadsheet frames using just one access to the Read Panel to construct multiple formulas for all required frames.
Reading data into the SpreadSheet is a two step
process: first one must construct a formula which includes a Read
function; then the formula must be evaluated. The first step can be
done via the Formula form, but there are two simpler methods which can
be used to construct Read formulas. When starting IISS from the UNIX
command line, the option -Qr,c can be followed by file names and
arguments describing those files, and the program will automatically
construct formulas using this information.
Also, the "Read" button on the Options Panel will pop up a
Read Panel, which can be used to construct a Read formula. There are
several data formats currently supported including RAW file format,
with variable header size, row/column dimensions, multiple channels,
region of interest subsetting, rotation and flipping, and support for
several different data types. When RAW three-channel frames are being
constructed, the file pattern must include a three way extension of the
form "[red_infix, green_infix, blue_infix]". Other formats
are HDF, HDF L2, Vis5D, GENERIC (such as JPEG, TIFF, GIF, SGI, SUN,
PNM, SLCCA, MPEG, VSLCCA) as described in Section 3 on Spreadsheet I/O.
Also, the "Read" button on the Options Panel will pop up a Read Panel, which can be used to construct a Read formula. There are several data formats currently supported including RAW file format, with variable header size, row/column dimensions, multiple channels, region of interest subsetting, rotation and flipping, and support for several different data types. When RAW three-channel frames are being constructed, the file pattern must include a three way extension of the form "[red_infix, green_infix, blue_infix]". Other formats are HDF, HDF L2, Vis5D, GENERIC (such as JPEG, TIFF, GIF, SGI, SUN, PNM, SLCCA, MPEG, VSLCCA) as described in Section 3 on Spreadsheet I/O.
6.6 Formula Panel & Evaluation
The Formula Panel associated with a frame (example shown below) can be displayed using the three methods described in Section 6.1.
Formulas using the IISS formula language (for syntax and operator list see Appendix C) can be constructed by typing in the formula directly into the text field of the Formula Panel as shown below. Formula operators operate on complex data types using operator overloading. Once the formula has been created then it can be stored without evaluation using the Store button, or evaluated using the Evaluate button.
The formula associated with a frame may not have been evaluated if the sheet is in manual evaluation mode. Or the frame data could become out of date, if for example the file has been updated after the frame was evaluated. In this case the image displayed in the IISS frame and the data in the newly modified file are out of date (since the IISS does not automatically check timestamps on files). In this situation the formula can simply be re-evaluated in which case the modified file would be re-read and the new data displayed in the frame as expected. The formula can be re-evaluated using three methods:
In the second method it is expected that the formula has not been previously evaluated. This is usually the situation if the sheet is in manual evaluation mode as shown in the pop-up menu above.
6.7 Dependencies and Evaluation Order
When one frame's formula holds a reference to another frame, the former frame is said to depend on the latter. Using dataflow terminology, if frame B is dependent on frame A, frame B is downstream of A, and A is upstream of B. Moreover, the order of evaluation should be in a downstream direction: if frame B gets the signal to evaluate, it will check to see if A needs evaluating, before starting its own computation. This is done recursively. Circular dependencies are possible, and evaluation will not get caught in an infinite loop; however saving and restoring a spreadsheet with such references is not possible at this time, since data is not stored, only formulas.
6.8 Delayed Evaluation
In traditional financial spreadsheets, computation is rapid and therefore usually automatic. However, in the Interactive Image SpreadSheet computations are done over images, and can therefore be expensive in terms of time as well as memory resources. Flexibility is provided to the user by offering three modes of evaluation within the IISS and granting the user control over the scheduling of frame evaluation. In automatic evaluation mode, changes to a frame's formula result in immediate recalculation, and all downstream frames are also updated. In manual evaluation mode, the user must initiate evaluation for each frame individually. And in the view driven mode, the first time a frame needs to be displayed, it is automatically evaluated. Subsequenct reevaluations which may be required with circular frame references, need to be initiated explicitly by the user in view driven mode .
6.9 Evaluation Status
A frame's evaluation status reflects whether its (displayed) data corresponds to its formula. If a frame's formula is updated but not re-evaluated, the frame's data is said to be out-of-date. This is also true if any upstream frames have been more recently evaluated, or they themselves are out of date. Upon evaluation, any upstream frames which are out of date get re-evaluated, and the frame itself does also. At that point, the frame is up-to-date. The exception to this would be a frame in a circular dependency loop, which is always out of date i.e. ready to be re-evaluated. The other evaluation status possibilities are 'currently evaluating' and 'error'.
6.10 Display After Evaluation
After a frame is successfully evaluated, it is set to be the current frame in its cell, and it is displayed in the cell window (if that window is in view). When cells are grouped together, post-evaluation display becomes more complicated, since setting the frame index for one cell may put that cell out of step with its partners in the group. Three options are available: frames in grouped cells can (a) not be shown, (b) be shown for a moment, or (c) be shown until the current phase of evaluation has finished. In the last two cases, the cell will return to its previously shown frame.
When a frame is evaluated due to a direct user action (not as a byproduct of another frame's evaluation), that frame is called a "root frame" of the evaluation. A root frame in a group can be set to display as described above, but it has two other options as well: (d) it can become the current frame in its cell with no effect on the other cells in the group, or (e) all cells in the group can be absolutely set to the root frame index.
The group dynamic behaviour of hidden frames and the root frame during formula evaluation and at the conclusion of the evaluation process can be specified using the pop-up menu items shown below.
6.11 Khoros Requirements
Formula evaluation currently depends on Khoros
binary executables running as separate processes, using temporary files
as a means for communicating intermediate data. Data is converted to
floating point representation for the maintaining precision during
computation, and then it is converted to 8-bit format for display.
Temporary floating point files must fit in
/usr/tmp). Khoros binary executables must be in
your path. Khoros cannot be distributed in conjunction with IISS, due
to copyright restrictions; instead it should be downloaded from
Internet and compiled.
6.12 Normalizing Data
Each frame computed using a formula that involves floating point calculations may need to be normalized into a one-byte dynamic range for display. The "Normalize" button in the Formula Panel when set will automatically normalize the resulting datasets before they are displayed. The normalization has global scope and needs to be reset for each new frame formula using the Formula Panel.