EOSDIS Science Data Processor

February 1995

Editor Introduction

Concentrated Efforts


Successful FOS Preliminary Design Review
NRA/CAN Workshop
Intercomparison, Visualization, and Analysis
Tools and Techniques for Automating Analysis
Intgelligent Information Fusion and Management
The Extension and Application of LinkWinds
Efforts of the HDF-EOS Team
Data Modeling Activity

Concentrated Efforts

Ted Meyer,Managing Editor
Lesley Knox, CSC, Production Editor

PDRs, Working Groups, and Workshops. The Project is making significant progress in EOSDIS development. The FOS PDR, held December 13-16, 1994, resulted in a total of 133 RIDs, with 24 received during the review. The CSMS PDR, held January 17-20, 1995, so far has resulted in 160 RIDs, with 104 received during the review. The last and final PDR will be held February 13-17 for SDPS. The wrap-up for the ECS preliminary design is February 22, 1995. The Design Working Group, Data Modeling Working Group (DMWG), and Ad Hoc Working Group on Production have been meeting, discussing, and resolving issues pertinent to requirements, data architecture, and data access. The recent CAN and NRA workshops provided the Project and ECS developers insight into on-going research that EOSDIS might benefit from in the future. All in all, the ball is rolling, and progress is being made on all fronts.

This issue of The Processor continues our new feature on NRA/CAN research abstracts. Four NRA abstracts are provided within this issue. A positive review of the FOS PDR is provided by Cal Moore, along with a summary of the NRA/CAN workshop. The DMWG is continuing its work in resolving issues related to Browse Package Specifications, Product Groupings in Data Servers, Earth Data Type Services, and compliance with the new Federal Geographic Data Committee Standards. The HDF team is in the midst of writing an HDF-EOS standard and developing an HDF-EOS library and a viewer (EOSView) for EOS HDF files.

Again, we hope you enjoy this issue of The Processor. Your comments, suggestions, and recommendations for future articles are always welcome.

Successful FOS Preliminary Design Review

Cal Moore, ECS FOS Manager, GSFC
The Preliminary Design Review (PDR) for the Flight Operations Segment (FOS) was held in Landover, MD, on December 13 and 14, 1995. The objective of the PDR was to present the preliminary design of the FOS, which will support the AM-1 spacecraft mission, and to receive approval to proceed to the Critical Design Review (CDR).The AM-1 spacecraft launch is scheduled for June 1998.

The FOS is one of three segments that, together with the Science Data Processing Segment (SDPS) and the Communications and System Management Segment (CSMS), make up ECS. ECS involves the collection and distribution of Earth science data from space and ground-based measurement systems. The FOS is responsible for providing the functionality to maintain spacecraft and instrument health and safety, monitor spacecraft performance, perform spacecraft engineering analysis, and perform high-level monitoring of mission performance. The FOS is required to support up to seven spacecraft concurrently.

The PDR was oriented toward presenting the FOS preliminary design from a scenario perspective to facilitate understanding by the diverse user community-i.e., the PDR addressed the FOS preliminary design from the scheduling activity phase, the real-time activity phase, and the analysis activity phase. The user community that participated in the PDR included representatives from NASA headquarters, the NASA ESDIS project office, the AM-1 project, the AM-1 spacecraft manufacturer, the instrument operations team (CERES, MOPITT, MISR, MODIS, ASTER, and AIRS), and external interfaces (EDOS, Ecom, and ETS).

The PDR was very well received by the Review Panel and the user community. A wrap-up session was held after the PDR at which members of the Review Panel presented their primary impressions of the PDR. Some of the superlatives from the review panel were as follows:

"Excellent job in presenting material," "very good flexible and scalable architecture," "a lot of very innovative processes," "advances a new standard," "major step in the right direction with ground scripts," "provides good automation for users," "sizing reasonable (talking about code)," "very pleased."

The review produced a good forum to validate the Level 4 requirements and to present the FOS design to the user community. Of the 133 RIDs received, only two major issues were identified: a concern of the FOS size in lines of code and the continuing coordination with the instrument and flight operations teams. The FOS team will incorporate the feedback received from the review into the PDR documentation deliverables and will respond to the RIDs by the end of January. As a result, the review board gave the FOS team the approval to proceed to CDR.


Lesley Knox, CSC
The first EOSDIS NRA/CAN workshop was held January 25-27, 1995, at the HAIS facility. The workshop was chaired by Richard McGinnis, NASA HQ (NRA); Gail McConaughy, ESDIS Project; Karen Moe, ESDIS Project; and Nand Lal, GSFC (CAN). The workshop brought together the ESDIS Project, ECS developers, and the researchers who recently received awards for NASA Research Announcement (NRA)-94-MTPE, "Information System Technology Applicable to EOSDIS," and Cooperative Agreement Notice (CAN)-OA-94, "Public Use of the Earth and Space Science Data Over the Internet." The focus of these two NRA and CAN awards is to foster technology for EOSDIS that can be useful in the next 3 to 5 years. A testbed capability currently exists that allows researchers to integrate the software products into a standalone subset of the ECS software for test and demonstration. Selected NRA ands CAN projects that show promise of potential benefit to ECS will be incorporated into a test environment to allow user test and evaluation of the technology.

During the 2 1/2 day workshop, ESDIS Project and Hughes personnel presented research briefings on each NRA/CAN project, along with overviews of the ECS and EOSDIS prototyping activities. Four parallel workshop groups were formed, and each NRA/CAN project was grouped into one of the following categories: Content-Based Search, Data Management, Architecture/ Performance, and User Environments/Analysis. Representatives from each NRA/CAN research project, along with an ESDIS Project facilitator, ECS contractor staff, and other participants, formed the basis of each group. These groups allowed researchers and ESDIS and ECS staff to discuss technology efforts and technology transfer processes into EOSDIS. Researchers discussed their views on where their research work falls into the EOSDIS environment, the products their research produces, what they need from EOSDIS/ECS/ESDIS, and how their work can be migrated into the EOSDIS environment.

The results of the workshop will be posted on a soon to be developed Mosaic home page, along with other information on NRAs and EOSDIS prototyping efforts. CAN information can be found via the World Wide Web at URL http:// sdcd.gsfc.nasa.gov/ISTO/DLT/abstracts.html.

NASA Research Activities: NRAs ands CANs

Lesley Knox, CSC
As presented in an earlier article, the NRA/CAN workshop, held January 25-27, 1995, provided a forum for the recent winners of NRA-94-MTPE and CAN-OA-94. The Processor continues to showcase summaries of these research projects. In this issue, we present abstracts of four NRAs. They are Allan Jacobsen's (budj@apex.jpl.nasa.gov) "Extension and Application of LinkWinds to EOSDIS"; William Campbell(Campbell@sauquoit.gsfc.nasa.gov), "Intelligent Information Fusion and Management Prototype Applicable to EOSDIS"; William Emery of the University of Chicago (emery@orbit.colorado.edu), "Tools and Techniques for Automating the Analysis of EOSDIS Data"; and K. Palaniappan, (palani@gloria.gsfc.nasa.gov), GSFC, "Intercomparison, Visualization, and Analysis Testbed System for EOS Global Assimilated Data Sets and Spacecraft."

Additional information on these NRAs can be obtained by contacting the Principal Investigators (PIs) or their coinvestigators via their email addresses. A complete list of the awards under the NRA will be available soon on the World Wide Web. The list of CAN awards can be found at Web URL http://sdcd.gsfc.nasa.gov/ISTO/DLT/abstracts.html.

Intercomparison, Visualization, and Analysis Testbed System for EOS Global Assimilated Datasets and Satellite Data

K. Palaniappan, NASA/GSFC
EOS will produce an enormous volume of observational and processed data each day. A subset of this satellite data and various other data, including in situ air and surface measurements, will be combined into a comprehensive climate dataset by the Data Assimilation Office (DAO) in the Goddard Laboratory for Atmospheres (GLA). The assimilated data, which include consistent estimates of quantities difficult to observe such as atmospheric heating rates, surface fluxes, and vertical motion, is expected to become a cornerstone dataset for future Earth science studies. The task of validating, inspecting, and analyzing the large volume of assimilated data requires the development of an integrated environment for managing, visualizing, manipulating, and intercomparing assimilated geophysical data, observational data, and derived products. The intercomparison and validation studies performed in such an integrated environment should also greatly enhance feedback for improving the assimilation system.

The objective of this proposal is to use the DAO-produced 5-year assimilated EOSDIS dataset and selected Pathfinder datasets to develop new approaches for integrating multiple datasets in an interactive environment for browsing, quality control, intercomparison of datasets, visualization, and analysis that facilitate global climatological research. The development of an integrated environment will be driven by scientists and users in the DAO and the Mesoscale Dynamics and Precipitation Branch (MDPB). The system validation will be done by using the prototype EOS 5-year assimilated dataset and several EOS Pathfinder datasets. These valuable science datasets will provide an important testbed for developing an appropriate environment to evaluate a number of requirements, including 2-D and 3-D visualization techniques, user interaction, scalability and adaptability issues for extending the capabilities to other Pathfinder datasets, and computational resources needed to meet science objectives. The proposed development work will build upon the visualization and data manipulation tools available at the University of Wisconsin and GLA.

The proposed testbed data system integrates model evaluation and data management with the VIS5D environment, a 3-D visualization tool for examining numerical model output, and the Interactive Image SpreadSheet (IISS), a 2-D visualization tool for manipulating and analyzing satellite data, observational data, and their derivative products. The important task of validating and inspecting assimilated datasets will be facilitated by building upon the development of these two robust visualization and data manipulation tools. The VIS5D system from the University of Wisconsin enables the volume visualization of numerical model outputs using a time series of 3-D grids for each variable. Typically, the evolution of atmospheric dynamics over surface topography can be studied using volume rendering, isosurfaces, colored slices, contour fields, wind vectors, trajectories, and data probes. The IISS system developed at Goddard facilitates the organization, interpretation, and analysis of extremely large and complex EOS-size observational and derived datasets. Satellite datasets are characterized by their global nature, providing consistent coverage over a long time period, and dense but irregular sampling in space and frequency. The integration and extension of the IISS and VIS5D will offer a powerful tool for interactively studying the assimilated datasets produced for the EOSDIS data archives.

Combining the IISS and VIS5D with a data management facility will facilitate the climatological study of assimilated model data with observational data. The data management approach will initially provide a catalog and book style digital library type interface to the assimilated datasets residing on the Goddard Cray or Unitree storage devices or on the DAO local network. The data access catalogs will offer selective retrieval of data based on geographical regions, atmospheric levels, time intervals, and assimilated variables.

Coinvestigators for this winning project are as follows: Siegfried D. Schubert, Data Assimilation Office, Laboratory for Atmospheres, NASA/Goddard Space Flight Center (schubert@albatross.gsfc.nasa.gov); William L. Hibbard, Space Science Engineering Center, University of Wisconsin-Madison (whibbard@macc. wisc.edu; and A. Fredrick Hasler, Mesoscale Dynamics and Precipitation Branch, Laboratory for Atmospheres, NASA/Goddard Space Flight Center (hasler@agnes.gsfc.nasa.gov).

Tools and Techniques for Automating Analysis of EOSDIS Data

William Emery, University of Colorado
EOSDIS will acquire, archive, process, and distribute unprecedented volumes of complex scientific data. To succeed, EOSDIS must automate its functions to a degree not previously achieved. Automation will be equally important to the scientists who must analyze this rapid stream of EOS data. Analysis of satellite data has always been a labor-intensive operation. Substantial involvement is required from scientists to select the best data for their research, to acquire all the needed data, to carry out the calibration, to select the proper algorithms, to prepare software for manipulating the data, and to verify the results. As designed, EOSDIS is not providing the general science community with all of the tools and techniques that will be needed to automate these data analysis tasks.

This proposal joins researchers in Earth science and computer science with experienced developers of scientific information systems to create, test, and demonstrate a set of tools and techniques for automating and simplifying the analysis of EOSDIS data. The capabilities that we propose will (1) capture the information about each EOSDIS data product that scientists need to conduct data analysis; (2) efficiently convey this information to users of EOSDIS; (3) store this information at the scientist's workplace so that it is available and useful to the scientist; and (4) provide data analysis programs with access to this information in ways that permit the programs to make many of the decisions that scientists formerly had to make when conducting data analysis. These capabilities will be embodied as the Science Data Analysis Automation Toolkit (SDAAT)-an integrated set of software tools that can reside at a scientist's own facility and ease the burden of acquiring and using new EOSDIS datasets.

All of the tools and techniques developed in this project will be tested and demonstrated by the scientists involved in this proposal. The capabilities will also be made widely available to other Earth scientists.

Coinvestigators are Randall Davis, UC; Frank LoPinto, CSC; James Maslaick, UC; Lewis Pinson, UC; Louis Reich, CSC; Carol Wessman, UC; and Richard Wiener, UCA.

Intelligent Information Fusion and Management Prototype Applicable to EOSDIS

William Campbell, NASA/GSFC
The Information Science and Technology office, Code 930.1, has developed an end-to-end scientific database management system called the Intelligent Information Fusion System (IIFS), with the express purpose of developing, incorporating, and evaluating state-of-the-art techniques for handling EOS-era scientific data challenges. Since 1989, the IIFS has been based on an object-oriented database, which is used to store metadata about remotely sensed images. The metadata itself is organized to enable fast, efficient access to the appropriate images with respect to typical image header data. It also incorporates innovative spatial data structures known as sphere quadtrees (SQTs) that more naturally represent data acquired globally. We propose to continue using, evaluating, constructing, and modifying our IIFS with respect to EOSDIS, and, in particular, we propose interacting with the appropriate EOSDIS representatives to transfer our techniques, methodologies, algorithms, knowledge, and justifications for our approaches.

The IIFS is an evolutionary system and has been implemented to support rapid augmentation and new approaches that do not impact its existing core functionality (which is expanding all the time). We propose to test the robustness and domain-independent nature of our approach by extending it to three scientific domains: land use/land cover, atmospheric, and biodiversity studies. Computationally, we propose to extend the IIFS to include specialty hardware and innovative approaches to automatically georegistering images.

Coinvestigators are: Robert F. Cromp, Nicholas Short, and James Tilton.

The Extension and Application of LinkWinds to EOSDIS

Allan S. Jacobson, NASA/JPL
LinkWinds is a visual data analysis and exploration system designed to rapidly and interactively investigate large multivariate and multidisciplinary datasets to detect trends, correlations, and anomalies. This proposal is a collaboration of physical scientists and computer scientists at the Jet Propulsion Laboratory and the University of Alabama, Huntsville, to extend and apply LinkWinds to problems of EOSDIS. We will (1) build visual science database management tools within the LinkWinds environment, (2) prototype and use these tools on a DAAC testbed to address current EOSDIS scientific problems involving the ingestion and intercomparison of disparate and unusual types of data, and (3) port LinkWinds to C++ to facilitate development of a user-accessible developmental environment and provide reusable LinkWinds software modules. We will carry out these technological developments in conjunction with at least three Earth science research activities. These include (1) data validation studies involving the intercomparison of atmospheric temperatures derived from the Microwave Sounder Unit (MSU) and temperature profiles measured by balloon-based radiosondes and ground stations, (2) scientific investigations into the physical relationships between various climatic forcing factors and atmospheric temperature moisture as derived from MSU and SSM/I, and (3) temporal studies and data ingestion related to MLS aboard UARS.

Coinvestigators are Andrew L. Berkin, JPL; Michael E. Botts, UAH/MSFC; John R. Christy, UAH; Lee S. Elson, JPL; and Thomas J. Handley, JPL.

Efforts of the HDF-EOS Team

Brand Fortner, ARC
In September 1993, HDF was adopted as the baseline standard for ECS development of standard data product generation archival, ingest, and distribution capabilities. At the most basic level, an individual HDF file consists of a directory and a collection of data objects, each of a particular datatype. Many of the NCSA-defined datatypes map very well to proposed EOS datatypes. Examples include raster images, multidimensional arrays, and text blocks. There are other EOS datatypes that do not map as well to NCSA datatypes. Examples include grids, swath structures, and point data.

There are two ways to deal with this: one way is to explicitly define the layout of every EOS standard data product and then incorporate these layout decisions in a subroutine library. The other method, and the one that we propose, is to define new EOS specific datatypes, such as Grid, Swath, and Point, that contain information in a specific structure. That way, our library has to know only about these structures and not about every single data product.

We are in the process of establishing these conventions for using HDF files to store EOS data. We plan to eventually supply a subroutine library that will encapsulate these conventions. We call the combination of the HDF standard and the EOS conventions the HDF-EOS standard, although, strictly speaking, it is not a new standard.

So far, our HDF-EOS efforts have resulted in a document, The HDF-EOS Primer, that describes our future plans in more detail. We also have completed a version of EOSView, our future HDF-EOS viewer application, for Evaluation Package 4. We are now in the process of distributing detailed proposals for grid, swath, and point structures.

Our timeline is as follows: in April 1995, we plan to release a draft of a proposal for an API for an HDF-EOS library. In July 1995, we plan to make available the first version of our HDF-EOS conventions document. We also plan to have another version of EOSView available, as well as some HDF-EOS library prototypes. Toward the end of 1995, we hope to release a second version of the conventions document, a third version of EOSView, and additional prototypes for the HDF-EOS library.

Teams working on Release A products may need conventions sooner than July. We will work closely with those groups to design interim conventions that we will then continue to support in the HDF-EOS standard and library.

Members of the HDF-EOS team currently include Brand Fortner, Doug Ilg, Dave Buto, Ray Milburn, David Wynne, and Anubha Singhal.

Data Modeling Activity

Lesley Knox, CSC
The EOSDIS Data Modeling Working Group (DMWG), cochaired by Ted Meyer and Denise Heller, has had two telecons over the past few months to discuss and resolve issues related to Product Groupings in Data Servers, Earth Data Type Services, the new Federal Geographic Data Standard, and the Browse Package Specification. Denise Heller's paper on PDR taxonomy of Data Servers lists 92 logical servers defined by HAIS that will process and archive EOS science data products. The paper lists the servers, the site responsible for that data, disciplines covered, Ad Hoc Working Group on Production file identifiers, science processing support office IDs, product names, platforms, and instruments. This is the first attempt to bring together products whose services are logically related. Feedback from the science and instrument team community will help refine data product groupings within servers. Mike Burnett's white paper on Earth Science Data Type Services gives an overview of the data services provided and an approach to the data types. It defines a three-point perspective on Browse and the five categories of services (i.e., selection, reduction, transformation, browse, and distribution). It references a 27-page matrixdescribing services in detail. Current versions of both documents can be found on the EDHS server at URL:http://edhs1.gsfc.nasa.gov or Info/pdr PDR page.html.

The ESDIS Project is fully committed to complying with Executive Order 12906 in its entirety. Compliance with the order will include FGDC Metadata Content Standard and a Clearinghouse. The order is applicable to the Global Change Master Directory, ECS, and Version 0. The clearinghouse will provide access to metadata systems through the Internet.

A working group has been established to develop a Browse Package Specification. This group is composed of representatives from ESDIS, EDC, the ECS contractor, and ESDIS support contractors. They have developed a strawman specification that is currently being reviewed by the DMWG. Major components of the strawman address browse assumptions, general browse specifications (for example, a 1-MB recommended size limitation for Browse packages), browse data objects, and a diagram depicting a browse package model. Numerous responses were received on browse specifications. The browse package specification and services will be refined and will evolve over time.

Return to ESDIS/EOSDIS home pagehere