Hugo Buddelmeijer, Danny Boxhoorn, Edwin A. Valentijn
We present the design of a novel way of handling astronomical catalogs in
Astro-WISE in order to achieve the scalability required for the data produced
by large scale surveys. A high level of automation and abstraction is achieved
in order to facilitate interoperation with visualization software for
interactive exploration. At the same time flexibility in processing is enhanced
and data is shared implicitly between scientists.
This is accomplished by using a data model that primarily stores how catalogs
are derived; the contents of the catalogs are only created when necessary and
stored only when beneficial for performance. Discovery of existing catalogs and
creation of new catalogs is done through the same process by directly
requesting the final set of sources (astronomical objects) and attributes
(physical properties) that is required, for example from within visualization
software.
New catalogs are automatically created to provide attributes of sources for
which no suitable existing catalogs can be found. These catalogs are defined to
contain the new attributes on the largest set of sources the calculation of the
attributes is applicable to, facilitating reuse for future data requests.
Subsequently, only those parts of the catalogs that are required for the
requested end product are actually processed, ensuring scalability.
The presented mechanisms primarily determine which catalogs are created and
what data has to be processed and stored: the actual processing and storage
itself is left to existing functionality of the underlying information system.
View original:
http://arxiv.org/abs/1111.3784
No comments:
Post a Comment