Data flow and processing
Operating the KITcube network of mostly not staffed stations requires a high degree of automation to ensure data collection preferably without interruptions. To achieve this, it is necessary to make the measurement data available for quality control mechanisms and surveillance immediately after the acquisition. The data flow in the KITcube is visualized in the following figure.
Usually, the software delivered by the manufacturer is used for data acquisition. This data is transmitted to the KITcube data management system where the files are archived in the proprietary format as level 0-data. In a next step, the heterogeneous data format of the different devices has to be harmonized. To this end, the data is fed into a powerful time series database (PostgreSQL/TimescalDB) in a standardized and interoperable format. The implemented system also enables the handling of extensive volume data sets from remote sensing devices such as Doppler lidars and radars.
The database provides the level 0 data to multiple applications promptly. In a first processing, a automatic quality control is carried out. If a system fault is detected, the principal investigator is notified immediately. Therefore, system faults are detected early and maintenance be initiated to minimize data gaps. Current and past measurements are provided by an interactive dashboard. The live-processing of quicklooks in the dashboard enables assessments of the current meteorological situation as well as understanding past weather situations. Post-processing algorithms can access level 0 data from the database as well as from the original level 0 files.
Time-critical applications as assimilations require a prompt processing of near-real time measurement data in an interoperable format like netCDF. For example, radial velocities measured in different directions by Doppler lidars have to be processed to get profiles of vertically and temporally resolved wind vectors (Fig. 1). This post-processing is conducted for all Doppler lidars in operation in regular intervals to enable time-critical follow-up processes. The algorithms developed for wind profiling from Doppler lidar measurements are made publicly available with extensions for the usage outside KITcube. The open-source software is available as Atmospheric Profile Processing toolKIT.
