O1. CABAHLA framework design. The CABAHLA project will provide an integrated framework for efficient real-time and in-memory data analysis for large-scale HPC infrastructures that focus on data intensive computation as well as an integrated development environment. Called CAIDE, this framework aims to facilitate the development of parallel, data-intensive application codes for large-scale systems, facilitating application development, hiding details of the underlying runtime systems and hardware. The objective of the framework is to integrate the complex infrastructure of Big Data systems that include everything from sensors to HPC simulations. To do so, the aim is to investigate components for programming models, application self-tuning, co-planning in heterogeneous systems, and large-scale data integration and management. In addition, multi-level components will be added to improve energy efficiency locally and globally. The different modules will provide services through interfaces that will ensure compatibility at both the local node and distributed system levels.
O2. Exploitation of parallelism at node level and accelerators. In this objective, we will specifically study some of the problems that arise for the exploration of intranode parallelism and the use of hardware accelerators contemplated in the CAIDE framework described in objective O1. We will focus on the development of techniques and methodologies for an efficient use of resources on homogeneous and heterogeneous platforms in both performance and consumption. Dedicated environments and cloud systems will be studied where the need to co-plan different applications is often frequent. Special emphasis will also be placed on facilitating programming for developers with abstractions that simplify the extraction and exploitation of parallelism. Among other technologies, we will mainly use task-oriented programming models, design patterns, specific domain languages, models based on hardware counters and new techniques for software synthesis in FPGAs. We will also explore the use of new accelerators, not only for the optimization of applications as usual, but we also consider their use to optimize our own framework. Many of the optimizations that are to be investigated are based on the use of predictive models, in some cases expensive, but which thanks to the new accelerators can be viable. To demonstrate the validity of the results, the optimization and evaluation at the node level of the data intensive applications proposed as a use case will be carried out, including the profiling and machine learning techniques that are intended to be investigated in the project.
O3. Management and capture of massive data integrating heterogeneous large-scale systems. The main objective is to investigate new techniques for data and I / O management that can provide the convergence of HPC and Big Data in large-scale systems, in order to improve their flexibility and performance, in addition to integrating heterogeneous sensor data capture systems. The project will focus on exploring the integration between the data flows of the Big Data and HPC applications and on the design of bridges between the programming of HPC and Big Data applying the paradigm of data parallelism for its integration in the CAIDE framework proposed in O1. To accomplish this, we will investigate to provide common middleware and efficient techniques for data transfer between different storage levels, with the ultimate goal of providing dynamic data migration between HPC and Big Data file systems, and exposing and exploiting the locality of data in the storage stack to optimize data placement for HPC environments. Considering that technology at computing nodes is changing rapidly, the project will explore techniques to process data at sensor nodes and prevent the anticipated flood of data from Internet of Things devices and data capture techniques. In addition, mechanisms for integrating heterogeneous data streams into HPC systems will be investigated.
O4. Development of global energy efficiency mechanisms. Due to the exascale trend, the employed supercomputing clusters will consume more energy because of both the number of nodes and the number of cores that each processor will have. Moreover, we must add the heterogeneity of the available nodes that will undoubtedly have accelerators such as GPU, FPGA, etc. as well as, perhaps, CPU processors of different types (Intel, AMD, ARM, etc.). As a consequence, consumption will be variable depending on the type of task that each processor executes, its physical location depending on the cooling system, the present or future computational load that may be queued, etc. This objective aims first to make a characterization with homogeneous CPU tasks that provide an energy and performance map of the consumption produced from traditional variables such as those indicated above, and then move on to heterogeneous systems. Thus it will be necessary to add new developments in order to have an efficient throughput in the cluster, such as advances in parallelism within the node (O2) or input/output operations in execution times (O3). The characterization study will be expanded taking into account these new developments so that they are equally efficient from the energy point of view. It will be integrated as a multi-level functionality in the framework proposed in O1.
O5. Capture and modeling of sensor data for the prediction of solar radiation with high spatio-temporal resolution. Improving weather forecasting in general is a major challenge that current numerical methods meet acceptably for time horizons ranging from a few hours to three days, spatial resolutions of kilometers, and time spans of hours. Beyond this spatiotemporal range of work, the results provided by these methods are far from adequate. In the case of solar radiation (as a source of energy production) the problem is further complicated because it is not a priority variable in operational forecasting schemes in institutional meteorological centers (AEMET, ECMWF); The information that these services provide is insufficient to build predictive models with a temporal resolution of minutes-seconds and spatial resolution of hundreds-tens of square meters. This “fine-tuning prediction” is of vital importance for the operation of solar plants (photovoltaic and solar thermal), for the management of the distribution network, etc.
O6. Processing massive data in medical imaging of the brain to detect abnormal patterns. Due to its high structural and functional complexity, the human brain is the least known organ of the human body. Brain morphology is constantly changing throughout life and its biological and functional implications remain unclear. Therefore, neuroimaging has become a key technique for the evaluation and quantification, in vivo, of changes in brain maturation related to neurodevelopment, aging, learning or disease. Features such as volume, cortical thickness, functional connectivity, surface area, or cortical gyrification can be mapped within the brain volume or on the brain surface using different imaging and deep learning suites. This requires being able to process high-resolution brain images in large-scale systems, optimizing data management and processing using computer accelerators, as well as a reliable data source. In this project we will use the Human Connectome Project (HCP) dataset containing structural and functional MR data from 900 subjects (3,600 images stored in 64 TeraBytes). All data is anonymous, so there are no privacy or ethical issues.