Performance Evaluation Analysis of Spark Streaming Backpressure for Data-Intensive Pipelines.

A significant rise in the adoption of streaming applications has changed the decision-making processes in the last decade. This movement has led to the emergence of several Big Data technologies for in-memory processing, such as the systems Apache Storm, Spark, Heron, Samza, Flink, and others. Spark Streaming, a widespread open-source implementation, processes data-intensive applications that often require large amounts of memory. However, Spark Unified Memory Manager cannot properly manage sudden or intensive data surges and their related in-memory caching needs, resulting in performance and throughput degradation, high latency, a large number of garbage collection operations, out-of-memory issues, and data loss. This work presents a comprehensive performance evaluation of Spark Streaming backpressure to investigate the hypothesis that it could support data-intensive pipelines under specific pressure requirements. The results reveal that backpressure is suitable only for small and medium pipelines for stateless and stateful applications. Furthermore, it points out the Spark Streaming limitations that lead to in-memory-based issues for data-intensive pipelines and stateful applications. In addition, the work indicates potential solutions.

Towards ubiquitous requirements engineering through recommendations based on context histories.

The growing technological advance is causing constant business changes. The continual uncertainties in project management make requirements engineering essential to ensure the success of projects. The usual exponential increase of stakeholders throughout the project suggests the application of intelligent tools to assist requirements engineers. Therefore, this article proposes Nhatos, a computational model for ubiquitous requirements management that analyses context histories of projects to recommend reusable requirements. The scientific contribution of this study is the use of the similarity analysis of projects through their context histories to generate the requirement recommendations. The implementation of a prototype allowed to evaluate the proposal through a case study based on real scenarios from the industry. One hundred fifty-three software projects from a large bank institution generated context histories used in the recommendations. The experiment demonstrated that the model achieved more than 70% stakeholder acceptance of the recommendations.

An Algorithm to Minimize Energy Consumption and Elapsed Time for IoT Workloads in a Hybrid Architecture.

Advances in communication technologies have made the interaction of small devices, such as smartphones, wearables, and sensors, scattered on the Internet, bringing a whole new set of complex applications with ever greater task processing needs. These Internet of things (IoT) devices run on batteries with strict energy restrictions. They tend to offload task processing to remote servers, usually to cloud computing (CC) in datacenters geographically located away from the IoT device. In such a context, this work proposes a dynamic cost model to minimize energy consumption and task processing time for IoT scenarios in mobile edge computing environments. Our approach allows for a detailed cost model, with an algorithm called TEMS that considers energy, time consumed during processing, the cost of data transmission, and energy in idle devices. The task scheduling chooses among cloud or mobile edge computing (MEC) server or local IoT devices to achieve better execution time with lower cost. The simulated environment evaluation saved up to 51.6% energy consumption and improved task completion time up to 86.6%.