ABSTRACT
This study introduces a novel methodology designed to assess the accuracy of data processing in the Lambda Architecture (LA), an advanced big-data framework qualified for processing streaming (data in motion) and batch (data at rest) data. Distinct from prior studies that have focused on hardware performance and scalability evaluations, our research uniquely targets the intricate aspects of data-processing accuracy within the various layers of LA. The salient contribution of this study lies in its empirical approach. For the first time, we provide empirical evidence that validates previously theoretical assertions about LA, which have remained largely unexamined due to LA's intricate design. Our methodology encompasses the evaluation of prospective technologies across all levels of LA, the examination of layer-specific design limitations, and the implementation of a uniform software development framework across multiple layers. Specifically, our methodology employs a unique set of metrics, including data latency and processing accuracy under various conditions, which serve as critical indicators of LA's accurate data-processing performance. Our findings compellingly illustrate LA's "eventual consistency". Despite potential transient inconsistencies during real-time processing in the Speed Layer (SL), the system ultimately converges to deliver precise and reliable results, as informed by the comprehensive computations of the Batch Layer (BL). This empirical validation not only confirms but also quantifies the claims posited by previous theoretical discourse, with our results indicating a 100% accuracy rate under various severe data-ingestion scenarios. We applied this methodology in a practical case study involving air/ground surveillance, a domain where data accuracy is paramount. This application demonstrates the effectiveness of the methodology using real-world data-intake scenarios, therefore distinguishing this study from hardware-centric evaluations. This study not only contributes to the existing body of knowledge on LA but also addresses a significant literature gap. By offering a novel, empirically supported methodology for testing LA, a methodology with potential applicability to other big-data architectures, this study sets a precedent for future research in this area, advancing beyond previous work that lacked empirical validation.
ABSTRACT
AIM: To describe a deep convolutional generative adversarial networks (DCGAN) model which learns normal brain MRI from normal subjects than finds distortions such as a glioma from a test subject while performing a segmentation at the same time. MATERIAL AND METHODS: MRIs of 300 healthy subjects were employed as training set. Additionally, test data were consisting anonymized T2-weigted MRIs of 27 healthy subjects and 27 HGG patients. Consecutive axial T2-weigted MRI slices of every subject were extracted and resized to 364x448 pixel resolution. The generative model produced random normal synthetic images and used these images for calculating residual loss to measure visual similarity between input MRIs and generated MRIs. RESULTS: The model correctly detected anomalies on 24 of 27 HGG patients? MRIs and marked them as abnormal. Besides, 25 of 27 healthy subjects? MRIs in the test dataset detected correctly as healthy MRI. The accuracy, precision, recall, and AUC were 0.907, 0.892, 0.923, and 0.907, respectively. CONCLUSION: Our proposed model demonstrates acceptable results can be achieved only by training with normal subject MRIs via using DCGAN model. This model is unique because it learns only from normal MRIs and it is able to find any abnormality which is different than the normal pattern.
