You are here

Web-scale distributed eScience AI search across disconnected and heterogeneous infrastructures

Authors: 

Lars Kotthoff, Thomas W. Kelsey, Martin McCaffery

Publication Type: 
Refereed Conference Meeting Proceeding
Abstract: 
We present a robust and generic framework for web-scale distributed e-Science Artificial Intelligence search. Our validation approach is to distribute constraint satisfaction problems that require perfect accuracy to 10, 12 and 15 digits. By checking solutions obtained using the framework against known results, we can ensure that no errors, duplications nor omissions are introduced. Unlike other approaches, we do not require dedicated machines, homogeneous infrastructure or the ability to communicate between nodes. We give special consideration to the robustness of the framework, minimising the loss of effort even after a total loss of infrastructure, and allowing easy verification of every step of the distribution process. The unique challenges our framework tackles are related to the combinatorial explosion of the space that contains the possible solutions, and the robustness of long-running computations. Not only is the time required to finish the computations unknown, but also the resource requirements may change during the course of the computation. We demonstrate the applicability of our framework by using it to solve challenging problems using two separate large-scale distribution paradigms. The results show that our approach scales to e-Science computations of a size that would have been impossible to tackle just a decade ago.
Proceedings: 
10th IEEE International Conference on eScience, Guaruj√°, Brazil
Digital Object Identifer (DOI): 
10.na
Publication Date: 
01/10/2014
Conference Location: 
Brazil
Research Group: 
Institution: 
National University of Ireland, Cork (UCC)
Open access repository: 
No