Typetheoretic approach to big data workflow composition

Document Type

Conference Proceeding

Publication Date

2024

Department/School

Computer Science

Publication Title

2024 IEEE 4th International Conference on Electronic Communications, Internet of Things and Big Data (ICEIB)

Abstract

Big data workflows have emerged as an important paradigm for analyzing increasingly large datasets generated in different areas of human activity, including healthcare, climate research, and manufacturing, just to name a few. The user, such as a medical doctor or a climatologist, relies on a specialized software, called Big Data Workflow Management System, to compose a big data workflow by selecting and linking together heterogeneous computational tasks, such as RESTful web services, Hadoop and Spark jobs, scripts, and other data processing and executing them in the cloud in a distributed fashion. Designing an elaborate multi-step, multi-branch directed graph of tasks is called workflow composition, which is a complex, labor-intensive process that has a high chance of human error. When composing a new workflow, for each new task, the user must (1) choose the correct task out of thousands of options, (2) insert the task into the correct position in the workflow, and (3) properly connect the inputs and outputs of the task to other workflow tasks and/or data products (such as files). Connecting tasks whose interfaces are not fully compatible leads to runtime errors during workflow execution and to a failure to produce data analysis results for the user. Big data workflow composition has been widely recognized as an important and challenging problem. To address this problem, a typetheoretic approach is proposed in this article to enable the software-assisted composition of big data workflows. Specifically, a set of subtyping rules was created to define compatibility between tasks in a precise and rigorous manner. A subtyping algorithm was written to determine whether two given tasks are compatible, and a task recommendation algorithm was developed to suggest the user tasks that fit into the current workflow design. An algorithm that translates workflow graphs into equivalent lambda calculus expressions for workflow verification was also created. The three algorithms were tested in the open-source big data workflow system called WorkfloWiz (kashliev.com), and the results validated the approach.

Comments

A. Kashliev is a faculty member in EMU's Department of Computer Science.

Link to Full Text

EMU Library Accessibility Request

Share

COinS