Introduction to Graph Computing

Graph computing is an innovative technology that allows developers to build applications and systems as directed acyclic graphs (DAGs). Graph computing offers generic solutions to some of the most fundamental challenges in enterprise computing such as scalability, transparency and lineage. In this workshop, we survey the available graph computing tools in Julia, then walk through a few hands-on examples of building real world applications and systems using graph computing.

Graph computing is an innovative and hyper-efficient technology for building large and distributed systems or applications, where common challenges include scalability, transparency, explainability, lineage, adaptability and reproducibility. We coined the acronym STELAR for these challenges.

In almost every organization, significant engineering resources and efforts are devoted to addressing the STELAR needs for their core enterprise systems. These efforts are not portable because they are specific to the particular organization and architecture. For example, the solution to improve the scalability of the trading system at JP Morgan is not applicable to Goldman Sachs, as their technology stacks are fundamentally different. There is an enormous waste of time, money, energy and human talents for re-creating bespoke solutions to the same STELAR problems across the industry. The world would be a much better place if we could solve these problems once and for all in enterprise systems. This is the promise of graph computing.

Instead of functions in the traditional programming paradigm, directed acyclic graphs (DAG) are the fundamental building blocks in graph computing. A DAG is a special type of graph, which consists of a collection of nodes and directional connections between them. Acyclic means that these connections do not form any loops. A DAG can also be used as a generic representation of any kind of computing or workflow. Conceptually, any computation, from the simplest formula in a spreadsheet to the most complex enterprise systems, reduces to a DAG. In graph computing, complex DAGs representing entire applications or systems are built by composing smaller and modular DAGs, analogous to function compositions in the traditional programming paradigm.

Compared to the function centric representation in traditional programming, DAG is a much better and more convenient representation for building generic solutions to STELAR. Once built, these solutions are applicable to any enterprise system as they all reduce to DAGs.

The outline of this workshop is as follows:

• Introduction to the key ideas and benefits of graph computing, and how DAGs can help solve the STELAR program generically.
• Survey of existing graph computing solutions, approaches and tools. We will cover both Python and Julia tools, as well as commercial and open source solutions
• Discuss the key challenges in graph computing and approaches, including graph creation and distribution
• Hands-on sessions in building real world applications/systems using graph computing. In these sessions, we will be using available graph computing tools like Dask, Dagger.jl and Julius etc.
  • Build a simple task graph and execute it
  • Query the graph data after execution
  • Build an ML data processing pipeline
  • Building generic and reusable patterns using graph composition
  • Graph distribution, for building distributed systems
  • End to end AAD (adjoint algorithmic differentiation) with graphs
  • Build streaming pipelines in graph *Plenty of time for questions.