Ladybug is a light-weight embedded graph database optimised for graph workloads

icebug is a high-performance open-source library, engineered for parallel processing and large-scale graph analysis. When paired with arrow-backed memory, it is 10x-100x faster than networkit / networkX

Proposal

Currently graphify is not particularly efficient in terms of storage and performance. Replacing it with ladybug + icebug would provide substantial improvements in storage, scalability and performance

Small graphs (in-memory)

Ladybug can be used instead of in-memory nxGraph & on-disk json and icebug for running complex algos like community detection. This enables faster build, serve and efficient storage

Build

Post extraction,

• An in-memory ladybug instance is created with node and rel tables (schema enforced). For incremental builds, prev exported graph can be imported using IMPORT DATABASE cmd
• nodes and rels from extraction phase can be directly written to the lbug instance
• For clustering, ladybug offers export_as_arrow_csr(which converts ladybug tables to arrow csr tables) which can then be used by icebug to run community detection
• During dedup or analysis phase either ladybug (algo ext or cypher queries can be used) or icebug (for faster parallel processing) can be used to run different algos
• Finally the graph can be persisted to disk in space-efficient parquet format using EXPORT DATABASE cmd in ladybug

PS: updates can be written to graph using ladybug instance during any of the phases

Serve

All the serve requests can be processed in ladybug using cypher queries efficiently

Large graphs

For large graphs which cannot fit in memory, an on-disk ladybug instance can be created (new Database('<path>/graph.lbdb')). Rest of the process is same except for importing / exporting graphs. Running explicit import and export cmds are not reqd as all the writes are persisted to a .lbdb file on disk

Analysis

As Arrow and Parquet are compressed columnar storage formats, memory and storage foot print would be a lot lesser

On graphify run on ladybug repo:

• graph.json : ~33 MB
• ladybug-export/ : ~2.5 MB

211 Jun 16 15:58 copy.cypher
1069880 Jun 16 15:58 edges_nodes_nodes.parquet
1522237 Jun 16 15:58 nodes.parquet
455 Jun 16 15:58 schema.cypher

which is a 13x reduction

Queries and algo runs would be certainly faster as ladybug and icebug are optimised for larger graphs. Take a look at the following benchmarks:

• https://github.com/Ladybug-Memory/icebug-benchmark-pagerank
• https://www.r-bloggers.com/2019/05/benchmark-of-popular-graph-network-packages/

References

• minimal notebook for a complete ladybug+icebug demonstration
• ladybug docs