Graph-based stream processing in Python
Details
After a long, covid induced hiatus Connected Data London's meetups are back! We have Memgraph visiting from Croatia who have agreed to give a fascinating talk on graph-based stream processing.
Doors will open at 6.30pm for talks starting at 7. As usual refreshments and pizza will be available courtesy of our sponsors Memgraph & Neural Alpha.
Talk description
The understanding of complex relationships and interdependencies between different data points is crucial to many decision-making processes.
Graph analytics have found their way into every major industry, from marketing and financial services to transportation. Fraud detection, recommendation engines, and process optimization are some of the use cases where real-time decisions are mission-critical, and the underlying domain can be easily modeled as a graph.
By ingesting data with Apache Kafka and applying graph-based stream processing in real-time, you can perform near-instantaneous graph analytics on vast amounts of data. When it comes to complex networks, it’s often necessary to perform graph algorithms such as calculating the PageRank, identifying communities, traversing relationships, etc. While solutions such as ksqlDB or Apache Spark are useful for processing relational data, Memgraph is an open-source streaming platform that can be used to analyze graph-based data models.
Graph analytics can provide insights into complex networks that would otherwise require resource-intensive computations. It is also much simpler to store streaming data in the form of graphs, as the graph model doesn't rely on predefined and rigid tables. When connecting a Kafka data stream to Memgraph, you only need to create a transformation module that will map incoming messages to the property graph model.
This data can then be traversed and analyzed using the Cypher query language without having to implement custom algorithms or relying on development-heavy solutions. MAGE (Memgraph Advanced Graph Extensions) is a graph library that works well with Kafka-powered systems and contains graph algorithms meant for analyzing streaming data. Besides stream processing, you can also utilize standard graph algorithms from the MAGE library to explore the stored data.We are going to showcase the Twitter Network Analysis demo, a web application with a Python Flask backend that uses Memgraph to ingest real-time data scraped from Twitter. You will learn how to use an Object Graph Mapper (OGM) in Python to connect to a graph database and perform stream processing on data streamed with Apache Kafka.
# Speaker bio
## Ivan Despot
Developer Relations Engineer, MemgraphIvan
Despot is a Developer Relations Engineer at Memgraph. His passion for mathematics and graph theory inspired him to become part of the Memgraph team and start contributing to the field of graph analytics. Besides graph-based technologies, he is also interested in streaming platforms, stream processing and event-driven development.
Twitter: https://twitter.com/ivan_g_despot
LinkedIn: https://www.linkedin.com/in/ivan-g-despot/
Medium: https://gdespot.medium.com/
## Katarina Šupe
Developer Relations Engineer, Memgraph
Katarina Šupe is a Developer Relations Engineer at Memgraph. Her journey there started with a summer internship, and her mathematics and computer science background was a perfect match to work in Memgraph. She enjoys contributing to different areas and exploring new real-time data visualization technologies. She sees the graph world as a future of data analytics due to the variety of algorithms used for stream processing.
Twitter: https://twitter.com/supe_katarina