Tutorial: Leader Election¶
Faust processes streams of data forming pipelines. Sometimes steps in the pipeline require synchronization, but instead of using mutexes, a better solution is to have one of the workers elected as the leader.
An example of such an application is a news crawler. We can elect one of the workers to be the leader, and the leader maintains all subscriptions (the sources to crawl), then periodically tells the other workers in the cluster to process them.
To demonstrate this we implement a straightforward example where we elect one of our workers as the leader. This leader then periodically sends out random greetings to be printed out by available workers.
As we did in the Tutorial: Count page views tutorial, we first define your application.
Create a module named
# examples/leader.py import faust app = faust.App( 'leader-example', broker='kafka://localhost:9092', value_serializer='raw', )
Next we define the
agent that will get greetings from the
leader and print them out to the console.
Create the agent:
@app.agent() async def say(greetings): async for greeting in greetings: print(greeting)
The Agents - Self-organizing Stream Processors guide – for more information about agents.
Now define a
timer with the
on_leader flag enabled
so it only executes on the leader.
timer will periodically send out a random greeting, to be printed
by one of the workers in the cluster.
Create the leader timer:
import random @app.timer(2.0, on_leader=True) async def publish_greetings(): print('PUBLISHING ON LEADER!') greeting = str(random.random()) await say.send(value=greeting)
The greeting could be picked up by the agent
say on any one of the
To run the project you first need to start Zookeeper and Kafka.
$ $KAFKA_HOME/bin/zookeeper-server-start $KAFKA_HOME/etc/kafka/zookeeper.properties
Then start Kafka:
$ $KAFKA_HOME/bin/kafka-server-start $KAFKA_HOME/etc/kafka/server.properties
Start the Faust worker, similarly to how we do it in the Quick Start tutorial:
$ faust -A leader worker -l info --web-port 6066
Let’s start two more workers in different terminals on the same machine:
$ faust -A leader worker -l info --web-port 6067
$ faust -A leader worker -l info --web-port 6068
Next try to arbitrary shut down (Control-c) some of the workers, to see how the leader stays at just one worker - electing a new leader upon killing a leader – and to see the greetings printed by the workers.