Build a one-to-many Pub/Sub system

This tutorial walks you through setting up a set of applications that communicate by sending messages through Pub/Sub rather than synchronous RPCs. By decoupling applications, messaging:

Makes applications more robust
Might simplify development

For example, the caller (publisher) does not need the receiver (subscriber) to be up and available. The publisher sends a message to Pub/Sub. The publisher doesn't need to know which and how many subscriber applications need to receive the message. As a result, the service can be relied upon to deliver the message to one or more subscriber applications whenever they are available.

System overview

In this tutorial, you start a publisher application that sends a "Hello, World!" message to two subscribers using one-to-many communication, as illustrated in the following diagram:

Diagram of the
topic, its attached subscriptions, and the publisher and subscriber
applications that send message to and receive messages from Cloud
Pub/Sub

The two subscriber applications use the same code, but you start them at different times. This process demonstrates how Pub/Sub enables asynchronous communication. To build this system, you complete the following steps:

Create an IAM service account that the applications use for authentication.
Set up IAM permissions.
Create a Pub/Sub topic and a subscription.
Start three independent applications: one publisher and two subscribers.

Before you begin

Sign in to your Google Cloud account. If you're new to Google Cloud, create an account to evaluate how our products perform in real-world scenarios. New customers also get $300 in free credits to run, test, and deploy workloads.

Install the Google Cloud CLI.

To initialize the gcloud CLI, run the following command:

gcloud init

Create or select a Google Cloud project.

Create a Google Cloud project:
```
gcloud projects create PROJECT_ID
```
Replace PROJECT_ID with a name for the Google Cloud project you are creating.
Select the Google Cloud project that you created:
```
gcloud config set project PROJECT_ID
```
Replace PROJECT_ID with your Google Cloud project name.

Make sure that billing is enabled for your Google Cloud project.

Enable the Pub/Sub API:

gcloud services enable pubsub.googleapis.com

Create local authentication credentials for your user account:

gcloud auth application-default login

Grant roles to your user account. Run the following command once for each of the following IAM roles: roles/pubsub.publisher, roles/pubsub.subscriber

gcloud projects add-iam-policy-binding PROJECT_ID --member="user:USER_IDENTIFIER" --role=ROLE

Replace PROJECT_ID with your project ID.
Replace USER_IDENTIFIER with the identifier for your user account. For example, user:myemail@example.com.
Replace ROLE with each individual role.

Install the Google Cloud CLI.

To initialize the gcloud CLI, run the following command:

gcloud init

Create or select a Google Cloud project.

Create a Google Cloud project:
```
gcloud projects create PROJECT_ID
```
Replace PROJECT_ID with a name for the Google Cloud project you are creating.
Select the Google Cloud project that you created:
```
gcloud config set project PROJECT_ID
```
Replace PROJECT_ID with your Google Cloud project name.

Make sure that billing is enabled for your Google Cloud project.

Enable the Pub/Sub API:

gcloud services enable pubsub.googleapis.com

Create local authentication credentials for your user account:

gcloud auth application-default login

Grant roles to your user account. Run the following command once for each of the following IAM roles: roles/pubsub.publisher, roles/pubsub.subscriber

gcloud projects add-iam-policy-binding PROJECT_ID --member="user:USER_IDENTIFIER" --role=ROLE

Replace PROJECT_ID with your project ID.
Replace USER_IDENTIFIER with the identifier for your user account. For example, user:myemail@example.com.
Replace ROLE with each individual role.

Install Python

This tutorial uses the Pub/Sub Client Libraries, which requires Python 3.7 or higher. Complete the instructions for installing Python.

Set up your Pub/Sub project

To manage message flow between publishing and subscribing applications, you create a topic and two different subscriptions.

Create a Pub/Sub topic

Create a topic with the ID hello_topic:

gcloud pubsub topics create hello_topic

Create Pub/Sub subscriptions

Create two subscriptions and attach them to your topic.

These subscriptions are a StreamingPull subscription, which is a type of pull subscription.

Subscription 1

Create a subscription with the ID sub_one and attach it to hello_topic.

gcloud pubsub subscriptions create sub_one --topic=hello_topic

Subscription 2

Create a subscription with the ID sub_two and attach it to hello_topic.

gcloud pubsub subscriptions create sub_two --topic=hello_topic

Build the one-to-many system

Download the publisher and subscriber code

Download the Pub/Sub Python files needed for this tutorial.

 git clone https://github.com/googleapis/python-pubsub.git

Close any open terminals before proceeding.

Set up three terminals

Start one terminal for each tutorial application (one publisher and two subscribers). For convenience, this tutorial calls these terminals:
- publisher terminal
- sub_one terminal
- sub_two terminal
Note: Make sure all three terminals are in the same directory relative to where you downloaded the Pub/Sub Python files in the previous step.
In the publisher terminal, create and activate a Python virtual environment named pyenv-qs.
Bash
```
python -m venv pyenv-qs
source pyenv-qs/bin/activate
```
PowerShell
```
py -m venv pyenv-qs
.\pyenv-qs\Scripts\activate
```
In the sub_one and sub_two terminals, run the following command:
Bash
```
source pyenv-qs/bin/activate
```
PowerShell
```
.\pyenv-qs\Scripts\activate
```
After you run the activate command, your command prompt includes the following value (pyenv-qs) $.
In the publisher terminal, install the Pub/Sub Python client library using pip:
```
python -m pip install --upgrade google-cloud-pubsub
```
In all three terminals, set up an environment variable with your current project ID. This gcloud command determines your selected project ID and sets it as a variable:
Bash
```
export PROJECT=`gcloud config get-value project`
```
PowerShell
```
$env:PROJECT=$(gcloud config get-value project)
```
In all three terminals, change to the project path that contains the sample code.
```
cd python-pubsub/samples/snippets/quickstart/
```

Start the apps and observe message flow

Start the Subscriber 1 application

In the sub_one terminal, start Subscriber 1:

Bash

python sub.py $PROJECT sub_one

PowerShell

py sub.py $env:PROJECT sub_one

Once started, this application opens a bidirectional streaming connection with the server. Pub/Sub delivers messages over the stream.

Subscriber 1
application begins listening for messages on the sub_one subscription.

Start the Publisher application

In the publisher terminal, start the Publisher application:

Bash

python pub.py $PROJECT hello_topic

PowerShell

py pub.py $env:PROJECT hello_topic

After the publisher application starts, the Pub/Sub system does the following:

The Publisher application sends a "Hello, World!" message to Pub/Sub while remaining unaware of any existing subscriptions. The server also assigns a message ID.
The Subscriber 1 application receives the 'Hello World' message, prints it, and sends an acknowledgment to Pub/Sub.
The Publisher application prints the acknowledgement. The acknowledgment tells Pub/Sub that the message has been processed successfully and does not need to be re-sent to this or any other sub_one subscriber.

Pub/Sub removes the message from sub_one.

The Publisher
application publishes the message and assigns a message ID. The Subscriber 1
application receives the 'Hello World' message and sends an
acknowledgment

Start the Subscriber 2 application

In the sub_two terminal, start Subscriber 2:

Bash

python sub.py $PROJECT sub_two

PowerShell

py sub.py $env:PROJECT sub_two

This subscriber receives messages delivered to the sub_two subscription. Subscriber 2 reuses the sub.py script. The difference is that Subscriber 2 isn't started until after the Publisher has sent the message to the topic and subscriptions. If Publisher were calling Subscriber 2 directly, the publishing application would either have to wait until Subscriber 2 comes up or it would have to time out. Pub/Sub manages this process by effectively saving the message for Subscriber 2.

Subscriber 2
starts listening and receives the message that was waiting for it in
sub_two

You are now ready to develop with Pub/Sub!

How did it go?

Additional resources and links are available on the Pub/Sub support page.

Clean up

Stop all running applications.
Delete the sample code directory from your local environment.
Delete the topic.
```
gcloud pubsub topics delete hello_topic
```

Delete the subscriptions.

gcloud pubsub subscriptions delete sub_one

gcloud pubsub subscriptions delete sub_two

Shut down the tutorial project in the IAM & admin section of the Google Cloud console.
Optional: Revoke the authentication credentials that you created, and delete the local credential file.
```
gcloud auth application-default revoke
```
Optional: Revoke credentials from the gcloud CLI.
```
gcloud auth revoke
```

What's next

Here are some things that you can try:

Examine the tutorial's pub.py and sub.py code and browse other Pub/Sub samples on GitHub. As an exercise, create a version of pub.py that publishes the local time every second.
Learn to batch messages.
Using Push subscriptions, receive messages that trigger App Engine endpoints or Cloud Functions.
Retrieve previously acknowledged messages using replay. By default, Pub/Sub removes acknowledged messages from subscriptions. In this tutorial, for instance, you wouldn't be able to rerun sub.py to receive the "Hello, World!" message again. The replay feature lets you set up subscriptions so that you can receive messages after they have been acknowledged.
Get started with client libraries in other languages.