Best practices to publish to a Pub/Sub topic

In publishing, a publisher client sends a message to a Pub/Sub topic. Here are some best practices for publishing messages to Pub/Sub.

This document assumes that you are already familiar with the process of publishing messages to a Pub/Sub topic.

If you're new to Pub/Sub, see one of the Quickstart guides and learn how to run Pub/Sub using the console, gCloud CLI, or the client libraries.

Attach a subscription or enable topic retention before you begin to publish

If you start publishing to a topic that does not have an attached subscriber, the messages are not retained. These messages can't be delivered to subsequently attached subscriptions. Hence, before you start publishing messages, do one of the following:

Configure batch messaging

Within Pub/Sub, batch messaging refers to the process of combining multiple messages into one batch which gets published in a single publish request. If you use client libraries to publish your messages, batching is enabled by default. The batching (or grouping) of messages helps the publisher to improve its efficiency and send messages at a higher throughput. Batching reduces the cost to publish data. However, batching also creates latency for individual messages because the publisher waits for the batch to be filled up before publishing the batch.

Latency in Pub/Sub can be of two types:

  • End-to-end latency is the time it takes for a message to be published by a publisher and delivered to the corresponding subscribers for processing.

  • Publish latency is the amount of time it takes to publish a message.

When using batching, increasing both types of latencies is a trade off for improving efficiency and throughput.

You can batch messages in a client library based on message request size, number of messages, and time. When configuring batch settings, you can find the right balance between cost, throughput, and latency to suit your use case.

The default values for the batch messaging variables and the names of the variables might differ across client libraries. You can specify one or all three values in the client library. If any one of the values for the batch messaging variables is met, the client library publishes the next batch of messages.

To configure batch messaging for a publisher client, see Batch messages in a publish request.

Configure flow control for transient message spikes

If the publisher client has to process a large number of messages, publish requests might start to accumulate in memory until messages fail to publish with a Deadline exceeded error.

To address transient spikes in publishing messages, you can use flow control in your publisher settings. Publisher-side flow control prevents the publisher client resources from becoming overwhelmed with too many outstanding requests. If the publisher client becomes constrained in terms of memory, CPU, or threads, a large number of Deadline exceeded errors are generated.

To configure flow control in the client library, set appropriate values for the maximum outstanding messages and the maximum outstanding message bytes variables. These values balance message throughput and system capacity.

To check if your client library supports publisher flow control and to configure it, see Flow control.

Understand your network bandwidth and latency

Your publisher throughput is constrained by your network bandwidth and the number of requests sent. If your bandwidth is good but your network latency is high, you don't want to overwhelm the system with many small requests. Publisher-side flow control can help with client-side network issues.

Your publisher throughput is also CPU and memory bound. More available machine cores lets you set higher thread count for better publish throughput. To further understand how to maximize streaming performance, see Testing Cloud Pub/Sub clients to maximize streaming performance.

Tweak the retry request variables for failed publishes

When a message is published by a publisher client, you might see publishing failures. These failures are typically caused by client-side bottlenecks, such as insufficient service CPUs, bad thread health, or network congestion. The publisher retry policy determines the behavior in case of message delivery failure. The retry policy defines the number of times Pub/Sub tries to deliver the message and the length of time between each attempt.

For example, in the Java client library for Pub/Sub, the publisher client contains the following values:

  • initialRetryDelay. The initial delay that the publisher waits before retrying a publish operation. The default value is 100 milliseconds.

  • retryDelayMultiplier. The multiplication factor used to calculate the delay between retries. The default value is 4. This means that the delay between retries is up to 100 milliseconds * 4 = 400 milliseconds for the second retry, and up to 400 milliseconds * 4 = 1600 milliseconds for the third retry.

  • maxRetryDelay. The maximum delay that the publisher waits before retrying a publish operation. The default value is 60 seconds.

  • initialRpcTimeout. The initial timeout that the publisher waits for the RPC call to complete. The default value is 5 seconds.

  • rpcTimeoutMultiplier. The multiplication factor used to calculate the RPC timeout. The default value is 4.0. This means that the timeout for the RPC call is up to 5 seconds * 4 = 20 seconds for the second retry, and up to 10 seconds * 4 = 40 seconds for the third retry.

  • maxRpcTimeout. The maximum timeout that the publisher waits for the RPC call to complete. The default value is 600 seconds.

  • totalTimeout. The total timeout for the publish operation. This includes the time spent waiting for the RPC call to complete and the time spent waiting between retries. The default value is 600 seconds.

Only make adjustments to the specified values if you find the default retry settings are not sufficient for your use case. For example, publishing a large number of messages wouldn't require you to increase the initialRetryDelay and maxRetryDelay values. However, you can adjust flow control and batching in such circumstances. If you're publishing from a flaky internet connection or a connection that is bandwidth constrained, you can experiment with the values for the initialRpcTimeout, maxRpcTimeout, and rpcTimeoutMultiplier variables. For recommended values, see Publish operations fail with DEADLINE_EXCEEDED.

Use message storage policy to ensure data locality

Pub/Sub's topic message storage policy offers a way to ensure that messages published to a topic are never persisted outside a set of Google Cloud regions that you specify, independent of where the publish requests originate.

Use the message storage policy to specify a list of Google Cloud regions where Pub/Sub is allowed to store message data on disk. When a message is published to a region not in this list, the request is forwarded to the nearest allowed region for processing. The policy can be configured on a topic or as an organizational policy for a project, project folder, or an entire organization. When an organization policy is configured, individual topic policy can be changed only in ways that don't violate the organization policy.

For example, a company that operates in Europe might use the message storage policy to ensure that all of the data is stored in EU regions to comply with local laws.

For more information, see Configure message storage policies.

Best practices for ordered messaging in publishing

If you use message ordering, ensure the following:

  • Use locational endpoints. The ordering of messages is preserved on the publish side and within a region. In other words, if you are publishing messages to multiple regions, only messages within the same region are delivered in a consistent order. If your messages are all published to the same region, but your subscribers are spread across regions, then the subscribers receive all messages in order. Use a locational endpoint to publish messages to the same region.

  • Configure a resume publishing function. When a client library retries a request and the message has an ordering key, the client library repeatedly retries the request, independent of the retry settings. If a non-retryable error occurs, the client library doesn't publish the message and stops publishing other messages with the same ordering key. When you are ready to continue publishing on an ordering key that has a failed publish, call the resumePublish method.

Summary of best practices

The following table summarizes the best practices recommended in this document:

Topic Task
Configure message retention Attach a subscription before you publish or enable message retention.
Batch messages in a publish request Batch or group messages to increase the efficiency of the publisher and send messages at a higher throughput.
Flow control Configure flow control in your publisher settings to handle transient traffic spikes.
Testing Pub/Sub clients to maximize streaming performance Scale publisher throughput with an increase in available machine cores and network bandwidth.
Retry requests Make adjustments to the specified values of the publisher retry policy only if you find the default settings are not sufficient for your use case.
Configure message storage policies Use the message storage policy to store message data on disk only in specific locations.
Use a locational endpoint when using ordering keys in publishing When using ordered messaging, use a locational endpoint and configure a resume publishing function for publishing failures.

What's next