Using the Indexing API

You can use the Indexing API to tell Google to update or remove job posting or livestreaming event pages from the Google index. The requests must specify the location of a web page. You can also get the status of notifications that you have sent to Google. The Indexing API can only be used to crawl pages with either JobPosting or BroadcastEvent embedded in a VideoObject.

Guidelines

The following guidelines apply when using the Indexing API.

  • Our spam policies apply to content submitted with the Indexing API.
  • All calls to https://indexing.googleapis.com/v3/UrlNotifications:publish MUST use "application/json" as the Content-Type header.
  • You can submit one URL only in the body of an update request, or combine up to 100 requests in a batch, as described in Send batch indexing requests. Don't circumvent our submission limits, such as by using multiple accounts.
  • The request body in these examples is the value of the content variable that is used in the access token examples.

What you can do with the API

When you send a request to the Indexing API, define the location of a standalone web page to notify Google that it can crawl or remove this page from its index.

The following examples show the actions that you can perform with the Indexing API:

Examples
Update a URL

Send the following HTTP POST request to the https://indexing.googleapis.com/v3/urlNotifications:publish endpoint. For example:

{
  "url": "https://careers.google.com/jobs/google/technical-writer",
  "type": "URL_UPDATED"
}
Remove a URL

Send the following HTTP POST request to the https://indexing.googleapis.com/v3/urlNotifications:publish endpoint. For example:

{
  "url": "https://careers.google.com/jobs/google/technical-writer",
  "type": "URL_DELETED"
}
Get notification status

Send a HTTP GET request to the https://indexing.googleapis.com/v3/urlNotifications/metadata endpoint.

Parameters

The following table describes the fields needed for all methods (update and remove a URL):

Fields
url

Required

The fully-qualified location of the item that you want to update or remove.
type

Required

The type of the notification that you submitted.

Update a URL

To notify Google of a new URL to crawl or that content at a previously-submitted URL has been updated, follow these steps:

  1. Submit an HTTP POST request to the following endpoint: 
    https://indexing.googleapis.com/v3/urlNotifications:publish
  2. In the body of the request, specify the location of the page using the following syntax: 
    {
  "url": "CONTENT_LOCATION",
  "type": "URL_UPDATED"
}
  3. Google responds to successful Indexing API calls with the an HTTP 200. An HTTP 200 response means that Google may try to recrawl this URL soon. The body of the response contains a UrlNotificationMetadata object, whose fields correspond to those returned by a notification status request.
  4. If you don't receive an HTTP 200 response, see the Indexing API specific errors.
  5. If the page's contents change, submit another update notification, which should trigger Google to recrawl the page.
  6. The Indexing API provides a default quota for testing. To use the API, request approval and quota.

Remove a URL

After you delete a page from your servers or add <meta name="robots" content="noindex" /> tag in the <head> section of a given page, notify Google so that we can remove the page from our index and so that we don't attempt to crawl and index the page again. Before you request removal, the URL must return a 404 or 410 status code or the page must contain <meta name="robots" content="noindex" /> meta tag.

To request removal from our index, follow these steps:

  1. Submit a POST request to the following endpoint: 
    https://indexing.googleapis.com/v3/urlNotifications:publish
  2. Specify the URL that you want to remove in the body of the request using the following syntax: 
    {
  "url": "CONTENT_LOCATION",
  "type": "URL_DELETED"
}

    For example:

    {
  "url": "https://careers.google.com/jobs/google/technical-writer",
  "type": "URL_DELETED"
}
  3. Google responds to successful Indexing API calls with the an HTTP 200. An HTTP 200 response means that Google may remove this URL from the index. The body of the response contains a UrlNotificationMetadata object, whose fields correspond to those returned by a notification status request.
  4. If you don't receive an HTTP 200 response, see the Indexing API specific errors.
  5. The Indexing API provides a default quota for testing. To use the API, request approval and quota.

Get notification status

You can use the Indexing API to check the last time Google received each kind of notification for a given URL. The GET request doesn't tell you when Google indexes or removes a URL; it only returns whether you successfully submitted a request.

To get the status of a notification, follow these steps:

  1. Submit a GET request to the following endpoint. The URLs you specify must be URL-encoded. For example, replace : (colons) with %3A and / (forward slashes) with %2F. 
    https://indexing.googleapis.com/v3/urlNotifications/metadata?url=ENCODED_URL

    For example:

    GET https://indexing.googleapis.com/v3/urlNotifications/metadata?url=https%3A%2F%2Fcareers.google.com%2Fjobs%2Fgoogle%2Ftechnical-writer
  2. The Indexing API responds with an HTTP 200 message, with a payload that contains details about the notification. The following example shows the body of a response that contains information about an update and delete notification: 
    {
  url: "http://foo.com",
  latest_update: {
    type: "URL_UPDATED",
    notify_time: "2017-07-31T19:30:54.524457662Z"
  },
  latest_remove: {
    type: "URL_DELETED",
    notify_time: "2017-08-31T19:30:54.524457662Z"
  }
}
  3. If you don't receive an HTTP 200 response, see the Indexing API specific errors.
  4. The Indexing API provides a default quota for testing. To use the API, request approval and quota.

Send batch indexing requests

To reduce the number of HTTP connections your client has to make, you can combine up to 100 calls to the Indexing API into a single HTTP request. You do this in a multi-part request called a batch.

When sending a batch request to the Indexing API, use the following endpoint:

https://indexing.googleapis.com/batch

The body of a batch request contains multiple parts. Each part is itself a complete HTTP request, with its own verb, URL, headers, and body. Each part within a batch request cannot exceed 1MB in size.

To make it easier for you to send batch requests, Google's API Client Libraries support batching. For more information about batching with the client libraries, see the following language-specific pages:

If you use the batching examples on these pages, you may need to update your code to reflect the implementation requirements described in Get an access token.

The following example batch request message body includes an update notification and a removal notification:

POST /batch HTTP/1.1
Host: indexing.googleapis.com
Content-Length: content_length
Content-Type: multipart/mixed; boundary="===============7330845974216740156=="
Authorization: Bearer oauth2_token

--===============7330845974216740156==
Content-Type: application/http
Content-Transfer-Encoding: binary
Content-ID: <b29c5de2-0db4-490b-b421-6a51b598bd22+2>

POST /v3/urlNotifications:publish [1]
Content-Type: application/json
accept: application/json
content-length: 58

{ "url": "http://example.com/jobs/42", "type": "URL_UPDATED" }
--===============7330845974216740156==
Content-Type: application/http
Content-Transfer-Encoding: binary
Content-ID: <b29c5de2-0db4-490b-b421-6a51b598bd22+1>

POST /v3/urlNotifications:publish [2]
Content-Type: application/json
accept: application/json
content-length: 75

{ "url": "http://example.com/widgets/1", "type": "URL_UPDATED" }
--===============7330845974216740156==
Content-Type: application/http
Content-Transfer-Encoding: binary
Content-ID: <b29c5de2-0db4-490b-b421-6a51b598bd22+3>

POST /v3/urlNotifications:publish [3]
Content-Type: application/json
accept: application/json
content-length: 58

{ "url": "http://example.com/jobs/43", "type": "URL_DELETED" }
--===============7330845974216740156==

For more information, see Sending Batch Requests.