Feeds can also be sent using a message queue to the Index API. However, this procedure is not common due to its complexity. The Index API consumes a configured queue, and any message received in this queue is processed and transformed into an index job that is queued and then processed within the indexing process.

Several cloud providers are supported, but Empathy's deployment pipeline is hosted in Amazon Web Services (AWS). This deployment uses a Simple Queue Service (SQS) queue and an S3 bucket, which requires permissions to write in both the S3 bucket and the Index queue. Consequently, this mechanism can be used only by the Index API clients that share the cloud provider with the Index pipeline deployment.

note

Request the permissions needed to your Key account manager if you need them to send a message to the Index API queue.

The message to the SQS should preferably be a JSON file with the following body content:

Key	Value
indexType	The index type: catalog, empathize, search configuration, etc.
opType	The operation that needs to be performed: full, partial, delta, etc.
siteId	Client name that is used as the client identifier.
generated_time	Time in format yyyy-MM-dd'T'HH:mm:ss'Z' used to generate the index version.
feeds_uri	Complete feed path, including the storage prefix (for example: s3://empathyco-test-index-data-feeds/path/to/file).
pivots	Pivots used to distinguish the type of feed. Example: "pivots": {"lang": "en"}

An example of a message without pivots is:

{
  "indexType": "BROWSECONFIGURATION",
  "opType": "FULL",
  "siteId": "empathy",
  "generated_time": "2023-08-15T06:35:57Z",
  "feeds_uri": "s3://empathyco-test-index-data-feeds/play/empathy-BROWSE_CONFIGURATION-2023-08-15T06:35:57Z-a2e45ed5-97be-45e0-8742-7f5cfd725349"
}

An example of a message with pivots is:

{
  "indexType": "CATALOG",
  "opType": "FULL",
  "siteId": "empathy",
  "generated_time": "2023-08-15T06:35:57Z",
  "feeds_uri": "s3://empathyco-test-index-data-feeds/play/empathy-CATALOG-en-2023-08-15T06:35:57Z-a2e45ed5-97be-45e0-8742-7f5cfd725349",
  "pivots": {
    "lang": "en"
  }
}