You can think of the indexing process as a journey of your product catalogue, from being described in the feed file in JSON format to being stored as an index in the search engine. In such a way, the stored information stays available for the Search microservice to browse and retrieve your catalogue's content at query time based on the shoppers' behavior.

Data journey: After you collect your product data into a feed file, the indexing process consists of both the automated transformation of your data, involving the feed's data validation and transformation into an index, and the automated storage of the index into the search engine. When the indexing process is completed, your product catalogue data is turned into data usable by the Search microservice to provide the search and discovery experience.

The indexing process starts when you send your feed file to the Index microservice. Afterward, the Index microservice performs the entire pipeline for indexing the feed: it manages the feed transformation in the Index Builder and the index deployment in Index Deployer.

Feed file transformation

The Index Builder is responsible for transforming the feed file into an index. The Index API receives your request for indexing the feed and saves the feed file in the Customer feeds storage.

To ensure the feed file is valid for being processed by the Index microservice, the Index Builder sends it to the Feed Validator. The validator checks if the feed file's format and codification meet the Index microservice's standards. In case of valid feed file, the Index API requests the Index Builder Worker to process the feed file. Otherwise, the indexing process stops, meaning that you need to modify the file to meet the required standards and start the process again. Information about the errors is provided to help you correct the file.

The Index Builder Worker processes the feed file from the Customer feeds database, retrieves information about the target search engine from your platform's implementation, and includes these pieces of data in an index package. When the index package is complete, the Index Builder Worker stores it in the Transformed feeds storage.

Lastly, the Index Builder Worker requests the Index Deployer for deploying the index in the search engine.

Index deployment

The Index Deployer is responsible for deploying the index in the search engine. The Index Deployer API receives a request for deploying the index via the Index Builder Worker, so it launches the deployment of the index via the Index Deployer Worker.

The Index Deployer Worker downloads the index package from the Transformed feeds storage and stores the index in the search engine specified in the package.

Eventually, the Index Deployer Worker downloads the index package from the Transformed feeds storage and stores the index in the search engine specified in the package.

To go futher

The Index API also provides endpoints to request information about the customers registered and the status of the jobs. The Index Deployer API provides endpoints to request information about the status of the jobs.