You can think of the indexing process as a journey of your product catalog, from being broken down in the feed file to being stored as an accessible index in the search engine. In such a way, the stored information stays available for the Search microservice to browse and retrieve your catalog's content based on the shoppers' behavior.

Data journey: After you collect your product data into a feed file, the indexing process consists of both the automated transformation of your data, involving the feed's data validation and transformation into one or several indexes and the automated storage of the index into the search engine. When the indexing process is completed, your product catalog information is turned into accessible data by the Search microservice to provide the search and discovery experiences.

The indexing process starts when you send your feed file to the Index microservice. Afterward, the Index microservice performs the entire pipeline for indexing the feed: it manages the feed transformation in the Index Builder and the index deployment in the Index Deployer.

Feed file transformation

The Index Builder is responsible for transforming the feed file into one or several indexes. The Index API receives your request to index the feed and saves the feed file in the Customer feeds storage.

To ensure the feed file is valid for being processed by the Index microservice, the Index Builder sends it to the Feed Validator. The Feed Validator checks if the feed file's format and codification meet the Index microservice's standards. In case of valid feed file, the Index API requests the Index Builder Worker to process the feed file. Otherwise, the indexing process stops, meaning that you need to modify the feed file to meet the required standards and start the process again. Information about the errors is provided to help you correct the feed file.

The Index Builder Worker processes the feed file from the Customer feeds database, retrieves information about the target search engine from your platform's implementation, and includes these pieces of data in an index package. When the index package is complete, the Index Builder Worker stores it in the Transformed feeds storage.

Lastly, the Index Builder Worker requests the Index Deployer to deploy the index in the search engine.

Index deployment

The Index Deployer is responsible for deploying the index in the search engine. The Index Deployer API receives a request for deploying the index via the Index Builder Worker, so it launches the deployment of the index via the Index Deployer Worker.

The Index Deployer Worker downloads the index package from the Transformed feeds storage and stores the index in the search engine specified in the package.

Eventually, the Index Deployer Worker downloads the index package from the Transformed feeds storage and stores the index in the search engine specified in the package.

To go further

The Index API also provides endpoints to request information about the customers registered and the status of the jobs.