Papers with Code has updated its search system based on Hugging Face infrastructure

The Papers with Code platform has migrated its search system to utilize Hugging Face tools, including Inference Endpoints, Jobs, and Buckets. The update has enabled the automation of scientific article indexing and accelerated user query processing.

The new architecture is based on Inference Endpoints, which provide hosting for models to generate vector representations (embeddings) in real-time. Jobs are used for batch data processing and regular index updates, allowing computational tasks to be run on a schedule without the need for maintaining a constant server infrastructure.

Data storage is organized through Buckets, which are integrated with the Hugging Face ecosystem for quick access to model weights and datasets. This combination of components has reduced the search system's response time, ensuring the relevance of search results when new publications are added to the project's database.

The use of the platform's cloud services has allowed developers to abandon managing their own GPU clusters. The search index update process is now fully automated: from extracting text from PDF files to vectorization and recording in the database, which minimizes delays between the publication of an article and its appearance in search results.

The transition to Hugging Face's infrastructure solutions demonstrates the scalability of specialized search engines for scientific data. The integration of components enables the efficient processing of the growing volume of publications while maintaining high semantic search accuracy for users of the resource.