Deep Search

Deep Search leverages state-of-the-art AI methods to continuously collect, convert, enrich, and link large document collections. You can use it for both public and proprietary PDF documents. Deep Search is offered commercially as-a-service.

documents have been converted by our public system so far. Try it out for free and contact us for a full access evaluation. Try Deep Search

Overview

Deep Search converts unstructured PDF documents into structured JSON files with accuracy and ease. It enables you to automate knowledge extraction as well as to fine-tune your proprietary Foundational Models and Large Language Models. We have already converted and collected the most common open-access repositories of technical documents for you to browse and search with ease. In the same way, you can point Deep Search to your proprietary document collections to create an Enterprise Search service.

Collect

Deep Search comes pre-loaded with millions of standard technical documents from many public data sources. We adhere to all licensing and only provide sources with permissive, open-access licenses. These sources are accessible after explicit license confirmation by the user.

Patents from

USA
Europe
Japan
Korea
China

Open-Access Publications

SemanticScholar
CrossRef
PLOS
MDPI

Public Repositories

arXiv
PubMed-Central

News articles that are curated and related to technical topics

Document collections are stored and indexed such that you can search and retrieve any document according to their contents, down to the values and physical units in tables. These large collections of documents are processed quickly and concurrently on scalable cloud infrastructure.

Convert

Deep Search converts large collections of PDF documents, such as scientific publications and patents, into JSON files. Our state-of-the-art AI detects and delimits document objects that contain information, such as paragraphs and tables. This information is then extracted from the objects, such as text from paragraphs and cells from tables.

Enrich

After parsing a document for its content, Deep Search enriches it. Paragraphs of text are passed through natural language models. These models identify language structures such as sentences and terms, which are then classified into entity types such as a country or a physical property of a material. Likewise, image objects are detected and interpreted by computer vision models.

Link

Use Deep Search to interlink your document collections into knowledge graphs. Knowledge graphs relate the entity types that have been discovered across documents. You can then query a graph to answer analytical questions that go beyond searching for keywords. For example:

For each material type that is mentioned in my collection of scientific papers, which of its physical properties have been tested and under which conditions?

For each company that is mentioned in my collection of annual reports, what was its total revenue per year?

Toolkit

Deep Search can be accessed programmatically to:

Convert documents in bulk.
Build document analysis tools on top.
Integrate with automation workflows.

Take a look at the Deep Search Toolkit, our Python development kit and command line interface.

Publications

People

At IBM Research we are always looking for scientists and engineers, be it as interns, PhD students, or staff. Contact us with your résumé and interests.