Deep Search
Deep Search leverages state-of-the-art AI methods to continuously collect, convert, enrich, and link large document collections. You can use it for both public and proprietary PDF documents. Deep Search is offered commercially as-a-service.
documents have been converted by our open access system. Try it yourself for free or request access to an enterprise system.
| Feature | Free | Enterprise |
|---|---|---|
| Convert documents |  |
|
| Extract components such as tables and figures | |
|
| Automate your process via the Toolkit | |
|
| Upload and search your own documents | |
|
| Search millions of pre-loaded documents | |
|
| Configure the document conversion process | |
|
| Search and extract chemical (sub-)structures | on request |
Overview
Deep Search converts unstructured PDF documents into structured JSON files with accuracy and ease. It enables you to automate knowledge extraction as well as to fine-tune your proprietary Foundational Models and Large Language Models. We have already converted and collected the most common open-access repositories of technical documents for you to browse and search with ease. In the same way, you can point Deep Search to your proprietary document collections to create an Enterprise Search service.
Collect
Deep Search comes pre-loaded with millions of standard technical documents from many public data sources. We adhere to all licensing and only provide sources with permissive, open-access licenses. These sources are accessible after explicit license confirmation by the user.
- USA
- Europe
- Japan
- Korea
- China
- SemanticScholar
- CrossRef
- PLOS
- MDPI
- arXiv
- PubMed-Central
Document collections are stored and indexed such that you can search and retrieve any document according to their contents, down to the values and physical units in tables. These large collections of documents are processed quickly and concurrently on scalable cloud infrastructure.
Convert
Deep Search converts large collections of PDF documents, such as scientific publications and patents, into JSON files. Our state-of-the-art AI detects and delimits document objects that contain information, such as paragraphs and tables. This information is then extracted from the objects, such as text from paragraphs and cells from tables.
Enrich
After parsing a document for its content, Deep Search enriches it. Paragraphs of text are passed through natural language models. These models identify language structures such as sentences and terms, which are then classified into entity types such as a country or a physical property of a material. Likewise, image objects are detected and interpreted by computer vision models.
Link
Use Deep Search to interlink your document collections into knowledge graphs. Knowledge graphs relate the entity types that have been discovered across documents. You can then query a graph to answer analytical questions that go beyond searching for keywords. For example:
For each material type that is mentioned in my collection of scientific papers, which of its physical properties have been tested and under which conditions?
For each company that is mentioned in my collection of annual reports, what was its total revenue per year?
Toolkit
Deep Search can be accessed programmatically to:
- Convert documents in bulk.
- Build document analysis tools on top.
- Integrate with automation workflows.
Take a look at the Deep Search Toolkit, our Python development kit and command line interface.
Use cases
PatCID: Patent Chemical-structure Image Discovery
PatCID is a collection of molecular structures to speed up discovery of new molecules, automatically and at scale with Deep Search. This large-scale collection was created by processing molecular images in United States, European, Japanese, Korean, and Chinese patents. We have applied a graph-based visual recognition model to obtain high-precision conversion of these molecular images (MolGrapher, ICCV2023, accepted). Contact us for early access.
Publications
2023
2022
2021 - 2016
People
At IBM Research we are always looking for scientists and engineers, be it as interns, PhD students, or staff. Contact us with your résumé and interests.
Team
Locations
