Deep Search converts unstructured PDF documents into structured JSON files with accuracy and ease. It enables you to automate knowledge extraction as well as to fine-tune your proprietary Foundational Models and Large Language Models. We have already converted and collected the most common open-access repositories of technical documents for you to browse and search with ease. In the same way, you can point Deep Search to your proprietary document collections to create an Enterprise Search service.
Deep Search comes pre-loaded with millions of standard technical documents from many public data sources. We adhere to all licensing and only provide sources with permissive, open-access licenses. These sources are accessible after explicit license confirmation by the user.
Document collections are stored and indexed such that you can search and retrieve any document according to their contents, down to the values and physical units in tables. These large collections of documents are processed quickly and concurrently on scalable cloud infrastructure.
Deep Search converts large collections of PDF documents, such as scientific publications and patents, into JSON files. Our state-of-the-art AI detects and delimits document objects that contain information, such as paragraphs and tables. This information is then extracted from the objects, such as text from paragraphs and cells from tables.
After parsing a document for its content, Deep Search enriches it. Paragraphs of text are passed through natural language models. These models identify language structures such as sentences and terms, which are then classified into entity types such as a country or a physical property of a material. Likewise, image objects are detected and interpreted by computer vision models.
Use Deep Search to interlink your document collections into knowledge graphs. Knowledge graphs relate the entity types that have been discovered across documents. You can then query a graph to answer analytical questions that go beyond searching for keywords. For example:
For each material type that is mentioned in my collection of scientific papers, which of its physical properties have been tested and under which conditions?
For each company that is mentioned in my collection of annual reports, what was its total revenue per year?
Deep Search can be accessed programmatically to:
- Convert documents in bulk.
- Build document analysis tools on top.
- Integrate with automation workflows.
Take a look at the Deep Search Toolkit, our Python development kit and command line interface.
PatCID: Patent Chemical-structure Image Discovery
PatCID is a collection of molecular structures to speed up discovery of new molecules, automatically and at scale with Deep Search. This large-scale collection was created by processing molecular images in United States, European, Japanese, Korean, and Chinese patents. We have applied a graph-based visual recognition model to obtain high-precision conversion of these molecular images (MolGrapher, ICCV2023, accepted). Contact us for early access.
DocQA: A question-answering conversational assistant on your documents
DocQA enables information extraction from documents via a question-answering conversational assistant. The system integrates several technologies from different AI disciplines consisting of document conversion to machine-readable format (via computer vision), finding relevant data (via natural language processing), and formulating an eloquent response (via large language models). A research paper describing this application is published at the AAAI 2024 conference. Contact us for early access.
2021 - 2016
At IBM Research we are always looking for scientists and engineers, be it as interns, PhD students, or staff. Contact us with your résumé and interests.