Create Knowledge from a Knowledge Pipeline

A knowledge pipeline is a document processing workflow that transforms raw data into searchable knowledge bases. Think of orchestrating a workflow, now you can visually combine and configure different processing nodes and tools to optimize data processing for better accuracy and relevance.

Every knowledge pipeline normally follows a structured flow through four key steps:

Data Sources → Data Extraction → Data Processing → Knowledge Storage

Each step serves a specific purpose: gathering content from various sources, converting it to processable text, refining it for search, and storing it in a format that enables fast, accurate retrieval.

Dify provides built-in pipeline templates that is optimized for certain use cases, or you can also create knowledge pipelines from scratch. In this session, we will go through creating options, general process of building knowledge pipelines, and how to manage it.

1. Step 1: Create Knowledge Pipeline

   Start from built-in templates, blank knowledge pipeline or import existing pipeline.

2. Step 2: Orchestrate Knowledge Pipeline

   Get to know how the knowledge pipeline works, orchestrate different nodes and make sure it’s ready to use.

3. Step 3: Publish Knowledge Pipeline

   Let’s make it ready for document processing.

4. Step 4: Upload Files

   Add documents and process them into the searchable knowledge base.

5. Step 5: Manage and Use Knowledge Base

   Maintain documents, test retrieval, modify settings, and more.