OpenDCAI/DataFlow

Generate, Clean, and Prepare LLM Data, All-in-One

Visual, low-code pipelines with flexible orchestration across domains and use cases.💪

Turn raw data into high-quality LLM training datasets.🔧

🎉 Get smarter LLMs cheaply — give us a star ⭐ on GitHub for the latest update.

Beginner-friendly learning resources (continuously updated): [🎬 Video Tutorials] [📚 Written Tutorials]

简体中文 | English

• [2026-09-12] 🎉 DataFlow has been accepted by ACM SIGMOD 2027! Our paper, DataFlow: An LLM-Driven Framework for Unified Data Preparation and Workflow Automation in the Era of Data-Centric AI, has been accepted by ACM SIGMOD 2027.

• [2026-07-18] DataFlow-Harness Released! 🔥DataFlow-Harness lets coding agents build DataFlow pipelines for you. Download and installation entry points are available in DataFlow-WebUI, where you can use DataFlow through either the visual interface or MCP.

• [2026-05-22] DataFlow-Skills is now available! A collection of skills and tutorials for working with DataFlow. 👉 DataFlow-Skills

• [2026-02-02] 🖥️ DataFlow WebUI is now available! Launch the visual pipeline builder with a single command: dataflow webui. Build and run DataFlow pipelines through an intuitive web interface. 👉 WebUI Docs

  

• [2026-01-20] 🌟 Awesome Works Using DataFlow is now live! A new section showcasing open-source projects and research built on DataFlow. Contributions are welcome! 👉 Awesome Works

• [2025-12-19] 🎉 Our DataFlow technical report is now available! Read and cite our work on arXiv: https://arxiv.org/abs/2512.16676

• [2025-11-20] 🤖 Introducing New Data Agents for DataFlow! Try them out and follow the tutorial on Bilibili: https://space.bilibili.com/3546929239689711/lists/6761342?type=season

• [2025-06-28] 🎉 DataFlow is officially released! Our data-centric AI system is now public. Stay tuned for future updates.

DataFlow is a data preparation and training system designed to generate, refine, evaluate, and filter high-quality data for AI from noisy sources (PDF, plain-text, low-quality QA), thereby improving the performance of large language models (LLMs) in specific domains through targeted training (Pre-training, Supervised Fine-tuning, RL training) or RAG system, in domains such as healthcare, finance, legal, and academic research.

Through an operator-based design, DataFlow turns the entire data cleaning workflow into a reproducible, reusable, and shareable pipeline, providing core infrastructure for the Data-Centric AI community. Additionally, we develop an intelligent DataFlow-agent capable of dynamically assembling new pipelines by recombining existing or creating new operators on demand.

• High-Quality Training Data Generation
  • Text, Math, and Code data generation (see DataFlow-Instruct-10K for results)
  • Data generation via tools like AgenticRAG and Text2SQL
• Structured Data Extraction
  • Large-scale PDF → QA conversion
  • Large-scale book PDF → Visual-QA conversion
• Scientific Data Workflow Management
  • Text2SQL workflow management (Accepted by ICDE 2026)
  • Math data workflows (Accepted by KDD 2026)

• 10+ core operators define interaction patterns and design principles
• 100+ pipeline-specific operators available for reuse or reference
• Full support for creating custom operators — plug-and-play, easily packaged and distributed via GitHub or PyPI

• Data governance algorithms are encapsulated as operator pipelines, enabling reproducibility and fair comparison of different data governance strategies (❤️research-friendly)
• Easily reuse swap underlying large models to analyze the relationship between model performance and data quality quickly
• Built on Python and Git ecosystems for easy distribution, management, and traceability of high-quality, user-defined data governance operators and pipelines (❤️enterprise-friendly)

The DataFlow Suite provides the essential infrastructure to automate and scale LLM data preparation with DataFlow main repository. It comprises four tightly integrated layers:

• DataFlow-Skills – A collection of skills and tutorials for working with DataFlow, covering operator development, pipeline construction, and best practices for data-centric AI.

• DataFlow-WebUI – A visual and AI Agent assisted workspace for building, managing, and running DataFlow pipelines. It combines drag-and-drop pipeline editing with natural-language interaction, enabling users to construct and execute data workflows more efficiently.

• DataFlow-Ecosystem – A modular distribution layer that standardizes operator registration. It enables domain-specific modules (e.g., DataFlow-MM, DataFlow-AI4S) to contribute extensible libraries under a unified abstraction.

• RayOrch – A high-performance orchestration layer built on Ray, providing distributed compute scheduling and resource management for massive-scale data tasks.

Together, these components form a unified, extensible environment that transforms raw data into model-ready intelligence.

Data generation and cleaning are crucial for high-quality models, but for both enterprises and individuals, these tasks are often time-consuming, labor-intensive, and costly. DataFlow provides a one-stop solution to tackle these challenges efficiently. DataFlow offers the following advantages:

• Enhanced Support for Data Synthesis Modules – Seamlessly integrates text, code, and math data generation pipeline for high-quality training datasets.
• PyTorch-like Programming Management – Clear Pipeline → Operator → Prompt hierarchical structure for workflow control.
• Principled and Multi-Category Operator Classification – Operators are systematically organized into multiple functional categories such as generation, evaluation, filtering, and refinement, forming a scientifically grounded, multi-dimensional taxonomy that reflects different stages of data preparation and enables precise operator selection and composition.
• User-Friendly Design for Easy Debugging and Onboarding – Simplified workflow patterns that reduce the learning curve and accelerate experimentation.

This section highlights open-source projects, research works, and applications built with DataFlow or deeply integrated with the DataFlow ecosystem.

• dataflow-speech-quality
  • Speech Quality · Data Cleaning · De-identification · Annotation
  • A DataFlow extension for filtering, cleaning, de-identifying, and annotating customer-service speech transcripts.
  • Maintainer: Jiapeng Mi · China Unicom Software Research Institute

Have a project built with DataFlow? We welcome community contributions via Pull Requests. See the DataFlow Ecosystem Guide for guidance on creating a DataFlow extension.

DataFlow operators are designed with simplicity and clarity in mind.

Operators take structured inputs (JSON, JSONL, CSV) and produce high-quality outputs after intelligent processing. Each operator encapsulates a specific data processing task, providing a clean and consistent API that is easy to understand and integrate. The PyTorch-like design makes them intuitive and ready to use, allowing you to quickly build, combine, and customize pipelines without dealing with complex boilerplate code.

For more details, refer to the Operator Documentation. Below is a minimal example demonstrating how to invoke the PromptedGenerator operator:

Example input data (json/jsonl-style):

// input.json
[
  {"problem": "What is 17 + 25?"},
  {"problem": "If x = 3, compute 2x^2 + 1."}
]

Operator invocation code:

from dataflow.operators.core_text import PromptedGenerator
from dataflow.utils.storage import FileStorage
from dataflow.serving import APILLMServing_request

# set input file to global storage class
storage = FileStorage(first_entry_file_name="./input.json",)

# configure LLM serving (e.g., OpenAI API)
# api key needs to be set via `export DF_API_KEY=sk-xxx`
llm_serving = APILLMServing_request(
    api_url="https://api.openai.com/v1/chat/completions",
)

prompted_generator = PromptedGenerator(
    llm_serving=llm_serving,  # pre-configured LLM backend
    system_prompt="Please solve this math problem."
)

prompted_generator.run(
    storage=self.storage.step(),  # data management (details omitted)
    input_key="problem",          # read from this column
    output_key="solution"         # write to this column
)

After running, the operator will append the generated results into output_key. For example, the output data (json/jsonl-style) becomes:

// dataflow_step1.json
[
    {"problem":"What is 17 + 25?","solution":"42"},
    {"problem":"If x = 3, compute 2x^2 + 1.","solution":"19"}
]

DataFlow supports Python>=3.10 environments, tested passed on Windows, Linux, and MacOS with Python 3.10, 3.11, and 3.12.

Please use the following commands for environment setup and installation👇

We recommend use uv to install DataFlow for speed up.

pip install uv
uv pip install open-dataflow

If you want to use your own GPU for local inference, please use:

pip install uv
uv pip install "open-dataflow[vllm]"

After installation, you can use the following command to check if dataflow has been installed correctly:

dataflow -v

If installed correctly, you should see:

open-dataflow codebase version: 1.0.0
        Checking for updates...
        Local version:  1.0.0
        PyPI newest version:  1.0.0
You are using the latest version: 1.0.0.

We also provide a Dockerfile for easy deployment and a pre-built Docker image for immediate use.

You can directly pull and use our pre-built Docker image:

# Pull the pre-built image
docker pull molyheci/dataflow:cu124

# Run the container with GPU support
docker run --gpus all -it molyheci/dataflow:cu124

# Inside the container, verify installation
dataflow -v

Alternatively, you can build the Docker image from the provided Dockerfile:

# Clone the repository (HTTPS)
git clone https://github.com/OpenDCAI/DataFlow.git
# Or use SSH
# git clone git@github.com:OpenDCAI/DataFlow.git

cd DataFlow

# Build the Docker image
docker build -t dataflow:custom .

# Run the container
docker run --gpus all -it dataflow:custom

# Inside the container, verify installation
dataflow -v

Note: The Docker image includes CUDA 12.4.1 support and comes with vLLM pre-installed for GPU acceleration. Make sure you have NVIDIA Container Toolkit installed to use GPU features.

You can start your first DataFlow translation project directly on Google Colab. By following the provided guidelines, you can seamlessly scale from a simple translation example to more complex DataFlow pipelines.

👉 Start DataFlow with Google Colab

For detailed usage instructions and getting started guide, please visit our DataFlow Documentation.

DataFlow provides a Web-based UI (WebUI) for visual pipeline construction and execution.

To start DataFlow-WebUI, simply run following command after install the DataFlow main repository:

dataflow webui

This will automatically download and launch the latest DataFlow-WebUI and open it in your browser (http://localhost:8000/ if it does not open automatically).

• Chinese: DataFlow-WebUI Documentation: https://wcny4qa9krto.feishu.cn/wiki/F4PDw76uDiOG42k76gGc6FaBnod
• English: DataFlow-WebUI Documentation: https://wcny4qa9krto.feishu.cn/wiki/SYELwZhh9ixcNwkNRnhcLGmWnEg

• https://github.com/OpenDCAI/DataFlow-webui

DataFlow-Instruct-10K is a unified multi-domain instruction dataset generated by the DataFlow framework. It is constructed through several automated data preparation pipelines spanning mathematical reasoning, code, and general text instructions. Each pipeline follows a generate–evaluate–filter–refine workflow to synthesize and curate high-quality instruction–response pairs. The resulting dataset contains approximately 10K samples and provides high-quality supervision for instruction tuning, enabling base models to approach the performance of fully trained instruct models with significantly fewer training examples.

For Detailed Experiments setting, please visit our DataFlow Technical Report.

Our team has published the following papers that form core components of the DataFlow system:

Contributing Institutions:

We are honored to have received first-place awards in two major international AI competitions, recognizing the excellence and robustness of DataFlow and its reasoning capabilities:

ICML 2025 Automated Math Reasoning Challenge — First Place Winner

BAAI Language & Intelligence Challenge 2025 — First Prize

We sincerely thank MinerU for their outstanding work, whose powerful PDF/document text extraction capabilities provided essential support for our data loading process. We also thank LLaMA-Factory for offering an efficient and user-friendly framework for large model fine-tuning, which greatly facilitated rapid iteration in our training and experimentation workflows. Our gratitude extends to all contributors in the open-source community—their efforts collectively drive the development of DataFlow. We thank Zhongguancun Academy for their API and GPU support.

Join the DataFlow open-source community to ask questions, share ideas, and collaborate with other developers!

• 📮 GitHub Issues: Report bugs or suggest features

• 🔧 GitHub Pull Requests: Contribute code improvements

• 💬 Join our community groups to connect with us and other contributors!

The DataFlow paper has been accepted for publication in the Proceedings of the ACM on Management of Data (PACMMOD) and presentation at ACM SIGMOD 2027. Until the official ACM citation becomes available, please cite the arXiv version:

@article{liang2025dataflow,
  title={DataFlow: An LLM-Driven Framework for Unified Data Preparation and Workflow Automation in the Era of Data-Centric AI},
  author={Liang, Hao and Ma, Xiaochen and Liu, Zhou and Wong, Zhen Hao and Zhao, Zhengyang and Meng, Zimo and He, Runming and Shen, Chengyu and Cai, Qifeng and Han, Zhaoyang and others},
  journal={arXiv preprint arXiv:2512.16676},
  year={2025}
}