Use with Diagrams

Let your AI agent work with the charts and diagrams of Diagrams. The document tools of the Server AI Toolkit read, add and change them like any other content. There are no extra tools to enable.

Alpha

Diagrams is in alpha and available through a pilot program. The API may change in future releases.

Continuation from the AI agent chatbot guide

This guide continues the AI agent chatbot guide. Read it first.

How it works

  1. When your editor has the chart and flowDiagram nodes of Diagrams, getEditorContext adds a description of them to the editor context. The Server AI Toolkit gives it to the AI in the systemPrompt of fetch-tools.
  2. A chart keeps its Chart Data in its data attribute, and its type, title and other settings, such as horizontal bars or a target line, in its Chart Spec, the spec attribute. A diagram keeps its Mermaid code in source and its caption in title. A chart's source attribute is different: it links the chart to a Data Source. The AI reads and writes these attributes in the format of your request.
  3. The AI changes charts and diagrams with the document tools, like tiptapQuery.

Set up the editor

Add Diagrams to the editor of your AI agent chatbot, see Install Diagrams. Then get the editor context from that editor, like in the AI agent chatbot guide:

import { getEditorContext, ServerAiToolkit } from '@tiptap/ai-toolkit'
import { useEditor } from '@tiptap/react'
import StarterKit from '@tiptap/starter-kit'
import { Diagrams } from '@tiptap-pro/extension-diagrams'
import { echartsRenderer } from '@tiptap-pro/extension-diagrams/echarts'

const editor = useEditor({
  immediatelyRender: false,
  extensions: [StarterKit, ServerAiToolkit, Diagrams.configure({ renderers: [echartsRenderer()] })],
})

const editorContext = editor ? getEditorContext(editor) : null

This editor draws charts. To draw flow diagrams too, add a Diagram Renderer, see A first diagram. Without a Mermaid Renderer, a diagram that the AI writes shows as unavailable.

You don't need to describe charts and diagrams with customNodes. getEditorContext adds their description.

Enable the tools

Enable tiptapQuery when you fetch the tool definitions:

tools: {
  tiptapQuery: true,
}

Its setAttrs action changes only the attributes that the AI names. So a chart keeps its other settings, and a diagram keeps its size, when the AI changes their data or code. tiptapRead and tiptapEdit also read and write charts and diagrams, but tiptapEdit replaces whole nodes. See Limitations.

Try prompts like these:

  • "Add a pie chart of the June revenue and costs after the first paragraph"
  • "Change the June revenue in the bar chart to 230"
  • "In the order flow diagram, add a Pack step between In stock? and Ship"

Check the result

The AI's changes reach your editor through collaboration, like the changes of any other user. To check a chart that the AI wrote, read it from the editor:

const charts = editor.getJSON().content?.filter((node) => node.type === 'chart') ?? []

console.log(charts.map((chart) => chart.attrs?.spec?.type))

Describe charts and diagrams yourself

The description names the chart types of Diagrams. Your Renderers may draw only some of them, or types of their own. To tell the AI which types your editor draws, add an item for chart with addJsonSchemaAwareness or the customNodes option of getEditorContext, and list the types in it. An item for chart or flowDiagram replaces the built-in description. See Custom nodes.

Limitations

  • The server doesn't check the data that the AI writes. A chart without rows, or with data that Diagrams can't read, shows as empty. A chart of a type that no registered Renderer draws shows as unavailable.
  • Replacing a chart or a diagram resets the attributes that the AI doesn't repeat, like the size of a diagram. setAttrs of tiptapQuery keeps them.
  • Text search and contains don't look inside attributes, so they don't find the numbers of a chart or the title of a diagram. To find charts and diagrams, the AI selects them by their type.
  • The AI can't link a chart to a Data Source or to a table. The Tools for a model of Diagrams can link a chart to a Data Source with insertLinkedChart. A CSV or JSON file at a URL needs allowSourceUrl, which is off by default.
  • The AI writes Mermaid diagrams. The server tells it to keep the code of draw.io and BPMN diagrams as it is, not to change the data of a chart that reads it from a Data Source, and to change the table that a chart or a diagram follows instead of the chart or the diagram. It doesn't enforce any of these.

Next steps