Use with Whiteboard

Let your AI agent work on a whiteboard. The board tools of the Server AI Toolkit add sticky notes, draw diagrams, group notes into columns, connect, change and delete shapes, and add doc frames for longer text. They change the board's Tiptap Cloud document, so all users on the board see the changes live.

How it works

1. The AI reads the board with readBoard: its shapes, the arrows between them and the shapes that the user selects.
2. The AI changes the board with the other board tools, like createShapes and createDiagram. It never gives coordinates. The tools size the new shapes, and place them next to the shapes that the AI names or the selection, or in the middle of the user's view, where they cover no other shape.
3. Each doc frame shows a document of its own, in its own Tiptap Cloud document. The AI writes it with tiptapRead and tiptapEdit.
4. With reviewOptions, the changes become suggestions of Tracked Changes, which users accept or reject on the board.

Set up the board

The board tools work on the board's Tiptap Cloud document. Set up the board's editor with collaboration, see Collaboration on the board. Then get the editor context from that editor, and send it with each chat message, like in the AI agent chatbot guide:

import { getEditorContext } from '@tiptap/ai-toolkit'

// The board's editor, with WhiteboardKit and BoardCollaboration
const boardEditorContext = getEditorContext(editor)

The board tools don't target nodes by their hash, so the board's editor doesn't need the ServerAiToolkit extension.

Enable the board tools

Enable the board tools when you fetch the tool definitions, with the board's editor context:

import { getAuthHeaders } from '@/lib/server-ai-toolkit/get-auth-headers'

const response = await fetch(`${apiBaseUrl}/v4/ai/toolkit/fetch-tools`, {
  method: 'POST',
  headers: await getAuthHeaders(),
  body: JSON.stringify({
    editorContext: boardEditorContext,
    tools: {
      readBoard: true,
      createShapes: true,
      arrangeShapes: true,
      createDiagram: true,
      updateShapes: true,
      deleteShapes: true,
      connectShapes: true,
      createDocFrame: true,
    },
  }),
})

Enable only the tools that your agent needs. Without deleteShapes, for example, the AI can't delete shapes.

The board tools take plain text, not document content, so they work the same in every format. Their descriptions tell the AI how to use them, so it doesn't need the systemPrompt of this request. The board tools aren't available through stream-tool.

Execute the board tools

Execute the board tools on the board's document. tool.config tells them what the user selects and sees:

const result = await fetch(`${apiBaseUrl}/v4/ai/toolkit/execute-tool`, {
  method: 'POST',
  headers: await getAuthHeaders(documentId),
  body: JSON.stringify({
    editorContext: boardEditorContext,
    document: { type: 'cloud', id: documentId },
    user: 'ai-assistant',
    tool: {
      name: toolName,
      input,
      // What the user selected and saw when they sent the message
      config: { selectedIds, viewport },
    },
  }),
})

• selectedIds are the shapes that the user selects. readBoard lists them first, with their full text, so the AI knows which shapes "these notes" means. New shapes go right of them, or below them when that is closer.
• viewport is the part of the board that the user sees, in board units. Without a selection, new shapes go in the middle of it.

Without either, new shapes go right of all shapes on the board, or to the origin of an empty board. A value that is missing, null or invalid falls back on its own. The AI can also name shapes in near, so a summary lands next to its diagram. See board placement.

Read both in the browser when the user sends a message, and send them along:

import { getSelectedShapeIds, visibleBoardRect } from '@tiptap-pro/extension-whiteboard'

sendMessage(
  { text: input },
  {
    body: {
      editorContext: boardEditorContext,
      selectedIds: getSelectedShapeIds(editor.state),
      viewport: visibleBoardRect({
        // Your app's camera, and the stage element around the board
        camera,
        viewport: { width: stage.clientWidth, height: stage.clientHeight },
      }),
    },
  },
)

The board keeps its selection while the user types in the chat, so you don't need to focus the board again.

When users change the board while a tool runs, execute-tool can answer 409 with concurrent_edit_conflict and change nothing. Run the tool again. The server checks this in the default field of a Tiptap Cloud document. In another field, it saves without the check. When the document or the editor context isn't a whiteboard, it answers 403 with board_schema_not_supported.

Write the documents of doc frames

Each doc frame shows a document of its own. createDocFrame adds a frame with a new, empty document and returns its docId, and readBoard lists the docId of each frame. The AI then reads and writes the document with tiptapRead and tiptapEdit.

Keep the documents in Tiptap Cloud

The AI server writes the documents in Tiptap Cloud. Keep the extensions of the frames' documents in a module that the browser and the server share, because the server needs the same schema. Add the ServerAiToolkit extension to them, because the document tools target nodes by their hash:

// lib/whiteboard-docs.ts, shared by the browser and the server
import { ServerAiToolkit } from '@tiptap/ai-toolkit'
import StarterKit from '@tiptap/starter-kit'

export const docExtensions = [StarterKit.configure({ undoRedo: false }), ServerAiToolkit]

// The name of the Tiptap Cloud document of a frame, on the board `documentId`
export const frameDocumentName = (documentId: string, docId: string) => `${documentId}/doc/${docId}`

In the browser, connect the editor of each frame to the Tiptap Cloud document of its docId:

import { Editor } from '@tiptap/core'
import { Collaboration } from '@tiptap/extension-collaboration'
import type { CreateDocEditorOptions } from '@tiptap-pro/extension-whiteboard'
import { TiptapCollabProvider } from '@tiptap-pro/provider'
import * as Y from 'yjs'
import { docExtensions, frameDocumentName } from '@/lib/whiteboard-docs'

function createEditor({ element, docId }: CreateDocEditorOptions) {
  const document = new Y.Doc()
  const provider = new TiptapCollabProvider({
    name: frameDocumentName(documentId, docId),
    appId,
    token,
    document,
  })

  return new Editor({
    element,
    extensions: [...docExtensions, Collaboration.configure({ document })],
    onDestroy: () => provider.destroy(),
  })
}

Use it as createEditor in the docs option of WhiteboardKit. See Doc frames for copy and start. Frames that the AI adds come through Yjs, so start doesn't run for them. createEditor opens their document by its docId, and shows what the AI writes there.

The browser's token must allow the documents of the frames too, not only the board's document. Allow the documents whose names start with ${documentId}/doc/, with a prefix constraint:

permissions: [
  { action: 'Documents:Write', resource: documentId },
  { action: 'Documents:Write', resource: '*', constraints: { prefix: `${documentId}/doc/` } },
],

Each provider opens a connection of its own. To open the documents of all frames over one connection, see Multiplexing.

Give the document tools a docId

The document tools work on one document. On the server, get the editor context of the frames' documents from an editor with docExtensions. Give it JSON content, because the default content is HTML, which needs a browser:

import { getEditorContext } from '@tiptap/ai-toolkit'
import { Editor } from '@tiptap/core'
import { docExtensions } from '@/lib/whiteboard-docs'

// Only for the schema of the frames' documents
const docEditor = new Editor({
  extensions: docExtensions,
  content: { type: 'doc', content: [{ type: 'paragraph' }] },
})
const docEditorContext = getEditorContext(docEditor)
docEditor.destroy()

Then fetch tiptapRead and tiptapEdit with that editor context. Add a docId to their input, and execute them on the document of that docId:

import { tool } from 'ai'
import z from 'zod'
import { executeTool } from '@/lib/server-ai-toolkit/execute-tool'
import { getToolDefinitions } from '@/lib/server-ai-toolkit/get-tool-definitions'
import { frameDocumentName } from '@/lib/whiteboard-docs'

// tiptapRead and tiptapEdit, and the prompt about the format of the documents
const { prompt: docsPrompt, tools: docToolDefinitions } = await getToolDefinitions(docEditorContext)

const docTools = Object.fromEntries(
  docToolDefinitions.map((definition) => [
    definition.name,
    tool({
      description: definition.description,
      inputSchema: z.fromJSONSchema({
        ...definition.inputSchema,
        properties: {
          docId: {
            type: 'string',
            description: 'The docId of the doc frame, from readBoard or createDocFrame.',
          },
          ...definition.inputSchema.properties,
        },
        required: ['docId', ...(definition.inputSchema.required ?? [])],
      }),
      strict: false,
      execute: async (toolInput) => {
        const { docId, ...input } = toolInput as Record<string, unknown>

        // The docId comes from the AI, and becomes part of a document name
        if (typeof docId !== 'string' || !/^[\w-]+$/.test(docId)) {
          return { success: false, error: 'Use the docId of a doc frame, from readBoard.' }
        }

        try {
          const result = await executeTool(
            definition.name,
            input,
            frameDocumentName(documentId, docId),
            docEditorContext,
          )
          return result.tool.output
        } catch (error) {
          console.error(`Failed to execute tool ${definition.name}:`, error)
          return { error: error instanceof Error ? error.message : 'Unknown error' }
        }
      },
    }),
  ]),
)

Give the AI these tools next to the board tools, and add docsPrompt to the instructions of your agent, so the AI knows the format of the documents.

The board draws a frame as tall as its document, but stores the height that createDocFrame gives it, from its lines input. Arrows, selection and placement use the stored height. The board updates it only after the changes of its own users, so it stays until a user edits the document.

Review the AI's changes

Add Tracked Changes to the board's editor, as in Tracked Changes on the board, and send the editor context of that editor. Then pass reviewOptions when you execute the board tools:

reviewOptions: {
  mode: 'trackedChanges',
  trackedChangesOptions: {
    userId: 'ai-assistant',
    userMetadata: { name: 'AI Assistant' },
  },
},

The board tools then suggest their changes, instead of making them:

• New shapes become add suggestions. Moves, colors and kinds become attributeChange suggestions with the old and new values, and a new text replaces the old paragraphs as a suggestion.
• Deleted shapes stay on the board with a delete suggestion, until a user accepts it. A shape that a pending suggestion adds goes right away.
• The shapes that one call adds or deletes share one suggestion, so one acceptSuggestion or rejectSuggestion reviews them all.
• Each changed shape and each new text get a suggestion of their own. suggestionIds lists the suggestions that the call made or went on with, so review all of them to review the whole call. A suggestion that the call went on with also holds an earlier change of the same shape.
• The tools leave a shape alone that waits for another user's review, and say why in skipped.
• readBoard marks a shape with suggestion when a suggestion adds, deletes or changes it, also when only its text waits for a review.

Without Tracked Changes in the editor context, execute-tool answers 403 with tracked_changes_unsupported_schema, like for the document tools, and changes nothing, instead of changing the board directly. Without reviewOptions, the tools change the board directly, and leave every shape alone that waits for a review.

For the documents of doc frames, add TrackedChanges to docExtensions, so that the frames' editors and docEditorContext both have it. Otherwise execute-tool answers 403 with tracked_changes_unsupported_schema. Then pass the same reviewOptions to the document tools, by adding them to the request body of the executeTool helper. See Use with Tracked Changes. A document that the AI creates in the same answer can skip the review, because its frame waits for one already.

Limitations

• The tools add only the whiteboard's own shapes: sticky notes, geometric shapes, free text, arrows and doc frames. readBoard lists every shape that has an id and a box.
• The server can't measure the fonts of the browser, so the tools estimate the height of titles, labels and doc frames. The board measures only the changes of its own users, so these heights stay until a user edits the text.
• Arrows are straight lines between the centers of their shapes, so a long arrow can cross other shapes. The label of an arrow is a free text shape, which stays in place when a user moves the arrow's shapes.
• readBoard lists at most 300 shapes and arrows, the selected shapes first. It cuts the text of the other shapes after 280 characters.
• The server writes a call into the board's Yjs document by comparing the whole board. When a call removes a shape and changes a later one, like a deleteShapes that frees arrow ends, a user's edit of the shapes in between at the same time can get lost or land on another shape.

Next steps

• See the board tools in the REST API reference
• Set up doc frames and collaboration on the board