Custom LLMs

• Introduction
• Setup
  • Install the extension
  • Using both custom LLM and Tiptap AI Cloud
• Resolver Functions
• Examples
  • Override a specific command resolution in completion mode
  • Register a new AI command and call a custom backend action
  • Use your custom backend in streaming mode

#Introduction

If you want to use a your own backend which provides access to a custom LLM, you can override the the resolver functions defined below on the extension configuration.

Make sure you’re returning the correct type of response and that you handle errors correctly.

🚨 We strongly advise against using e.g. OpenAI directly in your frontend as this will or could result in API token leakage!

#Setup

In order to use customized resolver functions, you need to install the advanced version of our Tiptap AI extension.

#Install the extension

Once done, you can install the extension from our private registry:

npm install @tiptap-pro/extension-ai-advanced

#Using both custom LLM and Tiptap AI Cloud

If you want to rely on our cloud in some cases, make sure that you setup your team as described here.

#Resolver Functions

You can define custom resolver functions on the extension options. Be aware that they expect the following return types.

#Examples

#Override a specific command resolution in completion mode

In this example we want to call our custom backend when the rephrase action/command is called. Everything else should be handled by the default backend in the Tiptap Cloud.

// ...
import Ai from '@tiptap-pro/extension-ai-advanced'
// ...

Ai.configure({
  appId: 'APP_ID_HERE',
  token: 'TOKEN_HERE',
  // ...
  // Define the resolver function for completions (attention: streaming and image have to be defined separately!)
  aiCompletionResolver: async ({
    action, text, textOptions, extensionOptions, defaultResolver,
  }) => {
    // Check against action, text, whatever you like
    // Decide to use custom endpoint
    if (action === 'rephrase') {
      const response = await fetch('https://dummyjson.com/quotes/random')
      const json = await response?.json()

      if (!response.ok) {
        throw new Error(`${response.status} ${json?.message}`)
      }

      return json?.quote
    }

    // Everything else is routed to the Tiptap AI service
    return defaultResolver({
      action, text, textOptions, extensionOptions,
    })
  },
})

#Register a new AI command and call a custom backend action

In this example, we register a new editor command named aiCustomTextCommand, use the Tiptap runAiTextCommand function to let Tiptap do the rest, and add a custom command resolution to call a custom backend (in completion mode).

// …
import { Ai, runAiTextCommand } from '@tiptap-pro/extension-ai-advanced'
// …

// Declare typings if TypeScript is used:
//
// declare module '@tiptap/core' {
//   interface Commands<ReturnType> {
//     ai: {
//       aiCustomTextCommand: () => ReturnType,
//     }
//   }
// }

const AiExtended = Ai.extend({
  addCommands() {
    return {
      ...this.parent?.(),

      aiCustomTextCommand: (options = {}) => props => {
        // Do whatever you want; e.g. get the selected text and pass it to the specific command resolution
        return runAiTextCommand(props, 'customCommand', options)
      },
    }
  },
})

// … this is where you initialize your Tiptap editor instance and register the extended extension

const editor = useEditor({
    extensions: [
        /* … add other extension */
        AiExtended.configure({
            /* … add configuration here (appId, token etc.) */
            aiCompletionResolver: async ({ action, text, textOptions, extensionOptions, defaultResolver }) => {
                if (action === 'customCommand') {
                    const response = await fetch('https://dummyjson.com/quotes/random')
                    const json = await response?.json()

                    if (!response.ok) {
                    throw new Error(`${response.status} ${json?.message}`)
                    }

                    return json?.quote
                }

                return defaultResolver({
                    action, text, textOptions, extensionOptions,
                })
            },
        }),
    ],
    content: '',
})

// … use this to run your new command:
// editor.chain().focus().aiCustomTextCommand().run()

#Use your custom backend in streaming mode

We’re entirely relying on a custom backend in this example.

Make sure that the function aiStreamResolver returns a ReadableStream<Uint8Array>.

And remember: If you want to use both streaming and the traditional completion mode, you’ll need to define a aiCompletionResolver, too!

// ...
import Ai from '@tiptap-pro/extension-ai-advanced'
// ...

Ai.configure({
  appId: 'APP_ID_HERE',
  token: 'TOKEN_HERE',
  // ...
  // Define the resolver function for streams
  aiStreamResolver: async ({
    action, text, textOptions, aborter
  }) => {
    const fetchOptions = {
      method: 'POST',
      headers: {
        accept: 'application/json',
        'content-type': 'application/json',
      },
      body: JSON.stringify({
        ...textOptions,
        text,
      }),
      signal: aborter.signal,
    }

    const response = await fetch(`<YOUR_STREAMED_BACKEND_ENDPOINT>`, fetchOptions)
    const json = await response?.json()

    if (!response.ok) {
      throw new Error(`${json?.error} ${json?.message}`)
    }

    return response?.body
  },
})