# ScribbleGPT: Building a Basic Handwriting-Driven Chatbot

Creating ScribbleGPT, a chatbot inspired by Tom Riddle's diary from the *Harry Potter* series, was a fun and exciting challenge. The bot’s user interface replicates the experience of writing in Riddle’s diary where messages appear and vanish after a few seconds. In this article, I’ll walk through the technical aspects of bringing this magical experience to life.

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1725856178247/f0991849-92b9-4ea1-aa23-672102be0ba8.gif align="center")

### HTML Canvas to Write On

The foundation of ScribbleGPT’s interface is the HTML Canvas where users write their messages. I started building one from scratch, seems pretty simple, listed to up and down events and track the move points, but the crude implementation was generating very pointy lines. So, to simplify the smoothing process, I opted for [Signature\_Pad](https://github.com/szimek/signature_pad) library. This library is ideal for smooth handwriting due to its built-in smoothing functions

* **Simple to Use**: [Signature\_Pad](https://github.com/szimek/signature_pad) offers a clean, straightforward way to capture input, which is perfect for this hack, it literally took 10 seconds to get it setup. Awesome documentation and really intuitive API methods
    
* **Programmatic Access**: Even though users interact with the [Signature\_Pad](https://github.com/szimek/signature_pad) component, the response should somehow be written back to the canvas. Since we have access to the canvas altogether, we can work with it directly. So, taking user input via the pad and write the AI response back on the canvas
    
* **Stroke Detection**: When a user finishes writing, an `endStroke` event is triggered, here we can trigger the text detection process. To avoid unnecessary recognitions, we can add **debounce** logic to minimize the GPT calls
    

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1725856830131/7fbceb5f-6dbf-4b0d-8d8e-2c4ec033232f.gif align="center")

```javascript
export default function Home() {
  const canvasRef = useRef();

  useEffect(() => {
    if (!canvasRef.current) {
      return;
    }
    const canvas = canvasRef.current;

    const writingPad = new SignaturePad(canvas);

    resizeCanvas(canvas);
  });

  function resizeCanvas(canvas) {
    const ratio = Math.max(window.devicePixelRatio || 1, 1);
    canvas.width = canvas.offsetWidth * ratio;
    canvas.height = canvas.offsetHeight * ratio;

    canvas.getContext("2d").scale(ratio, ratio);
  }

  return (
    <div style={{ height: "100vh" }}>
      <canvas ref={canvasRef} style={{ width: "100%", height: "100%", background: "#f3dfc1" }} />
    </div>
  );
}
```

### Handwriting Recognition

Initially, I explored [**Tesseract.js**](https://github.com/naptha/tesseract.js#tesseractjs) for text recognition. Unfortunately, it didn’t work well for handwriting, especially in the case of cursive or varied styles. It’s better suited for printed text, and training it for handwriting would have required significant customization.

Instead, I found **Google Input Tools** to be a great alternative. Though the API isn’t officially documented, and there’s a risk it could stop working at any moment, it does a pretty solid job of recognizing handwriting. I leveraged its handwriting input tool, and so far, it has proven reliable for this project.

* [Handwriting – Google Input Tools](https://www.google.com/inputtools/services/features/handwriting.html)
    
* [Try Google Input Tools online](https://www.google.com/inputtools/try/)
    

Once the `endStroke` event is fired, we get the points from the canvas and convert it to input format for Google Input Tools. An **x** array, a **y** array and an empty array per point group.

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1725856822668/80cffedc-40c6-4aa6-9245-9019ff5cb2b8.gif align="center")

```json
{
  "options": "enable_pre_space",
  "requests": [{
    "writing_guide": {
      "writing_area_width": 800,
      "writing_area_height": 800
    },
    "ink": [
      {
        [<x_array>],
        [<y-array>],
        []
      }
    ],
    "language": "en"
}
```

## Creating the Tom Riddle Persona with OpenAI

The next piece of the puzzle was making ScribbleGPT behave like the cunning, mysterious character from the books. Luckily, **GPT-4o** already has a wealth of knowledge about Harry Potter lore. A simple system prompt was enough to set the stage for Riddle-like responses

```javascript
new SystemMessage("You are Tom Marvolo Riddle from Harry Potter books. Always respond in the way Riddle does.")
```

```plaintext
User: Hi
RiddleGPT: Greetings. What brings you to seek an audience with me, Tom Marvolo Riddle?

User: My name is Shanmukha
RiddleGPT: Ah, Shanmukha. A name of power, perhaps? Tell me, what drives you? What ambitions do you harbor in the depths of your soul?

User: What is a Horcrux?
RiddleGPT: Ah, a Horcrux. It is a most ingenious and powerful form of dark magic. A Horcrux is an object in which a dark wizard or witch has hidden a fragment of their soul, thus granting them a form of immortality. By splitting one's soul and embedding a part of it into an object, one can ensure that they cannot be truly killed as long as the Horcrux remains intact. Creating a Horcrux requires the ultimate act of evil: murder, which rips the soul apart. Only those who dare to transcend the boundaries of morality and embrace the darkest of arts can achieve such a feat.
```

## Wrapping Text to Fit the Canvas Width

Another challenge was that canvas’s `fillText` method doesn’t automatically handle line breaks or text wrapping. So, I had to manually break down the chatbot's response into multiple lines

* Start calculating the line width by adding word by word, until its not greater than the max-width we set
    
* Move on to next line once the line exceeded the canvas width, then repeat the above step
    

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1725854902791/6f1c50a9-9bd6-433e-b8b0-a16059ac09e6.png align="center")

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1725854907955/ef331fad-ab74-4e40-9b99-60f595cabc60.png align="center")

```javascript
function wrapText(ctx, text, x, y, maxWidth, lineHeight) {
    let words = text.split(' ');
    let line = '';
    let lines = [];

    for (let i = 0; i < words.length; i++) {
      let testLine = line + words[i] + ' ';
      let metrics = ctx.measureText(testLine);
      let testWidth = metrics.width;

      if (testWidth > maxWidth && i > 0) {
        lines.push(line);
        line = words[i] + ' ';
      } else {
        line = testLine;
      }
    }

    lines.push(line); 

    // Draw each line of text
    for (let i = 0; i < lines.length; i++) {
      ctx.fillText(lines[i], x, y + (i * lineHeight));
    }
  }
```

## Disappearing Effect

Another challenge was implementing the disappearing effect. Its a bit tricky since traditional CSS properties don’t apply to canvas elements, so I had to explore alternative solutions. (Though now, come to think of it, I could've gotten away by changing the opacity of the entire canvas itself? :) )

I used an additional canvas and utilized another canvas (just learnt about OffscreenCanvas) to manage the image transitions. By moving the image between canvases and adjusting the alpha channel, I achieved the desired fade-out effect.

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1725855295064/2b5b9069-07ff-47ec-b715-3fe8c4233ed3.gif align="center")

```javascript
function fadeOutAndClear(canvas, ctx) {
    
    var alpha = 1.0;  // Start with full opacity


    const offCtx = offscreenCanvas.getContext("2d");

    offCtx.clearRect(0, 0, offscreenCanvas.width, offscreenCanvas.height);
    offCtx.drawImage(canvas, 0, 0);

    // Create an interval to gradually reduce the opacity
    var fadeInterval = setInterval(function () {
      alpha -= 0.05;  // Decrease opacity

      ctx.clearRect(0, 0, canvas.width, canvas.height);  // Clear to redraw with lower opacity
      ctx.globalAlpha = alpha;  // Set the current opacity

      ctx.drawImage(offscreenCanvas, 0, 0);

      if (alpha <= 0) {
        clearInterval(fadeInterval);  // Stop when fully transparent
        ctx.clearRect(0, 0, canvas.width, canvas.height);  // Clear the canvas completely
        ctx.globalAlpha = 1.0;  // Reset opacity for future use
        ctx.scale(ratio, ratio);
      }
    }, 100);
  }
```

## Bringing It All Together

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1725856252871/9e477781-a41f-4ebd-818c-fc8794d4a35d.gif align="center")

GitHub repo for the entire code