Skip to content

Commit

Permalink
docs: article about AI
Browse files Browse the repository at this point in the history
  • Loading branch information
@atinux
atinux committed Aug 25, 2024
1 parent 0a363cc commit e3f7865
Show file tree
Hide file tree
Showing 2 changed files with 286 additions and 17 deletions.
4 changes: 2 additions & 2 deletions docs/content/5.blog/2.drawing-app-with-nuxt-and-cloudflare-r2.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -467,8 +467,6 @@ Or go to https://admin.hub.nuxt.com and select your project.

Congratulations! You've now built a fully functional drawing application using Nuxt and Cloudflare R2 for storage. Users can create drawings, save them to the cloud, and access them from anywhere.

Next, we are going to leverage Cloudflare AI to generate the alternative text for the user drawings (accessibility & SEO) as well as generating an alternative drawing using AI.

Feel free to expand on this foundation and add your own unique features to make Atidraw yours!

::callout{to="https://github.com/atinux/atidraw" icon="i-simple-icons-github" color="gray" target="_blank"}
Expand All @@ -477,3 +475,5 @@ Feel free to expand on this foundation and add your own unique features to make
::note{to="https://draw.nuxt.dev" icon="i-ph-rocket-launch-duotone" target="_blank"}
The demo is available at **draw.nuxt.dev**.
::

Checkout the next article on how to leverage Cloudflare AI to generate the alternative text for the user drawings (accessibility & SEO) as well as generating an alternative drawing using AI: [Cloudflare AI for User Experience](/blog/cloudflare-ai-for-user-experience).
299 changes: 284 additions & 15 deletions docs/content/5.blog/3.cloudflare-ai-for-user-experience.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -8,39 +8,308 @@ authors:
src: https://avatars.githubusercontent.com/u/904724?v=4
to: https://x.com/atinux
username: atinux
date: 2024-08-12
date: 2024-08-26
category: Tutorial
draft: true
---

## Introduction

The application is [github.com/atinux/atidraw](https://github.com/atinux/atidraw), an open source collaborative drawing app made with Nuxt.
Let's improve [Atidraw](https://github.com/atinux/atidraw), an open source collaborative drawing app made with Nuxt.

It has basic features such as:
The application has basic features such as:
- Auth with Google, GitHub or Anonymously based on [`nuxt-auth-utils`](https://github.com/Atinux/nuxt-auth-utils)
- Draw with [`signature_pad`](https://github.com/szimek/signature_pad)
- Upload and store drawings with Cloudflare R2 using [`hubBlob()`](/docs/features/blob)
- Deploy to the Edge with [nuxthub deploy](https://github.com/nuxt-hub/cli) using a [GitHub action](./.github/workflows/deploy.yml)

You can play with it on [draw.nuxt.dev](https://draw.nuxt.dev).

::video{poster="https://res.cloudinary.com/nuxt/video/upload/v1723210615/nuxthub/344159247-85f79def-f633-40b7-97c2-3a8579e65af1_xyrfin.jpg" controls class="w-full h-auto" class="border rounded dark:border-gray-800 md:w-2/3"}
:source{src="https://res.cloudinary.com/nuxt/video/upload/v1723210615/nuxthub/344159247-85f79def-f633-40b7-97c2-3a8579e65af1_xyrfin.webm" type="video/webm"}
:source{src="https://res.cloudinary.com/nuxt/video/upload/v1723210615/nuxthub/344159247-85f79def-f633-40b7-97c2-3a8579e65af1_xyrfin.mp4" type="video/mp4"}
:source{src="https://res.cloudinary.com/nuxt/video/upload/v1723210615/nuxthub/344159247-85f79def-f633-40b7-97c2-3a8579e65af1_xyrfin.ogg" type="video/ogg"}
::tip
Checkout how we created Atidraw in the ["Code, Draw, Deploy: A drawing app with Nuxt & Cloudflare R2"](/blog/drawing-app-with-nuxt-and-cloudflare-r2) blog post.
::

Or deploy it directory on your NuxtHub account:
## Cloudflare AI Pricing

::a{href="https://hub.nuxt.com/new?repo=atinux/atidraw" target="_blank"}
:img{src="https://hub.nuxt.com/button.svg" alt="Deploy to NuxtHub" width="174" height="32"}
This is important to know how Cloudflare AI models are billed.

Cloudflare free allocation allows anyone to use a total of 10,000 Neurons per day at no charge.

Neurons are Cloudflare way of measuring AI outputs across different models. To give you a sense of what you can accomplish with 10,000 Neurons, you can generate:
- 100-200 LLM responses
- 500 translations
- 500 seconds of speech-to-text audio
- 10,000 text classifications
- 1,500 - 15,000 embeddings

Once you reach the free allocation, you can use the AI models with a pay-as-you-go model of $0.011 / 1,000 Neurons.

::tip
**AI Models in beta are free to use**, at this time, all the models we use in this tutorial are in beta, **so you we use them for free**.
::

::note{to="https://developers.cloudflare.com/workers-ai/platform/pricing" target="_blank"}
Read more about **Cloudflare AI pricing**.
::

## Add AI to Nuxt

To add AI to our Nuxt application, we need to enable the AI feature on our `nuxt.config.ts` file.

```ts [nuxt.config.ts]
export default defineNuxtConfig({
hub: {
ai: true, // <--- Enable AI
blob: true,
}
})
```

Then, we want to make sure our project is linked to our NuxtHub account.

```bash [Terminal]
npx nuxthub link
```

::note
This will allow the module to call AI models from your linked Cloudflare account in development mode.
::

::tip{icon="i-ph-rocket-launch"}
That's it! We can now use the AI models from Cloudflare using [`hubAI()`](/docs/features/ai).
::

## Adding AI
I've been thinking on two features:
- Generate the alternative text for the user drawings, improving the accessibility of the application & SEO.
- Generate an image based from the drawing and the alternative text, as a way to make the application more interactive.

Before starting, I took a look at [Cloudflare's multi modal playground](https://multi-modal.ai.cloudflare.com/) and played with some models.

:nuxt-img{alt="Atidraw AI models" dataZoomSrc="/images/blog/atidraw-ai-models.png" height="515" src="/images/blog/atidraw-ai-models.png" width="915"}

This guided me to the following models:
- [LLaVA](https://developers.cloudflare.com/workers-ai/models/llava-1.5-7b-hf/) for the alternative text
- [Stable Diffusion IMG2IMG](https://developers.cloudflare.com/workers-ai/models/stable-diffusion-v1-5-img2img/) for the alternative drawing

::note{to="https://developers.cloudflare.com/workers-ai/models/" target="_blank"}
See all **Cloudflare AI models** available.
::

## Image Alternative Text

To generate the alternative text for the user drawings, we use the [LLaVA](https://developers.cloudflare.com/workers-ai/models/llava-1.5-7b-hf/) model.

Let's see our current `/api/upload` route:

```ts [server/api/upload.post.ts]
export default eventHandler(async (event) => {
// Make sure the user is authenticated to upload
const { user } = await requireUserSession(event)

// Read the form data
const form = await readFormData(event)
const drawing = form.get('drawing') as File

// Make sure the drawing is a jpeg image and is not larger than 1MB
ensureBlob(drawing, { maxSize: '1MB', types: ['image/jpeg'] })

// Create a new pathname to be smaller than the last one uploaded (for a desc ordering)
const name = `${new Date('2050-01-01').getTime() - Date.now()}`
// Store the image in the R2 bucket with the `drawings/` prefix
return hubBlob().put(`${name}.jpg`, drawing, {
prefix: 'drawings/',
addRandomSuffix: true,
customMetadata: {
userProvider: user.provider,
userId: user.id,
userName: user.name,
userAvatar: user.avatar,
userUrl: user.url,
},
})
})
```

The `LLaVA` model is expecting a `Uint8Array` of the image and a `prompt` to generate the alternative text:

```ts
const { description } = await hubAI().run('@cf/llava-hf/llava-1.5-7b-hf', {
prompt: 'Describe this drawing in one sentence.',
// Convert the drawing from File to Uint8Array
image: [...new Uint8Array(await drawing.arrayBuffer())],
}))
```

We can add the description to the custom metadata of the drawing:

```diff [server/api/upload.post.ts]
export default eventHandler(async (event) => {
// ...
+ const { description } = await hubAI().run('@cf/llava-hf/llava-1.5-7b-hf', {
+ prompt: 'Describe this drawing in one sentence.',
+ image: [...new Uint8Array(await drawing.arrayBuffer())],
+ }))

While working on [`hubAI()`](/docs/features/ai), I wanted to add a feature to generate the alt text for the user drawings.
// ...
return hubBlob().put(`${name}.jpg`, drawing, {
// ...
customMetadata: {
// ...
userUrl: user.url,
+ description
},
})
})
```

- Image Alt text generation with [`hubAI()`](/docs/features/ai) and [LLaVA](https://developers.cloudflare.com/workers-ai/models/llava-1.5-7b-hf/)
- Gerate an image based from the drawing with [`hubAI()`](/docs/features/ai) and [Stable Diffusion](https://developers.cloudflare.com/workers-ai/models/stable-diffusion-v1-5-img2img/)
Lastly, we need to update our listing page to display the description in the `<img>` tag but also in the `title` attribute so the user can see the description when hovering the image.

```vue [app/pages/index.vue]
<script setup lang="ts">
const { data } = await useFetch('/api/drawings')
</script>
<template>
<div class="grid grid-cols-1 sm:grid-cols-2 lg:grid-cols-3 xl:grid-cols-4 gap-8">
<div v-for="drawing in data?.blobs" :key="drawing.pathname" class="flex flex-col gap-2">
<img
:src="`/drawings/${drawing.pathname}`"
:alt="drawing.customMetadata.description"
:title="drawing.customMetadata.description"
/>
<!-- ... -->
</div>
</div>
</template>
```

Let's see the result:

::video{poster="https://res.cloudinary.com/nuxt/video/upload/v1724609254/nuxthub/nuxt-ai-img-alt-text_zv0sx7.jpg" controls class="lg:w-2/3 h-auto border dark:border-gray-800 rounded"}
:source{src="https://res.cloudinary.com/nuxt/video/upload/v1724609254/nuxthub/nuxt-ai-img-alt-text_zv0sx7.webm" type="video/webm"}
:source{src="https://res.cloudinary.com/nuxt/video/upload/v1724609254/nuxthub/nuxt-ai-img-alt-text_zv0sx7.mp4" type="video/mp4"}
:source{src="https://res.cloudinary.com/nuxt/video/upload/v1724609254/nuxthub/nuxt-ai-img-alt-text_zv0sx7.ogg" type="video/ogg"}
::

## Alternative Drawing

To generate a new drawing based from the drawing and the alternative text, we need to use the [Stable Diffusion IMG2IMG](https://developers.cloudflare.com/workers-ai/models/stable-diffusion-v1-5-img2img/) model.

```ts
await hubAI().run('@cf/runwayml/stable-diffusion-v1-5-img2img', {
image: imageAsUint8Array,
prompt: imageAltText,
guidance: 8,
strength: 0.5,
})
```

It takes the following inputs:
- `image` (as `Uint8Array`) to use as a reference for the image generation
- `prompt` to guides the model in generating the image
- `guidance` to control how closely the generated image adheres to the given text prompt
- `strength` to controls how much the model changes the input image, with higher values creating bigger changes.

Let's update our `/api/upload` route to generate the alternative drawing and store it in the R2 bucket.

```ts [server/api/upload.post.ts]
export default eventHandler(async (event) => {
// ...

const aiImage = await hubAI().run('@cf/runwayml/stable-diffusion-v1-5-img2img', {
prompt: description || 'A drawing',
guidance: 8,
strength: 0.5,
image: [...new Uint8Array(await drawing.arrayBuffer())],
})
.then((blob: Blob | Uint8Array) => {
// Convert Uint8Array to Blob
if (blob instanceof Uint8Array) {
blob = new Blob([blob])
}
// Store the image in the R2 bucket with the `ai/` prefix
return hubBlob().put(`${name}.png`, blob, {
prefix: 'ai/',
addRandomSuffix: true,
contentType: 'image/png',
})
})

// ...
return hubBlob().put(`${name}.jpg`, drawing, {
// ...
customMetadata: {
// ...
aiImage: aiImage.pathname,
},
})
})
```

As you can see, we store the pathname of the AI generated image in the custom metadata of the drawing.

Now, we can display the AI generated image in the listing page when hovering the user's drawing:

```vue [app/pages/index.vue]
<script setup lang="ts">
const { data } = await useFetch('/api/drawings')
</script>
<template>
<div class="grid grid-cols-1 sm:grid-cols-2 lg:grid-cols-3 xl:grid-cols-4 gap-8">
<div v-for="drawing in data?.blobs" :key="drawing.pathname" class="flex flex-col gap-2">
<div
class="group relative max-w-[400px]"
:title="drawing.customMetadata.description"
>
<!-- User drawing -->
<img
:src="`/drawings/${drawing.pathname}`"
:alt="drawing.customMetadata.description"
class="w-full rounded aspect-1"
loading="lazy"
>
<!-- AI generated image, displayed on hover -->
<img
:src="`/drawings/${drawing.customMetadata?.aiImage}`"
:alt="`AI generated image of ${drawing.customMetadata?.description}`"
class="w-full rounded aspect-1 absolute inset-0 opacity-0 group-hover:opacity-100"
loading="lazy"
>
</div>
</div>
</div>
</template>
```

I am quite happy with the result:

::video{poster="https://res.cloudinary.com/nuxt/video/upload/v1724609261/nuxthub/nuxt-ai-generate-img_tdnyfq.jpg" controls class="lg:w-2/3 h-auto border dark:border-gray-800 rounded"}
:source{src="https://res.cloudinary.com/nuxt/video/upload/v1724609261/nuxthub/nuxt-ai-generate-img_tdnyfq.webm" type="video/webm"}
:source{src="https://res.cloudinary.com/nuxt/video/upload/v1724609261/nuxthub/nuxt-ai-generate-img_tdnyfq.mp4" type="video/mp4"}
:source{src="https://res.cloudinary.com/nuxt/video/upload/v1724609261/nuxthub/nuxt-ai-generate-img_tdnyfq.ogg" type="video/ogg"}
::

::note
Sometime the AI generated image is black, this is because the model is not able to generate an image from the description, most of the time because it is a sensitive content or misunderstood the description.
::

## Conclusion

This is the end of this tutorial on how to use Cloudflare AI models in a Nuxt application. I hope you enjoyed it and that it gave you some ideas on how to use AI in your Nuxt application.

Feel free to expand on this foundation and add your own unique features to make Atidraw yours!

::callout{to="https://github.com/atinux/atidraw" icon="i-simple-icons-github" color="gray" target="_blank"}
The source code of the app is available at **github.com/atinux/atidraw**.
::
::note{to="https://draw.nuxt.dev" icon="i-ph-rocket-launch-duotone" target="_blank"}
The demo is available at **draw.nuxt.dev**.
::

If you prefer, you can also deploy this project on your Cloudflare account by clicking on the button below:

::a{href="https://hub.nuxt.com/new?repo=atinux/atidraw" target="_blank"}
:img{src="https://hub.nuxt.com/button.svg" alt="Deploy to NuxtHub" width="174" height="32"}
::

Happy coding & drawing!

0 comments on commit e3f7865

Please sign in to comment.