diff --git a/docs/content/5.blog/2.drawing-app-with-nuxt-and-cloudflare-r2.md b/docs/content/5.blog/2.drawing-app-with-nuxt-and-cloudflare-r2.md index 04adb1b6..32aadd89 100644 --- a/docs/content/5.blog/2.drawing-app-with-nuxt-and-cloudflare-r2.md +++ b/docs/content/5.blog/2.drawing-app-with-nuxt-and-cloudflare-r2.md @@ -467,8 +467,6 @@ Or go to https://admin.hub.nuxt.com and select your project. Congratulations! You've now built a fully functional drawing application using Nuxt and Cloudflare R2 for storage. Users can create drawings, save them to the cloud, and access them from anywhere. -Next, we are going to leverage Cloudflare AI to generate the alternative text for the user drawings (accessibility & SEO) as well as generating an alternative drawing using AI. - Feel free to expand on this foundation and add your own unique features to make Atidraw yours! ::callout{to="https://github.com/atinux/atidraw" icon="i-simple-icons-github" color="gray" target="_blank"} @@ -477,3 +475,5 @@ Feel free to expand on this foundation and add your own unique features to make ::note{to="https://draw.nuxt.dev" icon="i-ph-rocket-launch-duotone" target="_blank"} The demo is available at **draw.nuxt.dev**. :: + +Checkout the next article on how to leverage Cloudflare AI to generate the alternative text for the user drawings (accessibility & SEO) as well as generating an alternative drawing using AI: [Cloudflare AI for User Experience](/blog/cloudflare-ai-for-user-experience). diff --git a/docs/content/5.blog/3.cloudflare-ai-for-user-experience.md b/docs/content/5.blog/3.cloudflare-ai-for-user-experience.md index e74499c8..5d57ff54 100644 --- a/docs/content/5.blog/3.cloudflare-ai-for-user-experience.md +++ b/docs/content/5.blog/3.cloudflare-ai-for-user-experience.md @@ -8,39 +8,308 @@ authors: src: https://avatars.githubusercontent.com/u/904724?v=4 to: https://x.com/atinux username: atinux -date: 2024-08-12 +date: 2024-08-26 category: Tutorial draft: true --- ## Introduction -The application is [github.com/atinux/atidraw](https://github.com/atinux/atidraw), an open source collaborative drawing app made with Nuxt. +Let's improve [Atidraw](https://github.com/atinux/atidraw), an open source collaborative drawing app made with Nuxt. -It has basic features such as: +The application has basic features such as: - Auth with Google, GitHub or Anonymously based on [`nuxt-auth-utils`](https://github.com/Atinux/nuxt-auth-utils) - Draw with [`signature_pad`](https://github.com/szimek/signature_pad) - Upload and store drawings with Cloudflare R2 using [`hubBlob()`](/docs/features/blob) -- Deploy to the Edge with [nuxthub deploy](https://github.com/nuxt-hub/cli) using a [GitHub action](./.github/workflows/deploy.yml) You can play with it on [draw.nuxt.dev](https://draw.nuxt.dev). -::video{poster="https://res.cloudinary.com/nuxt/video/upload/v1723210615/nuxthub/344159247-85f79def-f633-40b7-97c2-3a8579e65af1_xyrfin.jpg" controls class="w-full h-auto" class="border rounded dark:border-gray-800 md:w-2/3"} - :source{src="https://res.cloudinary.com/nuxt/video/upload/v1723210615/nuxthub/344159247-85f79def-f633-40b7-97c2-3a8579e65af1_xyrfin.webm" type="video/webm"} - :source{src="https://res.cloudinary.com/nuxt/video/upload/v1723210615/nuxthub/344159247-85f79def-f633-40b7-97c2-3a8579e65af1_xyrfin.mp4" type="video/mp4"} - :source{src="https://res.cloudinary.com/nuxt/video/upload/v1723210615/nuxthub/344159247-85f79def-f633-40b7-97c2-3a8579e65af1_xyrfin.ogg" type="video/ogg"} +::tip +Checkout how we created Atidraw in the ["Code, Draw, Deploy: A drawing app with Nuxt & Cloudflare R2"](/blog/drawing-app-with-nuxt-and-cloudflare-r2) blog post. :: -Or deploy it directory on your NuxtHub account: +## Cloudflare AI Pricing -::a{href="https://hub.nuxt.com/new?repo=atinux/atidraw" target="_blank"} - :img{src="https://hub.nuxt.com/button.svg" alt="Deploy to NuxtHub" width="174" height="32"} +This is important to know how Cloudflare AI models are billed. + +Cloudflare free allocation allows anyone to use a total of 10,000 Neurons per day at no charge. + +Neurons are Cloudflare way of measuring AI outputs across different models. To give you a sense of what you can accomplish with 10,000 Neurons, you can generate: +- 100-200 LLM responses +- 500 translations +- 500 seconds of speech-to-text audio +- 10,000 text classifications +- 1,500 - 15,000 embeddings + +Once you reach the free allocation, you can use the AI models with a pay-as-you-go model of $0.011 / 1,000 Neurons. + +::tip +**AI Models in beta are free to use**, at this time, all the models we use in this tutorial are in beta, **so you we use them for free**. +:: + +::note{to="https://developers.cloudflare.com/workers-ai/platform/pricing" target="_blank"} +Read more about **Cloudflare AI pricing**. +:: + +## Add AI to Nuxt + +To add AI to our Nuxt application, we need to enable the AI feature on our `nuxt.config.ts` file. + +```ts [nuxt.config.ts] +export default defineNuxtConfig({ + hub: { + ai: true, // <--- Enable AI + blob: true, + } +}) +``` + +Then, we want to make sure our project is linked to our NuxtHub account. + +```bash [Terminal] +npx nuxthub link +``` + +::note +This will allow the module to call AI models from your linked Cloudflare account in development mode. +:: + +::tip{icon="i-ph-rocket-launch"} +That's it! We can now use the AI models from Cloudflare using [`hubAI()`](/docs/features/ai). :: -## Adding AI +I've been thinking on two features: +- Generate the alternative text for the user drawings, improving the accessibility of the application & SEO. +- Generate an image based from the drawing and the alternative text, as a way to make the application more interactive. + +Before starting, I took a look at [Cloudflare's multi modal playground](https://multi-modal.ai.cloudflare.com/) and played with some models. + +:nuxt-img{alt="Atidraw AI models" dataZoomSrc="/images/blog/atidraw-ai-models.png" height="515" src="/images/blog/atidraw-ai-models.png" width="915"} + +This guided me to the following models: +- [LLaVA](https://developers.cloudflare.com/workers-ai/models/llava-1.5-7b-hf/) for the alternative text +- [Stable Diffusion IMG2IMG](https://developers.cloudflare.com/workers-ai/models/stable-diffusion-v1-5-img2img/) for the alternative drawing + +::note{to="https://developers.cloudflare.com/workers-ai/models/" target="_blank"} +See all **Cloudflare AI models** available. +:: + +## Image Alternative Text + +To generate the alternative text for the user drawings, we use the [LLaVA](https://developers.cloudflare.com/workers-ai/models/llava-1.5-7b-hf/) model. + +Let's see our current `/api/upload` route: + +```ts [server/api/upload.post.ts] +export default eventHandler(async (event) => { + // Make sure the user is authenticated to upload + const { user } = await requireUserSession(event) + + // Read the form data + const form = await readFormData(event) + const drawing = form.get('drawing') as File + + // Make sure the drawing is a jpeg image and is not larger than 1MB + ensureBlob(drawing, { maxSize: '1MB', types: ['image/jpeg'] }) + + // Create a new pathname to be smaller than the last one uploaded (for a desc ordering) + const name = `${new Date('2050-01-01').getTime() - Date.now()}` + // Store the image in the R2 bucket with the `drawings/` prefix + return hubBlob().put(`${name}.jpg`, drawing, { + prefix: 'drawings/', + addRandomSuffix: true, + customMetadata: { + userProvider: user.provider, + userId: user.id, + userName: user.name, + userAvatar: user.avatar, + userUrl: user.url, + }, + }) +}) +``` + +The `LLaVA` model is expecting a `Uint8Array` of the image and a `prompt` to generate the alternative text: + +```ts +const { description } = await hubAI().run('@cf/llava-hf/llava-1.5-7b-hf', { + prompt: 'Describe this drawing in one sentence.', + // Convert the drawing from File to Uint8Array + image: [...new Uint8Array(await drawing.arrayBuffer())], +})) +``` + +We can add the description to the custom metadata of the drawing: + +```diff [server/api/upload.post.ts] +export default eventHandler(async (event) => { + // ... ++ const { description } = await hubAI().run('@cf/llava-hf/llava-1.5-7b-hf', { ++ prompt: 'Describe this drawing in one sentence.', ++ image: [...new Uint8Array(await drawing.arrayBuffer())], ++ })) -While working on [`hubAI()`](/docs/features/ai), I wanted to add a feature to generate the alt text for the user drawings. + // ... + return hubBlob().put(`${name}.jpg`, drawing, { + // ... + customMetadata: { + // ... + userUrl: user.url, ++ description + }, + }) +}) +``` -- Image Alt text generation with [`hubAI()`](/docs/features/ai) and [LLaVA](https://developers.cloudflare.com/workers-ai/models/llava-1.5-7b-hf/) -- Gerate an image based from the drawing with [`hubAI()`](/docs/features/ai) and [Stable Diffusion](https://developers.cloudflare.com/workers-ai/models/stable-diffusion-v1-5-img2img/) +Lastly, we need to update our listing page to display the description in the `` tag but also in the `title` attribute so the user can see the description when hovering the image. + +```vue [app/pages/index.vue] + + + +``` + +Let's see the result: + +::video{poster="https://res.cloudinary.com/nuxt/video/upload/v1724609254/nuxthub/nuxt-ai-img-alt-text_zv0sx7.jpg" controls class="lg:w-2/3 h-auto border dark:border-gray-800 rounded"} + :source{src="https://res.cloudinary.com/nuxt/video/upload/v1724609254/nuxthub/nuxt-ai-img-alt-text_zv0sx7.webm" type="video/webm"} + :source{src="https://res.cloudinary.com/nuxt/video/upload/v1724609254/nuxthub/nuxt-ai-img-alt-text_zv0sx7.mp4" type="video/mp4"} + :source{src="https://res.cloudinary.com/nuxt/video/upload/v1724609254/nuxthub/nuxt-ai-img-alt-text_zv0sx7.ogg" type="video/ogg"} +:: + +## Alternative Drawing + +To generate a new drawing based from the drawing and the alternative text, we need to use the [Stable Diffusion IMG2IMG](https://developers.cloudflare.com/workers-ai/models/stable-diffusion-v1-5-img2img/) model. + +```ts +await hubAI().run('@cf/runwayml/stable-diffusion-v1-5-img2img', { + image: imageAsUint8Array, + prompt: imageAltText, + guidance: 8, + strength: 0.5, +}) +``` + +It takes the following inputs: +- `image` (as `Uint8Array`) to use as a reference for the image generation +- `prompt` to guides the model in generating the image +- `guidance` to control how closely the generated image adheres to the given text prompt +- `strength` to controls how much the model changes the input image, with higher values creating bigger changes. + +Let's update our `/api/upload` route to generate the alternative drawing and store it in the R2 bucket. + +```ts [server/api/upload.post.ts] +export default eventHandler(async (event) => { + // ... + + const aiImage = await hubAI().run('@cf/runwayml/stable-diffusion-v1-5-img2img', { + prompt: description || 'A drawing', + guidance: 8, + strength: 0.5, + image: [...new Uint8Array(await drawing.arrayBuffer())], + }) + .then((blob: Blob | Uint8Array) => { + // Convert Uint8Array to Blob + if (blob instanceof Uint8Array) { + blob = new Blob([blob]) + } + // Store the image in the R2 bucket with the `ai/` prefix + return hubBlob().put(`${name}.png`, blob, { + prefix: 'ai/', + addRandomSuffix: true, + contentType: 'image/png', + }) + }) + + // ... + return hubBlob().put(`${name}.jpg`, drawing, { + // ... + customMetadata: { + // ... + aiImage: aiImage.pathname, + }, + }) +}) +``` + +As you can see, we store the pathname of the AI generated image in the custom metadata of the drawing. + +Now, we can display the AI generated image in the listing page when hovering the user's drawing: + +```vue [app/pages/index.vue] + + + +``` + +I am quite happy with the result: + +::video{poster="https://res.cloudinary.com/nuxt/video/upload/v1724609261/nuxthub/nuxt-ai-generate-img_tdnyfq.jpg" controls class="lg:w-2/3 h-auto border dark:border-gray-800 rounded"} + :source{src="https://res.cloudinary.com/nuxt/video/upload/v1724609261/nuxthub/nuxt-ai-generate-img_tdnyfq.webm" type="video/webm"} + :source{src="https://res.cloudinary.com/nuxt/video/upload/v1724609261/nuxthub/nuxt-ai-generate-img_tdnyfq.mp4" type="video/mp4"} + :source{src="https://res.cloudinary.com/nuxt/video/upload/v1724609261/nuxthub/nuxt-ai-generate-img_tdnyfq.ogg" type="video/ogg"} +:: + +::note +Sometime the AI generated image is black, this is because the model is not able to generate an image from the description, most of the time because it is a sensitive content or misunderstood the description. +:: + +## Conclusion + +This is the end of this tutorial on how to use Cloudflare AI models in a Nuxt application. I hope you enjoyed it and that it gave you some ideas on how to use AI in your Nuxt application. + +Feel free to expand on this foundation and add your own unique features to make Atidraw yours! + +::callout{to="https://github.com/atinux/atidraw" icon="i-simple-icons-github" color="gray" target="_blank"} + The source code of the app is available at **github.com/atinux/atidraw**. +:: +::note{to="https://draw.nuxt.dev" icon="i-ph-rocket-launch-duotone" target="_blank"} + The demo is available at **draw.nuxt.dev**. +:: + +If you prefer, you can also deploy this project on your Cloudflare account by clicking on the button below: + +::a{href="https://hub.nuxt.com/new?repo=atinux/atidraw" target="_blank"} + :img{src="https://hub.nuxt.com/button.svg" alt="Deploy to NuxtHub" width="174" height="32"} +:: +Happy coding & drawing!