Analyze and generate images with Cognitive Services

This repository hosts the code of a smart Javascript application integrating image analysis and generation capabilities.

Empower your app with Computer Vision

Are you interested in how AI perceives the world around us? Do you know how AI can generate a picture of a seaside landscape in a photorealistic style? Look at this demo, to learn how you can embed computer vision capabilities into your applications using Azure services.

Pre-requisites

To run the app locally

To set up your local environment and run the web app you need to configure the following pre-requisites:

• Azure subscription or an Azure for students free trial.
  • Computer vision resource.
  • Azure OpenAI resource. Access to this service (and in particular to DALL-E 2 model) should be requested in advance through this form. If your application don't match the acceptance criteria, you can use OpenAI public APIs instead.
• Node.js and npm - installed to your local machine.
• Visual Studio Code - installed to your local machine.
• GitHub Codespaces - coming soon.

To deploy the app on the Cloud

In addition to the pre-requisites above, if you wish to deploy and run your app on Azure you'll need the following:

• Azure static Web App resource.
• Azure Static Web Apps VS Code extension - used to deploy React app to Azure Static Web app.
• GitHub account - to fork and push to a repo, which activates GitHub actions.

Description

The goal of this demo is to showcase how to empower your app with Computer Vision capabilities.

App functionalities

The React app of this demo provides the following functionalities:

• Displays message if Azure key and endpoint for Cognitive Services Computer Vision or Azure OpenAI service isn't found.

• Allows you to analyze an image with Cognitive Services Computer Vision, by entering a public image URL or analyzing an image from collection.

  • When analysis is complete: it displays the analyzed image and Computer Vision JSON results.

    

• Allows you to generate an image with DALL-E 2, by entering a textual prompt.

  • When generation process is complete: it displays the generated image and the corresponding url.

    

Step-by-step guide

To run the app locally

1. Fork this repository and clone it on your machine.

2. Install all the required dependencies by running npm install in a terminal or bash window, from the project folder.

3. Get your Computer Vision credentials and add them as environment variables in a terminal (use export rather than set if you are in a bash window).

   set REACT_APP_AZURE_COMPUTER_VISION_KEY=REPLACE-WITH-YOUR-KEY
   set REACT_APP_AZURE_COMPUTER_VISION_ENDPOINT=REPLACE-WITH-YOUR-ENDPOINT

4. Get your OpenAI instance credentials and add them as environment variables in a terminal (use export rather than set if you are in a bash window).

   set REACT_APP_AZURE_OPENAI_KEY=REPLACE-WITH-YOUR-KEY
   set REACT_APP_AZURE_OPENAI_ENDPOINT=REPLACE-WITH-YOUR-ENDPOINT

If you are using an Azure OpenAI endpoint, you can retrieve your credentials from the Keys and Endpoint tab of your resource in the Azure Portal.

On the other hand, if you are using an OpenAI instance, you can retrieve your key from your user page, while your endpoint is https://api.openai.com/v1/.

To deploy the app on the Cloud

1. Add environment variables to your remote environment, to enable GitHub actions to pass the secrets to the Static Web App. In a web browser, on your GitHub repository, select Settings -> Secrets and variables -> Actions -> New repository secrets. Enter the same name and value for the keys and endpoints you used in the previous section.

2. In Visual Studio Code, open the project folder and select the Azure icon in the left-hand side menu. Select Static Web Apps -> Create New Static Web App. Select the GitHub repository you forked and the branch you want to deploy. Select React as the build preset.

3. Pull down the changes made from creating the Azure resource to get the GitHub Action file. In the Visual Studio Code editor, edit the GitHub Action file found at ./.github/workflows/ to add the secrets (under the build_and_deploy_job).

   env:
     REACT_APP_AZURE_COMPUTER_VISION_ENDPOINT: ${{secrets.REACT_APP_AZURE_COMPUTER_VISION_ENDPOINT}}
     REACT_APP_AZURE_COMPUTER_VISION_KEY:  ${{secrets.REACT_APP_AZURE_COMPUTER_VISION_KEY}}
     REACT_APP_AZURE_OPENAI_ENDPOINT: ${{secrets.REACT_APP_AZURE_OPENAI_ENDPOINT}}
     REACT_APP_AZURE_OPENAI_KEY:  ${{secrets.REACT_APP_AZURE_OPENAI_KEY}} 
   

4. Add and commit the change to the local main branch. Push the change to the remote repository, starting a new build-and-deploy action to your Azure Static web app.

5. Monitor the build-and-deploy action in the Actions tab of your GitHub repository. Once the action is complete, you can view your deployed app by selecting the Browse to URL link in the Azure Static Web Apps section of the action.

Switching from an Azure OpenAI endpoint to OpenAI endpoint and viceversa

Azure OpenAI Services API and OpenAI API handles the construction of the endpoint and of the authentication header differently. This application can run with both services, but you need to set the api type to 'openai' in your environment variables if you are using non-Azure OpenAI endpoints (default is using Azure).

set REACT_APP_AZURE_OPENAI_API_TYPE=openai

Not sure what's the difference between the two services? Learn more here.

Troubleshooting

Are you getting ERR_OSSL_EVP_UNSUPPORTED error when running your app locally? This error message indicates that something within your web development stack isn’t compatible with the OpenSSL files you’re running.

This issue occurs due to an inconsistency with your development tools and the version of OpenSSL that you’re running. To fix this, you need to upgrade your development tools (Node.js, React.js, etc.) to the latest version. You can find a complete guide here.

If you don’t want to upgrade them for any reason, you can simply enable the OpenSSL 3.0 legacy provider as a workaround, with the command:

set NODE_OPTIONS=--openssl-legacy-provider

Useful Resources

• Create Computer Vision solutions with Azure Cognitive Services
• What is Image Analysis?
• Generating an image using Azure OpenAI service
• Build and deploy a Static Web app to Azure