[Vertex AI] DecodingError when Citation.uri is not included in response

[rfarrow1010]

Description

I wanted to pass an image along with a text prompt into VertexAI.vertexAI().generativeModel().generateContent() to generate a response. The call itself actually appeared to work fine; I have a log with the response formatted the way I wanted, along with citation metadata. But it appears the citations are causing issues for the SDK. In the logs below, see the first log with the successful response and the citations that do not all follow the same schema. Two of the citations have "startIndex", "endIndex", and "uri", but one of them has no "uri" and instead has a field called "title".
Judging from the second log, it looks like the SDK expects "uri" here, doesn't find it, and crashes before I can retrieve my response. Maybe the decoding can be made more flexible to prevent this?

Reproducing the issue

        let image = UIImage(data: <some photo data>)
        let config = GenerationConfig(temperature: 0.2)
        let model = vertex.generativeModel(modelName: "gemini-1.5-flash", generationConfig: config)
        // to set up the schema: https://firebase.google.com/docs/vertex-ai/function-calling?platform=ios

        let prompt = """
        Return a JSON object that uses the below schema to transcribe a recipe from the given photo. Do not format the JSON with backticks; simply write the JSON object itself in plain text. If there is information you are not sure about, make your best guess. When possible, transcribe ingredients and recipe steps exactly.
        {
            title: String
            creationDate: String (YYYY-MM-DD)
            description: String
            ingredients: [String]
            recipeSteps: [String]
            estimatedActiveTime: String (HH:MM:SS)
            estimatedTotalTime: String (HH:MM:SS)
            minimumServings: Int
            maximumServings: Int
            author: String
        }
        """

        let response = try await model.generateContent(image, prompt)

Firebase SDK Version

10.26.0

Xcode Version

15.4

Installation Method

Swift Package Manager

Firebase Product(s)

VertexAI

Targeted Platforms

iOS

Relevant Log Output

[FirebaseVertexAI] JSON response: {
  "candidates": [
    {
      "content": {
        "role": "model",
        "parts": [
          {
            "text": "{\n    \"title\": \"DIY Grain Salad\",\n    \"creationDate\": null,\n    \"description\": \"Every Sunday Elyse throws together a hearty grain salad for quick work lunches during the week. The base of her go-to protein-packed salads is typically quinoa or farro and kale and then she dresses it up with any assortment of seasonal veggies. She always tops it off with a generous amount of homemade vinaigrette.\",\n    \"ingredients\": [\n        \"1 cup whole grains, rinsed\",\n        \"1 bunch kale, stems removed, finely chopped\",\n        \"2 cups seasonal veggies, chopped or grated\",\n        \"1/2 cup something salty (Parmesan, feta, olives)\",\n        \"1/2 cup chopped roasted nuts or seeds (almonds, walnuts, pumpkin seeds, sunflower seeds)\",\n        \"1 recipe Apple Cider Vinaigrette (page 186)\",\n        \"olives\",\n        \"cheese\"\n    ],\n    \"recipeSteps\": [\n        \"Cook your grain of choice according to the package directions, being careful not to overcook (you want your grains to have enough texture to hold up to the dressing).\",\n        \"For quinoa, we recommend using less water than typically recommended (see our foolproof quinoa cooking method on page 90).\",\n        \"Set aside in a large salad bowl to cool.\",\n        \"Add the kale, veggies, cheese or olives, and nuts or seeds to the grains and toss to combine.\",\n        \"Add the dressing and salt and pepper to taste (go generous on the dressing as the grains will sop it up).\"\n    ],\n    \"estimatedActiveTime\": \"00:30:00\",\n    \"estimatedTotalTime\": \"00:30:00\",\n    \"minimumServings\": 5,\n    \"maximumServings\": 5,\n    \"author\": \"Elyse\"\n}"
          }
        ]
      },
      "finishReason": "STOP",
      "safetyRatings": [
        {
          "category": "HARM_CATEGORY_HATE_SPEECH",
          "probability": "NEGLIGIBLE",
          "probabilityScore": 0.13028541,
          "severity": "HARM_SEVERITY_NEGLIGIBLE",
          "severityScore": 0.15393034
        },
        {
          "category": "HARM_CATEGORY_DANGEROUS_CONTENT",
          "probability": "NEGLIGIBLE",
          "probabilityScore": 0.12940271,
          "severity": "HARM_SEVERITY_NEGLIGIBLE",
          "severityScore": 0.14057204
        },
        {
          "category": "HARM_CATEGORY_HARASSMENT",
          "probability": "NEGLIGIBLE",
          "probabilityScore": 0.1334152,
          "severity": "HARM_SEVERITY_NEGLIGIBLE",
          "severityScore": 0.091544405
        },
        {
          "category": "HARM_CATEGORY_SEXUALLY_EXPLICIT",
          "probability": "NEGLIGIBLE",
          "probabilityScore": 0.1041124,
          "severity": "HARM_SEVERITY_NEGLIGIBLE",
          "severityScore": 0.13050689
        }
      ],
      "citationMetadata": {
        "citations": [
          {
            "startIndex": 80,
            "endIndex": 392,
            "uri": "https://www.pinterest.com/pin/grain-salad--57843176451536516/"
          },
          {
            "startIndex": 854,
            "endIndex": 1202,
            "title": "Your prompt"
          },
          {
            "startIndex": 1218,
            "endIndex": 1424,
            "uri": "https://runfasteatslow.com/blogs/news/fall-inspired-diy-grain-salads"
          }
        ]
      }
    }
  ],
  "usageMetadata": {
    "promptTokenCount": 411,
    "candidatesTokenCount": 405,
    "totalTokenCount": 816
  }
}
Type: Error | Timestamp: 2024-08-12 00:15:27.319474-07:00 | Library: Sous | Subsystem: com.google.firebase.vertex-ai | Category: NetworkResponse | TID: 0x35cc63




[FirebaseVertexAI] Error decoding server JSON: Swift.DecodingError.keyNotFound(CodingKeys(stringValue: "uri", intValue: nil), Swift.DecodingError.Context(codingPath: [CodingKeys(stringValue: "candidates", intValue: nil), _JSONKey(stringValue: "Index 0", intValue: 0), CodingKeys(stringValue: "citationMetadata", intValue: nil), CodingKeys(stringValue: "citations", intValue: nil), _JSONKey(stringValue: "Index 1", intValue: 1)], debugDescription: "No value associated with key CodingKeys(stringValue: \"uri\", intValue: nil) (\"uri\").", underlyingError: nil))
Type: Error | Timestamp: 2024-08-12 00:15:27.322864-07:00 | Library: Sous | Subsystem: com.google.firebase.vertex-ai | Category:  | TID: 0x35cc63

If using Swift Package Manager, the project's Package.resolved

Expand Package.resolved snippet

{
  "originHash" : "aceced254f4af24930664d1a83076c5bafc9df681eccc6d9197ef56542e5907c",
  "pins" : [
    {
      "identity" : "abseil-cpp-binary",
      "kind" : "remoteSourceControl",
      "location" : "https://github.com/google/abseil-cpp-binary.git",
      "state" : {
        "revision" : "748c7837511d0e6a507737353af268484e1745e2",
        "version" : "1.2024011601.1"
      }
    },
    {
      "identity" : "app-check",
      "kind" : "remoteSourceControl",
      "location" : "https://github.com/google/app-check.git",
      "state" : {
        "revision" : "076b241a625e25eac22f8849be256dfb960fcdfe",
        "version" : "10.19.1"
      }
    },
    {
      "identity" : "firebase-ios-sdk",
      "kind" : "remoteSourceControl",
      "location" : "https://github.com/firebase/firebase-ios-sdk",
      "state" : {
        "revision" : "9d17b500cd98d9a7009751ad62f802e152e97021",
        "version" : "10.26.0"
      }
    },
    {
      "identity" : "fuse-swift",
      "kind" : "remoteSourceControl",
      "location" : "https://github.com/krisk/fuse-swift.git",
      "state" : {
        "revision" : "26ba868691b2d8b7bf2b1322951eb591be70ccca",
        "version" : "1.4.0"
      }
    },
    {
      "identity" : "googleappmeasurement",
      "kind" : "remoteSourceControl",
      "location" : "https://github.com/google/GoogleAppMeasurement.git",
      "state" : {
        "revision" : "16244d177c4e989f87b25e9db1012b382cfedc55",
        "version" : "10.25.0"
      }
    },
    {
      "identity" : "googledatatransport",
      "kind" : "remoteSourceControl",
      "location" : "https://github.com/google/GoogleDataTransport.git",
      "state" : {
        "revision" : "a637d318ae7ae246b02d7305121275bc75ed5565",
        "version" : "9.4.0"
      }
    },
    {
      "identity" : "googleutilities",
      "kind" : "remoteSourceControl",
      "location" : "https://github.com/google/GoogleUtilities.git",
      "state" : {
        "revision" : "57a1d307f42df690fdef2637f3e5b776da02aad6",
        "version" : "7.13.3"
      }
    },
    {
      "identity" : "grpc-binary",
      "kind" : "remoteSourceControl",
      "location" : "https://github.com/google/grpc-binary.git",
      "state" : {
        "revision" : "e9fad491d0673bdda7063a0341fb6b47a30c5359",
        "version" : "1.62.2"
      }
    },
    {
      "identity" : "gtm-session-fetcher",
      "kind" : "remoteSourceControl",
      "location" : "https://github.com/google/gtm-session-fetcher.git",
      "state" : {
        "revision" : "0382ca27f22fb3494cf657d8dc356dc282cd1193",
        "version" : "3.4.1"
      }
    },
    {
      "identity" : "interop-ios-for-google-sdks",
      "kind" : "remoteSourceControl",
      "location" : "https://github.com/google/interop-ios-for-google-sdks.git",
      "state" : {
        "revision" : "2d12673670417654f08f5f90fdd62926dc3a2648",
        "version" : "100.0.0"
      }
    },
    {
      "identity" : "leveldb",
      "kind" : "remoteSourceControl",
      "location" : "https://github.com/firebase/leveldb.git",
      "state" : {
        "revision" : "a0bc79961d7be727d258d33d5a6b2f1023270ba1",
        "version" : "1.22.5"
      }
    },
    {
      "identity" : "nanopb",
      "kind" : "remoteSourceControl",
      "location" : "https://github.com/firebase/nanopb.git",
      "state" : {
        "revision" : "b7e1104502eca3a213b46303391ca4d3bc8ddec1",
        "version" : "2.30910.0"
      }
    },
    {
      "identity" : "promises",
      "kind" : "remoteSourceControl",
      "location" : "https://github.com/google/promises.git",
      "state" : {
        "revision" : "540318ecedd63d883069ae7f1ed811a2df00b6ac",
        "version" : "2.4.0"
      }
    },
    {
      "identity" : "swift-protobuf",
      "kind" : "remoteSourceControl",
      "location" : "https://github.com/apple/swift-protobuf.git",
      "state" : {
        "revision" : "9f0c76544701845ad98716f3f6a774a892152bcb",
        "version" : "1.26.0"
      }
    },
    {
      "identity" : "swiftsoup",
      "kind" : "remoteSourceControl",
      "location" : "https://github.com/scinfu/SwiftSoup.git",
      "state" : {
        "revision" : "e2d11208519549c2e5798d70190472045633f22f",
        "version" : "2.7.3"
      }
    },
    {
      "identity" : "tagcloud",
      "kind" : "remoteSourceControl",
      "location" : "https://github.com/yarspirin/TagCloud.git",
      "state" : {
        "revision" : "81c1019541c2748cfecff4eabc05ca0bdcaa3713",
        "version" : "1.0.4"
      }
    }
  ],
  "version" : 3
}

If using CocoaPods, the project's Podfile.lock

Expand Podfile.lock snippet

Replace this line with the contents of your Podfile.lock!

[google-oss-bot]

I couldn't figure out how to label this issue, so I've labeled it for a human to triage. Hang tight.

[andrewheard]

Thank you very much for the detailed prompt and full response, @rfarrow1010! We hadn't yet seen a case where the citation had a title instead of a uri. This real-world example provides good justification to make uri optional and to add a title: String? field too. Working on a fix now.

[andrewheard]

@rfarrow1010 This fix is now merged in the main branch and will go out in the next release. Sorry about the bug and thanks again for the detailed response output!

[rfarrow1010]

Thanks @andrewheard! Any idea when the next release will be available?

[andrewheard]

Thanks @andrewheard! Any idea when the next release will be available?

@rfarrow1010 The next release should be available in approximately 3 weeks.