Create Item with image is stuck at partial processing

[ajturner]

Attempting to create a new Item with an Image. The addItem is successful but the processing says 'partial' and the actual item is not accessible via the Portal REST API

Portal.createItem({
      item: {
        title: `SharingImage`,
        type: "Image",
        tags: ["dev","api"],
        typeKeywords: ["Data","Image","png"]
      },
      file: file,
      overwrite: false,
      multipart: true, 
      async: true,
      filename: file.name,
      authentication
    }).then((response) => {
      console.log("Created Item", response)
      this.checkStatus(response.id);
    })
  }

responds {success: true, id: "7f58e9931f394d0894d079b45062c0fe"}

Here is the request Form data network

Request URL: https://www.arcgis.com/sharing/rest/content/users/aturner/addItem
Request Method: POST
Status Code: 200 

Referrer Policy: no-referrer-when-downgrade
:authority: www.arcgis.com
:method: POST
:path: /sharing/rest/content/users/aturner/addItem
:scheme: https
accept: */*
accept-encoding: gzip, deflate, br
accept-language: en-US,en;q=0.9,es;q=0.8,he;q=0.7
cache-control: no-cache
content-length: 99560
content-type: multipart/form-data; boundary=----WebKitFormBoundaryMmhFiPDClwvxio9L
origin: http://localui.arcgis.com:9001
pragma: no-cache
referer: http://localui.arcgis.com:9001/iframe.html?id=hub-components--hubupload&viewMode=story
sec-fetch-dest: empty
sec-fetch-mode: cors
sec-fetch-site: cross-site
user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.113 Safari/537.36

Form data
------WebKitFormBoundaryMmhFiPDClwvxio9L
Content-Disposition: form-data; name="f"

json
------WebKitFormBoundaryMmhFiPDClwvxio9L
Content-Disposition: form-data; name="title"

Item_77250
------WebKitFormBoundaryMmhFiPDClwvxio9L
Content-Disposition: form-data; name="type"

Image
------WebKitFormBoundaryMmhFiPDClwvxio9L
Content-Disposition: form-data; name="tags"

dev,api
------WebKitFormBoundaryMmhFiPDClwvxio9L
Content-Disposition: form-data; name="typeKeywords"

Data,Image,png
------WebKitFormBoundaryMmhFiPDClwvxio9L
Content-Disposition: form-data; name="file"; filename="Sharing.png"
Content-Type: image/png


------WebKitFormBoundaryMmhFiPDClwvxio9L
Content-Disposition: form-data; name="async"

true
------WebKitFormBoundaryMmhFiPDClwvxio9L
Content-Disposition: form-data; name="multipart"

true
------WebKitFormBoundaryMmhFiPDClwvxio9L
Content-Disposition: form-data; name="filename"

Sharing.png
------WebKitFormBoundaryMmhFiPDClwvxio9L
Content-Disposition: form-data; name="overwrite"

false
------WebKitFormBoundaryMmhFiPDClwvxio9L
Content-Disposition: form-data; name="token"

{token removed}
------WebKitFormBoundaryMmhFiPDClwvxio9L--

Multipart commit?

I tried to commitItemUpload...

Portal.commitItemUpload({
        id: response.id,
        authentication
      })

but receive the REST API error response
{"error":{"code":400,"messageCode":"CONT_0031","message":"Item does not have any parts to commit","details":[]}}

[ajturner]

I think there are two issues.

• Multipart is not required for addItem. It works sync, but Arcgis-rest.js blocks this
• When multipart and aaddItemPart is used, these ignore other item parameters like name + type. These should be sent in commitItemUpload but arcgis-rest.js requires requestOptions to only be the authentication but should allow for title and type

[ajturner]

ok - I got this working by sending in params. This should be doc'd or even lifted some item parameters. What do you think @patrickarlt ?

    Portal.createItem({
      file: file,
      multipart: true, 
      async: true,
      filename: file.name,
      authentication
    }).then((response) => {
      console.log("Created Item", response)
      Portal.addItemPart({
        id: response.id,
        file,
        partNum: 1,
        authentication
      }).then((partResponse) =>  {
        console.log("Created Item", partResponse)
        Portal.commitItemUpload({
          id: response.id,
          authentication,
          params: {
            title: `Item_${Math.floor(Math.random()*100000)}`,
            type: itemType,
          }
        }).then((statusResponse) => {
          console.log("Created Item", statusResponse)
          this.checkStatus(response.id);          
        })
          

[jesseadamsnpgs]

I'm trying to get a multiPart upload function to work with large files in node.js. Most of the workflow seems to work, however, every time I send a file in multiple parts using "createReadStream" the result is a corrupt zip file. Here is the workflow I'm using and the code for the addItemPart function.

1. Add Item with the mulitPart parameter.
2. Split file into parts and upload each using the following function:

`require('cross-fetch/polyfill');
require('isomorphic-form-data');

let fs = require('fs'),
arcgisAuth = require('@esri/arcgis-rest-auth'),
arcgisPortal = require('@esri/arcgis-rest-portal');

const RESTORE_DIR = './test/';
let fileName = 'testFile.zip';

const fileSize = fs.lstatSync(RESTORE_DIR + fileName).size;
uploadPart(userId, token, addItemId, 1, fileName, 0, (fileSize / 2));
uploadPart(userId, token, addItemId, 2, fileName, (fileSize / 2), Infinity);

function uploadPart(userId, token, addItemId, partNum, fileName, rangeStart, end) {
return new Promise(function (resolve, reject) {
const authentication = new arcgisAuth.UserSession({
username: userId,
token: token
});

// upload the part
arcgisPortal.addItemPart({
  id: addItemId,
  file: fs.createReadStream(RESTORE_DIR + fileName, {
    start: rangeStart,
    end: end,
    autoClose: true
  }),
  partNum: partNum,
  authentication
}).then(res => {
  resolve(res);
});

});
}`

3. Commit the multipart upload.

Once that workflow completes if I try publishing a service from the zip file or downloading the file it reports that the resulting file is corrupt.

Any thoughts/suggestions on what format the addItemPart function is looking for with the file parameter?

Thank you.

[AviCueva]

Man... this was a fun one to figure out. For those of you still having issues. This is how I was able to publish utilizing GeoJson.

  const {UserSession} = require("@esri/arcgis-rest-auth")
  const {
  createItem,
  commitItemUpload,
  addItemPart
  } = require("@esri/arcgis-rest-portal")
  let fs = require('fs')

  exports.publishGeoJson2FeatureService = async (data, authentication) => {

  const {username, password} = authentication

  // data = {
  //     "type": "FeatureCollection",
  //     "crs": {"type": "name", "properties": {"name": "EPSG:4326"}},
  //     "features": [{
  //         "type": "Feature",
  //         "id": 1,
  //         "geometry": {"type": "Point", "coordinates": [-122.682207049, 45.526159497]},
  //         "properties": {
  //             "FID": 1,
  //             "PlaceName": "Boyds Coffee",
  //             "Place_addr": "404 NW 11th Ave Portland Oregon",
  //             "Score": 100,
  //             "lat": 45.526159497,
  //             "lon": -122.682207049
  //         }
  //     }]
  // }
  const session = new UserSession({
      username: username,
      password: password,
      portal: "https://<PORTAL>/<PORTAL_ENDPOINT>/sharing/rest"
  })

  const str = JSON.stringify(data)
  const bytes = new TextEncoder().encode(str)

  let itemNum = Math.floor(Math.random() * 100000)

  await new fs.writeFile(`Item_${itemNum}.geoJson`, bytes, (res, err) => {
      console.log(err | res)
  })
  let file = fs.createReadStream(`Item_${itemNum}.geoJson`)

  createItem({
      item: {
          title: `Item_${itemNum}`,
          type: "GeoJson",
      },
      file: file.toString(),
      multipart: true,
      async: true,
      filename: `Item_${itemNum}.geoJson`,
      overwrite: true,
      authentication: session
  }).then(async (response) => {
      console.log(response)
      console.log("Pre Timer")
     // Could probably add a checkStatus() and then do addItemPart.  I just wait 15 seconds...
      await new Promise(resolve => setTimeout(resolve, 15000))
      console.log("Post Timer")

      addItemPart({
          id: response.id,
          file: file,
          partNum: 1,
          authentication: session
      }).then(results => {
          console.log("AddItemPart: ")
          console.log(results)

          commitItemUpload({
              item: {
                  title: `Item_${itemNum}`,
                  type: "GeoJson",
              }, id: results.id,
              authentication: session
          }).then((res) => console.log("Successfully published item: " + res.id))

      })

  })
}

[addy]

@AviCueva

Man... this was a fun one to figure out. For those of you still having issues. This is how I was able to publish utilizing GeoJson.

  let file = fs.createReadStream(`Item_${itemNum}.geoJson`)

Were you actually able to upload a Stream through addItemPart? We are getting 400s on our side.

[AviCueva]

@addy - Yes. One thing to note. I believe I was utilizing version 3.5ish of the API. Not sure about your 400 errors, but if I recall correctly, it's because the item had not yet been created completely yet. This is why I added a manual 15 second delay. I'd try to access the REST end point and see if the item exists and is just waiting for data.

Version 4.0 - Esri provided examples
https://github.com/Esri/arcgis-rest-js/blob/main/demos/attachments-node/index.js

https://github.com/Esri/arcgis-rest-js/blob/main/demos/stream-response-to-file/index.js