[#171182749] Created Azure DevOps pipeline

[raicastino]

This is an Azure DevOps pipeline to build, check source codes, run tests, and more.
This pipeline has been implemented to be run on hosted agent pools based both on 'windows-2019' and 'ubuntu-latest' virtual machine images and using the scripts defined in the package.json file.

[gunzip]

Suggested change

	vmImage: 'ubuntu-latest'
	vmImage: 'windows-2019'

since we are deploying on azure functions windows runtime, probably it's better to use a windows vm

[raicastino]

@gunzip I tried to run the pipeline with vmImage: 'windows-2019' but several syntax errors occurred.
For instance, in the Build step of the compile job (stage: Build_check), this is the result of the execution of the yarn build command:

Starting: Build
==============================================================================
Task         : Command line
Description  : Run a command line script using Bash on Linux and macOS and cmd.exe on Windows
Version      : 2.163.0
Author       : Microsoft Corporation
Help         : https://docs.microsoft.com/azure/devops/pipelines/tasks/utility/command-line
==============================================================================
Generating script.
Script contents:
yarn build
========================== Starting Command Output ===========================
"C:\windows\system32\cmd.exe" /D /E:ON /V:OFF /S /C "CALL "d:\a\_temp\79a3529b-5427-4b37-80f3-c82ead3bf936.cmd""
yarn run v1.21.1
$ yarn generate:definitions && tsc
$ rimraf ./generated/definitions && mkdir -p ./generated/definitions && gen-api-models --api-spec ./openapi/index.yaml --out-dir ./generated/definitions
The syntax of the command is incorrect.
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
##[error]Cmd.exe exited with code '1'.
Finishing: Build

Do you have any suggestions?

[gunzip]

can you try using "bash" instead of "script" ?
ie.

steps:
- bash: |
    yarn ...

[raicastino]

I already tried using bash but without success! It's not enough...

After many attempts, I have found the root cause of the problems... In the packege.json file there is the definition of some scripts used in the pipeline:

{
  "name": "io-functions-admin",
  "version": "0.1.0",
  "license": "MIT",
  "scripts": {
    "build": "yarn generate:definitions && tsc",
    "build:production": "npm run prestart && npm prune --production",
    "watch": "tsc --w",
    "prestart": "npm run build && func extensions install",
    "start:host": "func start",
    "start": "npm-run-all --parallel start:host watch",
    "test": "jest",
    "test:coverage": "jest --coverage",
    "lint": "tslint -p .",
    "deploy": "npm run build && func azure functionapp publish agid-functions-admin-test",
    "generate:definitions": "rimraf ./generated/definitions && mkdir -p ./generated/definitions && gen-api-models --api-spec ./openapi/index.yaml --out-dir ./generated/definitions"
  },

          - bash: |
              yarn build
            displayName: 'Build'

The yarn build command fails when running in a Windows bash for the following reasons:

• mkdir -p is not supported
• && can be used only one time (i.e. cmd1 && cmd2 is ok but cmd1 && cmd2 && cmd3 fails)

So, it seems that there are some limitations when executing a bash script on Azure pipelines... I could easily solve the problem avoiding re-using the scripts defined in the packege.json, for instance changing the above job in the pipeline with the following:

        - bash: |
            mkdir ./generated && mkdir ./generated/definitions 
            yarn gen-api-models --api-spec ./openapi/index.yaml --out-dir ./generated/definitions 
            yarn tsc
          displayName: 'Build'

In this way, the pipeline runs without problems both in 'ubuntu-latest' and 'windows-2019' and we only need to take into account that developers could use little different scripts to build the code in their local environments.

I also notice that the pipeline execution time is different:

• vmImage: 'ubuntu-latest' takes about 50s
• vmImage: 'windows-2019' takes about 1m 20s

If all the above limitations are reasonable I can proceed with the specified changes to run the pipeline in Windows vm.
@gunzip what do you think?

[gunzip]

Thank you for the insight @raicastino ! IMHO it's a duty of the package owners to maintain compatibility. As we run everything inside a windows VM we must make package.json tasks cross platform. There are some techniques:

• using shx to run shell commands
• cross-env to use enviroment variables
• use npm-run-all with subtasks instead of &&

If you want, you may try to do this inside another PR

[raicastino]

@gunzip Following your suggestions:

• I changed vmImage from 'ubuntu-latest' to 'windows-2019'
• Changed the package.json file to make a script cross-platform (using shx)
• For the build job I used in the pipeline the script defined in the package.json (yarn build)

[gunzip]

great ! can we test a deploy into the staging environment ? (how ? do we have to create an azure pipeline service connection ? in which account ?)

[raicastino]

Actually, this pipeline will be a replace of the current pipeline based on CircleCI. I need permissions on GitHub to enable this pipeline. By the way, only the CI part is currently implemented...

[digitalcitizenship]

	Warnings
⚠️	Please include a Pivotal story at the beginning of the PR title (see below).
⚠️	Please include a description of your PR changes.

Example of PR titles that include pivotal stories:

• single story: [#123456] my PR title
• multiple stories: [#123456,#123457,#123458] my PR title

New dependencies added: shx.

shx

Author: Unknown

Description: Portable Shell Commands for Node

Homepage: https://github.com/shelljs/shx#readme

Created	almost 4 years ago
Last Updated	over 1 year ago
License	MIT
Maintainers	2
Releases	11
Direct Dependencies	es6-object-assign, minimist and shelljs
Keywords	shelljs, shell, unix, bash, sh, exec, cli and zsh

README

Shx

shx is a wrapper around ShellJS Unix
commands, providing an easy solution for simple Unix-like, cross-platform
commands in npm package scripts.

Difference Between ShellJS and shx

• ShellJS: Good for writing long scripts, all in JS, running via NodeJS (e.g. node myScript.js).
• shx: Good for writing one-off commands in npm package scripts (e.g. "clean": "shx rm -rf out/").

Install

npm install shx --save-dev

This will allow using shx in your package.json scripts.

Usage

Command Line

If you'd like to use shx on the command line, install it globally with the -g flag.
The following code can be run either a Unix or Windows command line:

$ shx pwd                       # ShellJS commands are supported automatically
/home/username/path/to/dir

$ shx ls                        # files are outputted one per line
file.txt
file2.txt

$ shx rm *.txt                  # a cross-platform way to delete files!

$ shx ls

$ shx echo "Hi there!"
Hi there!

$ shx touch helloworld.txt

$ shx cp helloworld.txt foobar.txt

$ shx mkdir sub

$ shx ls
foobar.txt
helloworld.txt
sub

$ shx rm -r sub                 # options work as well

$ shx --silent ls fakeFileName  # silence error output

All commands internally call the ShellJS corresponding function, guaranteeing
cross-platform compatibility.

package.json

ShellJS is good for writing long scripts. If you want to write bash-like,
platform-independent scripts, we recommend you go with that.

However, shx is ideal for one-liners inside package.json:

{
  "scripts": {
    "clean": "shx rm -rf build dist && shx echo Done"
  }
}

Unsupported Commands

Due to the differences in execution environments between ShellJS and shx (JS vs CLI) some commands are not supported:

Unsupported command	Recommend workaround
shx cd	Just use plain old cd (it's the same on windows too)
shx pushd	Just use plain old pushd. Use forward slashes and double-quote the path. (e.g. pushd "../docs". This would fail on Windows without the quotes)
shx popd	Just use plain old popd
shx dirs	No workaround
shx set	See below
shx exit	Just use plain old exit
shx exec	Instead of shx exec cmd, just use plain old cmd
shx ShellString	No workaround (but why would you want this?)

Shx options

Shx allows you to modify its behavior by passing arguments. Here's a list of
supported options:

set flag	shell.config setting	shx command	Effect
-e	config.fatal = true	Not supported	Exit upon first error
-v	config.verbose = true	shx --verbose cd foo	Log the command as it's run
-f	config.noglob = true	shx --noglob cat '*.txt'	Don't expand wildcards
N/A	config.silent = true	shx --silent cd noexist	Don't show error output

Team

Nate Fischer	Ari Porad	Levi Thomason

Generated by 🚫 dangerJS