Skip to content

Commit

Permalink
[Cosmos] Pipelines for cosmos sdk (Azure#4881)
Browse files Browse the repository at this point in the history
* Pipelines and setup for cosmos sdk  (#16)

* resolve merge conflict - tsconfig.json

* add cosmos in rush.json

* update dependencies - sync-versions

* add script "extract-api"

* delete .prettierrc

* update package.json script commands

* npm run bundle command

* Fix typo in samples/package.json

* npm run clean command

* remove unrelated file .gitattributes

* Update package.json repo links

* add ci.yml from cosmos v2

* Node 6 to Node 8 - integration step

* azure-cosmos-js -> azure-sdk-for-js

* Update License document

* update tool chain from npm to rush

* add package name

* delete package-lock.json

* add cosmos-additional-steps.yml file

* add PostIntegrationSteps variable

* condition: succeededOrFailed() for additional steps

* add parameters PackageName and PackagePath

* Smaller Matrix for Unit tests as per Daniel's idea

* parameters - fix typo

* regenerate pnpm-lock file

* test:consumer -> test-consumer

* remove matrix for unit-tests

* checking if bash task affects the build

* update test-consumer command

* script over one-line setup

* remove endpoint in env vars

* disable IndexManagement sample

* add mocha-junit-reporter

* update pipeline script (#18)

* add await for cosmosdb to start

* use java emulator install procedure

* add variables for emulator msi

* use parameters as would be expected

* use correct quotes

* remove java-specific steps, use accepted emulator msi url

* without the single quote

* update start-process step

* ProgramFiles -> Temp
  • Loading branch information
HarshaNalluru authored Aug 29, 2019
1 parent df3a7b2 commit 9ed6846
Show file tree
Hide file tree
Showing 11 changed files with 169 additions and 51 deletions.
28 changes: 14 additions & 14 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -165,20 +165,20 @@ If you want to get back to a completely clean state, you can instead run `rush r

Generally speaking, the following commands are roughly equivalent:

| NPM command | Rush command | Rush command effect |
| ------------------------------------ | ------------------------------------ | ---------------------------------------------------------------- |
| `npm install` | `rush update` | Install dependencies for all projects in the Rush workspace |
| `npm install --save[-dev] <package>` | `rush add -p <package> --caret [--dev]` | Add or update a dependency in the current project |
| `npm build` | `rush [re]build` | Build all projects in the Rush workspace |
| | `rush [re]build -t <package>` | Build named project and any projects it depends on |
| | `rushx build` | Build the current project only |
| `npm test` | `rush test` | Run dev tests in all projects in the Rush workspace |
| | `rush test -t <packagename>` | Run dev tests in named project and any projects it depends on |
| | `rushx test` | Run dev tests in the current project only |
| `npm run <scriptname>` | `rush <scriptname>` | Run named script in all projects in the Rush workspace |
| | `rush <scriptname> -t <packagename>` | Run named script in named project and any projects it depends on |
| | `rushx <scriptname>` | Run named script in the current project only |
| `npx <command>` | `node_modules/.bin/<command>` | Run named command provided by installed dependency package |
| NPM command | Rush command | Rush command effect |
| ------------------------------------ | --------------------------------------- | ---------------------------------------------------------------- |
| `npm install` | `rush update` | Install dependencies for all projects in the Rush workspace |
| `npm install --save[-dev] <package>` | `rush add -p <package> --caret [--dev]` | Add or update a dependency in the current project |
| `npm build` | `rush [re]build` | Build all projects in the Rush workspace |
| | `rush [re]build -t <package>` | Build named project and any projects it depends on |
| | `rushx build` | Build the current project only |
| `npm test` | `rush test` | Run dev tests in all projects in the Rush workspace |
| | `rush test -t <packagename>` | Run dev tests in named project and any projects it depends on |
| | `rushx test` | Run dev tests in the current project only |
| `npm run <scriptname>` | `rush <scriptname>` | Run named script in all projects in the Rush workspace |
| | `rush <scriptname> -t <packagename>` | Run named script in named project and any projects it depends on |
| | `rushx <scriptname>` | Run named script in the current project only |
| `npx <command>` | `node_modules/.bin/<command>` | Run named command provided by installed dependency package |

## Onboarding a new library

Expand Down
13 changes: 13 additions & 0 deletions common/config/rush/command-line.json
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,13 @@
"ignoreMissingScript": false,
"allowWarningsInSuccessfulBuild": true
},
{
"commandKind": "bulk",
"name": "build:samples",
"summary": "Build the samples only in projects",
"enableParallelism": true,
"allowWarningsInSuccessfulBuild": true
},
{
"commandKind": "bulk",
"name": "build:test",
Expand Down Expand Up @@ -100,6 +107,12 @@
"summary": "Execute browser dev tests defined in projects",
"enableParallelism": true
},
{
"commandKind": "bulk",
"name": "test-consumer",
"summary": "Execute consumer tests defined in projects",
"enableParallelism": true
},
{
"commandKind": "bulk",
"name": "test:node",
Expand Down
59 changes: 31 additions & 28 deletions eng/pipelines/templates/jobs/archetype-sdk-client.yml
Original file line number Diff line number Diff line change
@@ -1,3 +1,33 @@
parameters:
Matrix:
Linux_Node8:
OSVmImage: "ubuntu-16.04"
NodeVersion: "8.x"
Linux_Node10:
OSVmImage: "ubuntu-16.04"
NodeVersion: "10.x"
Linux_Node12:
OSVmImage: "ubuntu-16.04"
NodeVersion: "12.x"
macOS_Node8:
OSVmImage: "macOS-10.13"
NodeVersion: "8.x"
macOS_Node10:
OSVmImage: "macOS-10.13"
NodeVersion: "10.x"
macOS_Node12:
OSVmImage: "macOS-10.13"
NodeVersion: "12.x"
Windows_Node8:
OSVmImage: "vs2017-win2016"
NodeVersion: "8.x"
Windows_Node10:
OSVmImage: "vs2017-win2016"
NodeVersion: "10.x"
Windows_Node12:
OSVmImage: "vs2017-win2016"
NodeVersion: "12.x"

jobs:
- job: "Build"
variables:
Expand Down Expand Up @@ -119,34 +149,7 @@ jobs:
- job: "UnitTest"

strategy:
matrix:
Linux_Node8:
OSVmImage: "ubuntu-16.04"
NodeVersion: "8.x"
Linux_Node10:
OSVmImage: "ubuntu-16.04"
NodeVersion: "10.x"
Linux_Node12:
OSVmImage: "ubuntu-16.04"
NodeVersion: "12.x"
macOS_Node8:
OSVmImage: "macOS-10.13"
NodeVersion: "8.x"
macOS_Node10:
OSVmImage: "macOS-10.13"
NodeVersion: "10.x"
macOS_Node12:
OSVmImage: "macOS-10.13"
NodeVersion: "12.x"
Windows_Node8:
OSVmImage: "vs2017-win2016"
NodeVersion: "8.x"
Windows_Node10:
OSVmImage: "vs2017-win2016"
NodeVersion: "10.x"
Windows_Node12:
OSVmImage: "vs2017-win2016"
NodeVersion: "12.x"
matrix: ${{parameters.Matrix}}

pool:
vmImage: "$(OSVmImage)"
Expand Down
2 changes: 2 additions & 0 deletions eng/pipelines/templates/jobs/archetype-sdk-integration.yml
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@ parameters:
PackageName: "" # For Rush toolchain only
PackagePath: "" # For NPM toolchain only
PreIntegrationSteps: ""
PostIntegrationSteps: ""
ResourceFileLocation: ""
EnvVars: []
MaxParallel: 0
Expand Down Expand Up @@ -86,6 +87,7 @@ jobs:
PackageName: ${{parameters.PackageName}}
PackagePath: ${{parameters.PackagePath}}
PreIntegrationSteps: ${{parameters.PreIntegrationSteps}}
PostIntegrationSteps: ${{parameters.PostIntegrationSteps}}
TestType: $(TestType)
EnvVars: ${{parameters.EnvVars}}

Expand Down
20 changes: 20 additions & 0 deletions eng/pipelines/templates/steps/cosmos-additional-steps.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
parameters:
PackagePath: "sdk/cosmosdb/cosmos/"
PackageName: "@azure/cosmos"

steps:
- bash: |
node common/scripts/install-run-rush.js test-consumer -t "${{parameters.PackageName}}" --verbose
condition: succeededOrFailed()
displayName: "TypeScript Consumer Tests - Cosmos"
- script: |
node common/scripts/install-run-rush.js build:samples -t "${{parameters.PackageName}}" --verbose
condition: succeededOrFailed()
displayName: "Typecheck Samples - Cosmos"
- script: |
npm run execute:samples
condition: succeededOrFailed()
displayName: "Run Samples - Cosmos"
workingDirectory: "${{parameters.PackagePath}}"
37 changes: 33 additions & 4 deletions eng/pipelines/templates/steps/cosmos-integration-public.yml
Original file line number Diff line number Diff line change
@@ -1,5 +1,34 @@
parameters:
EmulatorMsiUrl: "https://aka.ms/cosmosdb-emulator"

steps:
- task: azure-cosmosdb.emulator-public-preview.run-cosmosdbemulatorcontainer.CosmosDbEmulator@2
inputs:
defaultPartitionCount: 25
displayName: "Run Azure Cosmos DB Emulator container (public)"
- powershell: |
Write-Host "Downloading Cosmos Emulator - ${{ parameters.EmulatorMsiUrl }}"
wget "${{ parameters.EmulatorMsiUrl }}" -outfile "$env:temp\azure-cosmosdb-emulator.msi"
Write-Host "Finished Downloading Cosmos Emulator - $env:temp\azure-cosmosdb-emulator.msi"
dir "$env:temp"
displayName: "Download Public Cosmos DB Emulator"
- script: |
choco install lessmsi
choco upgrade lessmsi
mkdir "%TEMP%\Azure Cosmos DB Emulator"
lessmsi x "%TEMP%\azure-cosmosdb-emulator.msi" "%TEMP%\Azure Cosmos DB Emulator\"
displayName: "Install Public Cosmos DB Emulator"
- powershell: |
Write-Host "Starting Comsos DB Emulator"
Start-Process "$env:Temp\Azure Cosmos DB Emulator\SourceDir\Azure Cosmos DB Emulator\CosmosDB.Emulator.exe" "/NoExplorer /NoUI" -Verb RunAs
displayName: "Run Public Cosmos DB Emulator"
- powershell: |
Import-Module "$env:Temp\Azure Cosmos DB Emulator\SourceDir\Azure Cosmos DB Emulator\PSModules\Microsoft.Azure.CosmosDB.Emulator"
Do {
sleep 5
$cosmosStatus = Get-CosmosDbEmulatorStatus
Write-Host "Cosmos Status: $cosmosStatus"
} While ($cosmosStatus -ne 'Running')
Write-Host "Done"
displayName: "Check Public Cosmos DB Emulator Status"
5 changes: 4 additions & 1 deletion eng/pipelines/templates/steps/integration-npm.yml
Original file line number Diff line number Diff line change
Expand Up @@ -21,10 +21,13 @@ steps:
workingDirectory: "${{parameters.PackagePath}}"
- ${{if ne(parameters.PreIntegrationSteps, '')}}:
- template: ../steps/${{parameters.PreIntegrationSteps}}.yml
- template: ../steps/${{parameters.PreIntegrationSteps}}.yml

- script: |
npm run integration-test:${{parameters.TestType}}
displayName: "Integration test libraries"
workingDirectory: "${{parameters.PackagePath}}"
env: ${{parameters.EnvVars}}
- ${{if ne(parameters.PostIntegrationSteps, '')}}:
- template: ../steps/${{parameters.PostIntegrationSteps}}.yml
5 changes: 4 additions & 1 deletion eng/pipelines/templates/steps/integration-rush.yml
Original file line number Diff line number Diff line change
Expand Up @@ -18,9 +18,12 @@ steps:
displayName: "Build test assets"
- ${{if ne(parameters.PreIntegrationSteps, '')}}:
- template: ../steps/${{parameters.PreIntegrationSteps}}.yml
- template: ../steps/${{parameters.PreIntegrationSteps}}.yml

- script: |
node common/scripts/install-run-rush.js integration-test:${{parameters.TestType}} -t "${{parameters.PackageName}}" --verbose
displayName: "Integration test libraries"
env: ${{parameters.EnvVars}}
- ${{if ne(parameters.PostIntegrationSteps, '')}}:
- template: ../steps/${{parameters.PostIntegrationSteps}}.yml
43 changes: 43 additions & 0 deletions sdk/cosmosdb/ci.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,43 @@
# DO NOT EDIT THIS FILE
# This file is generated automatically and any changes will be lost.

trigger:
branches:
include:
- master
- feature/*
paths:
include:
- sdk/cosmosdb/

pr:
branches:
include:
- master
- feature/*
paths:
include:
- sdk/cosmosdb/

jobs:
- template: ../../eng/pipelines/templates/jobs/archetype-sdk-client.yml
parameters:
# Unit-tests are being skipped. So, matrix is not needed.
Matrix: {}
ServiceDirectory: cosmosdb

- template: ../../eng/pipelines/templates/jobs/archetype-sdk-integration.yml
parameters:
PackagePath: "sdk/cosmosdb/cosmos/"
PackageName: "@azure/cosmos"
Toolchain: rush
Matrix:
Windows_Node8:
OSVmImage: "vs2017-win2016"
NodeVersion: "8.x"
TestType: "node"
PreIntegrationSteps: cosmos-integration-public
PostIntegrationSteps: cosmos-additional-steps
EnvVars:
MOCHA_TIMEOUT: 100000
NODE_TLS_REJECT_UNAUTHORIZED: 0
5 changes: 3 additions & 2 deletions sdk/cosmosdb/cosmos/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -43,10 +43,11 @@
"check-format": "prettier --list-different --config ../../.prettierrc.json \"src/**/*.ts\" \"test/**/*.ts\" \"*.{js,json}\"",
"clean": "rimraf dist dist-esm temp types *.tgz *.html *.log *.tsbuildinfo test/**/*.{js,js.map,d.ts}",
"docs": "typedoc --tsconfig ./src/tsconfig.json --excludePrivate --mode file --out ./dist/docs ./src",
"execute:samples": "cd samples && npm run all-samples",
"extract-api": "npm run build:src && api-extractor run --local",
"format": "prettier --write --config ../../.prettierrc.json \"src/**/*.ts\" \"test/**/*.ts\" \"*.{js,json}\"",
"integration-test:browser": "echo skipped",
"integration-test:node": "mocha -r esm -r dotenv/config -r ./test/common/setup.js \"./test/**/*.spec.js\" --timeout 100000",
"integration-test:node": "mocha -r esm -r dotenv/config -r ./test/common/setup.js \"./test/**/*.spec.js\" --timeout 100000 --reporter mocha-multi --reporter-options spec=-,mocha-junit-reporter=-",
"integration-test": "npm run integration-test:node && npm run integration-test:browser",
"lint:fix": "eslint \"src/**/*.ts\" \"test/**/*.ts\" -c ../../.eslintrc.json --fix --fix-type [problem,suggestion]",
"lint": "eslint -c ../../.eslintrc.json src test --ext .ts -f html -o cosmos-lintReport.html || exit 0",
Expand All @@ -55,7 +56,7 @@
"test:browser": "npm run build:test && npm run unit-test:browser && npm run integration-test:browser",
"test:node": "npm run build:test && npm run unit-test:node && npm run integration-test:node",
"test-browser": "karma start ./karma.config.js --single-run",
"test-consumer": "node consumer-test.js",
"test-consumer": "node consumer-test.js 2>&1",
"test": "npm run build:test && npm run unit-test && npm run integration-test",
"unit-test:browser": "echo skipped",
"unit-test:node": "echo skipped",
Expand Down
3 changes: 2 additions & 1 deletion sdk/cosmosdb/cosmos/samples/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@
"ItemManagement": "npx ts-node ./ItemManagement",
"DatabaseManagement": "npx ts-node ./DatabaseManagement",
"IndexManagement": "npx ts-node ./IndexManagement",
"ChangeFeed": "npx ts-node ./ChangeFeed"
"ChangeFeed": "npx ts-node ./ChangeFeed",
"all-samples": "npm run ContainerManagement && npm run UserManagement && npm run ServerSideScripts && npm run ItemManagement && npm run DatabaseManagement && npm run ChangeFeed"
}
}

0 comments on commit 9ed6846

Please sign in to comment.