Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore(tests): stress testing data generation scripts #2000

Open
wants to merge 19 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ testData=${1:-"./testdata_template/environments"}
for filename in "$testData"/*; do
configFile="$filename"/config.json
env=$(basename -- "$filename")
env=$(echo "$env" | awk '{print tolower($0)}')
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Upper case environments are not accepted by kuberpult

echo Writing $env...
DATA=$(cat $configFile)
curl -f -X POST -H "multipart/form-data" \
Expand Down
63 changes: 63 additions & 0 deletions infrastructure/scripts/create-testdata/stress_testing/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,63 @@
# Stress Tests - Data Generation Scripts

## Generating the environments

The generate_environments.sh script is setup to receive two parameters:

1. The base name of the environments that you want to create, BASE_ENV_NAME,
2. The number of environments you want to create, NUMBER_ENVS,
3. Upstream environment name, UPSTREAM_ENV.

Running the script as such:
```shell
./generate_environments.sh qa 32 testing
```
Creates 32 environments under the qa environment group (each name qa-${COUNTRY_CODE}, where
the country codes are listed under the country_codes.csv file). All these environments have as upstream
environment the testing environment, also created by the script.

Running this script not only generates the necessary files under ./environments, but also publishes them to a
local instance of kuberpult.


## Generating applications data

For generating application data, we have the create_releases.sh script. It is setup to accept three parameters:
1. The base app name of the applications that you want to create,
2. The team of these applications,
3. The total number of applications you want to create.

Running the script as such:
```shell
./create-releases.sh stress-testing-app sreteam 1000
```
creates 1000 applications name stress-testing-app-{0-999} under the sreteam. If you use the generate_environments.sh
these applications will be automatically deployed to testing and have a manifest for all the qa environments.

For each app, the script also creates a random number of releases, between 10 and 20.


## Know Issues

### Time

First of all, creating this much data takes a long time. The manifest-repo-export-service takes and increasingly
bigger amount of time processing one event the greater the number of apps. This is because of the [UpdateArgoApps](https://github.com/freiheit-com/kuberpult/blob/127d6ec37801b0c420688847f0c0ee113459eb77/services/manifest-repo-export-service/pkg/repository/repository.go#L700) steps
that is run after each transformer. Our recommendation is to comment out this step if your desire is to simply generate
a lot of data.

### Space

It is not uncommon for the postgres container to throw an error akin to something like:
```
no space left on device
```

This can usually be solved by running:
```shell
docker system prune
```
It is recommended that you run this command before you start generating a lot of data.



Original file line number Diff line number Diff line change
@@ -0,0 +1,249 @@
AF
AL
DZ
AS
AD
AO
AI
AQ
AG
AR
AM
AW
AU
AT
AZ
BS
BH
BD
BB
BY
BE
BZ
BJ
BM
BT
BO
BQ
BA
BW
BV
BR
IO
BN
BG
BF
BI
KH
CM
CA
CV
KY
CF
TD
CL
CN
CX
CC
CO
KM
CG
CD
CK
CR
HR
CU
CW
CY
CZ
CI
DK
DJ
DM
DO
EC
EG
SV
GQ
ER
EE
ET
FK
FO
FJ
FI
FR
GF
PF
TF
GA
GM
GE
DE
GH
GI
GR
GL
GD
GP
GU
GT
GG
GN
GW
GY
HT
HM
VA
HN
HK
HU
IS
IN
ID
IR
IQ
IE
IM
IL
IT
JM
JP
JE
JO
KZ
KE
KI
KP
KR
KW
KG
LA
LV
LB
LS
LR
LY
LI
LT
LU
MO
MK
MG
MW
MY
MV
ML
MT
MH
MQ
MR
MU
YT
MX
FM
MD
MC
MN
ME
MS
MA
MZ
MM
NA
NR
NP
NL
NC
NZ
NI
NE
NG
NU
NF
MP
NO
OM
PK
PW
PS
PA
PG
PY
PE
PH
PN
PL
PT
PR
QA
RO
RU
RW
RE
BL
SH
KN
LC
MF
PM
VC
WS
SM
ST
SA
SN
RS
SC
SL
SG
SX
SK
SI
SB
SO
ZA
GS
SS
ES
LK
SD
SR
SJ
SZ
SE
CH
SY
TW
TJ
TZ
TH
TL
TG
TK
TO
TT
TN
TR
TM
TC
TV
UG
UA
AE
GB
US
UM
UY
UZ
VU
VE
VN
VG
VI
WF
EH
YE
ZM
ZW
AX
Loading
Loading