Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Sustainable Kibana Architecture: auto PR #2

Closed
wants to merge 1 commit into from
+547 −4
Closed

Sustainable Kibana Architecture: auto PR #2

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
101 changes: 101 additions & 0 deletions TODO.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,101 @@

## PHASE 1 - ESLint constraints

- Manifests (plugin export function + setup + start contracts)
- Import statements
- group: 'platform' | 'observability' | 'security' | 'search'
- visibility: 'private' | 'shared' # platform only, solutions are 'private'

## PHASE 2 - Categorising modules (packages and plugins)

- Not moving, only tagging
- Reveal inter-group dependencies


## PHASE 3 - Actually moving stuff
```
'src/platform/(packages|plugins)/(private|shared)',
'x-pack/platform/(packages|plugins)/(private|shared)',

'x-pack/solutions/observability/plugins',
'x-pack/solutions/observability/packages',
'x-pack/solutions/security/plugins',
'x-pack/solutions/security/packages',
'x-pack/solutions/search/plugins',
'x-pack/solutions/search/packages',
```

- Command-line tool to relocate modules, by owner
- 38 teams => 38 big PRs



---



















node scripts/manifest --relocate @elastic/appex-ai-infra

Manually search for [\/\\]

const TEAMS = [
'@elastic/kibana-core', https://github.com/elastic/kibana/pull/201653
'@elastic/appex-ai-infra', https://github.com/elastic/kibana/pull/202410
'@elastic/appex-sharedux', ai-infra
'@elastic/docs', https://github.com/elastic/kibana/pull/202416
'@elastic/fleet', shallow-2 https://github.com/elastic/kibana/pull/202422
* x-pack/platform/plugins/shared/fleet/cypress/tasks/login.ts
* x-pack/platform/plugins/shared/fleet/cypress/reporter_config.json
'@elastic/kibana-cloud-security-posture',
'@elastic/kibana-data-discovery',
'@elastic/kibana-esql',
'@elastic/kibana-localization',
'@elastic/kibana-management',
'@elastic/kibana-operations',
'@elastic/kibana-presentation',
'@elastic/kibana-reporting-services',
'@elastic/kibana-security',
'@elastic/kibana-visualizations',
'@elastic/logstash',
'@elastic/ml-ui',
'@elastic/obs-ai-assistant',
'@elastic/obs-entities',
'@elastic/obs-knowledge-team',
'@elastic/obs-ux-infra_services-team',
'@elastic/obs-ux-logs-team',
'@elastic/obs-ux-management-team',
'@elastic/obs-ux-onboarding-team',
'@elastic/observability-ui',
'@elastic/response-ops',
'@elastic/search-kibana',
'@elastic/security-asset-management',
'@elastic/security-defend-workflows',
'@elastic/security-detection-engine',
'@elastic/security-detection-rule-management',
'@elastic/security-detections-response',
'@elastic/security-generative-ai',
'@elastic/security-scalability',
'@elastic/security-solution',
'@elastic/security-threat-hunting-explore',
'@elastic/security-threat-hunting-investigations',
'@elastic/security-threat-hunting',
'@elastic/stack-monitoring',
'@simianhacker @flash1293 @dgieselaar',
'@vigneshshanmugam',
];
32 changes: 32 additions & 0 deletions launcher.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@




cd /Users/gsoldevila/Work/kibana-presentation && git pull && cp ../kibana/packages/kbn-manifest/* ./packages/kbn-manifest/ && rm -fR node_modules && yarn kbn bootstrap && node scripts/manifest --max-old-space-size=16384 --relocate @elastic/kibana-presentation > relocate.out 2>relocate.err && git branch kbn-team-1309-move-kibana-presentation && git add . && git restore --staged ./packages/kbn-manifest/*
cd /Users/gsoldevila/Work/kibana-reporting-services && git pull && cp ../kibana/packages/kbn-manifest/* ./packages/kbn-manifest/ && rm -fR node_modules && yarn kbn bootstrap && node scripts/manifest --max-old-space-size=16384 --relocate @elastic/kibana-reporting-services > relocate.out 2>relocate.err && git branch kbn-team-1309-move-kibana-reporting-services && git add . && git restore --staged ./packages/kbn-manifest/*
cd /Users/gsoldevila/Work/kibana-security && git pull && cp ../kibana/packages/kbn-manifest/* ./packages/kbn-manifest/ && rm -fR node_modules && yarn kbn bootstrap && node scripts/manifest --max-old-space-size=16384 --relocate @elastic/kibana-security > relocate.out 2>relocate.err && git branch kbn-team-1309-move-kibana-security && git add . && git restore --staged ./packages/kbn-manifest/*
cd /Users/gsoldevila/Work/kibana-visualizations && git pull && cp ../kibana/packages/kbn-manifest/* ./packages/kbn-manifest/ && rm -fR node_modules && yarn kbn bootstrap && node scripts/manifest --max-old-space-size=16384 --relocate @elastic/kibana-visualizations > relocate.out 2>relocate.err && git branch kbn-team-1309-move-kibana-visualizations && git add . && git restore --staged ./packages/kbn-manifest/*
cd /Users/gsoldevila/Work/kibana-logstash && git pull && cp ../kibana/packages/kbn-manifest/* ./packages/kbn-manifest/ && rm -fR node_modules && yarn kbn bootstrap && node scripts/manifest --max-old-space-size=16384 --relocate @elastic/logstash > relocate.out 2>relocate.err && git branch kbn-team-1309-move-kibana-logstash && git add . && git restore --staged ./packages/kbn-manifest/*
cd /Users/gsoldevila/Work/kibana-ml-ui && git pull && cp ../kibana/packages/kbn-manifest/* ./packages/kbn-manifest/ && rm -fR node_modules && yarn kbn bootstrap && node scripts/manifest --max-old-space-size=16384 --relocate @elastic/ml-ui > relocate.out 2>relocate.err && git branch kbn-team-1309-move-kibana-ml-ui && git add . && git restore --staged ./packages/kbn-manifest/*
cd /Users/gsoldevila/Work/kibana-obs-ai-assistant && git pull && cp ../kibana/packages/kbn-manifest/* ./packages/kbn-manifest/ && rm -fR node_modules && yarn kbn bootstrap && node scripts/manifest --max-old-space-size=16384 --relocate @elastic/obs-ai-assistant > relocate.out 2>relocate.err && git branch kbn-team-1309-move-kibana-obs-ai-assistant && git add . && git restore --staged ./packages/kbn-manifest/*
cd /Users/gsoldevila/Work/kibana-obs-entities && git pull && cp ../kibana/packages/kbn-manifest/* ./packages/kbn-manifest/ && rm -fR node_modules && yarn kbn bootstrap && node scripts/manifest --max-old-space-size=16384 --relocate @elastic/obs-entities > relocate.out 2>relocate.err && git branch kbn-team-1309-move-kibana-obs-entities && git add . && git restore --staged ./packages/kbn-manifest/*
cd /Users/gsoldevila/Work/kibana-obs-knowledge-team && git pull && cp ../kibana/packages/kbn-manifest/* ./packages/kbn-manifest/ && rm -fR node_modules && yarn kbn bootstrap && node scripts/manifest --max-old-space-size=16384 --relocate @elastic/obs-knowledge-team > relocate.out 2>relocate.err && git branch kbn-team-1309-move-kibana-obs-knowledge-team && git add . && git restore --staged ./packages/kbn-manifest/*
cd /Users/gsoldevila/Work/kibana-obs-ux-infra_services-team && git pull && cp ../kibana/packages/kbn-manifest/* ./packages/kbn-manifest/ && rm -fR node_modules && yarn kbn bootstrap && node scripts/manifest --max-old-space-size=16384 --relocate @elastic/obs-ux-infra_services-team > relocate.out 2>relocate.err && git branch kbn-team-1309-move-kibana-obs-ux-infra_services-team && git add . && git restore --staged ./packages/kbn-manifest/*
cd /Users/gsoldevila/Work/kibana-obs-ux-logs-team && git pull && cp ../kibana/packages/kbn-manifest/* ./packages/kbn-manifest/ && rm -fR node_modules && yarn kbn bootstrap && node scripts/manifest --max-old-space-size=16384 --relocate @elastic/obs-ux-logs-team > relocate.out 2>relocate.err && git branch kbn-team-1309-move-kibana-obs-ux-logs-team && git add . && git restore --staged ./packages/kbn-manifest/*
cd /Users/gsoldevila/Work/kibana-obs-ux-management-team && git pull && cp ../kibana/packages/kbn-manifest/* ./packages/kbn-manifest/ && rm -fR node_modules && yarn kbn bootstrap && node scripts/manifest --max-old-space-size=16384 --relocate @elastic/obs-ux-management-team > relocate.out 2>relocate.err && git branch kbn-team-1309-move-kibana-obs-ux-management-team && git add . && git restore --staged ./packages/kbn-manifest/*
cd /Users/gsoldevila/Work/kibana-obs-ux-onboarding-team && git pull && cp ../kibana/packages/kbn-manifest/* ./packages/kbn-manifest/ && rm -fR node_modules && yarn kbn bootstrap && node scripts/manifest --max-old-space-size=16384 --relocate @elastic/obs-ux-onboarding-team > relocate.out 2>relocate.err && git branch kbn-team-1309-move-kibana-obs-ux-onboarding-team && git add . && git restore --staged ./packages/kbn-manifest/*
cd /Users/gsoldevila/Work/kibana-observability-ui && git pull && cp ../kibana/packages/kbn-manifest/* ./packages/kbn-manifest/ && rm -fR node_modules && yarn kbn bootstrap && node scripts/manifest --max-old-space-size=16384 --relocate @elastic/observability-ui > relocate.out 2>relocate.err && git branch kbn-team-1309-move-kibana-observability-ui && git add . && git restore --staged ./packages/kbn-manifest/*
cd /Users/gsoldevila/Work/kibana-response-ops && git pull && cp ../kibana/packages/kbn-manifest/* ./packages/kbn-manifest/ && rm -fR node_modules && yarn kbn bootstrap && node scripts/manifest --max-old-space-size=16384 --relocate @elastic/response-ops > relocate.out 2>relocate.err && git branch kbn-team-1309-move-kibana-response-ops && git add . && git restore --staged ./packages/kbn-manifest/*
cd /Users/gsoldevila/Work/kibana-search-kibana && git pull && cp ../kibana/packages/kbn-manifest/* ./packages/kbn-manifest/ && rm -fR node_modules && yarn kbn bootstrap && node scripts/manifest --max-old-space-size=16384 --relocate @elastic/search-kibana > relocate.out 2>relocate.err && git branch kbn-team-1309-move-kibana-search-kibana && git add . && git restore --staged ./packages/kbn-manifest/*
cd /Users/gsoldevila/Work/kibana-security-asset-management && git pull && cp ../kibana/packages/kbn-manifest/* ./packages/kbn-manifest/ && rm -fR node_modules && yarn kbn bootstrap && node scripts/manifest --max-old-space-size=16384 --relocate @elastic/security-asset-management > relocate.out 2>relocate.err && git branch kbn-team-1309-move-kibana-security-asset-management && git add . && git restore --staged ./packages/kbn-manifest/*
cd /Users/gsoldevila/Work/kibana-security-defend-workflows && git pull && cp ../kibana/packages/kbn-manifest/* ./packages/kbn-manifest/ && rm -fR node_modules && yarn kbn bootstrap && node scripts/manifest --max-old-space-size=16384 --relocate @elastic/security-defend-workflows > relocate.out 2>relocate.err && git branch kbn-team-1309-move-kibana-security-defend-workflows && git add . && git restore --staged ./packages/kbn-manifest/*
cd /Users/gsoldevila/Work/kibana-security-detection-engine && git pull && cp ../kibana/packages/kbn-manifest/* ./packages/kbn-manifest/ && rm -fR node_modules && yarn kbn bootstrap && node scripts/manifest --max-old-space-size=16384 --relocate @elastic/security-detection-engine > relocate.out 2>relocate.err && git branch kbn-team-1309-move-kibana-security-detection-engine && git add . && git restore --staged ./packages/kbn-manifest/*
cd /Users/gsoldevila/Work/kibana-security-detection-rule-management && git pull && cp ../kibana/packages/kbn-manifest/* ./packages/kbn-manifest/ && rm -fR node_modules && yarn kbn bootstrap && node scripts/manifest --max-old-space-size=16384 --relocate @elastic/security-detection-rule-management > relocate.out 2>relocate.err && git branch kbn-team-1309-move-kibana-security-detection-rule-management && git add . && git restore --staged ./packages/kbn-manifest/*
cd /Users/gsoldevila/Work/kibana-security-detections-response && git pull && cp ../kibana/packages/kbn-manifest/* ./packages/kbn-manifest/ && rm -fR node_modules && yarn kbn bootstrap && node scripts/manifest --max-old-space-size=16384 --relocate @elastic/security-detections-response > relocate.out 2>relocate.err && git branch kbn-team-1309-move-kibana-security-detections-response && git add . && git restore --staged ./packages/kbn-manifest/*
cd /Users/gsoldevila/Work/kibana-security-generative-ai && git pull && cp ../kibana/packages/kbn-manifest/* ./packages/kbn-manifest/ && rm -fR node_modules && yarn kbn bootstrap && node scripts/manifest --max-old-space-size=16384 --relocate @elastic/security-generative-ai > relocate.out 2>relocate.err && git branch kbn-team-1309-move-kibana-security-generative-ai && git add . && git restore --staged ./packages/kbn-manifest/*
cd /Users/gsoldevila/Work/kibana-security-scalability && git pull && cp ../kibana/packages/kbn-manifest/* ./packages/kbn-manifest/ && rm -fR node_modules && yarn kbn bootstrap && node scripts/manifest --max-old-space-size=16384 --relocate @elastic/security-scalability > relocate.out 2>relocate.err && git branch kbn-team-1309-move-kibana-security-scalability && git add . && git restore --staged ./packages/kbn-manifest/*
cd /Users/gsoldevila/Work/kibana-security-solution && git pull && cp ../kibana/packages/kbn-manifest/* ./packages/kbn-manifest/ && rm -fR node_modules && yarn kbn bootstrap && node scripts/manifest --max-old-space-size=16384 --relocate @elastic/security-solution > relocate.out 2>relocate.err && git branch kbn-team-1309-move-kibana-security-solution && git add . && git restore --staged ./packages/kbn-manifest/*
cd /Users/gsoldevila/Work/kibana-security-threat-hunting-explore && git pull && cp ../kibana/packages/kbn-manifest/* ./packages/kbn-manifest/ && rm -fR node_modules && yarn kbn bootstrap && node scripts/manifest --max-old-space-size=16384 --relocate @elastic/security-threat-hunting-explore > relocate.out 2>relocate.err && git branch kbn-team-1309-move-kibana-security-threat-hunting-explore && git add . && git restore --staged ./packages/kbn-manifest/*
cd /Users/gsoldevila/Work/kibana-security-threat-hunting-investigations && git pull && cp ../kibana/packages/kbn-manifest/* ./packages/kbn-manifest/ && rm -fR node_modules && yarn kbn bootstrap && node scripts/manifest --max-old-space-size=16384 --relocate @elastic/security-threat-hunting-investigations > relocate.out 2>relocate.err && git branch kbn-team-1309-move-kibana-security-threat-hunting-investigations && git add . && git restore --staged ./packages/kbn-manifest/*
cd /Users/gsoldevila/Work/kibana-security-threat-hunting && git pull && cp ../kibana/packages/kbn-manifest/* ./packages/kbn-manifest/ && rm -fR node_modules && yarn kbn bootstrap && node scripts/manifest --max-old-space-size=16384 --relocate @elastic/security-threat-hunting > relocate.out 2>relocate.err && git branch kbn-team-1309-move-kibana-security-threat-hunting && git add . && git restore --staged ./packages/kbn-manifest/*
cd /Users/gsoldevila/Work/kibana-stack-monitoring && git pull && cp ../kibana/packages/kbn-manifest/* ./packages/kbn-manifest/ && rm -fR node_modules && yarn kbn bootstrap && node scripts/manifest --max-old-space-size=16384 --relocate @elastic/stack-monitoring > relocate.out 2>relocate.err && git branch kbn-team-1309-move-kibana-stack-monitoring && git add . && git restore --staged ./packages/kbn-manifest/*
8 changes: 6 additions & 2 deletions packages/kbn-manifest/index.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@

import { run } from '@kbn/dev-cli-runner';
import { listManifestFiles, printManifest, updateManifest } from './manifest';
import { relocateModules } from './relocate';

/**
* A CLI to manipulate Kibana package manifest files
Expand All @@ -17,7 +18,9 @@ export const runKbnManifestCli = () => {
run(
async ({ log, flags }) => {
if (flags.list === 'all') {
listManifestFiles(flags, log);
await listManifestFiles(flags, log);
} else if (typeof flags.relocate === 'string' && flags.relocate!.length > 0) {
await relocateModules(flags.relocate, log);
} else {
if (!flags.package && !flags.plugin) {
throw new Error('You must specify the identifer of the --package or --plugin to update.');
Expand All @@ -31,10 +34,11 @@ export const runKbnManifestCli = () => {
defaultLevel: 'info',
},
flags: {
string: ['list', 'package', 'plugin', 'set', 'unset'],
string: ['list', 'relocate', 'package', 'plugin', 'set', 'unset'],
help: `
Usage: node scripts/manifest --package <packageId> --set group=platform --set visibility=private
--list all List all the manifests
--relocate <owner> Relocate all modules (packages and plugins) belonging to the specified owner
--package [packageId] Select a package to update.
--plugin [pluginId] Select a plugin to update.
--set [property]=[value] Set the desired "[property]": "[value]"
Expand Down
71 changes: 69 additions & 2 deletions packages/kbn-manifest/manifest.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,13 +8,14 @@
*/

import { join } from 'path';
import { writeFile } from 'fs/promises';
import { readFile, writeFile } from 'fs/promises';
import { flatMap, unset } from 'lodash';
import { set } from '@kbn/safer-lodash-set';
import type { ToolingLog } from '@kbn/tooling-log';
import type { Flags } from '@kbn/dev-cli-runner';
import { type Package, getPackages } from '@kbn/repo-packages';
import { type Package, getPackages, Jsonc } from '@kbn/repo-packages';
import { REPO_ROOT } from '@kbn/repo-info';
import { existsSync, readFileSync } from 'fs';

const MANIFEST_FILE = 'kibana.jsonc';

Expand Down Expand Up @@ -42,6 +43,72 @@ const getKibanaJsonc = (flags: Flags, log: ToolingLog): Package[] => {
);
};

interface TsConfigObject {
kbn_references: string[];
}

export const readTsConfig = async (directory: string): Promise<any | undefined> => {
const tsConfigPath = join(directory, 'tsconfig.json');
if (existsSync(tsConfigPath)) {
const tsConfigString = await readFile(tsConfigPath);
const tsConfig = Jsonc.parse(tsConfigString.toString()) as TsConfigObject;
return tsConfig;
}
};

export const writeTsConfig = async (directory: string, data: object) => {
const tsConfigPath = join(directory, 'tsconfig.json');
return await writeFile(tsConfigPath, JSON.stringify(data, null, 2));
};

export const dependsOn = (potentialDependency: Package, module: Package): boolean => {
const tsConfigPath = join(potentialDependency.directory, 'tsconfig.json');
if (existsSync(tsConfigPath)) {
const tsConfigString = readFileSync(tsConfigPath);
const tsConfig = Jsonc.parse(tsConfigString.toString()) as TsConfigObject;
return tsConfig?.kbn_references?.includes(module.id);
} else {
return false;
}
};

export const noDependants = (current: Package): boolean => {
const modules = getPackages(REPO_ROOT);
const dependants = modules.filter((module) => dependsOn(module, current));
return dependants.length === 0;
};

export const checkDependants = (current: Package): boolean => {
const modules = getPackages(REPO_ROOT);
const dependants = modules.filter((module) => dependsOn(module, current));
const solutionDependants = dependants
.filter(
(module) =>
module.group === 'observability' || module.group === 'security' || module.group === 'search'
)
.map((module) => module.group);

if (solutionDependants.length) {
console.log(
'Module',
current.id,
'has the following dependant(s)',
dependants.map((module) => `${module.id} - ${module.group}/${module.visibility}`)
);
}
return true;
// dependants.length > 0 &&
// dependants.every(
// (module) =>
// module.group !== 'observability' && module.group !== 'security' && module.group !== 'search'
// ) &&
// dependants.some(
// (module) =>
// module.group === 'observability' || module.group === 'security' || module.group === 'search'
// )
// dependants.some((module) => module.group === 'platform')
};

export const listManifestFiles = (flags: Flags, log: ToolingLog) => {
const modules = getPackages(REPO_ROOT);
modules
Expand Down
Loading