Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Security Solution] improve endpoint metadata tests #125883

Merged
merged 3 commits into from
May 19, 2022
Merged

[Security Solution] improve endpoint metadata tests #125883

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
f669de0
[Security Solution] improve endpoint metadata tests
joeypoon Feb 16, 2022
ca23845
Merge branch 'main' into chore/better-integ-tests
kibanamachine May 12, 2022
cd4c0f9
Merge branch 'main' into chore/better-integ-tests
kibanamachine May 19, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion x-pack/plugins/security_solution/common/endpoint/data_loaders/index_fleet_agent.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ import { wrapErrorAndRejectPromise } from './utils';
const defaultFleetAgentGenerator = new FleetAgentGenerator();

export interface IndexedFleetAgentResponse {
agents: Agent[];
agents: Array<Agent & FleetServerAgent>;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why did you have to add FleetServerAgent here?

Copy link
Member Author

@joeypoon joeypoon Feb 22, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agent type doesn't contain the agent field which FleetServerAgent does. United transform uses the agent.id field to join on so it's required. Agent type is actually what the fleet agents endpoint is currently typed to return but it actually does also contain the agent field. My guess is that when we asked fleet team to add agent.id to the index for the united transform recently, this api's return type wasn't updated since we were the only ones using the new field and it was via transform.

Ideally, I think we also update the api's return type but that is fleet code so I didn't want to scope creep this PR. I think it'd be best for that to be its own PR since it's potentially going to end up needing other changes.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ahh. Ok. thanks for that explanation. I would suggest asking the Fleet team about this and then creating an issue to track the update if they are good with it.

fleetAgentsIndex: string;
}

Expand Down
68 changes: 54 additions & 14 deletions x-pack/test/security_solution_endpoint/services/endpoint.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ import {
metadataCurrentIndexPattern,
metadataTransformPrefix,
METADATA_UNITED_INDEX,
METADATA_UNITED_TRANSFORM,
} from '@kbn/security-solution-plugin/common/endpoint/constants';
import {
deleteIndexedHostsAndAlerts,
Expand Down Expand Up @@ -77,6 +78,27 @@ export class EndpointTestResources extends FtrService {
await this.transform.api.updateTransform(transform.id, { frequency }).catch(catchAndWrapError);
}

private async stopTransform(transformId: string) {
const stopRequest = {
transform_id: `${transformId}*`,
force: true,
wait_for_completion: true,
allow_no_match: true,
};
return this.esClient.transform.stopTransform(stopRequest);
}

private async startTransform(transformId: string) {
const transformsResponse = await this.esClient.transform.getTransform({
transform_id: `${transformId}*`,
});
return Promise.all(
transformsResponse.transforms.map((transform) => {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I assume the number of transforms here is small and thus we don't need to throttle the calls to *.startTranform() - right?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep. We control these transforms and it's only 2 so no worries there.

return this.esClient.transform.startTransform({ transform_id: transform.id });
})
);
}

/**
* Loads endpoint host/alert/event data into elasticsearch
* @param [options]
Expand All @@ -86,6 +108,8 @@ export class EndpointTestResources extends FtrService {
* @param [options.enableFleetIntegration=true] When set to `true`, Fleet data will also be loaded (ex. Integration Policies, Agent Policies, "fake" Agents)
* @param [options.generatorSeed='seed`] The seed to be used by the data generator. Important in order to ensure the same data is generated on very run.
* @param [options.waitUntilTransformed=true] If set to `true`, the data loading process will wait until the endpoint hosts metadata is processed by the transform
* @param [options.waitTimeout=60000] If waitUntilTransformed=true, number of ms to wait until timeout
* @param [options.customIndexFn] If provided, will use this function to generate and index data instead
*/
async loadEndpointData(
options: Partial<{
Expand All @@ -95,6 +119,8 @@ export class EndpointTestResources extends FtrService {
enableFleetIntegration: boolean;
generatorSeed: string;
waitUntilTransformed: boolean;
waitTimeout: number;
customIndexFn: () => Promise<IndexedHostsAndAlertsResponse>;
}> = {}
): Promise<IndexedHostsAndAlertsResponse> {
const {
Expand All @@ -104,25 +130,39 @@ export class EndpointTestResources extends FtrService {
enableFleetIntegration = true,
generatorSeed = 'seed',
waitUntilTransformed = true,
waitTimeout = 60000,
customIndexFn,
} = options;

if (waitUntilTransformed) {
// need this before indexing docs so that the united transform doesn't
// create a checkpoint with a timestamp after the doc timestamps
await this.stopTransform(METADATA_UNITED_TRANSFORM);
}

// load data into the system
const indexedData = await indexHostsAndAlerts(
this.esClient as Client,
this.kbnClient,
generatorSeed,
numHosts,
numHostDocs,
'metrics-endpoint.metadata-default',
'metrics-endpoint.policy-default',
'logs-endpoint.events.process-default',
'logs-endpoint.alerts-default',
alertsPerHost,
enableFleetIntegration
);
const indexedData = customIndexFn
? await customIndexFn()
: await indexHostsAndAlerts(
this.esClient as Client,
this.kbnClient,
generatorSeed,
numHosts,
numHostDocs,
'metrics-endpoint.metadata-default',
'metrics-endpoint.policy-default',
'logs-endpoint.events.process-default',
'logs-endpoint.alerts-default',
alertsPerHost,
enableFleetIntegration
);

if (waitUntilTransformed) {
await this.waitForEndpoints(indexedData.hosts.map((host) => host.agent.id));
const metadataIds = Array.from(new Set(indexedData.hosts.map((host) => host.agent.id)));
await this.waitForEndpoints(metadataIds, waitTimeout);
await this.startTransform(METADATA_UNITED_TRANSFORM);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If I follow what is going on here correctly, you seem to be missing a .startTransform() at the end of this entire function, since you immediately stopped the transform when this function executes. If waitUntilTransformed is false, then you never restart the transform.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ah yes, good catch. will fix.

const agentIds = Array.from(new Set(indexedData.agents.map((agent) => agent.agent!.id)));
await this.waitForUnitedEndpoints(agentIds, waitTimeout);
}

return indexedData;
Expand Down
9 changes: 0 additions & 9 deletions x-pack/test/security_solution_endpoint_api_int/apis/endpoint_authz.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,6 @@
* 2.0.
*/

import { IndexedHostsAndAlertsResponse } from '@kbn/security-solution-plugin/common/endpoint/index_data';
import { wrapErrorAndRejectPromise } from '@kbn/security-solution-plugin/common/endpoint/data_loaders/utils';
import { FtrProviderContext } from '../ftr_provider_context';
import {
Expand All @@ -15,23 +14,15 @@ import {
} from '../../common/services/security_solution';

export default function ({ getService }: FtrProviderContext) {
const endpointTestResources = getService('endpointTestResources');
const supertestWithoutAuth = getService('supertestWithoutAuth');

describe('When attempting to call an endpoint api with no authz', () => {
let loadedData: IndexedHostsAndAlertsResponse;

before(async () => {
// create role/user
await createUserAndRole(getService, ROLES.t1_analyst);
loadedData = await endpointTestResources.loadEndpointData();
Copy link
Member Author

@joeypoon joeypoon Feb 17, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

since we're only testing for 403s here, the data load seems unnecessary.

});

after(async () => {
if (loadedData) {
await endpointTestResources.unloadEndpointData(loadedData);
}

// delete role/user
await deleteUserAndRole(getService, ROLES.t1_analyst);
});
Expand Down
49 changes: 25 additions & 24 deletions x-pack/test/security_solution_endpoint_api_int/apis/metadata.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,8 @@ import {
import { AGENTS_INDEX } from '@kbn/fleet-plugin/common';
import { indexFleetEndpointPolicy } from '@kbn/security-solution-plugin/common/endpoint/data_loaders/index_fleet_endpoint_policy';
import { TRANSFORM_STATES } from '@kbn/security-solution-plugin/common/constants';
import type { IndexedHostsAndAlertsResponse } from '@kbn/security-solution-plugin/common/endpoint/index_data';

import { generateAgentDocs, generateMetadataDocs } from './metadata.fixtures';
import {
deleteAllDocsFromMetadataCurrentIndex,
Expand Down Expand Up @@ -47,38 +49,37 @@ export default function ({ getService }: FtrProviderContext) {
const numberOfHostsInFixture = 2;

before(async () => {
await stopTransform(getService, `${METADATA_UNITED_TRANSFORM}*`);
await deleteAllDocsFromFleetAgents(getService);
await deleteAllDocsFromMetadataDatastream(getService);
await deleteAllDocsFromMetadataCurrentIndex(getService);
await deleteAllDocsFromIndex(getService, METADATA_UNITED_INDEX);

// generate an endpoint policy and attach id to agents since
// metadata list api filters down to endpoint policies only
const policy = await indexFleetEndpointPolicy(
getService('kibanaServer'),
`Default ${uuid.v4()}`,
'1.1.1'
);
const policyId = policy.integrationPolicies[0].policy_id;
const currentTime = new Date().getTime();
const customIndexFn = async (): Promise<IndexedHostsAndAlertsResponse> => {
// generate an endpoint policy and attach id to agents since
// metadata list api filters down to endpoint policies only
const policy = await indexFleetEndpointPolicy(
getService('kibanaServer'),
`Default ${uuid.v4()}`,
'1.1.1'
);
const policyId = policy.integrationPolicies[0].policy_id;
const currentTime = new Date().getTime();

const agentDocs = generateAgentDocs(currentTime, policyId);
const agentDocs = generateAgentDocs(currentTime, policyId);
const metadataDocs = generateMetadataDocs(currentTime);

await Promise.all([
bulkIndex(getService, AGENTS_INDEX, agentDocs),
bulkIndex(getService, METADATA_DATASTREAM, generateMetadataDocs(currentTime)),
]);
await Promise.all([
bulkIndex(getService, AGENTS_INDEX, agentDocs),
bulkIndex(getService, METADATA_DATASTREAM, metadataDocs),
]);

await endpointTestResources.waitForEndpoints(
agentDocs.map((doc) => doc.agent.id),
60000
);
await startTransform(getService, METADATA_UNITED_TRANSFORM);
await endpointTestResources.waitForUnitedEndpoints(
agentDocs.map((doc) => doc.agent.id),
60000
);
return {
agents: agentDocs,
hosts: metadataDocs,
} as unknown as IndexedHostsAndAlertsResponse;
};

await endpointTestResources.loadEndpointData({ customIndexFn });
});

after(async () => {
Expand Down