[UNR-2003] Create CookAndGenerateSchemaCommandlet #1342

mattyoung-improbable · 2019-09-10T16:01:14Z

Description

This PR Includes:

A new CookAndGenerateSchemaCommentlet that will run a regular Unreal Cook but also generate schema for any supported classes that get created during the cook.
Changed what is considered a supported for schema generation to be only classes that are explicitly SpatialTypes. (All AActor, UActorComponent, UGameplayAbility, UAttributeSet) subclasses are explicitly SpatialTypes.
NOTE Any users that have custom replicated subobjects will need to make sure they are marked as SpatialType (either with UClass specifier or in the blueprint class properties window).

Engine PR: https://github.com/improbableio/UnrealEngine/pull/250

… schema but appears to be missing Data-Only Blueprints

… of things missed. Also filtering UObjects to replicate based on IsSupportedForNetworking

…nent are explicit by default). Also use objectiterator with FObjectListener directly.

… explicit flag is present as a workaround for stale blueprints

improbable-prow-robot · 2019-09-10T16:01:16Z

Corresponding JIRA ticket: https://improbableio.atlassian.net/browse/UNR-2003

SpatialGDK/Source/SpatialGDKEditor/Private/SchemaGenerator/SpatialGDKEditorSchemaGenerator.cpp

m-samiec · 2019-09-12T14:15:21Z

SpatialGDK/Source/SpatialGDKEditor/Private/SchemaGenerator/SpatialGDKEditorSchemaGenerator.cpp

-	for (TObjectIterator<UClass> ClassIt; ClassIt; ++ClassIt)
+	if (SupportedClass->IsEditorOnly())
+	{
+		UE_LOG(LogSpatialGDKSchemaGenerator, Verbose, TEXT("[%s] Editor-only Class Not supported for schema gen"), *GetPathNameSafe(SupportedClass));


The capitalization of this message is triggering me.

SpatialGDK/Source/SpatialGDKEditor/Private/SchemaGenerator/SpatialGDKEditorSchemaGenerator.cpp

m-samiec · 2019-09-12T14:16:50Z

SpatialGDK/Source/SpatialGDKEditor/Private/SchemaGenerator/SpatialGDKEditorSchemaGenerator.cpp

+	return Classes;
+}
+
+TSet<UClass*> FilterClasses(TSet<UClass*> Classes)


Suggested change

TSet<UClass*> FilterClasses(TSet<UClass*> Classes)

TSet<UClass*> FilterClasses(const TSet<UClass*>& Classes)

m-samiec · 2019-09-12T14:19:19Z

SpatialGDK/Source/SpatialGDKEditor/Private/SchemaGenerator/SpatialGDKEditorSchemaGenerator.cpp

+	return true;
+}
+
+SPATIALGDKEDITOR_API bool SpatialGDKGenerateSchemaForClasses(TSet<UClass*> Classes)


Suggested change

SPATIALGDKEDITOR_API bool SpatialGDKGenerateSchemaForClasses(TSet<UClass*> Classes)

bool SpatialGDKGenerateSchemaForClasses(TSet<UClass*> Classes)

m-samiec · 2019-09-12T14:19:40Z

SpatialGDK/Source/SpatialGDKEditor/Private/SchemaGenerator/SpatialGDKEditorSchemaGenerator.cpp

 	{
+		if (SchemaGeneratedClasses.Contains(Class)) continue;


nit: New line please

m-samiec · 2019-09-12T14:21:42Z

SpatialGDK/Source/SpatialGDKEditor/Private/SchemaGenerator/SpatialGDKEditorSchemaGenerator.cpp

 	{
+		if (SchemaGeneratedClasses.Contains(Class)) continue;


Should SchemaGeneratedClasses get cleared before this loop?

Generate Schema is being called in batches, so it's useful to cache generated classes between calls. This code really needs cleaning up to encapsulate this state properly, most likely as part of the testing work.

m-samiec · 2019-09-12T14:37:57Z

...DK/Source/SpatialGDKEditorCommandlet/Private/Commandlets/CookAndGenerateSchemaCommandlet.cpp

+	}
+
+	UE_LOG(LogCookAndGenerateSchemaCommandlet, Display, TEXT("Generate Schema for C++ and in-memory Classes."));
+	if (!SpatialGDKGenerateSchema(false /* bSaveSchemaDatabase */, false /* bRunSchemaCompiler */))


Why not do this at the end with the blueprint classes?

Instead of doing this I've used a class iterator to add the classes to ReferencedClasses upfront.

…nerate-schema-commandlet

m-samiec · 2019-09-23T12:01:31Z

SpatialGDK/Source/SpatialGDKEditor/Private/SchemaGenerator/SpatialGDKEditorSchemaGenerator.cpp

@@ -410,47 +410,90 @@ bool SaveSchemaDatabase()
 	return true;
 }

-TArray<UClass*> GetAllSupportedClasses()
+bool IsSupportedClass(UClass* SupportedClass)


Suggested change

bool IsSupportedClass(UClass* SupportedClass)

bool IsSupportedClass(const UClass* SupportedClass)

m-samiec · 2019-09-23T12:58:11Z

SpatialGDK/Source/SpatialGDKEditor/Private/SchemaGenerator/SpatialGDKEditorSchemaGenerator.cpp

@@ -684,21 +727,65 @@ bool RunSchemaCompiler()
 	}
 }

-bool SpatialGDKGenerateSchema()
+bool SpatialGDKGenerateSchema(bool bSaveSchemaDatabase, bool bRunSchemaCompiler)


Worth considering removing parameter options

m-samiec · 2019-09-23T13:01:06Z

SpatialGDK/Source/SpatialGDKEditor/Private/SchemaGenerator/SpatialGDKEditorSchemaGenerator.cpp


-	return RunSchemaCompiler();
+	return true;
 }

 #undef LOCTEXT_NAMESPACE


Move outside namespaces

m-samiec · 2019-09-23T13:05:23Z

SpatialGDK/Source/SpatialGDKEditor/Private/SchemaGenerator/SpatialGDKEditorSchemaGenerator.cpp

+		{
+			if (UClass* NestedClass = Cast<UClass>(TypeNode->Type))
+			{
+				if (!Classes.Contains(NestedClass) && !SchemaGeneratedClasses.Contains(NestedClass) && IsSupportedClass(NestedClass))


Suggested change

if (!Classes.Contains(NestedClass) && !SchemaGeneratedClasses.Contains(NestedClass) && IsSupportedClass(NestedClass))

if (!SchemaGeneratedClasses.Contains(NestedClass) && IsSupportedClass(NestedClass))

…e KismetCompiler

…lType now that semantics have changed

m-samiec · 2019-09-24T14:29:26Z

SpatialGDK/Source/SpatialGDK/Private/EngineClasses/SpatialNetDriver.cpp

@@ -994,7 +994,8 @@ void USpatialNetDriver::ServerReplicateActors_ProcessPrioritizedActors(UNetConne
 				// or it's an editor placed actor and the client hasn't initialized the level it's in
 				if (Channel == nullptr && GuidCache->SupportsObject(Actor->GetClass()) && GuidCache->SupportsObject(Actor->IsNetStartupActor() ? Actor : Actor->GetArchetype()))
 				{
-					if (Actor->GetClass()->HasAnySpatialClassFlags(SPATIALCLASS_NotSpatialType))
+					if (Actor->GetClass()->HasAnySpatialClassFlags(SPATIALCLASS_NotSpatialType) ||
+						!Actor->GetClass()->HasAnySpatialClassFlags(SPATIALCLASS_ExplicitSpatialType))


I believe these flags are mutually exclusive. So we should just be able to check for the existence of ExplicitSpatialType.

Good Spot. Done

…it's mutually exclusive to SPATIALCLASS_NotSpatialType

…com:spatialos/UnrealGDK into feature/cook-and-generate-schema-commandlet

m-samiec · 2019-09-25T09:54:35Z

SpatialGDK/Source/SpatialGDKEditor/Private/SchemaGenerator/SpatialGDKEditorSchemaGenerator.cpp

+		return false;
+	}
+
+	// User told us to ignore this class


Remove this section

…ove redundant check for NotSpatialType

m-samiec · 2019-09-25T16:30:18Z

...DK/Source/SpatialGDKEditorCommandlet/Private/Commandlets/CookAndGenerateSchemaCommandlet.cpp

+
+	UE_LOG(LogCookAndGenerateSchemaCommandlet, Display, TEXT("Schema Generation Finished in %.2f seconds"), Duration.GetTotalSeconds());
+
+	if (!SaveSchemaDatabase())


Do we have a ticket to iterate on this process so we don't have to run the cook command multiple times?

...DK/Source/SpatialGDKEditorCommandlet/Private/Commandlets/CookAndGenerateSchemaCommandlet.cpp

m-samiec

Nice!

…ts/CookAndGenerateSchemaCommandlet.cpp Co-Authored-By: improbable-valentyn <32096431+improbable-valentyn@users.noreply.github.com>

mattyoung-improbable added 12 commits August 20, 2019 17:00

Initial commit of CookAndGenerateSchemaCommandlet, generating correct…

73d917c

… schema but appears to be missing Data-Only Blueprints

Wrap generate call in an async task to avoid blocking.

c54c9a0

New Implementation using UObjectArray callback instead, catches a lot…

1e81ed0

… of things missed. Also filtering UObjects to replicate based on IsSupportedForNetworking

Avoid adding classes twice during generate. Tidy up

01a64e9

Only generate schema for explicit spatial types (AActor & UActorCompo…

c39eed9

…nent are explicit by default). Also use objectiterator with FObjectListener directly.

Check for explicit Spatial flag, also double check parent class if no…

e219ba9

… explicit flag is present as a workaround for stale blueprints

Downgrade logs to verbose. Add Schema Gen Timer

6354a23

Cleanup

1098e5f

More Cleanup

994ba4d

Merge branch 'master' into feature/cook-and-generate-schema-commandlet

8baa1a4

Add back in merge error

7850ac7

Update UnrealEngine Version for CI

ea60fc8

improbable-prow-robot added jira/UNR size/XL Denotes a PR that changes 300-599 lines, ignoring generated files. labels Sep 10, 2019

mattyoung-improbable added 5 commits September 10, 2019 17:11

Merge branch 'master' into feature/cook-and-generate-schema-commandlet

de65361

Merge branch 'master' into feature/cook-and-generate-schema-commandlet

6181d43

Remove Newline

62d744d

Merge branch 'master' into feature/cook-and-generate-schema-commandlet

c9bb1a5

Fix merge issues

75dfdba

m-samiec reviewed Sep 12, 2019

View reviewed changes

SpatialGDK/Source/SpatialGDKEditor/Private/SchemaGenerator/SpatialGDKEditorSchemaGenerator.cpp Outdated Show resolved Hide resolved

m-samiec reviewed Sep 12, 2019

View reviewed changes

SpatialGDK/Source/SpatialGDKEditor/Private/SchemaGenerator/SpatialGDKEditorSchemaGenerator.cpp Show resolved Hide resolved

m-samiec reviewed Sep 12, 2019

View reviewed changes

mattyoung-improbable added 2 commits September 16, 2019 11:34

Merge remote-tracking branch 'origin/master' into feature/cook-and-ge…

a666306

…nerate-schema-commandlet

PR Comments

3a88165

m-samiec reviewed Sep 23, 2019

View reviewed changes

mattyoung-improbable added 3 commits September 23, 2019 15:40

PR Comments

d2d850b

Remove recursive check now that blueprint class flags are fixed in th…

d3dc19a

…e KismetCompiler

Update checks for NotSpatialType to include checks for ExplicitSpatia…

73dc6d9

…lType now that semantics have changed

mattyoung-improbable requested a review from m-samiec September 24, 2019 14:25

Merge branch 'master' into feature/cook-and-generate-schema-commandlet

b1f06ff

m-samiec reviewed Sep 24, 2019

View reviewed changes

mattyoung-improbable added 2 commits September 24, 2019 15:37

Only check for the absence of SPATIALCLASS_ExplicitSpatialType since …

8efafd4

…it's mutually exclusive to SPATIALCLASS_NotSpatialType

Merge branch 'feature/cook-and-generate-schema-commandlet' of github.…

fe18624

…com:spatialos/UnrealGDK into feature/cook-and-generate-schema-commandlet

mattyoung-improbable requested a review from m-samiec September 24, 2019 14:38

m-samiec reviewed Sep 25, 2019

View reviewed changes

mattyoung-improbable added 2 commits September 25, 2019 15:15

rename SPATIALCLASS_ExplicitSpatialType SPATIALCLASS_SpatialType, rem…

fb7181e

…ove redundant check for NotSpatialType

Merge branch 'master' into feature/cook-and-generate-schema-commandlet

25b3e64

mattyoung-improbable requested a review from m-samiec September 25, 2019 14:18

mattyoung-improbable added 2 commits September 25, 2019 15:56

Add CHANGELOG

df40810

Increment RequiresSetup

cee0d59

m-samiec reviewed Sep 25, 2019

View reviewed changes

improbable-valy reviewed Sep 25, 2019

View reviewed changes

...DK/Source/SpatialGDKEditorCommandlet/Private/Commandlets/CookAndGenerateSchemaCommandlet.cpp Outdated Show resolved Hide resolved

Better error when no generated schema for class

9c808e1

m-samiec approved these changes Sep 25, 2019

View reviewed changes

mattyoung-improbable and others added 3 commits September 25, 2019 17:41

Update engine-version

fed4611

Update SpatialGDK/Source/SpatialGDKEditorCommandlet/Private/Commandle…

c4fca92

…ts/CookAndGenerateSchemaCommandlet.cpp Co-Authored-By: improbable-valentyn <32096431+improbable-valentyn@users.noreply.github.com>

remove newline from engine version

cf916e4

improbable-valy approved these changes Sep 25, 2019

View reviewed changes

Actually remove the newline this time

2fe8e34

m-samiec merged commit e3e8469 into master Sep 26, 2019

m-samiec deleted the feature/cook-and-generate-schema-commandlet branch September 26, 2019 09:49

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[UNR-2003] Create CookAndGenerateSchemaCommandlet #1342

[UNR-2003] Create CookAndGenerateSchemaCommandlet #1342

mattyoung-improbable commented Sep 10, 2019 •

edited

Loading

improbable-prow-robot commented Sep 10, 2019

m-samiec Sep 12, 2019

mattyoung-improbable Sep 16, 2019

m-samiec Sep 12, 2019 •

edited

Loading

mattyoung-improbable Sep 16, 2019

m-samiec Sep 12, 2019

mattyoung-improbable Sep 16, 2019

m-samiec Sep 12, 2019

mattyoung-improbable Sep 16, 2019

m-samiec Sep 12, 2019

mattyoung-improbable Sep 16, 2019

m-samiec Sep 12, 2019

mattyoung-improbable Sep 16, 2019

m-samiec Sep 23, 2019

mattyoung-improbable Sep 24, 2019

m-samiec Sep 23, 2019

mattyoung-improbable Sep 24, 2019

m-samiec Sep 23, 2019

mattyoung-improbable Sep 24, 2019

m-samiec Sep 23, 2019

mattyoung-improbable Sep 24, 2019

m-samiec Sep 24, 2019

mattyoung-improbable Sep 24, 2019

m-samiec Sep 25, 2019

mattyoung-improbable Sep 25, 2019

m-samiec Sep 25, 2019

m-samiec left a comment

	TSet<UClass> FilterClasses(TSet<UClass> Classes)
	TSet<UClass> FilterClasses(const TSet<UClass>& Classes)

	SPATIALGDKEDITOR_API bool SpatialGDKGenerateSchemaForClasses(TSet<UClass*> Classes)
	bool SpatialGDKGenerateSchemaForClasses(TSet<UClass*> Classes)

	bool IsSupportedClass(UClass* SupportedClass)
	bool IsSupportedClass(const UClass* SupportedClass)

	if (!Classes.Contains(NestedClass) && !SchemaGeneratedClasses.Contains(NestedClass) && IsSupportedClass(NestedClass))
	if (!SchemaGeneratedClasses.Contains(NestedClass) && IsSupportedClass(NestedClass))


		UE_LOG(LogCookAndGenerateSchemaCommandlet, Display, TEXT("Schema Generation Finished in %.2f seconds"), Duration.GetTotalSeconds());

		if (!SaveSchemaDatabase())

[UNR-2003] Create CookAndGenerateSchemaCommandlet #1342

[UNR-2003] Create CookAndGenerateSchemaCommandlet #1342

Conversation

mattyoung-improbable commented Sep 10, 2019 • edited Loading

Description

improbable-prow-robot commented Sep 10, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

m-samiec Sep 12, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

m-samiec left a comment

Choose a reason for hiding this comment

mattyoung-improbable commented Sep 10, 2019 •

edited

Loading

m-samiec Sep 12, 2019 •

edited

Loading