Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SNAP 3104] Showing external hive metastore tables as part of HIVETABLES VTI #530

Open
wants to merge 14 commits into
base: snappy/master
Choose a base branch
from

Conversation

vatsalmevada
Copy link

@vatsalmevada vatsalmevada commented Nov 20, 2019

Changes proposed in this pull request

Showing external hive metastore tables as part of HIVETABLES VTI

Approach

Keeping a flag hiveSessionInitialized as part of the profile of the lead node
with the default value set to false.
When the hive session is initialized on the lead node, we set this flag to
true and distribute the updated profile.

When SYS.HIVETABLES VIT is executed on a server node, the server node checks
whether hiveSessionInitialized is set to true in lead's profile. If it is
set to true it executes ExternalHiveTablesCollectorFunction on the lead node
to fetch the metadata of hive tables available on external hive metastore.

ExternalHiveTablesCollectorFunction is only executed when
hiveSessionInitialized is set to true to avoid unnecessary remote function
call.


Note that server need to fetch these details from the leader node as the hive
session is available only one leader.

Patch testing

precheckin -Pstore
Manually tested. Will add automated coverage.

Other PRs

TIBCOSoftware/snappydata#1472

Vatsal Mevada added 6 commits November 18, 2019 13:21
as part of SYS.HIVETABLES VTI.

Pending: Changes are tested by picking first available snappysession
from ConnectionIdToSnappySession map instead of picking the correct
SnappySession for given connection id.
`ExternalHiveTablesCollectorFunction`.
- added copyright headers
…tion

which will be used to retrieve hive tables available in current schema.
@vatsalmevada vatsalmevada requested a review from kneeraj November 20, 2019 13:31
Comment on lines +58 to +64
@Override
public boolean optimizeForWrite() {
return false;
}

//todo[vatsal] : verify this return value
@Override
public boolean isHA() {
return false;
}
Copy link
Author

@vatsalmevada vatsalmevada Nov 20, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Need to verify these return values.


@Override
public void toData(DataOutput out) throws IOException {
InternalDataSerializer.writeArrayLength(tablesMetadata.size(), out);
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Serializing only those fields from ExternalTableMetadata which are required for HiveTablesVTI.

@sumwale sumwale force-pushed the snappy/master branch 2 times, most recently from 337768c to ec453a6 Compare April 8, 2022 19:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant