Added new Kusto ServiceLayer (#1009)

* Copy smoModel some rename

* Copy entire service layer

* Building copy

* Fixing some references

* Launch profile

* Resolve namespace issues

* Compiling tests. Correct manifest.

* Fixing localization resources

* ReliableKustoClient

* Some trimming of extra code and Kusto code

* Kusto client creation in bindingContent

* Removing Smo and new Kusto classes

* More trimming

* Kusto schema hookup

* Solidying DataSource abstraction

* Solidifying further

* Latest refatoring

* More refactoring

* Building and launching Kusto service layer

* Working model which enumerates databases

* Refactoring to pass IDataSource to all tree nodes

* Removing some dependencies on the context

* Working with tables and schema

* Comment checkin

* Refactoring to give out select script

* Query created and sent back to ADS

* Fix query generation

* Fix listing of databases

* Tunneling the query through.

* Successful query execution

* Return only results table

* Deleting Cms

* Delete DacFx

* Delete SchemaCompare and TaskServices

* Change build definition to not stop at launch

* Fix error after merge

* Save Kusto results in different formats (#935)

* save results as csv etc

* some fixes

Co-authored-by: Monica Gupta <mogupt@microsoft.com>

* 2407 Added OrderBy clause in KustoDataSource > GetDatabaseMetaData and GetColumnMetadata (#959)

* 2405 Defaulted Options when setting ServerInfo in ConnectionService > GetConnectionCompleteParams (#965)

* 2747 Fixed IsUnknownType error for Kusto (#989)

* 2747 Removed unused directives in Kusto > DbColumnWrapper. Refactored IsUnknownType to handle null DataTypeName

* 2747 Reverted IsUnknownType change in DbColumnWrapper. Changed DataTypeName to get calue from ColumnType. Refactored SafeGetValue to type check before hard casting to reduce case exceptions.

* Added EmbeddedResourceUseDependentUponConvention to Microsoft.Kusto.ServiceLayer.csproj. Also renamed DACfx to match Microsoft.SqlTools.ServiceLayer. Added to compile Exclude="**/obj/**/*.cs"

* Srahman cleanup sql code (#992)

* Removed Management and Security Service Code.

* Remove FileBrowser service

* Comment why we are using SqlServer library

* Remove SQL specific type definitions

* clean up formatter service (#996)

Co-authored-by: Monica Gupta <mogupt@microsoft.com>

* Code clean up and Kusto intellisense (#994)

* Code clean up and Kusto intellisense

* Addressed few comments

* Addressed few comments

* addressed comments

Co-authored-by: Monica Gupta <mogupt@microsoft.com>

* Return multiple tables for Kusto

* Changes required for Kusto manage dashboard (#1039)

* Changes required for manage dashboard

* Addressed comments

Co-authored-by: Monica Gupta <mogupt@microsoft.com>

* 2728 Kusto function support (#1038)

* loc update (#914)

* loc update

* loc updates

* 2728 moved ColumnInfo and KustoResultsReader to separate files. Added Folder and Function to TreeNode.cs

* 2728 Added FunctionInfo. Added Folder to ColumnInfo. Removed partial class from KustoResultsReader. Set Function.IsAlwaysLeaf=true in TreeNode.cs. In KustoDataSource changed tableMetadata type to TableMetaData. Added folder and function dictionaries. Refactored GetSchema function. Renamed GenerateColumnMetadataKey to GenerateMetadataKey

* 2728 Added FunctionInfo. Added Folder to ColumnInfo. Removed partial class from KustoResultsReader. Set Function.IsAlwaysLeaf=true in TreeNode.cs. In KustoDataSource changed tableMetadata type to TableMetaData. Added folder and function dictionaries. Refactored GetSchema function. Renamed GenerateColumnMetadataKey to GenerateMetadataKey

* 2728 Created new SqlConnection within using block. Refactored KustoDataSource > columnmetadata to sort on get instead of insert.

* 2728 Added GetFunctionInfo function to KustoDataSource.

* 2728 Reverted change to Microsoft.Kusto.ServiceLayer.csproj from merge

* 2728 Reverted change to SqlTools.ServiceLayer\Localization\transXliff

* 2728 Reverted change to sr.de.xlf and sr.zh-hans.xlf

* 2728 Refactored KustoDataSource Function folders to support subfolders

* 2728 Refactored KustoDataSource to use urn for folders, functions, and tables instead of name.

* Merge remote-tracking branch 'origin/main' into feature-ADE

# Conflicts:
#	Packages.props

* 2728 Moved metadata files into Metadata subdirectory. Added GenerateAlterFunction to IDataSource and DataSourceBase.

* 2728 Added summary information to SafeAdd in SystemExtensions. Renamed local variable in SetTableMetadata

* 2728 Moved SafeAdd from SystemExtensions to KustoQueryUtils. Added check when getting database schema to return existing records before querying again. Added AddRange function to KustoQueryUtils. Created SetFolderMetadataForFunctions method.

* 2728 Added DatabaseKeyPrefix to only return tables to a database for the dashboard. Added logic to store all database tables within the tableMetadata dictionary for the dashboard.

* 2728 Created TableInfo and moved info objects into Models directory. Refactored KustoDataSource to lazy load columns for tables. Refactored logic to load tables using cslschema instead of schema.

* 2728 Renamed LoadColumnSchema to GetTableSchema to be consistent.

Co-authored-by: khoiph1 <khoiph@microsoft.com>

* Addressed comments

Co-authored-by: Shafiq Rahman <srahman@microsoft.com>
Co-authored-by: Monica Gupta <mogupt@microsoft.com>
Co-authored-by: Justin M <63619224+JustinMDotNet@users.noreply.github.com>
Co-authored-by: rkselfhost <rkselfhost@outlook.com>
Co-authored-by: khoiph1 <khoiph@microsoft.com>

.vscode/launch.json

@@ -18,6 +18,20 @@
             "stopAtEntry": false,
             "internalConsoleOptions": "openOnSessionStart"
         },
+        {
+            "name": "Kusto service Launch (console)",
+            "type": "coreclr",
+            "request": "launch",
+            "preLaunchTask": "build",
+            // If you have changed target frameworks, make sure to update the program path.
+            "program": "${workspaceFolder}/src/Microsoft.Kusto.ServiceLayer/bin/Debug/netcoreapp3.1/MicrosoftKustoServiceLayer.dll",
+            "args": [],
+            "cwd": "${workspaceFolder}/src/Microsoft.Kusto.ServiceLayer",
+            // For more information about the 'console' field, see https://github.com/OmniSharp/omnisharp-vscode/blob/master/debugger-launchjson.md#console-terminal-window
+            "console": "externalTerminal",
+            "stopAtEntry": false,
+            "internalConsoleOptions": "openOnSessionStart"
+        },
         {
             "name": ".NET Core Attach",
             "type": "coreclr",
@@ -26,6 +40,15 @@
             "requireExactSource": false,
             "justMyCode": false,
             "enableStepFiltering": false
+        },
+        {
+            "name": "Kusto Core Attach",
+            "type": "coreclr",
+            "request": "attach",
+            "processId": "${command:pickProcess}",
+            "requireExactSource": false,
+            "justMyCode": false,
+            "enableStepFiltering": false
         }
     ,]
 }

Packages.props

@@ -20,6 +20,8 @@
 		<PackageReference Update="Microsoft.Data.SqlClient" Version="2.0.0"/>
 		<PackageReference Update="Microsoft.SqlServer.SqlManagementObjects" Version="160.2004021.0" />
 		<PackageReference Update="Microsoft.SqlServer.DACFx" Version="150.4870.3-preview" GeneratePathProperty="true" />
+		<PackageReference Update="Microsoft.Azure.Kusto.Data" Version="8.0.2" />
+		<PackageReference Update="Microsoft.Azure.Kusto.Language" Version="8.1.2"/>
 
 		<PackageReference Update="Moq" Version="4.8.2" />
 		<PackageReference Update="NUnit" Version="3.12.0" />

sqltoolsservice.sln

@@ -104,6 +104,8 @@ Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "Microsoft.SqlTools.ManagedB
 EndProject
 Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "Microsoft.SqlTools.Test.CompletionExtension", "test\Microsoft.SqlTools.Test.CompletionExtension\Microsoft.SqlTools.Test.CompletionExtension.csproj", "{0EC2B30C-0652-49AE-9594-85B3C3E9CA21}"
 EndProject
+Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "Microsoft.Kusto.ServiceLayer", "src\Microsoft.Kusto.ServiceLayer\Microsoft.Kusto.ServiceLayer.csproj", "{E0C941C8-91F2-4BE1-8B79-AC88EDB78729}"
+EndProject
 Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "azure-pipelines", "azure-pipelines", "{48D5C134-4091-438D-9D35-6EB8CF0D1DCC}"
 	ProjectSection(SolutionItems) = preProject
 		azure-pipelines\build.yml = azure-pipelines\build.yml
@@ -249,6 +251,12 @@ Global
 		{0EC2B30C-0652-49AE-9594-85B3C3E9CA21}.Integration|Any CPU.Build.0 = Debug|Any CPU
 		{0EC2B30C-0652-49AE-9594-85B3C3E9CA21}.Release|Any CPU.ActiveCfg = Release|Any CPU
 		{0EC2B30C-0652-49AE-9594-85B3C3E9CA21}.Release|Any CPU.Build.0 = Release|Any CPU
+		{E0C941C8-91F2-4BE1-8B79-AC88EDB78729}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
+		{E0C941C8-91F2-4BE1-8B79-AC88EDB78729}.Debug|Any CPU.Build.0 = Debug|Any CPU
+		{E0C941C8-91F2-4BE1-8B79-AC88EDB78729}.Integration|Any CPU.ActiveCfg = Debug|Any CPU
+		{E0C941C8-91F2-4BE1-8B79-AC88EDB78729}.Integration|Any CPU.Build.0 = Debug|Any CPU
+		{E0C941C8-91F2-4BE1-8B79-AC88EDB78729}.Release|Any CPU.ActiveCfg = Release|Any CPU
+		{E0C941C8-91F2-4BE1-8B79-AC88EDB78729}.Release|Any CPU.Build.0 = Release|Any CPU
 	EndGlobalSection
 	GlobalSection(SolutionProperties) = preSolution
 		HideSolutionNode = FALSE
@@ -278,6 +286,7 @@ Global
 		{3F82F298-700A-48DF-8A69-D048DFBA782C} = {2BBD7364-054F-4693-97CD-1C395E3E84A9}
 		{D3696EFA-FB1E-4848-A726-FF7B168AFB96} = {AB9CA2B8-6F70-431C-8A1D-67479D8A7BE4}
 		{0EC2B30C-0652-49AE-9594-85B3C3E9CA21} = {AB9CA2B8-6F70-431C-8A1D-67479D8A7BE4}
+		{E0C941C8-91F2-4BE1-8B79-AC88EDB78729} = {2BBD7364-054F-4693-97CD-1C395E3E84A9}
 	EndGlobalSection
 	GlobalSection(ExtensibilityGlobals) = postSolution
 		SolutionGuid = {B31CDF4B-2851-45E5-8C5F-BE97125D9DD8}

src/Microsoft.Kusto.ServiceLayer/Admin/AdminService.cs

@@ -0,0 +1,136 @@
+//
+// Copyright (c) Microsoft. All rights reserved.
+// Licensed under the MIT license. See LICENSE file in the project root for full license information.
+//
+
+using System;
+using System.Collections.Generic;
+using System.Linq;
+using System.Threading.Tasks;
+using Microsoft.SqlTools.Hosting.Protocol;
+using Microsoft.Kusto.ServiceLayer.Admin.Contracts;
+using Microsoft.Kusto.ServiceLayer.Connection;
+using Microsoft.Kusto.ServiceLayer.DataSource;
+using Microsoft.Kusto.ServiceLayer.DataSource.Metadata;
+using Microsoft.Kusto.ServiceLayer.Hosting;
+using Microsoft.Kusto.ServiceLayer.Utility;
+
+namespace Microsoft.Kusto.ServiceLayer.Admin
+{
+    /// <summary>
+    /// Admin task service class
+    /// </summary>
+    public class AdminService
+    {
+        private static readonly Lazy<AdminService> instance = new Lazy<AdminService>(() => new AdminService());
+
+        private static ConnectionService connectionService = null;
+
+        /// <summary>
+        /// Default, parameterless constructor.
+        /// </summary>
+        internal AdminService()
+        {
+        }
+
+        /// <summary>
+        /// Internal for testing purposes only
+        /// </summary>
+        internal static ConnectionService ConnectionServiceInstance
+        {
+            get
+            {
+                if (AdminService.connectionService == null)
+                {
+                    AdminService.connectionService = ConnectionService.Instance;
+                }
+                return AdminService.connectionService;
+            }
+
+            set
+            {
+                AdminService.connectionService = value;
+            }
+        }
+
+        /// <summary>
+        /// Gets the singleton instance object
+        /// </summary>
+        public static AdminService Instance
+        {
+            get { return instance.Value; }
+        }
+
+        /// <summary>
+        /// Initializes the service instance
+        /// </summary>
+        public void InitializeService(ServiceHost serviceHost)
+        {
+            serviceHost.SetRequestHandler(GetDatabaseInfoRequest.Type, HandleGetDatabaseInfoRequest);
+        }
+
+        /// <summary>
+        /// Handle get database info request
+        /// </summary>
+        internal static async Task HandleGetDatabaseInfoRequest(
+            GetDatabaseInfoParams databaseParams,
+            RequestContext<GetDatabaseInfoResponse> requestContext)
+        {
+            try
+            {
+                Func<Task> requestHandler = async () =>
+                {
+                    ConnectionInfo connInfo;
+                    AdminService.ConnectionServiceInstance.TryFindConnection(
+                            databaseParams.OwnerUri,
+                            out connInfo);
+                    DatabaseInfo info = null;
+
+                    if (connInfo != null)
+                    {
+                        info = GetDatabaseInfo(connInfo);
+                    }
+
+                    await requestContext.SendResult(new GetDatabaseInfoResponse()
+                    {
+                        DatabaseInfo = info
+                    });
+                };
+
+                Task task = Task.Run(async () => await requestHandler()).ContinueWithOnFaulted(async t =>
+                {
+                    await requestContext.SendError(t.Exception.ToString());
+                });
+
+            }
+            catch (Exception ex)
+            {
+                await requestContext.SendError(ex.ToString());
+            }
+        }
+
+        /// <summary>
+        /// Return database info for a specific database
+        /// </summary>
+        /// <param name="connInfo"></param>
+        /// <returns></returns>
+        internal static DatabaseInfo GetDatabaseInfo(ConnectionInfo connInfo)
+        {
+            if(!string.IsNullOrEmpty(connInfo.ConnectionDetails.DatabaseName)){
+                ReliableDataSourceConnection connection;
+                connInfo.TryGetConnection("Default", out connection);
+                IDataSource dataSource = connection.GetUnderlyingConnection();
+                DataSourceObjectMetadata objectMetadata = DataSourceFactory.CreateClusterMetadata(connInfo.ConnectionDetails.ServerName);
+
+                List<DataSourceObjectMetadata> metadata = dataSource.GetChildObjects(objectMetadata, true).ToList();
+                var databaseMetadata = metadata.Where(o => o.Name == connInfo.ConnectionDetails.DatabaseName);
+
+                List<DatabaseInfo> databaseInfo = DataSourceFactory.ConvertToDatabaseInfo(databaseMetadata);
+
+                return databaseInfo.ElementAtOrDefault(0);
+            }
+
+            return null;
+        }
+    }
+}

src/Microsoft.Kusto.ServiceLayer/Admin/Contracts/DatabaseInfo.cs

@@ -0,0 +1,23 @@
+//
+// Copyright (c) Microsoft. All rights reserved.
+// Licensed under the MIT license. See LICENSE file in the project root for full license information.
+//
+
+using System.Collections.Generic;
+
+namespace Microsoft.Kusto.ServiceLayer.Admin.Contracts
+{
+    public class DatabaseInfo
+    {
+        /// <summary>
+        /// Gets or sets the options
+        /// </summary>
+        public Dictionary<string, object> Options { get; set; }
+
+        public DatabaseInfo()
+        {
+            Options = new Dictionary<string, object>();
+        }
+
+    }
+}

src/Microsoft.Kusto.ServiceLayer/Admin/Contracts/GetDatabaseInfoRequest.cs

@@ -0,0 +1,41 @@
+//
+// Copyright (c) Microsoft. All rights reserved.
+// Licensed under the MIT license. See LICENSE file in the project root for full license information.
+//
+
+using Microsoft.SqlTools.Hosting.Protocol.Contracts;
+
+namespace Microsoft.Kusto.ServiceLayer.Admin.Contracts
+{
+    /// <summary>
+    /// Params for a get database info request
+    /// </summar>
+    public class GetDatabaseInfoParams
+    {
+        /// <summary>
+        /// Uri identifier for the connection to get the database info for
+        /// </summary>
+        public string OwnerUri { get; set; }
+    }
+
+    /// <summary>
+    /// Response object for get database info
+    /// </summary>
+    public class GetDatabaseInfoResponse
+    {
+        /// <summary>
+        /// The object containing the database info
+        /// </summary>
+        public DatabaseInfo DatabaseInfo { get; set; }
+    }
+
+    /// <summary>
+    /// Get database info request mapping
+    /// </summary>
+    public class GetDatabaseInfoRequest
+    {
+        public static readonly
+            RequestType<GetDatabaseInfoParams, GetDatabaseInfoResponse> Type =
+                RequestType<GetDatabaseInfoParams, GetDatabaseInfoResponse>.Create("admin/getdatabaseinfo");
+    }
+}

src/Microsoft.Kusto.ServiceLayer/Connection/CancelTokenKey.cs

@@ -0,0 +1,36 @@
+using System;
+using System.Collections.Generic;
+using System.Linq;
+using System.Threading.Tasks;
+using Microsoft.Kusto.ServiceLayer.Connection.Contracts;
+
+namespace Microsoft.Kusto.ServiceLayer.Connection
+{
+    /// <summary>
+    /// Used to uniquely identify a CancellationTokenSource associated with both
+    /// a string URI and a string connection type.
+    /// </summary>
+    public class CancelTokenKey : CancelConnectParams, IEquatable<CancelTokenKey>
+    {
+        public override bool Equals(object obj)
+        {
+            CancelTokenKey other = obj as CancelTokenKey;
+            if (other == null)
+            {
+                return false;
+            }
+
+            return other.OwnerUri == OwnerUri && other.Type == Type;
+        }
+
+        public bool Equals(CancelTokenKey obj)
+        {
+            return obj.OwnerUri == OwnerUri && obj.Type == Type;
+        }
+
+        public override int GetHashCode()
+        {
+            return OwnerUri.GetHashCode() ^ Type.GetHashCode();
+        }
+    }
+}