English | 中文
DataSphere Studio (DSS for short) is WeDataSphere, a big data platform of WeBank, a self-developed one-stop data application development management portal.
Based on Linkis computation middleware, DSS can easily integrate upper-level data application systems, making data application development simple and easy to use.
DataSphere Studio is positioned as a data application development portal, and the closed loop covers the entire process of data application development. With a unified UI, the workflow-like graphical drag-and-drop development experience meets the entire lifecycle of data application development from data import, desensitization cleaning, data analysis, data mining, quality inspection, visualization, scheduling to data output applications, etc.
With the connection, reusability, and simplification capabilities of Linkis, DSS is born with financial-grade capabilities of high concurrency, high availability, multi-tenant isolation, and resource management.
Please be patient, it will take some time to load gif.
DSS is highly integrated. Currently integrated systems include:
a. Scriptis - Data Development IDE Tool.
b. Visualis - Data Visualization Tool(Based on the open source project Davinci contributed by CreditEase)
c. Qualitis - Data Quality Management Tool
d. Schedulis - Batch workflow job scheduler
f. Exchangis - Data Exchange Tool
AppConn——application connector, defining unified front-end and back-end integration specifications, can quickly and easily integrate with external data application systems, making them as part of DSS data application development.
DSS arranges multiple AppConns in series to form a workflow that supports real-time execution and scheduled execution. Users can complete the entire process development of data applications with simple drag and drop operations.
Since AppConn is integrated with Linkis, the external data application system shares the capabilities of resource management, concurrent limiting, and high performance. AppConn also allows sharable context across system level and completely gets away from application silos.
With Project as the management unit, DSS organizes and manages the business applications of each data application system, and defines a set of common standards for collaborative development of projects across data application systems.
a. Schedulis AppConn —— Batch workflow job scheduler
Many data applications developed by users usually require periodic scheduling capability.
At present, the open source scheduling system in the community is pretty unfriendly to integrate with other data application systems.
DSS implements Schedulis AppConn, which allows users to publish DSS workflows to Azkaban for regular scheduling.
DSS also defines standard and generic workflow parsing and publishing specifications for scheduling systems, allowing other scheduling systems to easily achieve low-cost integration with DSS.
b. Scriptis AppConn —— Data Development IDE Tool
What is Scriptis?
Scriptis is for interactive data analysis with script development(SQL, Pyspark, HiveQL), task submission(Spark, Hive), UDF, function, resource management and intelligent diagnosis.
Scriptis AppConn integrates the data development capabilities of Scriptis to DSS, and allows various script types of Scriptis to serve as nodes in the DSS workflow to participate in the application development process.
Currently supports HiveSQL, SparkSQL, Pyspark, Scala and other script node types.
c. Visualis AppConn —— Data Visualization Tool
What is Visualis?
Visualis is a BI tool for data visualization. It provides financial-grade data visualization capabilities on the basis of data security and permissions, based on the open source project Davinci contributed by CreditEase.
Visualis AppConn integrates data visualization capabilities to DSS, and allows displays and dashboards, as nodes of DSS workflows, to be associated with upstream data market.
d. Qualitis AppConn —— Data quality management Tool
Qualitis AppConn integrates data quality verification capabilities for DSS, allows Qualitis as a node in DSS workflow
e. Data Sender——Sender AppConn
Sender AppConn provides data delivery capability for DSS. Currently it supports the SendEmail node type, and the result sets of all other nodes can be sent via email.
For example, the SendEmail node can directly send the screen shot of a display as an email.
f. Signal AppConn —— Signal Nodes
Signal AppConn is used to strengthen the correlation between business and process while keeping them decoupled.
DataChecker Node：Checks whether a table or partition exists.
EventSender Node: Messaging nodes across workflows and projects.
EventReceiver: Receive nodes for messages across workflows and projects.
g. Function node
Empty nodes, sub workflow nodes.
DSS is an open source project leading the direction of data application development and management. The open source community currently does not have similar products.
DataSphere Studio is suitable for the following scenarios:
1. Scenarios in which big data platform capability is being prepared or initialized but no data application tools are available.
2. Scenarios in which users already have big data foundation platform capabilities but with only a few data application tools.
3. Scenarios in which users have the ability of big data foundation platform and comprehensive data application tools, but suffers strong isolation and and high learning costs because those tools have not been integrated together.
4. Scenarios in which users have the capabilities of big data foundation platform and comprehensive data application tools. but lacks unified and standardized specifications, while a part of these tools have been integrated.
Click to Quick start
Quick integration with DSS for external systems
DSS is under the Apache 2.0 license. See the License file for details.