Skip to content

Latest commit

 

History

History
259 lines (249 loc) · 40.2 KB

CHANGES.md

File metadata and controls

259 lines (249 loc) · 40.2 KB

InLong Changelog

Release InLong 1.10.0 - Released (as of 2023-12-05)

Agent

ISSUE Summary
INLONG-9089 [Improve][Agent] Add enums for task and instance
INLONG-9091 [Improve][Agent] Add offset profile
INLONG-9094 [Umbrella][Agent] Reconfiguration of task management
INLONG-9102 [Improve][Agent] Add file utils
INLONG-9112 [Improve][Agent] Add task and instance profile
INLONG-9117 [Improve][Agent] Rewrite class RocksDbImp to enable it to be constructed with a child path
INLONG-9120 [Improve][Agent] Add offset db to store the offset data
INLONG-9122 [Improve][Agent] Add task and instance action
INLONG-9124 [Improve][Agent] Add task and instance db
INLONG-9125 [Improve][Agent] Add offset manager
INLONG-9132 [Improve][Agent] Add file used message cache
INLONG-9134 [Improve][Agent] Add file related utils
INLONG-9136 [Improve][Agent] Add instance manager
INLONG-9138 [Improve][Agent] Add task manager
INLONG-9143 [Improve][Agent] Add log file collect task
INLONG-9149 [Improve][Agent] Add sender manager for file collect
INLONG-9151 [Improve][Agent] Add log file source and source related modification
INLONG-9155 [Improve][Agent] Add file used proxy
INLONG-9159 [Improve][Agent] Add file instance
INLONG-9161 [Improve][Agent] Modify left sink
INLONG-9163 [Improve][Agent] Delete trigger related file
INLONG-9165 [Improve][Agent] Delete job related file
INLONG-9182 [Improve][Agent] Delete useless code
INLONG-9187 [Improve][Agent] Delete useless memory manager
INLONG-9190 [Bug][Agent] Log file source clear buffer queue does not take effect
INLONG-9194 [Bug][Agent] Calc time offset failed if the param is 0
INLONG-9200 [Bug][Agent] Duplicate file collect instance
INLONG-9207 [Bug][Agent] Task manager stuck
INLONG-9214 [Improve][Agent] Limit max file count to collect once
INLONG-9215 [Improve][Agent] Add predefine fields
INLONG-9233 [Bug][Agent] Fix bug: source, proxy, sender get stuck
INLONG-9237 [Improve][Agent] Move addictive fields to package attributes
INLONG-9241 [Improve][Agent] Print task and instance detail every ten seconds
INLONG-9244 [Bug][Agent] Fix bug: miss file from next data time
INLONG-9253 [Bug][Agent] Get byte position of file by line count offset failed
INLONG-9263 [Improve][Agent] Print the statistics of task and instance not detail
INLONG-9265 [Improve][Agent] Add audit of agent send success
INLONG-9267 [Bug][Agent] Data loss when there are many files to read once
INLONG-9284 [Improve][Agent] Report audit by data time not real time
INLONG-9286 [Improve][Agent] Adjust the time offset calculation function
INLONG-9289 [Improve][Agent] Improve the completion judgment logic of collecting instances
INLONG-9300 [Improve][Agent] Divide data time into source time and sink time
INLONG-9308 [Improve][Agent] The sink end of the file instance supports sending data with different streamIds
INLONG-9310 [Improve][Agent] Add extended handler in file source
INLONG-9312 [Improve][Agent] Add data content style
INLONG-9315 [Improve][Agent] Convert data time from source time zone to sink time zone.
INLONG-9317 [Improve][Agent] Print basic info of hearbeat
INLONG-9335 [Improve][Agent] Bring cycle parameters when creating an instance
INLONG-9338 [Improve][Agent] Real time file collection uses the current time as the data time
INLONG-9347 [Improve][Agent] Check task profile before save into db
INLONG-9364 [Improve][Agent] Remove expired instance from db
INLONG-9366 [Improve][Agent] Remove useless offset record
INLONG-9369 [Improve][Agent] Increase sending failure audit and real-time audit
INLONG-9375 [Improve][Agent] Modify the agent's real-time audit id to prevent duplication
INLONG-9390 [Improve][Agent] Collect supplementary data in chronological order
INLONG-9397 [Improve][Agent] Do not directly delete the instance records of the local db when stopping the instances

Audit

ISSUE Summary
INLONG-9225 [Improve][Audit] Automatically create audit topic after service startup

Dashboard

ISSUE Summary
INLONG-8983 [Improve][Dashboard] Log component query optimization
INLONG-9016 [Improve][Dashboard] Change the wrapWithInlongMsg to wrapType in data stream
INLONG-9053 [Improve][Dashboard] Query list shows creators and modifiers
INLONG-9055 [Improve][Dashboard] The approval management list displays the name of the consumer group
INLONG-9107 [Improve][Dashboard] Navigation widget optimization
INLONG-9114 [Improve][Dashboard] Stream log query parameter optimization
INLONG-9185 [Feature][Dashboard] Cluster management supports sort_cls, sort_es, sort_pulsar types
INLONG-9196 [Improve][Dashboard] Navigation widget user manual optimization
INLONG-9219 [Improve][Dashboard] Stream advanced options parameter optimization
INLONG-9229 [Improve][Dashboard] Data access transmission delay optimization
INLONG-9256 [Improve][Dashboard] Data synchronization iceberg sink supports append Mode
INLONG-9260 [Improve][Dashboard] Support batch import in Synchronize dashboard and detail page
INLONG-9302 [Improve][Dashboard] Elasticsearch sink optimization
INLONG-9322 [Improve][Dashboard] Sink field mapping title format is unified
INLONG-9354 [Improve][Dashboard] Data access File sources optimization
INLONG-9368 [Improve][Dashboard] Creation time or other time display in an easily readable format
INLONG-9383 [Improve][Dashboard] Resource details display cluster label information
INLONG-9385 [Improve][Dashboard] Create a new cluster and remove the agent and dataproxy types
INLONG-9427 [Improve][Dashboard] Data access supports viewing operation logs

Manager

ISSUE Summary
INLONG-8958 [Bug][Manager]Create cls topic fail
INLONG-8960 [Feature][Manager] Automatically assign sort cluster after creating stream sinks
INLONG-8964 [Feature][Manager] Support Sortstandalone cluster management
INLONG-8975 [Improve][Manager] The manager issues an audit IDs based on the group mode
INLONG-8986 [Bug][Manager] Sort standalone get params from manager error
INLONG-8988 [Improve][Manager] Supports multiple wrap types for message body
INLONG-8990 [Improve][Manager][DataProxy][SDK] Rename DataProxyMsgEncType to Message WrapType
INLONG-8992 [Bug][Manager] NPE caused by not using transform
INLONG-8995 [Improve][Manager] Add an interface for querying used task information for agent
INLONG-9003 [Feature][DataProxy][Manager] Support extend fields
INLONG-9017 [Improve][Manager] Add stop interface in stream source API
INLONG-9023 [Bug][Manager] Failed to stop job task
INLONG-9027 [Bug][Manager] Fix SQL error in ClusterSetMapper.selectInlongStreamId
INLONG-9036 [Improve][Manager] Support create tables for dataSync
INLONG-9041 [Improve][Manager] Support saving schema information when saving iceberg source
INLONG-9051 [Improve][Manager] Add consumer group in the approval form
INLONG-9060 [Bug][Manager] Manager return wrong sink configuration to sort standalone
INLONG-9062 [Improve][Manager] Add default values for stream related interface parameters
INLONG-9066 [Improve][Manager] Renaming group related states
INLONG-9069 [Improve][Manager] Filter out invalid configs when organize SortStandalone configuration
INLONG-9070 [Improve][Manager] Add MessageWrapType.forType method for MessageWrapType
INLONG-9073 [Improve][Manager] Add getStreamBriefInfo method for InlongStreamClient
INLONG-9092 [Improve][Manager] Tube supports inlong-msg format
INLONG-9098 [Improve][Manager] Support to save additional info for the Iceberg field
INLONG-9106 [Improve][Manager] Support configuring multiple job tasks under a group
INLONG-9109 [Bug][Manager] Incorrect URL address for getbrief mapping in streamApi
INLONG-9147 [Improve][Manager] The group details are displayed with the tag of the cluster to which the group belongs
INLONG-9192 [Feature][Manager] Flat Sort Cluster types
INLONG-9203 [Bug][Manager][Sort] Failed to use Iceberg sink
INLONG-9205 [Bug][Manager] Failed to use Hudi sink
INLONG-9209 [Improve][Manager] Support configuring predefined fields and issuing agents
INLONG-9211 [Bug][Manager] Redis cannot get the clusterMode from sink
INLONG-9240 [Improve][Sort][Manager] Add options for Iceberg connector
INLONG-9248 [Improve][Manager] Supports configuring builtIn fields for tube source and pulsar source
INLONG-9250 [Improve][Manager] Add auditId for tube, pulsar, and mysql
INLONG-9259 [Feature][Manager] Optimize Elasticsearch sink and datanode
INLONG-9269 [Bug][Manager] Get SortClusterConfig is empty,when sink params include non-string type
INLONG-9280 [Feature][Manager] Support different size of extended fields of InlongStream
INLONG-9285 [Bug][Manager] When creating KafkaSource, the autoOffsetReset param cannot be empty
INLONG-9297 [Feature][Manager] Support configuring multiple sink types of tasks under a single stream
INLONG-9303 [Feature][Manager]Support Tube MQ sink
INLONG-9314 [Feature][Manager] Support cluster switch for InlongGroup
INLONG-9318 [Improve][Manager] ManagerClient supports pulling clusters based on tenant roles
INLONG-9328 [Improve][Manager] Add parameters validation for the updateAuditSource method
INLONG-9330 [Improve][Manager] Add encoding check to the StarRocks JDBC URL
INLONG-9337 [Improve][Manager] Support querying operation records
INLONG-9343 [Improve][Manager] Support configuring timeZone related parameters for fileSource
INLONG-9351 [Improve][Manager] Support querying audit data size
INLONG-9358 [Bug][Manager] The creation time of the information in the database differs from the creation time in the returned information by eight hours
INLONG-9362 [Improve][Manager] Support the configuration of parameters related to the migration of the entire Iceberg database
INLONG-9373 [Bug][Manager] Failed to create namespace
INLONG-9400 [Bug][Manager] Error obtaining sort task type
INLONG-9433 [Bug][Manager] NPE exception encountered while querying audit information
INLONG-9444 [Bug][Manager] ES sink unsuccessful flow status configuration successful
INLONG-9447 [Bug][Manager] Suspend group failed

SDK

ISSUE Summary
INLONG-8990 [Improve][Manager][DataProxy][SDK] Rename DataProxyMsgEncType to Message WrapType
INLONG-9007 [Improve][SDK] ClientList out of bounds
INLONG-9044 [Improve][SDK] Add server response log to facilitate troubleshooting
INLONG-9058 [Improve][SDK] Limit the number of inlong-groupid and inlong-streamid of a single SDK instance
INLONG-9079 [Improve][SDK] Shaded some dependency to reduce conflicts with other sdk
INLONG-9083 [Improve][SDK] Bump up Golang SDK x/net modules
INLONG-9167 [Improve][SDK] Use UUID as the batch ID instead of snowflake ID for DataProxy Golang SDK
INLONG-9170 [Improve][SDK] Use pointer instead of object for chans in Golang SDK
INLONG-9172 [Improve][SDK] comment/delete some debug log in some frequently called methods for Golang SDK
INLONG-9174 [Improve][SDK] Improve response attr parsing in Golang SDK
INLONG-9176 [Improve][SDK] Fail fast when worker is unavailable in Golang SDK
INLONG-9178 [Improve][SDK] Update the default values of the config options of Golang SDK
INLONG-9180 [Improve][SDK] Cache up batchReq.dataReqs
INLONG-9184 [Improve][SDK] Update README.md in Golang SDK
INLONG-9213 [Improve][SDK] Support isolation by inlong groupid
INLONG-9228 [Improve][SDK] CPP SDK supports dynamic load balancing
INLONG-9277 [Feature][SDK] Optimize multi-region nearby access
INLONG-9293 [Improve][SDK] Optimize the problem that the more inlong grouids there are, the more memory is consumed.
INLONG-9307 [Improve][SDK] Improve DataProxy SDK code readability
INLONG-9320 [Improve][SDK] Support local disaster recovery manager configuration
INLONG-9324 [Improve][SDK] Supports automatic creation of custom log directories
INLONG-9341 [Improve][SDK] Optimize obtaining local IP information
INLONG-9345 [Bug][SDK] DataProxy SDK throws java.lang.NoClassDefFoundErrorException
INLONG-9355 [Improve][SDK] Optimize resource isolation for CPP SDK
INLONG-9378 [Improve][SDK] Optimize proxy configuration update

Sort

ISSUE Summary
INLONG-8959 [Feature][Sort]Add mongodb connector on flink 1.15
INLONG-8973 [Bug][Sort] Fix PRIMARY KEY desc error in redis readme.
INLONG-8979 [Bug][Sort] Wrong desc about redis-conenctor in flink 1.13 module about command option
INLONG-8982 [Feature][Sort] Iceberg sink on flink 1.15
INLONG-8997 [Improve][Sort] Keep the logic the same with flink v1.15 mysql in flink v1.13 end to end test
INLONG-8998 [Improve][Sort] Add sqlserver connector of flink 1.15
INLONG-8999 [Feature][Sort] Add sqlserver connector of flink 1.15
INLONG-9009 [Feature][Sort] Add HBase connector on flink 1.15
INLONG-9038 [Bug][Sort] Fail to parse InlongMessage when exporting Pulsar
INLONG-9075 [Improve][Sort] TubeMQSource support InlongMsg format
INLONG-9077 [Bug][Sort] TubeMQ connector fail to subscribe streamId
INLONG-9084 [Bug][Sort] Fail to parse InlongGroupId when report Inlong Audit
INLONG-9087 [Improve][Sort] TubeMQ Connector use latest offset mode
INLONG-9128 [Bug][Sort] Failed to init TubeMQ source with InlongMsg type message
INLONG-9203 [Bug][Manager][Sort] Failed to use Iceberg sink
INLONG-9223 [Feature][Sort] TubeMQ souce support InlongAudit
INLONG-9231 [Bug][Sort] Find no audit time field when the filed is in upper case
INLONG-9240 [Improve][Sort][Manager] Add options for Iceberg connector
INLONG-9246 [Improve][Sort] Pulsar source support audit when the deserialized type is not InlongMsg
INLONG-9247 [Improve][Sort] TubeMQ source support audit when the deserialized type is not InlongMsg
INLONG-9273 [Bug][Sort] IcebergSingleFileCommiter will throw exception for error code
INLONG-9333 [Bug][Sort] Related problem with attribute exceptions when creating 'hudiSink'
INLONG-9371 [Bug][Sort] elasticsearch sort shade relocation error
INLONG-9377 [Bug][Sort] Failed to init iceberg sink with upsert mode
INLONG-9380 [Bug][Sort] Audit lost when stop job immediately after checkpoint
INLONG-9388 [Improve][Sort] Updated version of embedded-redis in sort-connector-redis
INLONG-9392 [Bug][Sort] MySqlContainer class getJdbcUrl() method splicing querystring error
INLONG-9394 [Improve][Sort] Incorrect use of maven plugin on integration test among sort-end-to-end-tests
INLONG-9417 [Bug][Sort] Missing MySQL CDC connector for Flink 1.15

Other

ISSUE Summary
INLONG-8961 [Improve] Server-side request forgery attack prevention in some classes
INLONG-8967 [Feature] Add Mysql connector on flink 1.15
INLONG-8971 [Improve][manager] Missing audit id for MQ
INLONG-8977 [Feature] Add tube source connector on flink 1.15
INLONG-8994 [Feature] Add hudi connector on flink 1.15
INLONG-9013 [Improve][DataProxy] Inconsistent annotations
INLONG-9025 [Improve][Build] Avoid deploying the distribution packages to repositories
INLONG-9034 [Improve] Fix sort redis test with incorrect use of sleep
INLONG-9046 [Improve][CVE] snappy-java's missing upper bound check on chunk length can lead to Denial of Service (DoS) impact
INLONG-9064 [Feature] Add Audit report for Pulsar connector in flink 1.15
INLONG-9081 [Bug] Pulsar connector in flink 1.15 should running in exclusive mode
INLONG-9095 [Feature] Support inlong-msg in pulsar flink 1.15 connector
INLONG-9104 [Improve][ASF] Add a protected branches strategy for the master branch
INLONG-9202 [Bug] Fix audit report error when running pulsar -> iceberg in flink1.15
INLONG-9221 [Bug]When sink is MySQL, there is no 'DataNode' when first selected
INLONG-9225 [Improve][Audit] Automatically create audit topic after service startup
INLONG-9271 [Bug] When creating 'StreamField', 'isMetaField' must be initialized
INLONG-9281 [Bug] Fix pulsar flink connector cannot support batch messages
INLONG-9296 [Bug] Cluster type 'DATAPROXY' not supported
INLONG-9299 [Feature] Iceberg support all migrate and auto create table
INLONG-9359 [Bug] Fix iceberg all migrate connector stack overflow error
INLONG-9384 [Bug] Fix audi report loss in pulsar connector