distributed-system

introduction

• Starting Points in Distributed Systems, Pt. I: Background
• Distributed computing
• A Thorough Introduction to Distributed Systems
• Distributed systems for fun and profit
• Fallacies of distributed computing
• Notes on Distributed Systems for Young Bloods
• An introduction to distributed systems
• awesome-distributed-systems
• 分布式系统(Distributed System)资料
• The System Design Primer
• BigData Notes
• Designing Data-Intensive Applications
• 设计数据密集型应用
• Distributed systems theory for the distributed systems engineer
• Readings in distributed systems
• 分布式系统前沿技术

tutorial

• Distributed Algorithms
• Principles of Distributed Computing
• DISTRIBUTED ALGORITHMS
• 分布式系统原理
• CSE5306: Distributed Systems
• Distributed Computing: Principles, Algorithms, and Systems
• CS5620: Distributed Systems and Algorithms
• W4995-1: Distributed Systems
• G22.3033-001: Distributed Systems
• 15-440: Distributed Systems
• COS-418: Distributed Systems
• COS-418: Distributed Systems

reference

• arxiv
• 系统
• 数据库
• LESLIE LAMPORT'S HOME PAGE
• Martin Kleppmann
• The Paper Trail
• the morning paper
• Metadata
• Read, Write, Execute
• Marc's Blog
• Peter Bailis :: Highly Available, Seldom Consistent
• db ranking
• 阿里中间件
• 美团技术
• 数据库内核月报
• 阿里云服务产品目录
• NoSQL Distilled
• 数据中台之结构化大数据存储设计
• 百万节点数据库扩展之道

conference

sosp

• 2021

summary

impossibility

• flp
• cap

formal verification

• coq
• tla

clock

• lamport clock
• vector clock
• matrix clock
• dotted version vectors
• interval tree clock
• concensus clock
• hlc
• hvc

gossip

conflict

• git
• ot
• mpds
• mrdt
• crdt

quorum

concensus

CFT

• viewstamped replication
• paxos
• raft
• zab
• chubby
• kafka
• bookeeper
• pacifica

BFT

BFT (probability)

• bitcoin
• hotstuff

replication

partition

consistency

• linearizability
• sequential consistency
• causal consistency
• pram consistency

shared memory

transaction

• scheduler
• crash recovery

storage

column storage

query optimizer

query execution

database

atomic commit

• 2PC
• paxos commit

distributed transaction

• determinism
• percolator
• omid
• maat

htap

snapshot

architecture

• shared
  • shared nothing
  • shared disk
  • shared memory
  • shared everything
• message
  • 1
    • udp/tcp
    • queue/channel
    • message broker
  • 2
    • block/nonblock
    • sync/async
• log
  • application log
  • database
    • log structured kv database
    • wal log
    • replication log
  • log-based message broker
  • consensus/atomic broadcast
  • cdc/etl
• state
  • stateless
  • stateful
• stream
  • immutable
  • mutable
• process
  • batch
  • stream

存储

分类

• 数据库
• KV存储/对象存储
• 文件存储/块存储
• 日志

存算分离

• 计算的一致性
• 存储的一致性

数据库

• distributed file system
• gfs
• hdfs
• curve
• ceph
• cache
• redis
• memcached
• kv cluster
• dynamo
• voldemort
• riak
• gobeansdb
• document database
• mongodb
• rethinkdb
• couchdb
• couchbase/membase
• hbase
• cassandra
• boltdb
• badger
• leveldb
• rocksdb
• agatedb
• terarkdb
• pebble
• bitcask
• sqlite
• oracle
• mysql
• myrocks
• postgres
• gluesql
• peloton
• noisepage
• duckdb
• velox
• chiselstore
• rqlite
• toydb
• vitess
• tdsql
• kunlun
• aurora
• cockroachdb
• yugabytedb
• baikaldb
• polardb
• oceanbase
• pingcap
• matrixorigin
• map reduce
• pig
• hive
• dremel
• bigquery
• arrow-datafusion
• arrow-ballista
• drill
• kylin
• impala
• presto
• kudu
• starrocks
• doris
• snowflake
• clickhouse
• datafuse
• hologres
• greenplum

message broker

design

• exact once message passing
  • atomic commit
  • idempotent
• message order
• fault tolerance
  • drop message
  • apply backpressure
  • buffer message
    • ring buffer

comparison

	tcp	message broker
exact once message passing	sequence number	log-based
message order	sequence number	log-based
falut tolerance	slide window, traffic control	log-based

data process

design

• expressive: sql
• expressive: fork-join, dag
• expressive: language, framework
• throughoutput: partition, parallel
• fault tolerance

comparison

	spark / flink	tensorflow / pytorch	pregel
parallel	dag + cluster parallel	dag + standalone parallel

the future of data systems

• data integration
  • distributed transaction vs log-based system
  • kappa architecture vs lambda architecture vs lambda plus architecture
    • kappa-architecture.com
    • Lambda Architecture
    • From Lambda to Kappa: A Guide on Real-time Big Data Architectures
  • batch process vs stream process
• aiming for correctness
  • the end-to-end argument
    • inner vs outer
  • correctness and fault-tolerance
  • timeliness and integrity