cess-cahcer

Introduction

Cacher is an important part of CESS CDN, which is used to improve the speed of users downloading files. The cacher is built between the user and the CESS storage miner. The user creates the cache order by indexer, and then downloads the cache file from the cache miner.

Get start

1. First, you need to make a simple configuration. The configuration file is config.toml under the config directory,Please fill in all configuration options.

#Directory where the file cache is stored 
CacheDir=""
#There are some data for configuring the cache function
#MaxCacherSize represents the maximum cache space you allow cacher to use(byte)
MaxCacheSize=107374182400
#MaxCacheRate indicates the maximum utilization of cache space. If this threshold is exceeded, files will be cleaned up according to the cache obsolescence policy
MaxCacheRate=0.95
#Threshold indicates the target threshold when cache obsolescence occurs, that is, when cache space utilization reaches this value, cache clean will be stopped
Threshold=0.8
#FreqWeight represents the weight of file usage frequency, which is used in cache obsolescence strategy
FreqWeight=0.3
#cacher IP address,please ensure external accessibility
ServerIp=""
#cacher server port
ServerPort="8080"
#the key used to encrypt the token, which is generated randomly by default
TokenKey=""
#you CESS account and seed
AccountSeed="lunar talent spend shield blade when dumb toilet drastic unique taxi water"
AccountID="cXgZo3RuYkAGhhvCHjAcc9FU13CG44oy8xW6jN39UYvbBaJx5"
#CESS network ws address
RpcAddr="wss://devnet-rpc.cess.cloud/ws/"
#unit price of bytes downloaded from file cache
BytePrice=1000

2. Before starting the cache service, you need to register the cache miner,you need to go back to the project main directory and run:

   go run main.go register

3. You can run the following command to update the registration information:

   go run main.go update

4. And you can run the following command to logout:

   go run main.go logout

Run Cache Server

You only need to start the cache service with one line of command, and the subsequent tasks should be handed to the indexer. Of course, cache miners also provide a series of rich APIs for developers to use, which will be explained later.

go run main.go run

Unit Test

You can use the test samples in the test directory for unit testing. Note that you should set the configuration file before testing

cd test 
# test cacher chain client
go test chain_test.go
# test cacher init
go test init_test.go
# test cacher query api
go test query_test.go

Code Walkthrough

1. When the user uses the register command, the transaction will be send through the register method under the chain directory to complete the registration on the blockchain, and the registration data uses the content configured in config.toml

   cess-cacher/base/chain/transaction.go

   Lines 122 to 135 in 1c28cba

   	func (c *chainClient) Register(ip, port string, price uint64) (string, error) {
   	info, err := NewCacherInfo(c.IncomeAcc, ip, port, price)
   	if err != nil {
   	return "", errors.Wrap(err, "register cacher error")
   	}
   	txhash, err := c.SubmitExtrinsic(
   	CACHER_REGISTER,
   	func(events CacheEventRecords) bool {
   	return len(events.Cacher_Register) > 0
   	},
   	info,
   	)
   	return txhash, errors.Wrap(err, "register cacher error")
   	}

2. User can update the information in the configuration file first, and then use the update command to update the cacher data on the blockchain. This method is still in the transaction.go file under the chain directory.

   cess-cacher/base/chain/transaction.go

   Lines 137 to 153 in 1c28cba

   	func (c *chainClient) Update(ip, port string, price uint64) (string, error) {
   	info, err := NewCacherInfo(c.IncomeAcc, ip, port, price)
   	if err != nil {
   	return "", errors.Wrap(err, "update cacher info error")
   	}
   	if _, err = c.GetMinerInfo(); err != nil {
   	return "", errors.Wrap(err, "update cacher info error")
   	}
   	txhash, err := c.SubmitExtrinsic(
   	CACHER_UPDATE,
   	func(events CacheEventRecords) bool {
   	return len(events.Cacher_Update) > 0
   	},
   	info,
   	)
   	return txhash, errors.Wrap(err, "update cacher info error")
   	}

3. Similarly, when the user uses the logout command, the logout method will be called to log out the cacher information.

   cess-cacher/base/chain/transaction.go

   Lines 155 to 163 in 1c28cba

   	func (c *chainClient) Logout() (string, error) {
   	txhash, err := c.SubmitExtrinsic(
   	CACHER_LOGOUT,
   	func(events CacheEventRecords) bool {
   	return len(events.Cacher_Logout) > 0
   	},
   	)
   	return txhash, errors.Wrap(err, "logout cacher error")
   	}

4. If the user executes the run command, the cache service will be started, which is an HTTP service. Then the indexer can call the query service in the query.go file under the service directory to obtain the information about the cacher.

   cess-cacher/server/service/query.go

   Lines 29 to 78 in 1c28cba

   	func QueryMinerStats() (MinerStats, resp.Error) {
   	var (
   	mstat MinerStats
   	err error
   	)
   	mstat.MinerStatus = "active"
   	mstat.NetStats = cache.GetNetInfo()
   	mstat.CacheStat = cache.GetCacheHandle().GetCacheStats()
   	mstat.BytePrice = config.GetConfig().BytePrice
   	mstat.MemoryStats, err = cache.GetMemoryStats()
   	if err != nil {
   	return mstat, resp.NewError(500, errors.Wrap(err, "query miner stats error"))
   	}
   	mstat.CPUStats, err = cache.GetCPUStats()
   	if err != nil {
   	return mstat, resp.NewError(500, errors.Wrap(err, "query miner stats error"))
   	}
   	mstat.DiskStats = cache.GetCacheDiskStats()
   	extIp, err := utils.GetExternalIp()
   	if err != nil {
   	return mstat, resp.NewError(500, errors.Wrap(err, "query miner stats error"))
   	}
   	country, city, err := utils.ParseCountryFromIp(extIp)
   	if err != nil {
   	return mstat, resp.NewError(500, errors.Wrap(err, "query miner stats error"))
   	}
   	mstat.GeoLocation = country + "," + city
   	return mstat, nil
   	}
   	func QueryCachedFiles() []string {
   	return cache.GetCacheHandle().GetHashList()
   	}
   	func QueryFileInfo(hash string) FileStat {
   	var stat FileStat
   	info, ok := cache.GetCacheHandle().QueryFile(hash)
   	if !ok {
   	//query info from chain
   	return stat
   	}
   	stat.Cached = true
   	stat.Price = uint64(info.Size) * config.GetConfig().BytePrice
   	stat.Size = uint64(info.Size)
   	return stat
   	}
   	func QueryBytePrice() uint64 {
   	return config.GetConfig().BytePrice
   	}

5. When the indexer requests to generate a file download token, it will call the GenerateToken service, which will check the validity of the cache bill and warm up the cache download.

   cess-cacher/server/service/auth.go

   Lines 24 to 57 in 1c28cba

   	func GenerateToken(hash, bid string, sign []byte) (string, resp.Error) {
   	var token string
   	t, err := PraseTicketByBID(hash, bid)
   	if err != nil {
   	return token, resp.NewError(400, errors.Wrap(err, "generate token error"))
   	}
   	if !utils.VerifySign(t.Account, []byte(hash+bid), sign) {
   	return token, resp.NewError(400, errors.Wrap(err, "generate token error"))
   	}
   	if ticketBeUsed(bid, t.Expires) {
   	err := errors.New("invalid bill")
   	return token, resp.NewError(400, errors.Wrap(err, "generate token error"))
   	}
   	if aesHandle.Enc == nil {
   	aesHandle.Enc = []byte(utils.GetRandomcode(32))
   	}
   	hash58, err := utils.HexStringToBase58(hash)
   	if err != nil {
   	return token, resp.NewError(400, errors.Wrap(err, "generate token error"))
   	}
   	bid58, err := utils.HexStringToBase58(bid)
   	if err != nil {
   	return token, resp.NewError(400, errors.Wrap(err, "generate token error"))
   	}
   	cipText, err := aesHandle.SymmetricEncrypt([]byte(hash58 + "-" + bid58))
   	if err != nil {
   	return token, resp.NewError(400, errors.Wrap(err, "generate token error"))
   	}
   	token = base58.Encode(cipText)
   	//data preheating: prepare the files not downloaded
   	cache.GetCacheHandle().HitOrLoad(t.FileHash + "-" + t.SliceHash)
   	deleteTicket(bid)
   	return token, nil
   	}

6. When user download a file, the DownloadService will be called. The service will record the used bill and automatically destroy it after it expires. Since the validity of the bill has been verified in the token generation stage, it is only necessary to verify whether it expires when downloading the file.

   cess-cacher/server/service/download.go

   Lines 41 to 75 in 1c28cba

   	func DownloadService(t Ticket) (string, resp.Error) {
   	var slicePath string
   	if time.Since(t.Expires) >= 0 {
   	err := errors.New("The ticket has expired")
   	return slicePath, resp.NewError(400, errors.Wrap(err, "download service error"))
   	}
   	if ticketBeUsed(t.BID, t.Expires) {
   	err := errors.New("The ticket has been used")
   	return slicePath, resp.NewError(400, errors.Wrap(err, "download service error"))
   	}
   	if ok, err := cache.GetCacheHandle().HitOrLoad(t.FileHash + "-" + t.SliceHash); !ok {
   	if err != nil {
   	tickets.Delete(t.BID)
   	return slicePath, resp.NewError(500, errors.Wrap(err, "download service error"))
   	}
   	if count, ok := cache.GetCacheHandle().LoadFailedFile(t.SliceHash); ok && count >= 1 {
   	err = errors.New("cache file failed,remote miner offline or refused")
   	tickets.Delete(t.BID)
   	return slicePath, resp.NewError(500, errors.Wrap(err, "download service error"))
   	}
   	progress, ect := cache.DownloadProgressBar(t.FileHash, t.SliceHash, t.Size)
   	slicePath = fmt.Sprintf(
   	"file %s is being cached %2.2f %%,it will take about %d s",
   	t.SliceHash, progress*100, ect,
   	)
   	tickets.Delete(t.BID)
   	return slicePath, resp.NewError(0, nil)
   	}
   	slicePath = path.Join(cache.FilesDir, t.FileHash, t.SliceHash)
   	if _, err := os.Stat(slicePath); err != nil {
   	tickets.Delete(t.BID)
   	return slicePath, resp.NewError(500, errors.Wrap(err, "download service error"))
   	}
   	return slicePath, nil
   	}

7. When the cache service is started, it will start a series of services through goroutine. These services ensure that the cacher can provide stable and reliable cache services in the background. The specific implementation of cache is in the ./base/cache directory, due to the complexity of the content, it will not be introduced here.