🍀 Proposal: Cloudflare IP List Monitor

[IronCore864]

What would you like to add? Why is this needed?

Background

We use S3 to store published plugins from v0.6.0. And to speed things up, we have Cloudflare in front of the S3 bucket. The thing is, we grant access to S3 by IP addresses.

Cloudflare publishes its IP range here: https://www.cloudflare.com/ips/. It's also available from a text file: https://www.cloudflare.com/ips-v4.

If Cloudflare IP changes, we won't be able to dtm init anymore because S3 doesn't know if Cloudflare changes its IP or not and is still using the old list.

IP Change Monitoring

It would be nice if we can have a daily cronjob that checks if the IP list of Cloudflare has changed or not.

If yes, send a notification (email/feishu/slack) to DevStream PMC.

Functional Requirements

The script can be a simple python one, or a simple go app. We can even utilize github actions or something to that effect. The goal is to have a service like "hascloudflarechangedip.com" or something.

Non-Functional Requirements

Maybe figure out a way to ensure 24 hours SLA.

I.E., at least run the job once per day so that we can be notified within 24 hours (maximum) after IP changed.

[iyear]

This issue might be a good first issue, as the tool does not require going deep into the devstream to understand the details. If someone wants to try it out they can give it a go first.

If the need is urgent at the moment, I can develop it.

This should require a new repo to place the code.

My solution is outlined as follows:

• Use https://github.com/patrickmn/go-cache as a cache and a simple local message store.
  Considering it's just a small tool, I'm not going to use sqlite/boltdb for persistence, which is of course optional.
• Use https://github.com/deckarep/golang-set as the set calculation library, each time a message is sent it will indicate which ips have been added and which have been removed.
• Use https://github.com/chyroc/lark to send messages, but I don't know the usage, so someone familiar with lark can help with this part.
• If we don't use systemd to register the daemon manually, we can use https://github.com/kardianos/service to register the daemon.
• If we don't use cronjob, we can use https://github.com/robfig/cron as the cron job lib.

The above libraries are very lightweight, so they are not overly bloated.

If we intend to develop a cross-platform widget, my advice is not to rely on OS services, cron and daemon should be handled by the program.

[algobot76]

I have a simpler solution in minde: everything can be done using a bash script.

• Store IP addresses in a simple plain text.
• Read the text file using read.
• Send API requests using curl.
• Schedule the task using crontab.

[iyear]

I have a simpler solution in minde: everything can be done using a bash script.

• Store IP addresses in a simple plain text.
• Read the text file using read.
• Send API requests using curl.
• Schedule the task using crontab.

Yes, this is a much simpler solution. I started out with the idea of open sourcing as an widget, providing docker images and cross-platform binaries to make it possible to run everywhere and without relying on a specific OS.

That way, others could start it up immediately if they needed to use it.

How to do this depends on what the requirement ends up being.

[KeHaohaoke]

This issue might be a good first issue, as the tool does not require going deep into the devstream to understand the details. If someone wants to try it out they can give it a go first.

If the need is urgent at the moment, I can develop it.

This should require a new repo to place the code.

My solution is outlined as follows:

• Use https://github.com/chyroc/lark to send messages, but I don't know the usage, so someone familiar with lark can help with this part.

Lark： I suggest not to use sdk, just use Feishu official website's development guide and use package resty to do it. Using lark's official guide can facilitate iteration, and others' libraries may not follow up and modify.

[iyear]

@KeHaohaoke Okay, I'll do that. Thanks for the help and reply!

[IronCore864]

Both designs are great. Thanks, @iyear, and @algobot76.

I think we have two choices here:

1. Go with @algobot76's simple design, run it somewhere in a VM with corn. This way, it will only be used by DevStream.
2. Go with @iyear's design, run it as a pod in a K8s cluster, maybe even build a frontend for it, as simple as "hascloudflareipchanged.io", and it returns a simple JSON. We can have another app calling this API periodically, and if it has changed, send a message to Feishu. That is to say, create a "has cloudflare changed its IP" service, and create a client app that consumes that service and notifies.

I personally wouldn't future-proof too much, so my vote is on the first solution.

@daniel-hutao @KeHaohaoke what do you think? Need you guys' input here.

[daniel-hutao]

This is a difficult requirement to test, and it's likely that we won't receive an alert message for years after we're done. Maybe after a few months, our group names have changed and we won't receive any alerts. In short, I think this requirement is not necessary to spend too much effort, there is no need to do too fancy. Just a simple script that can output useful logs after running. As for this script, maybe we can use actions to run it, and when something goes wrong, actions will report the error. The simpler, the better.

[daniel-hutao]

Also the script only needs to be put into the hack/, and there is no need to consider a separate repo, that repo, if created, will soon be forgotten.

[KeHaohaoke]

Both designs are great. Thanks, @iyear, and @algobot76.

I think we have two choices here:

1. Go with @algobot76's simple design, run it somewhere in a VM with corn. This way, it will only be used by DevStream.

@daniel-hutao @KeHaohaoke what do you think? Need you guys' input here.

Maybe it's a typo. It's cron. @IronCore864

[KeHaohaoke]

Using scripts and cron is a lightweight scheme. Easy to implement.

[iyear]

So I will write a go file as a simple script and put it in the /hack dir, is that ok?

Or give it to a new contributor to complete? This issue has been a good fisrt issue.

[iyear]

Looks like this is a good first issue, please unassign me to avoid misunderstandings, thanks!

Or do I complete the issue?

[IronCore864]

@iyear sorry for the late reply.

Since this is an advanced feature (a standalone script/app,) and you are already a contributor, I removed the good first issue label. I hope you won't mind.

As we communicated over our Slack channel, let's proceed with a simple design without too many extra features.

In the future, we might consider running it as a pod or a job in K8s and add Slack notification feature; but for now, we do not need to have it at the moment.

[IronCore864]

@iyear sorry for the late reply.

Since this is an advanced feature (a standalone script/app,) and you are already a contributor, I removed the good first issue label. I hope you won't mind.

As we communicated over our Slack channel, let's proceed with a simple design without too many extra features.

In the future, we might consider running it as a pod or a job in K8s and add a Slack notification feature; but for now, we do not need to have it at the moment.

@daniel-hutao please also have a look at our decision.