feat: Recommendation framework #445

xieydd · 2022-07-28T09:25:39Z

What type of PR is this?

Feature

What this PR does / why we need it:

Define recommendation framework interfaces .
Implement replicas recommendation via framework.

Special notes for your reviewer:

@qmhu

…ommendation via framework(WIP)

xieydd · 2022-07-28T09:25:55Z

/hold

github-actions · 2022-07-28T09:26:12Z

♻️ PR Preview b7228dd has been successfully destroyed since this PR has been closed.

_{🤖 By surge-preview}

xieydd · 2022-08-10T08:14:31Z

done. pls review this pr @qmhu

github-actions · 2022-08-10T08:49:36Z

🎉 Successfully Build Images.
Now Support ARM Platforms.
Comment Post Time: 2022-08-19 01:53
Git Version: b7228dd

Docker Registry

Overview: https://hub.docker.com/u/gocrane

Image	Pull Command
crane-agent:pr-445-b7228dd	docker pull gocrane/crane-agent:pr-445-b7228dd
dashboard:pr-445-b7228dd	docker pull gocrane/dashboard:pr-445-b7228dd
metric-adapter:pr-445-b7228dd	docker pull gocrane/metric-adapter:pr-445-b7228dd
craned:pr-445-b7228dd	docker pull gocrane/craned:pr-445-b7228dd

Quick Deploy - Helm

helm repo add crane https://finops-helm.pkg.coding.net/gocrane/gocrane
helm install crane -n crane-system --create-namespace \
                   --set craned.image.repository=gocrane/craned \
                   --set craned.image.tag=pr-445-b7228dd \
                   --set metricAdapter.image.repository=gocrane/metric-adapter \
                   --set metricAdapter.image.tag=pr-445-b7228dd \
                   --set craneAgent.image.repository=gocrane/crane-agent \
                   --set craneAgent.image.tag=pr-445-b7228dd \
                   --set cranedDashboard.image.repository=gocrane/dashboard \
                   --set cranedDashboard.image.tag=pr-445-b7228dd crane/crane

Coding Registry

Overview: https://finops.coding.net/public-artifacts/gocrane/crane/packages

Image	Pull Command
crane-agent:pr-445-b7228dd	docker pull finops-docker.pkg.coding.net/gocrane/crane/crane-agent:pr-445-b7228dd
dashboard:pr-445-b7228dd	docker pull finops-docker.pkg.coding.net/gocrane/crane/dashboard:pr-445-b7228dd
metric-adapter:pr-445-b7228dd	docker pull finops-docker.pkg.coding.net/gocrane/crane/metric-adapter:pr-445-b7228dd
craned:pr-445-b7228dd	docker pull finops-docker.pkg.coding.net/gocrane/crane/craned:pr-445-b7228dd

Quick Deploy - Helm

helm repo add crane https://finops-helm.pkg.coding.net/gocrane/gocrane
helm install crane -n crane-system --create-namespace \
                   --set craned.image.repository=finops-docker.pkg.coding.net/gocrane/crane/craned \
                   --set craned.image.tag=pr-445-b7228dd \
                   --set metricAdapter.image.repository=finops-docker.pkg.coding.net/gocrane/crane/metric-adapter \
                   --set metricAdapter.image.tag=pr-445-b7228dd \
                   --set craneAgent.image.repository=finops-docker.pkg.coding.net/gocrane/crane/crane-agent \
                   --set craneAgent.image.tag=pr-445-b7228dd \
                   --set cranedDashboard.image.repository=finops-docker.pkg.coding.net/gocrane/crane/dashboard \
                   --set cranedDashboard.image.tag=pr-445-b7228dd crane/crane

Ghcr Registry

Overview: https://github.com/orgs/gocrane/packages?repo_name=crane

Image	Pull Command
crane-agent:pr-445-b7228dd	docker pull ghcr.io/gocrane/crane/crane-agent:pr-445-b7228dd
dashboard:pr-445-b7228dd	docker pull ghcr.io/gocrane/crane/dashboard:pr-445-b7228dd
metric-adapter:pr-445-b7228dd	docker pull ghcr.io/gocrane/crane/metric-adapter:pr-445-b7228dd
craned:pr-445-b7228dd	docker pull ghcr.io/gocrane/crane/craned:pr-445-b7228dd

Quick Deploy - Helm

helm repo add crane https://finops-helm.pkg.coding.net/gocrane/gocrane
helm install crane -n crane-system --create-namespace \
                   --set craned.image.repository=ghcr.io/gocrane/crane/craned \
                   --set craned.image.tag=pr-445-b7228dd \
                   --set metricAdapter.image.repository=ghcr.io/gocrane/crane/metric-adapter \
                   --set metricAdapter.image.tag=pr-445-b7228dd \
                   --set craneAgent.image.repository=ghcr.io/gocrane/crane/crane-agent \
                   --set craneAgent.image.tag=pr-445-b7228dd \
                   --set cranedDashboard.image.repository=ghcr.io/gocrane/crane/dashboard \
                   --set cranedDashboard.image.tag=pr-445-b7228dd crane/crane

pkg/recommendation/replicas/fliter.go

pkg/recommendation/replicas/prepare.go

qmhu · 2022-08-10T09:05:00Z

pkg/recommendation/replicas/recommend.go

+func (rr *ReplicasRecommender) PreRecommend(ctx *framework.RecommendationContext) error {
+	// we load algorithm config in this phase
+	// TODO(chrisydxie) support configuration
+	config := &config.Config{


What if we have multiple algo config

Can you describe the specific user case? Do we need hyper-parameter search?

pkg/recommendation/replicas/recommend.go

xieydd · 2022-08-17T02:38:56Z

pkg/recommendation/framework/context.go

@@ -73,3 +79,33 @@ func NewRecommendationContext(context context.Context, identity analytics.Object
 //		return false
 //	}
 //}
+
+func ToK8SObject(object interface{}, target interface{}) error {


we can move those func to a separate file, not context.go.

suggest a file name?

xieydd · 2022-08-17T02:47:07Z

pkg/recommendation/framework/context.go

+	return json.Unmarshal(bytes, target)
+}
+
+func GetDaemonSetPods(kubeClient client.Client, namespace string, name string) ([]corev1.Pod, error) {


duplicated with utils GetDaemonSetPods.

xieydd · 2022-08-17T02:47:42Z

pkg/recommendation/framework/context.go

+//	}
+//}
+
+func ObjectConversion(object interface{}, target interface{}) error {


We need change Object and ObjectIdenty to seperate file, not context.go.

suggest a filename ?

xieydd · 2022-08-17T02:52:00Z

pkg/recommendation/recommender/base/prepare.go

+	}
+
+	// will be use in future.
+	// if no data provider config set, use default history data provider


we need grpc data source.

we can implement it in future.

pkg/recommendation/recommender/hpa/observe.go

xieydd · 2022-08-17T02:56:43Z

pkg/recommendation/recommender/hpa/recommend.go

+		return fmt.Errorf("%s checkFluctuation failed: %v", rr.Name(), err)
+	}
+
+	targetUtilization, requestTotal, err := rr.proposeTargetUtilization(ctx)


fix lint error.

xieydd · 2022-08-17T02:57:30Z

pkg/recommendation/recommender/hpa/recommend.go

+}
+
+// checkFluctuation check if the time series fluctuation is reach to replicas.fluctuation-threshold
+func (rr *HPARecommender) checkFluctuation(medianMin, medianMax float64) error {


look like same as replicas function?

this is a hpa recommendation function. Already remove from replicas recommender.

xieydd · 2022-08-17T03:00:54Z

pkg/recommendation/recommender/resource/prepare.go

+	return nil
+}
+
+func (rr *ResourceRecommender) CollectData(ctx *framework.RecommendationContext) error {


resource no need collect data?

currently on.

xieydd · 2022-08-17T03:01:57Z

pkg/recommendation/recommender/resource/recommend.go

+	"github.com/gocrane/crane/pkg/utils"
+)
+
+const callerFormat = "RecommendationCaller-%s-%s"


same as hpa recommender, need change to ResourceRecommederCaller-%s-%s.

xieydd · 2022-08-17T03:02:09Z

pkg/recommendation/recommender/resource/recommend.go

+const callerFormat = "RecommendationCaller-%s-%s"
+
+func (rr *ResourceRecommender) PreRecommend(ctx *framework.RecommendationContext) error {
+	return nil


no need load algoritjm config?

every container has two algoritjm config: cpu and memory ,so cannot use context's algoritjm config. we may need to reconsider it.

Signed-off-by: qmhu <brillantroad@gmail.com>

xieydd · 2022-08-19T03:00:45Z

link: #433

qmhu and others added 12 commits July 6, 2022 19:16

proposel for recommend v2

9e27d1c

crane recommendation framework proposal v1

6cae230

add phase detile

f2c1621

fix image

fb8b9b2

fix

a701ba7

delete extender proposal

31eef14

Define recommendation framework interface

cb12b75

Merge crane/main

ba1defe

add recommenderMgr in manager.go

2f1b4a9

refector the controller of recommendation controller

cc18cf2

Merge crane/main

7b475a3

Implement recommendation framework interfaces; Implement replicas rec…

61a7fa7

…ommendation via framework(WIP)

xieydd requested a review from qmhu July 28, 2022 09:25

implement replicas recommendation

2f38aeb

xieydd changed the title ~~(WIP) Recommendation framework~~ Recommendation framework Aug 10, 2022

qmhu reviewed Aug 10, 2022

View reviewed changes

xieydd and others added 5 commits August 12, 2022 09:56

Merge gocrane/main

68f7a16

format import

b99fc32

add filter logic

e4daaf4

add resource recommender

10a0fff

add hpa recommender

e5ed0a3

qmhu force-pushed the recommendation-framework branch from 23e7ab6 to e5ed0a3 Compare August 16, 2022 09:39

xieydd commented Aug 17, 2022

View reviewed changes

recommender bugfix

b7228dd

Signed-off-by: qmhu <brillantroad@gmail.com>

qmhu merged commit 940f57f into main Aug 18, 2022

qmhu changed the title ~~Recommendation framework~~ feat: Recommendation framework Aug 18, 2022

This was referenced Aug 19, 2022

Define Recommendation Framework in-tree interfaces. #433

Closed

Define new controller for framework. #434

Closed

Transfer the craned default logic to framework. #435

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Recommendation framework #445

feat: Recommendation framework #445

xieydd commented Jul 28, 2022

xieydd commented Jul 28, 2022

github-actions bot commented Jul 28, 2022 •

edited

Loading

xieydd commented Aug 10, 2022

github-actions bot commented Aug 10, 2022 •

edited

Loading

qmhu Aug 10, 2022

xieydd Aug 11, 2022

xieydd Aug 17, 2022

qmhu Aug 18, 2022

xieydd Aug 17, 2022

xieydd Aug 17, 2022

qmhu Aug 18, 2022

xieydd Aug 17, 2022

qmhu Aug 18, 2022

xieydd Aug 17, 2022

xieydd Aug 17, 2022

qmhu Aug 18, 2022

xieydd Aug 17, 2022

qmhu Aug 18, 2022

xieydd Aug 17, 2022

xieydd Aug 17, 2022

qmhu Aug 18, 2022

xieydd commented Aug 19, 2022

feat: Recommendation framework #445

feat: Recommendation framework #445

Conversation

xieydd commented Jul 28, 2022

What type of PR is this?

What this PR does / why we need it:

Special notes for your reviewer:

xieydd commented Jul 28, 2022

github-actions bot commented Jul 28, 2022 • edited Loading

xieydd commented Aug 10, 2022

github-actions bot commented Aug 10, 2022 • edited Loading

Docker Registry

Coding Registry

Ghcr Registry

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

xieydd commented Aug 19, 2022

github-actions bot commented Jul 28, 2022 •

edited

Loading

github-actions bot commented Aug 10, 2022 •

edited

Loading