MF-552 - Use event sourcing to keep Bootstrap service in sync with Things service #603

dborovcanin · 2019-02-18T16:32:28Z

What does this do?

This PR adds the use of event sourcing from Things service to keep corresponding Bootstrap service entities in sync.

Which issue(s) does this PR fix/relate to?

This PR resolves #552.

List any changes that modify/break current functionality

There are 2 major API changes:
- List of Configs does not return Channels
- Update Config is split to Config fields update and Config connections update

Have you included tests for your changes?

Yes.

Did you document any new/modified functionality?

Yes, docs are up to date.

drasko

LGTM

Signed-off-by: Dusan Borovcanin <dusan.borovcanin@mainflux.com>

codecov-io · 2019-02-24T22:23:21Z

Codecov Report

Merging #603 into master will decrease coverage by 0.41%.
The diff coverage is 82.17%.

@@            Coverage Diff             @@
##           master     #603      +/-   ##
==========================================
- Coverage   87.13%   86.72%   -0.42%     
==========================================
  Files          62       62              
  Lines        3437     3623     +186     
==========================================
+ Hits         2995     3142     +147     
- Misses        304      327      +23     
- Partials      138      154      +16

Impacted Files	Coverage Δ
bootstrap/state.go	`100% <ø> (ø)`	⬆️
bootstrap/api/responses.go	`100% <100%> (ø)`	⬆️
bootstrap/api/transport.go	`97.89% <100%> (+0.15%)`	⬆️
bootstrap/api/endpoint.go	`100% <100%> (ø)`	⬆️
bootstrap/postgres/init.go	`92.85% <100%> (+3.11%)`	⬆️
bootstrap/postgres/configs.go	`73.07% <72.97%> (-2.34%)`	⬇️
bootstrap/api/requests.go	`95.55% <80%> (-4.45%)`	⬇️
bootstrap/service.go	`84.42% <90.47%> (+2.65%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a60a5c1...e3cea73. Read the comment docs.

Signed-off-by: Dusan Borovcanin <dusan.borovcanin@mainflux.com>

drasko

LGTM

Signed-off-by: Dusan Borovcanin <dusan.borovcanin@mainflux.com>

anovakovic01 · 2019-02-28T16:24:50Z

bootstrap/api/endpoint_test.go

-		assert.Equal(t, tc.res, data, fmt.Sprintf("%s: expected response '%s' got '%s'", tc.desc, tc.res, data))
+		var view config
+		err = json.NewDecoder(res.Body).Decode(&view)
+		if err != io.EOF {


anovakovic01 · 2019-02-28T16:26:51Z

bootstrap/api/logging.go

+
+func (lm *loggingMiddleware) RemoveConfigHandler(id string) (err error) {
+	defer func(begin time.Time) {
+		message := fmt.Sprintf("Method removeConfig for config %s took %s to complete", id, time.Since(begin))


anovakovic01 · 2019-02-28T16:27:17Z

bootstrap/api/logging.go

+
+func (lm *loggingMiddleware) UpdateChannelHandler(channel bootstrap.Channel) (err error) {
+	defer func(begin time.Time) {
+		message := fmt.Sprintf("Method updateChannel for channel %s took %s to complete", channel.ID, time.Since(begin))


anovakovic01 · 2019-02-28T16:32:26Z

bootstrap/postgres/configs.go

+
+	for rows.Next() {
+		c := bootstrap.Channel{}
+		if err = rows.Scan(&c.ID, &c.Name, &c.Metadata); err != nil {


anovakovic01 · 2019-02-28T16:32:56Z

bootstrap/postgres/configs.go

+
+	for rows.Next() {
+		c := bootstrap.Channel{}
+		if err = rows.Scan(&c.ID, &c.Name, &c.Metadata); err != nil {


anovakovic01 · 2019-02-28T16:33:35Z

bootstrap/postgres/configs.go

+		return err
+	}
+
+	if err = insertChannels(key, channels, tx); err != nil {


anovakovic01 · 2019-02-28T16:33:40Z

bootstrap/postgres/configs.go

+		return err
+	}
+
+	if err = updateConnections(key, id, connections, tx); err != nil {


anovakovic01 · 2019-02-28T16:33:55Z

bootstrap/postgres/configs.go

+	var channels []bootstrap.Channel
+	for rows.Next() {
+		var ch bootstrap.Channel
+		if err = rows.Scan(&ch.ID, &ch.Name, &ch.Metadata); err != nil {


Signed-off-by: Dusan Borovcanin <dusan.borovcanin@mainflux.com>

drasko · 2019-03-03T02:58:06Z

bootstrap/mocks/configs.go

@@ -205,6 +242,75 @@ func (crm *configRepositoryMock) RemoveUnknown(key, id string) error {
 }

 func (crm *configRepositoryMock) SaveUnknown(key, id string) error {
+	crm.mu.Lock()
+	crm.mu.Unlock()


Isn't defer needed here?

drasko · 2019-03-03T02:58:38Z

bootstrap/mocks/configs.go

+
+func (crm *configRepositoryMock) DisconnectThing(channelID, thingID string) error {
+	crm.mu.Lock()
+	crm.mu.Unlock()


Same here - defer

Signed-off-by: Dusan Borovcanin <dusan.borovcanin@mainflux.com>

anovakovic01

LGTM!

drasko

LGTM

…ings service (absmach#603) * Use separate table for Channels Signed-off-by: Dusan Borovcanin <dusan.borovcanin@mainflux.com> * Add inital event sourcing subscription Signed-off-by: Dusan Borovcanin <dusan.borovcanin@mainflux.com> * Add Channel update sync Signed-off-by: Dusan Borovcanin <dusan.borovcanin@mainflux.com> * Add Thing remove sync Signed-off-by: Dusan Borovcanin <dusan.borovcanin@mainflux.com> * Add Channel remove sync Signed-off-by: Dusan Borovcanin <dusan.borovcanin@mainflux.com> * Update service add method marshalling metadata Signed-off-by: Dusan Borovcanin <dusan.borovcanin@mainflux.com> * Make separate methods for connection update Signed-off-by: Dusan Borovcanin <dusan.borovcanin@mainflux.com> * Add diconnect event sync Signed-off-by: Dusan Borovcanin <dusan.borovcanin@mainflux.com> * Update Configs repository mock Signed-off-by: Dusan Borovcanin <dusan.borovcanin@mainflux.com> * Fix service tests Signed-off-by: Dusan Borovcanin <dusan.borovcanin@mainflux.com> * Update repository tests Signed-off-by: Dusan Borovcanin <dusan.borovcanin@mainflux.com> * Update API docs Signed-off-by: Dusan Borovcanin <dusan.borovcanin@mainflux.com> * Update Location header Signed-off-by: Dusan Borovcanin <dusan.borovcanin@mainflux.com> * Update README.md Signed-off-by: Dusan Borovcanin <dusan.borovcanin@mainflux.com> * Fix tests mutex lock Signed-off-by: Dusan Borovcanin <dusan.borovcanin@mainflux.com> * Fix method names in logs Signed-off-by: Dusan Borovcanin <dusan.borovcanin@mainflux.com>

…ings service (#603) * Use separate table for Channels Signed-off-by: Dusan Borovcanin <dusan.borovcanin@mainflux.com> * Add inital event sourcing subscription Signed-off-by: Dusan Borovcanin <dusan.borovcanin@mainflux.com> * Add Channel update sync Signed-off-by: Dusan Borovcanin <dusan.borovcanin@mainflux.com> * Add Thing remove sync Signed-off-by: Dusan Borovcanin <dusan.borovcanin@mainflux.com> * Add Channel remove sync Signed-off-by: Dusan Borovcanin <dusan.borovcanin@mainflux.com> * Update service add method marshalling metadata Signed-off-by: Dusan Borovcanin <dusan.borovcanin@mainflux.com> * Make separate methods for connection update Signed-off-by: Dusan Borovcanin <dusan.borovcanin@mainflux.com> * Add diconnect event sync Signed-off-by: Dusan Borovcanin <dusan.borovcanin@mainflux.com> * Update Configs repository mock Signed-off-by: Dusan Borovcanin <dusan.borovcanin@mainflux.com> * Fix service tests Signed-off-by: Dusan Borovcanin <dusan.borovcanin@mainflux.com> * Update repository tests Signed-off-by: Dusan Borovcanin <dusan.borovcanin@mainflux.com> * Update API docs Signed-off-by: Dusan Borovcanin <dusan.borovcanin@mainflux.com> * Update Location header Signed-off-by: Dusan Borovcanin <dusan.borovcanin@mainflux.com> * Update README.md Signed-off-by: Dusan Borovcanin <dusan.borovcanin@mainflux.com> * Fix tests mutex lock Signed-off-by: Dusan Borovcanin <dusan.borovcanin@mainflux.com> * Fix method names in logs Signed-off-by: Dusan Borovcanin <dusan.borovcanin@mainflux.com>

dborovcanin added the work-in-progress label Feb 18, 2019

dborovcanin self-assigned this Feb 18, 2019

drasko previously approved these changes Feb 18, 2019

View reviewed changes

dborovcanin dismissed drasko’s stale review via 0fd7897 February 19, 2019 08:40

dborovcanin force-pushed the MF-552 branch 8 times, most recently from c39bc67 to 469bedb Compare February 22, 2019 08:34

manuio previously approved these changes Feb 22, 2019

View reviewed changes

dborovcanin dismissed manuio’s stale review via cc126bb February 22, 2019 15:04

dborovcanin force-pushed the MF-552 branch from 469bedb to cc126bb Compare February 22, 2019 15:04

dborovcanin added 9 commits February 22, 2019 16:07

Use separate table for Channels

65d2517

Signed-off-by: Dusan Borovcanin <dusan.borovcanin@mainflux.com>

Add inital event sourcing subscription

b833755

Signed-off-by: Dusan Borovcanin <dusan.borovcanin@mainflux.com>

Add Channel update sync

d0c8cd6

Signed-off-by: Dusan Borovcanin <dusan.borovcanin@mainflux.com>

Add Thing remove sync

849ffb6

Signed-off-by: Dusan Borovcanin <dusan.borovcanin@mainflux.com>

Add Channel remove sync

a6d294a

Signed-off-by: Dusan Borovcanin <dusan.borovcanin@mainflux.com>

Update service add method marshalling metadata

23810de

Signed-off-by: Dusan Borovcanin <dusan.borovcanin@mainflux.com>

Make separate methods for connection update

4529711

Signed-off-by: Dusan Borovcanin <dusan.borovcanin@mainflux.com>

Add diconnect event sync

3c7890e

Signed-off-by: Dusan Borovcanin <dusan.borovcanin@mainflux.com>

Update Configs repository mock

377da5f

Signed-off-by: Dusan Borovcanin <dusan.borovcanin@mainflux.com>

dborovcanin force-pushed the MF-552 branch from cc126bb to 318515a Compare February 24, 2019 12:45

Fix service tests

a7b06a7

Signed-off-by: Dusan Borovcanin <dusan.borovcanin@mainflux.com>

dborovcanin force-pushed the MF-552 branch 4 times, most recently from b8255a2 to f463c0c Compare February 24, 2019 22:18

dborovcanin force-pushed the MF-552 branch from f463c0c to d67b178 Compare February 24, 2019 23:12

Update repository tests

7ea664a

Signed-off-by: Dusan Borovcanin <dusan.borovcanin@mainflux.com>

dborovcanin force-pushed the MF-552 branch 3 times, most recently from c38d20f to 2743bbf Compare February 25, 2019 11:16

Update API docs

321f0f2

Signed-off-by: Dusan Borovcanin <dusan.borovcanin@mainflux.com>

dborovcanin force-pushed the MF-552 branch 2 times, most recently from c82982c to 3ee8e1f Compare February 25, 2019 16:17

dborovcanin removed the work-in-progress label Feb 25, 2019

drasko previously approved these changes Feb 25, 2019

View reviewed changes

manuio previously approved these changes Feb 25, 2019

View reviewed changes

Update Location header

247b49a

Signed-off-by: Dusan Borovcanin <dusan.borovcanin@mainflux.com>

dborovcanin dismissed stale reviews from manuio and drasko via 247b49a February 26, 2019 08:32

dborovcanin force-pushed the MF-552 branch from 3ee8e1f to 247b49a Compare February 26, 2019 08:32

anovakovic01 suggested changes Feb 28, 2019

View reviewed changes

Update README.md

b6848c9

Signed-off-by: Dusan Borovcanin <dusan.borovcanin@mainflux.com>

dborovcanin force-pushed the MF-552 branch from e2e25d7 to b6848c9 Compare March 1, 2019 14:46

drasko requested changes Mar 3, 2019

View reviewed changes

dborovcanin added 2 commits March 4, 2019 16:35

Fix tests mutex lock

1a578a6

Signed-off-by: Dusan Borovcanin <dusan.borovcanin@mainflux.com>

Fix method names in logs

e3cea73

Signed-off-by: Dusan Borovcanin <dusan.borovcanin@mainflux.com>

anovakovic01 approved these changes Mar 4, 2019

View reviewed changes

drasko approved these changes Mar 4, 2019

View reviewed changes

dborovcanin merged commit 3300814 into absmach:master Mar 4, 2019

dborovcanin deleted the MF-552 branch March 4, 2019 16:42

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

MF-552 - Use event sourcing to keep Bootstrap service in sync with Things service #603

MF-552 - Use event sourcing to keep Bootstrap service in sync with Things service #603

dborovcanin commented Feb 18, 2019 •

edited

Loading

drasko left a comment

codecov-io commented Feb 24, 2019 •

edited

Loading

drasko left a comment

anovakovic01 Feb 28, 2019

anovakovic01 Feb 28, 2019

anovakovic01 Feb 28, 2019

anovakovic01 Feb 28, 2019

anovakovic01 Feb 28, 2019

anovakovic01 Feb 28, 2019

anovakovic01 Feb 28, 2019

anovakovic01 Feb 28, 2019

drasko Mar 3, 2019

drasko Mar 3, 2019

anovakovic01 left a comment

drasko left a comment

MF-552 - Use event sourcing to keep Bootstrap service in sync with Things service #603

MF-552 - Use event sourcing to keep Bootstrap service in sync with Things service #603

Conversation

dborovcanin commented Feb 18, 2019 • edited Loading

What does this do?

Which issue(s) does this PR fix/relate to?

List any changes that modify/break current functionality

Have you included tests for your changes?

Did you document any new/modified functionality?

drasko left a comment

Choose a reason for hiding this comment

codecov-io commented Feb 24, 2019 • edited Loading

Codecov Report

drasko left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

anovakovic01 left a comment

Choose a reason for hiding this comment

drasko left a comment

Choose a reason for hiding this comment

dborovcanin commented Feb 18, 2019 •

edited

Loading

codecov-io commented Feb 24, 2019 •

edited

Loading