Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SM-4 - device_data VIN from fingerprint #55

Merged
merged 16 commits into from
Oct 19, 2023
Merged

SM-4 - device_data VIN from fingerprint #55

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
54eaccc
VIN Verified
ricardovivanco Oct 11, 2023
5b1f839
Fix comment
ricardovivanco Oct 11, 2023
bb6190a
Fix comment
ricardovivanco Oct 11, 2023
08db1a4
Update test
ricardovivanco Oct 11, 2023
7b93c06
Fix gomock
ricardovivanco Oct 11, 2023
9f7d906
Update Test
ricardovivanco Oct 12, 2023
5200148
use different consumer group
JamesReate Oct 13, 2023
3e3e643
add transaction, go 1.21, update deps
JamesReate Oct 13, 2023
08bcd8f
fix tests, tweak implementation
JamesReate Oct 14, 2023
722c71a
remove vin from status ingest if it comes through
JamesReate Oct 15, 2023
408b87c
fix gomock to uber version change, set missing generic
JamesReate Oct 15, 2023
df91075
remove unused code
JamesReate Oct 15, 2023
a5a9361
remove mocks that caused import cycle in test error
JamesReate Oct 17, 2023
a74f5c1
wip test
JamesReate Oct 18, 2023
f6684f7
Update test
ricardovivanco Oct 19, 2023
25dd992
Update Test
ricardovivanco Oct 19, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 7 additions & 1 deletion README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -131,4 +131,10 @@ Add a migrations:
`$ goose -dir migrations create <migration_name> sql`

Migrate DB to latest:
`$ go run ./cmd/device-data-api migrate`
`$ go run ./cmd/device-data-api migrate`

# Go Ethereum issues

if get ambiguous reference issue, try:
- make sure latest go ethereum version: `go get github.com/ethereum/go-ethereum`
- `go get github.com/btcsuite/btcd/chaincfg/chainhash`
2 changes: 2 additions & 0 deletions charts/device-data-api/values-prod.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,8 @@ env:
AWS_BUCKET_NAME: dimo-network-device-data-export-prod
NATS_URL: nats-prod:4222
KAFKA_BROKERS: kafka-prod-dimo-kafka-kafka-brokers:9092
DEVICE_FINGERPRINT_TOPIC: topic.device.fingerprint
JamesReate marked this conversation as resolved.
Show resolved Hide resolved
DEVICE_FINGERPRINT_CONSUMER_GROUP: device-fingerprint-vin-data
ingress:
enabled: true
className: nginx
Expand Down
2 changes: 2 additions & 0 deletions charts/device-data-api/values.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,8 @@ env:
KAFKA_BROKERS: kafka-dev-dimo-kafka-kafka-brokers:9092
DEVICE_STATUS_TOPIC: topic.device.status
EVENTS_TOPIC: topic.event
DEVICE_FINGERPRINT_TOPIC: topic.device.fingerprint
JamesReate marked this conversation as resolved.
Show resolved Hide resolved
DEVICE_FINGERPRINT_CONSUMER_GROUP: device-fingerprint-vin-data
service:
type: ClusterIP
ports:
Expand Down
11 changes: 11 additions & 0 deletions cmd/device-data-api/main.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,8 @@ import (
"os/signal"
"syscall"

"github.com/DIMO-Network/device-data-api/internal/services/fingerprint"

"github.com/grpc-ecosystem/go-grpc-middleware/v2/interceptors/recovery"

pb "github.com/DIMO-Network/users-api/pkg/grpc"
Expand Down Expand Up @@ -96,6 +98,7 @@ func main() {
if settings.IsKafkaEnabled(&logger) {
eventService := services.NewEventService(&logger, &settings, deps.getKafkaProducer())
startDeviceStatusConsumer(logger, &settings, pdb, eventService, deviceDefsSvc, devicesSvc)
startDeviceFingerprint(logger, &settings, pdb, devicesSvc)
}
if settings.IsWebAPIEnabled(&logger) {
usersConn, err := grpc.Dial(settings.UsersAPIGRPCAddr, grpc.WithTransportCredentials(insecure.NewCredentials()))
Expand Down Expand Up @@ -192,6 +195,14 @@ func startDeviceStatusConsumer(logger zerolog.Logger, settings *config.Settings,
logger.Info().Msg("Device status update consumer started")
}

func startDeviceFingerprint(logger zerolog.Logger, settings *config.Settings, pdb db.Store, deviceAPIService services.DeviceAPIService) {
ctx := context.Background()

if err := fingerprint.RunConsumer(ctx, settings, &logger, pdb, deviceAPIService); err != nil {
logger.Fatal().Err(err).Msg("Failed to create vin credentialer listener")
}
}

// ErrorHandler custom handler to log recovered errors using our logger and return json instead of string
func ErrorHandler(c *fiber.Ctx, err error, logger zerolog.Logger) error {
code := fiber.StatusInternalServerError // Default 500 statuscode
Expand Down
80 changes: 42 additions & 38 deletions go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,27 +1,26 @@
module github.com/DIMO-Network/device-data-api

go 1.20
go 1.21

require (
github.com/DIMO-Network/device-definitions-api v1.0.7
github.com/DIMO-Network/devices-api v1.18.2
github.com/DIMO-Network/shared v0.9.53
github.com/DIMO-Network/device-definitions-api v1.0.23
github.com/DIMO-Network/devices-api v1.23.2
github.com/DIMO-Network/shared v0.10.4
github.com/Shopify/sarama v1.38.1
github.com/aws/aws-sdk-go-v2 v1.19.0
github.com/aws/aws-sdk-go-v2/config v1.18.28
github.com/aws/aws-sdk-go-v2/service/s3 v1.37.0
github.com/burdiyan/kafkautil v0.0.0-20190131162249-eaf83ed22d5b
github.com/docker/go-connections v0.4.0
github.com/elastic/go-elasticsearch/v8 v8.6.0
github.com/ethereum/go-ethereum v1.12.0
github.com/ethereum/go-ethereum v1.13.3
github.com/friendsofgo/errors v0.9.2
github.com/gofiber/contrib/jwt v1.0.2
github.com/gofiber/fiber/v2 v2.47.0
github.com/gofiber/fiber/v2 v2.48.0
github.com/golang-jwt/jwt/v5 v5.0.0
github.com/golang/mock v1.6.0
github.com/google/subcommands v1.2.0
github.com/grpc-ecosystem/go-grpc-middleware v1.4.0
github.com/grpc-ecosystem/go-grpc-middleware/v2 v2.0.0-rc.5
github.com/grpc-ecosystem/go-grpc-middleware/v2 v2.0.0
github.com/grpc-ecosystem/go-grpc-prometheus v1.2.0
github.com/lib/pq v1.10.8
github.com/lovoo/goka v1.1.7
Expand All @@ -32,22 +31,24 @@ require (
github.com/rs/zerolog v1.29.1
github.com/segmentio/ksuid v1.0.4
github.com/smartcar/go-sdk v1.4.0
github.com/stretchr/testify v1.8.2
github.com/stretchr/testify v1.8.4
github.com/swaggo/swag v1.16.1
github.com/testcontainers/testcontainers-go v0.19.0
github.com/testcontainers/testcontainers-go v0.23.0
github.com/tidwall/gjson v1.14.4
github.com/tidwall/sjson v1.2.5
github.com/volatiletech/null/v8 v8.1.2
github.com/volatiletech/sqlboiler/v4 v4.14.2
github.com/volatiletech/strmangle v0.0.4
golang.org/x/exp v0.0.0-20230321023759-10a507213a29
google.golang.org/grpc v1.55.0
github.com/volatiletech/sqlboiler/v4 v4.15.0
github.com/volatiletech/strmangle v0.0.5
go.uber.org/mock v0.2.0
golang.org/x/exp v0.0.0-20230905200255-921286631fa9
google.golang.org/grpc v1.57.0
)

require (
dario.cat/mergo v1.0.0 // indirect
github.com/Azure/go-ansiterm v0.0.0-20230124172434-306776ec8161 // indirect
github.com/MicahParks/keyfunc/v2 v2.1.0 // indirect
github.com/Microsoft/go-winio v0.6.0 // indirect
github.com/Microsoft/go-winio v0.6.1 // indirect
github.com/avast/retry-go v3.0.0+incompatible // indirect
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.4.10 // indirect
github.com/aws/aws-sdk-go-v2/credentials v1.13.27 // indirect
Expand All @@ -65,26 +66,32 @@ require (
github.com/aws/aws-sdk-go-v2/service/ssooidc v1.14.13 // indirect
github.com/aws/aws-sdk-go-v2/service/sts v1.19.3 // indirect
github.com/aws/smithy-go v1.13.5 // indirect
github.com/cenkalti/backoff/v4 v4.2.0 // indirect
github.com/containerd/containerd v1.6.19 // indirect
github.com/btcsuite/btcd/btcec/v2 v2.3.2 // indirect
github.com/btcsuite/btcd/chaincfg/chainhash v1.0.2 // indirect
github.com/cenkalti/backoff/v4 v4.2.1 // indirect
github.com/containerd/containerd v1.7.3 // indirect
github.com/cpuguy83/dockercfg v0.3.1 // indirect
github.com/davecgh/go-spew v1.1.1 // indirect
github.com/docker/distribution v2.8.1+incompatible // indirect
github.com/docker/docker v23.0.1+incompatible // indirect
github.com/decred/dcrd/dcrec/secp256k1/v4 v4.2.0 // indirect
github.com/docker/distribution v2.8.2+incompatible // indirect
github.com/docker/docker v24.0.5+incompatible // indirect
github.com/docker/go-units v0.5.0 // indirect
github.com/eapache/go-resiliency v1.3.0 // indirect
github.com/eapache/go-xerial-snappy v0.0.0-20230111030713-bf00bc1b83b6 // indirect
github.com/eapache/queue v1.1.0 // indirect
github.com/elastic/elastic-transport-go/v8 v8.1.0 // indirect
github.com/ericlagergren/decimal v0.0.0-20211103172832-aca2edc11f73 // indirect
github.com/go-stack/stack v1.8.1 // indirect
github.com/goccy/go-json v0.10.2 // indirect
github.com/gofrs/uuid v4.3.0+incompatible // indirect
github.com/gogo/protobuf v1.3.2 // indirect
github.com/golang/mock v1.6.0 // indirect
github.com/golang/snappy v0.0.5-0.20220116011046-fa5810519dcb // indirect
github.com/google/uuid v1.3.0 // indirect
github.com/hashicorp/errwrap v1.1.0 // indirect
github.com/hashicorp/go-multierror v1.1.1 // indirect
github.com/hashicorp/go-uuid v1.0.3 // indirect
github.com/holiman/uint256 v1.2.3 // indirect
github.com/jcmturner/aescts/v2 v2.0.0 // indirect
github.com/jcmturner/dnsutils/v2 v2.0.0 // indirect
github.com/jcmturner/gofork v1.7.6 // indirect
Expand All @@ -93,35 +100,33 @@ require (
github.com/magiconair/properties v1.8.7 // indirect
github.com/mattn/go-colorable v0.1.13 // indirect
github.com/mattn/go-isatty v0.0.19 // indirect
github.com/mattn/go-runewidth v0.0.14 // indirect
github.com/mattn/go-runewidth v0.0.15 // indirect
github.com/mitchellh/mapstructure v1.5.0 // indirect
github.com/moby/patternmatcher v0.5.0 // indirect
github.com/moby/sys/sequential v0.5.0 // indirect
github.com/moby/term v0.0.0-20221205130635-1aeaba878587 // indirect
github.com/moby/term v0.5.0 // indirect
github.com/morikuni/aec v1.0.0 // indirect
github.com/nats-io/nkeys v0.4.4 // indirect
github.com/nats-io/nuid v1.0.1 // indirect
github.com/opencontainers/go-digest v1.0.0 // indirect
github.com/opencontainers/image-spec v1.1.0-rc2 // indirect
github.com/opencontainers/runc v1.1.4 // indirect
github.com/opencontainers/image-spec v1.1.0-rc4 // indirect
github.com/opencontainers/runc v1.1.5 // indirect
github.com/philhofer/fwd v1.1.2 // indirect
github.com/pierrec/lz4/v4 v4.1.17 // indirect
github.com/pmezard/go-difflib v1.0.0 // indirect
github.com/rcrowley/go-metrics v0.0.0-20201227073835-cf1acfcdf475 // indirect
github.com/rivo/uniseg v0.4.4 // indirect
github.com/savsgio/dictpool v0.0.0-20221023140959-7bf2e61cea94 // indirect
github.com/savsgio/gotils v0.0.0-20230208104028-c358bd845dee // indirect
github.com/sirupsen/logrus v1.9.0 // indirect
github.com/spf13/cast v1.5.0 // indirect
github.com/swaggo/files/v2 v2.0.0 // indirect
github.com/syndtr/goleveldb v1.0.1-0.20210819022825-2ae1ddf74ef7 // indirect
github.com/tinylib/msgp v1.1.8 // indirect
github.com/volatiletech/inflect v0.0.1 // indirect
github.com/volatiletech/randomize v0.0.1 // indirect
golang.org/x/mod v0.10.0 // indirect
golang.org/x/sync v0.2.0 // indirect
golang.org/x/mod v0.12.0 // indirect
golang.org/x/sync v0.3.0 // indirect
golang.org/x/xerrors v0.0.0-20220907171357-04be3eba64a2 // indirect
gotest.tools/v3 v3.4.0 // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20230525234030-28d5490b6b19 // indirect
)

require (
Expand All @@ -131,14 +136,14 @@ require (
github.com/avast/retry-go/v4 v4.3.3 // indirect
github.com/beorn7/perks v1.0.1 // indirect
github.com/cespare/xxhash/v2 v2.2.0 // indirect
github.com/go-openapi/jsonpointer v0.19.6 // indirect
github.com/go-openapi/jsonpointer v0.20.0 // indirect
github.com/go-openapi/jsonreference v0.20.2 // indirect
github.com/go-openapi/spec v0.20.9 // indirect
github.com/go-openapi/swag v0.22.3 // indirect
github.com/go-openapi/swag v0.22.4 // indirect
github.com/gofiber/swagger v0.1.12
github.com/golang/protobuf v1.5.3 // indirect
github.com/josharian/intern v1.0.0 // indirect
github.com/klauspost/compress v1.16.5 // indirect
github.com/klauspost/compress v1.16.7 // indirect
github.com/mailru/easyjson v0.7.7 // indirect
github.com/matttproud/golang_protobuf_extensions v1.0.4 // indirect
github.com/nats-io/nats.go v1.25.0
Expand All @@ -148,14 +153,13 @@ require (
github.com/tidwall/match v1.1.1 // indirect
github.com/tidwall/pretty v1.2.1 // indirect
github.com/valyala/bytebufferpool v1.0.0 // indirect
github.com/valyala/fasthttp v1.47.0 // indirect
github.com/valyala/fasthttp v1.48.0 // indirect
github.com/valyala/tcplisten v1.0.0 // indirect
golang.org/x/crypto v0.9.0 // indirect
golang.org/x/net v0.10.0 // indirect
golang.org/x/sys v0.9.0 // indirect
golang.org/x/text v0.9.0 // indirect
golang.org/x/tools v0.9.1 // indirect
google.golang.org/genproto v0.0.0-20230410155749-daa745c078e1 // indirect
golang.org/x/crypto v0.13.0 // indirect
golang.org/x/net v0.15.0 // indirect
golang.org/x/sys v0.12.0 // indirect
golang.org/x/text v0.13.0 // indirect
golang.org/x/tools v0.13.0 // indirect
google.golang.org/protobuf v1.31.0
gopkg.in/yaml.v3 v3.0.1 // indirect
)
Loading
Loading