Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Vector Type #8215

Merged
merged 18 commits into from
Aug 12, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions .changeset/nice-eyes-tan.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
---
"@firebase/firestore": minor
"firebase": minor
---

Add support for reading and writing Firestore vectors.
10 changes: 10 additions & 0 deletions common/api-review/firestore-lite.api.md
Original file line number Diff line number Diff line change
Expand Up @@ -460,6 +460,16 @@ export function updateDoc<AppModelType, DbModelType extends DocumentData>(refere
// @public
export function updateDoc<AppModelType, DbModelType extends DocumentData>(reference: DocumentReference<AppModelType, DbModelType>, field: string | FieldPath, value: unknown, ...moreFieldsAndValues: unknown[]): Promise<void>;

// @public
export function vector(values?: number[]): VectorValue;

// @public
export class VectorValue {
/* Excluded from this release type: __constructor */
isEqual(other: VectorValue): boolean;
toArray(): number[];
}

// @public
export function where(fieldPath: string | FieldPath, opStr: WhereFilterOp, value: unknown): QueryFieldFilterConstraint;

Expand Down
10 changes: 10 additions & 0 deletions common/api-review/firestore.api.md
Original file line number Diff line number Diff line change
Expand Up @@ -745,6 +745,16 @@ export function updateDoc<AppModelType, DbModelType extends DocumentData>(refere
// @public
export function updateDoc<AppModelType, DbModelType extends DocumentData>(reference: DocumentReference<AppModelType, DbModelType>, field: string | FieldPath, value: unknown, ...moreFieldsAndValues: unknown[]): Promise<void>;

// @public
export function vector(values?: number[]): VectorValue;

// @public
export class VectorValue {
/* Excluded from this release type: __constructor */
isEqual(other: VectorValue): boolean;
toArray(): number[];
}

// @public
export function waitForPendingWrites(firestore: Firestore): Promise<void>;

Expand Down
555 changes: 0 additions & 555 deletions docs-devsite/_toc.yaml

This file was deleted.

27 changes: 27 additions & 0 deletions docs-devsite/firestore_.md
Original file line number Diff line number Diff line change
Expand Up @@ -124,6 +124,8 @@ https://github.com/firebase/firebase-js-sdk
| [endBefore(snapshot)](./firestore_.md#endbefore_9a4477f) | Creates a [QueryEndAtConstraint](./firestore_.queryendatconstraint.md#queryendatconstraint_class) that modifies the result set to end before the provided document (exclusive). The end position is relative to the order of the query. The document must contain all of the fields provided in the orderBy of the query. |
| [startAfter(snapshot)](./firestore_.md#startafter_9a4477f) | Creates a [QueryStartAtConstraint](./firestore_.querystartatconstraint.md#querystartatconstraint_class) that modifies the result set to start after the provided document (exclusive). The starting position is relative to the order of the query. The document must contain all of the fields provided in the orderBy of the query. |
| [startAt(snapshot)](./firestore_.md#startat_9a4477f) | Creates a [QueryStartAtConstraint](./firestore_.querystartatconstraint.md#querystartatconstraint_class) that modifies the result set to start at the provided document (inclusive). The starting position is relative to the order of the query. The document must contain all of the fields provided in the <code>orderBy</code> of this query. |
| <b>function(values, ...)</b> |
| [vector(values)](./firestore_.md#vector_0dbdaf2) | Creates a new <code>VectorValue</code> constructed with a copy of the given array of numbers. |

## Classes

Expand Down Expand Up @@ -155,6 +157,7 @@ https://github.com/firebase/firebase-js-sdk
| [SnapshotMetadata](./firestore_.snapshotmetadata.md#snapshotmetadata_class) | Metadata about a snapshot, describing the state of the snapshot. |
| [Timestamp](./firestore_.timestamp.md#timestamp_class) | A <code>Timestamp</code> represents a point in time independent of any time zone or calendar, represented as seconds and fractions of seconds at nanosecond resolution in UTC Epoch time.<!-- -->It is encoded using the Proleptic Gregorian Calendar which extends the Gregorian calendar backwards to year one. It is encoded assuming all minutes are 60 seconds long, i.e. leap seconds are "smeared" so that no leap second table is needed for interpretation. Range is from 0001-01-01T00:00:00Z to 9999-12-31T23:59:59.999999999Z.<!-- -->For examples and further specifications, refer to the [Timestamp definition](https://github.com/google/protobuf/blob/master/src/google/protobuf/timestamp.proto)<!-- -->. |
| [Transaction](./firestore_.transaction.md#transaction_class) | A reference to a transaction.<!-- -->The <code>Transaction</code> object passed to a transaction's <code>updateFunction</code> provides the methods to read and write data within the transaction context. See [runTransaction()](./firestore_.md#runtransaction_6f03ec4)<!-- -->. |
| [VectorValue](./firestore_.vectorvalue.md#vectorvalue_class) | Represents a vector type in Firestore documents. Create an instance with . VectorValue |
| [WriteBatch](./firestore_.writebatch.md#writebatch_class) | A write batch, used to perform multiple writes as a single atomic unit.<!-- -->A <code>WriteBatch</code> object can be acquired by calling [writeBatch()](./firestore_.md#writebatch_231a8e0)<!-- -->. It provides methods for adding writes to the write batch. None of the writes will be committed (or visible locally) until [WriteBatch.commit()](./firestore_.writebatch.md#writebatchcommit) is called. |

## Interfaces
Expand Down Expand Up @@ -2452,6 +2455,30 @@ export declare function startAt<AppModelType, DbModelType extends DocumentData>(

A [QueryStartAtConstraint](./firestore_.querystartatconstraint.md#querystartatconstraint_class) to pass to `query()`<!-- -->.

## function(values, ...)

### vector(values) {:#vector_0dbdaf2}

Creates a new `VectorValue` constructed with a copy of the given array of numbers.

<b>Signature:</b>

```typescript
export declare function vector(values?: number[]): VectorValue;
```

#### Parameters

| Parameter | Type | Description |
| --- | --- | --- |
| values | number\[\] | Create a <code>VectorValue</code> instance with a copy of this array of numbers. |

<b>Returns:</b>

[VectorValue](./firestore_.vectorvalue.md#vectorvalue_class)

A new `VectorValue` constructed with a copy of the given array of numbers.

## CACHE\_SIZE\_UNLIMITED

Constant used to indicate the LRU garbage collection should be disabled. Set this value as the `cacheSizeBytes` on the settings passed to the [Firestore](./firestore_.firestore.md#firestore_class) instance.
Expand Down
27 changes: 27 additions & 0 deletions docs-devsite/firestore_lite.md
Original file line number Diff line number Diff line change
Expand Up @@ -89,6 +89,8 @@ https://github.com/firebase/firebase-js-sdk
| [endBefore(snapshot)](./firestore_lite.md#endbefore_9a4477f) | Creates a [QueryEndAtConstraint](./firestore_.queryendatconstraint.md#queryendatconstraint_class) that modifies the result set to end before the provided document (exclusive). The end position is relative to the order of the query. The document must contain all of the fields provided in the orderBy of the query. |
| [startAfter(snapshot)](./firestore_lite.md#startafter_9a4477f) | Creates a [QueryStartAtConstraint](./firestore_.querystartatconstraint.md#querystartatconstraint_class) that modifies the result set to start after the provided document (exclusive). The starting position is relative to the order of the query. The document must contain all of the fields provided in the orderBy of the query. |
| [startAt(snapshot)](./firestore_lite.md#startat_9a4477f) | Creates a [QueryStartAtConstraint](./firestore_.querystartatconstraint.md#querystartatconstraint_class) that modifies the result set to start at the provided document (inclusive). The starting position is relative to the order of the query. The document must contain all of the fields provided in the <code>orderBy</code> of this query. |
| <b>function(values, ...)</b> |
| [vector(values)](./firestore_lite.md#vector_0dbdaf2) | Creates a new <code>VectorValue</code> constructed with a copy of the given array of numbers. |

## Classes

Expand Down Expand Up @@ -117,6 +119,7 @@ https://github.com/firebase/firebase-js-sdk
| [QueryStartAtConstraint](./firestore_lite.querystartatconstraint.md#querystartatconstraint_class) | A <code>QueryStartAtConstraint</code> is used to exclude documents from the start of a result set returned by a Firestore query. <code>QueryStartAtConstraint</code>s are created by invoking [startAt()](./firestore_.md#startat_9a4477f) or [startAfter()](./firestore_.md#startafter_9a4477f) and can then be passed to [query()](./firestore_.md#query_9f7b0f4) to create a new query instance that also contains this <code>QueryStartAtConstraint</code>. |
| [Timestamp](./firestore_lite.timestamp.md#timestamp_class) | A <code>Timestamp</code> represents a point in time independent of any time zone or calendar, represented as seconds and fractions of seconds at nanosecond resolution in UTC Epoch time.<!-- -->It is encoded using the Proleptic Gregorian Calendar which extends the Gregorian calendar backwards to year one. It is encoded assuming all minutes are 60 seconds long, i.e. leap seconds are "smeared" so that no leap second table is needed for interpretation. Range is from 0001-01-01T00:00:00Z to 9999-12-31T23:59:59.999999999Z.<!-- -->For examples and further specifications, refer to the [Timestamp definition](https://github.com/google/protobuf/blob/master/src/google/protobuf/timestamp.proto)<!-- -->. |
| [Transaction](./firestore_lite.transaction.md#transaction_class) | A reference to a transaction.<!-- -->The <code>Transaction</code> object passed to a transaction's <code>updateFunction</code> provides the methods to read and write data within the transaction context. See [runTransaction()](./firestore_.md#runtransaction_6f03ec4)<!-- -->. |
| [VectorValue](./firestore_lite.vectorvalue.md#vectorvalue_class) | Represents a vector type in Firestore documents. Create an instance with . VectorValue |
| [WriteBatch](./firestore_lite.writebatch.md#writebatch_class) | A write batch, used to perform multiple writes as a single atomic unit.<!-- -->A <code>WriteBatch</code> object can be acquired by calling [writeBatch()](./firestore_.md#writebatch_231a8e0)<!-- -->. It provides methods for adding writes to the write batch. None of the writes will be committed (or visible locally) until [WriteBatch.commit()](./firestore_.writebatch.md#writebatchcommit) is called. |

## Interfaces
Expand Down Expand Up @@ -1568,6 +1571,30 @@ export declare function startAt<AppModelType, DbModelType extends DocumentData>(

A [QueryStartAtConstraint](./firestore_.querystartatconstraint.md#querystartatconstraint_class) to pass to `query()`<!-- -->.

## function(values, ...)

### vector(values) {:#vector_0dbdaf2}

Creates a new `VectorValue` constructed with a copy of the given array of numbers.

<b>Signature:</b>

```typescript
export declare function vector(values?: number[]): VectorValue;
```

#### Parameters

| Parameter | Type | Description |
| --- | --- | --- |
| values | number\[\] | Create a <code>VectorValue</code> instance with a copy of this array of numbers. |

<b>Returns:</b>

[VectorValue](./firestore_lite.vectorvalue.md#vectorvalue_class)

A new `VectorValue` constructed with a copy of the given array of numbers.

## AddPrefixToKeys

Returns a new map where every key is prefixed with the outer key appended to a dot.
Expand Down
5 changes: 4 additions & 1 deletion packages/firestore/lite/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -127,7 +127,8 @@ export {
arrayRemove,
arrayUnion,
serverTimestamp,
deleteField
deleteField,
vector
} from '../src/lite-api/field_value_impl';

export {
Expand All @@ -138,6 +139,8 @@ export {
snapshotEqual
} from '../src/lite-api/snapshot';

export { VectorValue } from '../src/lite-api/vector_value';

export { WriteBatch, writeBatch } from '../src/lite-api/write_batch';

export { TransactionOptions } from '../src/lite-api/transaction_options';
Expand Down
5 changes: 4 additions & 1 deletion packages/firestore/src/api.ts
Original file line number Diff line number Diff line change
Expand Up @@ -172,9 +172,12 @@ export {
arrayUnion,
deleteField,
increment,
serverTimestamp
serverTimestamp,
vector
} from './api/field_value_impl';

export { VectorValue } from './lite-api/vector_value';

export { LogLevelString as LogLevel, setLogLevel } from './util/log';

export { Bytes } from './api/bytes';
Expand Down
3 changes: 2 additions & 1 deletion packages/firestore/src/api/field_value_impl.ts
Original file line number Diff line number Diff line change
Expand Up @@ -20,5 +20,6 @@ export {
arrayRemove,
arrayUnion,
serverTimestamp,
deleteField
deleteField,
vector
} from '../lite-api/field_value_impl';
27 changes: 26 additions & 1 deletion packages/firestore/src/index/firestore_index_value_writer.ts
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,11 @@ import {
normalizeNumber,
normalizeTimestamp
} from '../model/normalize';
import { isMaxValue } from '../model/values';
import {
isVectorValue,
VECTOR_MAP_VECTORS_KEY,
isMaxValue
} from '../model/values';
import { ArrayValue, MapValue, Value } from '../protos/firestore_proto_api';
import { fail } from '../util/assert';
import { isNegativeZero } from '../util/types';
Expand All @@ -41,6 +45,7 @@ const INDEX_TYPE_BLOB = 30;
const INDEX_TYPE_REFERENCE = 37;
const INDEX_TYPE_GEOPOINT = 45;
const INDEX_TYPE_ARRAY = 50;
const INDEX_TYPE_VECTOR = 53;
const INDEX_TYPE_MAP = 55;
const INDEX_TYPE_REFERENCE_SEGMENT = 60;

Expand Down Expand Up @@ -121,6 +126,8 @@ export class FirestoreIndexValueWriter {
} else if ('mapValue' in indexValue) {
if (isMaxValue(indexValue)) {
this.writeValueTypeLabel(encoder, Number.MAX_SAFE_INTEGER);
} else if (isVectorValue(indexValue)) {
this.writeIndexVector(indexValue.mapValue!, encoder);
} else {
this.writeIndexMap(indexValue.mapValue!, encoder);
this.writeTruncationMarker(encoder);
Expand Down Expand Up @@ -160,6 +167,24 @@ export class FirestoreIndexValueWriter {
}
}

private writeIndexVector(
mapIndexValue: MapValue,
encoder: DirectionalIndexByteEncoder
): void {
const map = mapIndexValue.fields || {};
this.writeValueTypeLabel(encoder, INDEX_TYPE_VECTOR);

// Vectors sort first by length
const key = VECTOR_MAP_VECTORS_KEY;
const length = map[key].arrayValue?.values?.length || 0;
this.writeValueTypeLabel(encoder, INDEX_TYPE_NUMBER);
encoder.writeNumber(normalizeNumber(length));

// Vectors then sort by position value
this.writeIndexString(key, encoder);
this.writeIndexValueAux(map[key], encoder);
}

private writeIndexArray(
arrayIndexValue: ArrayValue,
encoder: DirectionalIndexByteEncoder
Expand Down
12 changes: 12 additions & 0 deletions packages/firestore/src/lite-api/field_value_impl.ts
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@ import {
NumericIncrementFieldValueImpl,
ServerTimestampFieldValueImpl
} from './user_data_reader';
import { VectorValue } from './vector_value';

/**
* Returns a sentinel for use with {@link @firebase/firestore/lite#(updateDoc:1)} or
Expand Down Expand Up @@ -97,3 +98,14 @@ export function arrayRemove(...elements: unknown[]): FieldValue {
export function increment(n: number): FieldValue {
return new NumericIncrementFieldValueImpl('increment', n);
}

/**
* Creates a new `VectorValue` constructed with a copy of the given array of numbers.
*
* @param values - Create a `VectorValue` instance with a copy of this array of numbers.
*
* @returns A new `VectorValue` constructed with a copy of the given array of numbers.
*/
export function vector(values?: number[]): VectorValue {
return new VectorValue(values);
}
44 changes: 42 additions & 2 deletions packages/firestore/src/lite-api/user_data_reader.ts
Original file line number Diff line number Diff line change
Expand Up @@ -41,12 +41,17 @@ import {
NumericIncrementTransformOperation,
ServerTimestampTransform
} from '../model/transform_operation';
import {
TYPE_KEY,
VECTOR_MAP_VECTORS_KEY,
VECTOR_VALUE_SENTINEL
} from '../model/values';
import { newSerializer } from '../platform/serializer';
import {
MapValue as ProtoMapValue,
Value as ProtoValue
} from '../protos/firestore_proto_api';
import { toNumber } from '../remote/number_serializer';
import { toDouble, toNumber } from '../remote/number_serializer';
import {
JsonProtoSerializer,
toBytes,
Expand All @@ -69,6 +74,7 @@ import {
WithFieldValue
} from './reference';
import { Timestamp } from './timestamp';
import { VectorValue } from './vector_value';

const RESERVED_FIELD_REGEX = /^__.*__$/;

Expand Down Expand Up @@ -901,13 +907,46 @@ function parseScalarValue(
value._key.path
)
};
} else if (value instanceof VectorValue) {
return parseVectorValue(value, context);
} else {
throw context.createError(
`Unsupported field value: ${valueDescription(value)}`
);
}
}

/**
* Creates a new VectorValue proto value (using the internal format).
MarkDuckworth marked this conversation as resolved.
Show resolved Hide resolved
*/
export function parseVectorValue(
value: VectorValue,
context: ParseContextImpl
): ProtoValue {
const mapValue: ProtoMapValue = {
fields: {
[TYPE_KEY]: {
stringValue: VECTOR_VALUE_SENTINEL
},
[VECTOR_MAP_VECTORS_KEY]: {
arrayValue: {
values: value.toArray().map(value => {
if (typeof value !== 'number') {
throw context.createError(
'VectorValues must only contain numeric values.'
);
}

return toDouble(context.serializer, value);
})
}
}
}
};

return { mapValue };
}

/**
* Checks whether an object looks like a JSON object that should be converted
* into a struct. Normal class/prototype instances are considered to look like
Expand All @@ -925,7 +964,8 @@ function looksLikeJsonObject(input: unknown): boolean {
!(input instanceof GeoPoint) &&
!(input instanceof Bytes) &&
!(input instanceof DocumentReference) &&
!(input instanceof FieldValue)
!(input instanceof FieldValue) &&
!(input instanceof VectorValue)
);
}

Expand Down
18 changes: 17 additions & 1 deletion packages/firestore/src/lite-api/user_data_writer.ts
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ import {
getPreviousValue
} from '../model/server_timestamps';
import { TypeOrder } from '../model/type_order';
import { typeOrder } from '../model/values';
import { VECTOR_MAP_VECTORS_KEY, typeOrder } from '../model/values';
import {
ApiClientObjectMap,
ArrayValue as ProtoArrayValue,
Expand All @@ -48,6 +48,7 @@ import { forEach } from '../util/obj';

import { GeoPoint } from './geo_point';
import { Timestamp } from './timestamp';
import { VectorValue } from './vector_value';

export type ServerTimestampBehavior = 'estimate' | 'previous' | 'none';

Expand Down Expand Up @@ -85,6 +86,8 @@ export abstract class AbstractUserDataWriter {
return this.convertArray(value.arrayValue!, serverTimestampBehavior);
case TypeOrder.ObjectValue:
return this.convertObject(value.mapValue!, serverTimestampBehavior);
case TypeOrder.VectorValue:
return this.convertVectorValue(value.mapValue!);
default:
throw fail('Invalid value type: ' + JSON.stringify(value));
}
Expand All @@ -111,6 +114,19 @@ export abstract class AbstractUserDataWriter {
return result;
}

/**
* @internal
*/
convertVectorValue(mapValue: ProtoMapValue): VectorValue {
const values = mapValue.fields?.[
VECTOR_MAP_VECTORS_KEY
].arrayValue?.values?.map(value => {
return normalizeNumber(value.doubleValue);
});

return new VectorValue(values);
}

private convertGeoPoint(value: ProtoLatLng): GeoPoint {
return new GeoPoint(
normalizeNumber(value.latitude),
Expand Down
Loading
Loading