use prefix index on value instead of separate string_value column

CHANGELOG.md

@@ -25,19 +25,18 @@ Version 6 contains a number of changes to improve the security and performance o
 - `ModelHandler` will no longer throw a model not found exception if the model no longer exists. Instead, the meta value will return `null`. This is more in line with the existing behavior of the `ModelCollectionHandler`.
 - `ModelCollectionHandler` will now validate that the encoded collection class is a valid Eloquent collection before attempting to instantiate it during unserialization. If the class is invalid,  an instance of `Illuminate\Database\Eloquent\Collection` will be used instead.
 - `ModelCollectionHandler` will now validate that the encoded class of each entry is a valid Eloquent Model before attempting to instantiate it during unserialization. If the class is invalid, that entry in the collection will be omitted.
-- Added `getStringValue(): ?string` and `getNumericValue(): null|int|float` methods to `HandlerInterface` which should convert the original value into a format that can be indexed, if possible.
-- Added `isIdempotent(): bool` method to `HandlerInterface` which should indicate whether multiple calls to the `serialize()` method with the same value will produce the same serialized output. This is used to determine if the complete serialized value can be used when searching for meta values.
+- Added `getNumericValue(): null|int|float` method to `HandlerInterface` which should convert the original value into a numeric format for indexing, if relevant for the data type.
 - Added `useHmacVerification(): bool` method to `HandlerInterface` which should indicate whether the integrity of the serialized data should be verified with an HMAC.
 
 ### New Commands
 
 - Added `metable:refresh` artisan command which will decode and re-encode all meta values in the database. This is useful if you have changed the data type handlers and need to update the serialized data and indexes in the database.
 
-### Searching Metables By Meta Value 
+### Efficient Value Search 
 
-- the Metable `whereMeta()`, `whereMetaIn()`, and `orderByMeta()` query scopes will now scan the indexed `string_value` column instead of the serialized `value` column. This greatly improves performance when searching for meta values against larger datasets.
+- The Metable `whereMeta()`, `whereMetaIn()`, and `orderByMeta()` query scopes can now leverage a prefix index on the ``meta.value`` column. This greatly improves performance when searching for meta values against larger datasets when using applicable operators, e.g. `=`, `%`, `>`, `>=`, `<`, `<=`, `<>`, `LIKE` (no leading wildcard).
 - `whereMetaNumeric()` and `orderByMetaNumeric()` query scopes will now scan the indexed `numeric_value` column instead of the serialized `value` column. This greatly improves performance when searching for meta values against larger datasets.
-- `whereMetaNumeric()` query scope will now accept a value of any type. It will be converted to an integer or float by the handler. This is more consistent with the behaviour of the other query scopes.  
+- `whereMetaNumeric()` query scope will now accept a value of any type. It will be converted to an integer or float by the handler. This is more consistent with the behaviour of the other query scopes.
 - Added additional query scopes to more easily search meta values based on different criteria:
   - `whereMetaInNumeric()`
   - `whereMetaNotIn()`
@@ -48,21 +47,24 @@ Version 6 contains a number of changes to improve the security and performance o
   - `whereMetaNotBetweenNumeric()`
   - `whereMetaIsNull()`
   - `whereMetaIsModel()`
-- If the data type handlers cannot convert the search value provided to a whereMeta* query scope to a string or numeric value (as appropriate for the method), then an exception will be thrown.
+- If the data type handlers cannot convert the search value provided to a ``whereMeta*Numeric()`` query scope to a numeric value, then an exception will be thrown.
 
 ### Metable Casting
 
 - Added support for casting meta values to specific types by defining the `$castMeta` property or `castMeta(): array` method on the model. This is similar to the `casts` property of Eloquent Models used for attributes. All cast types supported by Eloquent Models are also available for Meta values.Values will be cast before values are stored in the database to ensure that they are indexed consistently
-- the `encrypted:` cast prefix can be combined with any other cast type to cast to the desired type before the value is encrypted to be stored it in the database. Encrypted values are not searchable.
-- A value of `null` is ignored by all cast types. 
 - Added `mergeMetaCasts()` method which can be used to override the defined cast on a meta key at runtime.
 
-### Meta Attributes
+### Encrypt Meta
+
+- Added the `setMetaEncrypted()` method which will encrypt data before storing it in the database and decrypt it when retrieving it. This is useful for storing sensitive data in the meta table. 
+- prefixing a meta cast with `encrypted:` will automatically encrypt all values for that meta key.
+
+### Metable Attributes
 
 - Added optional trait `MetableAttributes` which can further extend the `Metable` trait allowing access to meta values as model attributes using a `meta_` prefix. This can be useful for type hinting, IDE autocompletion, static analysis, and usage in Blade templates. 
 
 ### Meta
-- Added `$meta->string_value` and `$meta->numeric_value` attributes, which are used for optimizing queries filtering by meta value
+- Added `$meta->numeric_value` attributes, which are used for optimizing queries filtering by meta value
 - Added `$meta->hmac` attribute, which is used by some data type handlers to validate that the payload has not been tampered with.
 - Added `$meta->raw_value` virtual attribute, which exposes the raw serialized value of the meta key. This is useful for debugging purposes.
 - Added `encrypt()` method, used internally by the `Metable::setMetaEncrypted()` method

UPGRADING.md

@@ -22,8 +22,7 @@
 ### Handlers
 
 * If you have any custom data types, you will need to implement the new methods from the `HandlerInterface`:
-  * `getStringValue(): ?string` and `getNumericValue(): null|int|float`: These are used to populate the new indexed search columns. You may return `null` if the value cannot be converted into the specified format or does not need to be searchable.
-  * `isIdempotent(): bool`: This method should indicate whether multiple calls to the `serialize()` method with the same value will produce the same serialized output. This is used to determine if the complete serialized value can be used when searching for meta values.  
+  * `getNumericValue(): null|int|float`: used to populate the new indexed numeric search column. You may return `null` if the value cannot be converted into a meaningful numeric value or does not need to be searchable.
   * `useHmacVerification(): bool`: if the integrity of the serialized data should be verified with a HMAC, return `true`. If unserializing this data type is safe without HMAC verification, you may return `false`. 
 
 ### Update Existing Data

config/metable.php

@@ -73,21 +73,13 @@
     ],
 
     /**
-     * Whether to index complex data types (arrays, objects, etc).
-     * If enabled the value will be serialized and the first 255 characters will be indexed.
-     * This allows for using whereMeta*() query scopes on serialized values, but may have
-     * performance and disk usage implications for large data sets.
+     * Number of bytes of the to index for strings
+     * This value is used to determine the length of the prefix index on the value column in the database.
+     * Higher values allow for better precision when querying, but will use more disk space in the database.
      *
-     * If you do not intend to query meta values containing complex data types, you should leave this disabled.
-     * If you change this value, it may be necessary to refresh the meta table with the `artisan metable:refresh` command.
-     */
-    'indexComplexDataTypes' => false,
-
-    /**
-     * Number of bytes to index for strings and complex data types.
-     * This value is used to determine the length of the index column in the database.
-     * Higher values allow for better precision when querying,
-     * but will use more disk space in the database.
+     * Prefix index is only supported on the 'mysql', 'mariadb', 'pgsql', and 'sqlite' database drivers.
+     *
+     * Set to 0 before running the migration to disable the index.
      */
     'stringValueIndexLength' => 255,
 ];

docs/source/datatypes.rst

@@ -74,7 +74,7 @@ String
 ^^^^^^^^
 +----------------------+-----+
 | Handler              | ``\Plank\Metable\DataType\StringHandler`` |
-| String Query Scopes  | Yes, first ``metable.stringValueIndexLength`` characters indexed |
+| String Query Scopes  | Yes |
 | Numeric Query Scopes | if string is numeric |
 | Other Query Scopes   |     |
 +----------------------+-----+
@@ -126,7 +126,7 @@ Eloquent Collections
 
 +----------------------+-----+
 | Handler              | ``\Plank\Metable\DataType\ModelCollectionHandler`` |
-| String Query Scopes  | No  |
+| String Query Scopes  | Yes |
 | Numeric Query Scopes | No  |
 | Other Query Scopes   |     |
 +----------------------+-----+
@@ -145,7 +145,7 @@ DateTime & Carbon
 ^^^^^^^^^^^^^^^^^^
 +----------------------+-----+
 | Handler              | ``\Plank\Metable\DataType\DateTimeHandler`` |
-| String Query Scopes  | Yes (UTC format) |
+| String Query Scopes  | Yes |
 | Numeric Query Scopes | Yes (timestamp) |
 | Other Query Scopes   |     |
 +----------------------+-----+
@@ -162,7 +162,7 @@ DateTimeImmutable & CarbonImmutable
 
 +----------------------+-----+
 | Handler              | ``\Plank\Metable\DataType\DateTimeImmutableHandler`` |
-| String Query Scopes  | Yes (UTC format) |
+| String Query Scopes  | Yes |
 | Numeric Query Scopes | Yes (timestamp) |
 | Other Query Scopes   |     |
 +----------------------+-----+
@@ -209,7 +209,7 @@ Objects and Arrays
 
 +----------------------+-----+
 | Handler              | ``\Plank\Metable\DataType\SignedSerializeHandler`` |
-| String Query Scopes  | if ``metable.indexComplexDataTypes`` is enabled  |
+| String Query Scopes  | Yes |
 | Numeric Query Scopes | No  |
 | Other Query Scopes   |     |
 +----------------------+-----+
@@ -236,7 +236,7 @@ Array
 
 +----------------------+-----+
 | Handler              | ``\Plank\Metable\DataType\ArrayHandler`` |
-| String Query Scopes  | if ``metable.indexComplexDataTypes`` is enabled  |
+| String Query Scopes  | Yes |
 | Numeric Query Scopes | No  |
 | Other Query Scopes   |     |
 +----------------------+-----+
@@ -268,7 +268,7 @@ Serializable
 
 +----------------------+-----+
 | Handler              | ``\Plank\Metable\DataType\ArrayHandler`` |
-| String Query Scopes  | if ``metable.indexComplexDataTypes`` is enabled  |
+| String Query Scopes  | Yes |
 | Numeric Query Scopes | No  |
 | Other Query Scopes   |     |
 +----------------------+-----+
@@ -296,7 +296,7 @@ Plain Objects
 
 +----------------------+-----+
 | Handler              | ``\Plank\Metable\DataType\ArrayHandler`` |
-| String Query Scopes  | if ``metable.indexComplexDataTypes`` is enabled  |
+| String Query Scopes  | Yes |
 | Numeric Query Scopes | No  |
 | Other Query Scopes   |     |
 +----------------------+-----+

docs/source/querying_meta.rst

@@ -43,9 +43,6 @@ String Value Query Scopes
 
 All string-based query scopes using lexicographic comparison to look up values. This means that the values are compared alphabetically as strings. This can lead to unexpected results when comparing numbers, e.g. ``'11'`` is greater than ``'100'``.
 
-By default, only the first 255 characters of a string are indexed (can be adjusted with the ``metable.stringValueIndexLength`` config). When querying by longer values, characters exceeding the limit will be ignored when determining if the criteria matches. The ``whereMeta()`` method will attempt to work around this by comparing the entire serialized value after the results have been filtered by the indexed portion (other query scopes will not do this).
-
-
 The ``whereMeta()`` method can be used to compare the value using any of the operators accepted by the Laravel query builder's ``where()`` method.
 
 ::

migrations/2024_04_14_000000_add_meta_search_columns.php

@@ -16,15 +16,26 @@ public function up()
     {
         Schema::table('meta', function (Blueprint $table) {
             $table->decimal('numeric_value', 36, 16)->nullable();
-            $table->string(
-                'string_value',
-                config('metable.stringValueIndexLength', 255)
-            )->nullable();
             $table->string('hmac', 64)->nullable();
+
             $table->dropIndex(['key', 'metable_type']);
             $table->dropIndex(['key']);
             $table->index(['key', 'metable_type', 'numeric_value']);
-            $table->index(['key', 'metable_type', 'string_value']);
+
+            $stringIndexLength = (int)config('metable.stringValueIndexLength', 255);
+            if ($stringIndexLength > 0 && $driver = $this->detectDriverName()) {
+                if (in_array($driver, ['mysql', 'mariadb'])) {
+                    $table->rawIndex(
+                        "metable_type, key, value($stringIndexLength)",
+                        'value_string_prefix_index'
+                    );
+                } elseif (in_array($driver, ['pgsql', 'sqlite'])) {
+                    $table->rawIndex(
+                        "metable_type, key, substr(value, 1, $stringIndexLength)",
+                        'value_string_prefix_index'
+                    );
+                }
+            }
         });
     }
 
@@ -36,12 +47,35 @@ public function up()
     public function down()
     {
         Schema::table('meta', function (Blueprint $table) {
-            $table->dropIndex(['key', 'metable_type', 'string_value']);
+            $stringIndexLength = (int)config('metable.stringValueIndexLength', 255);
+            if ($stringIndexLength > 0
+                && in_array($this->detectDriverName(), ['mysql', 'mariadb', 'pgsql', 'sqlite'])
+            ) {
+                $table->dropIndex('value_string_prefix_index');
+            }
+
             $table->dropIndex(['key', 'metable_type', 'numeric_value']);
             $table->index(['key']);
             $table->index(['key', 'metable_type']);
-            $table->dropColumn('string_value');
+            $table->dropColumn('hmac');
             $table->dropColumn('numeric_value');
         });
     }
+
+    private function detectDriverName(): ?string
+    {
+        /** @var \Illuminate\Database\Migrations\Migrator $migrator */
+        $migrator = app('migrator');
+        $repository = $migrator->getRepository();
+
+        if (method_exists($repository, 'getConnectionResolver')) {
+            $resolver = $repository->getConnectionResolver();
+        } else {
+            $resolver = DB::getFacadeRoot();
+        }
+
+        return $resolver->connection(
+            $this->getConnection() ?? $migrator->getConnection()
+        )->getDriverName();
+    }
 }

src/DataType/ArrayHandler.php

@@ -50,24 +50,6 @@ public function getNumericValue(mixed $value): null|int|float
         return null;
     }
 
-    public function getStringValue(mixed $value): null|string
-    {
-        if (!config('metable.indexComplexDataTypes', false)) {
-            return null;
-        }
-
-        return substr(
-            json_encode($value, JSON_THROW_ON_ERROR),
-            0,
-            config('metable.stringValueIndexLength', 255)
-        );
-    }
-
-    public function isIdempotent(): bool
-    {
-        return true;
-    }
-
     public function useHmacVerification(): bool
     {
         return false;

src/DataType/BackedEnumHandler.php

@@ -44,16 +44,6 @@ public function getNumericValue(mixed $value): null|int|float
         return null;
     }
 
-    public function getStringValue(mixed $value): null|string
-    {
-        return (string)$value->value;
-    }
-
-    public function isIdempotent(): bool
-    {
-        return true;
-    }
-
     public function useHmacVerification(): bool
     {
         return false;

src/DataType/BooleanHandler.php

@@ -16,9 +16,4 @@ public function getNumericValue(mixed $value): null|int|float
     {
         return $value ? 1 : 0;
     }
-
-    public function getStringValue(mixed $value): null|string
-    {
-        return $value ? 'true' : 'false';
-    }
 }

src/DataType/DateTimeHandler.php

@@ -56,18 +56,6 @@ public function getNumericValue(mixed $value): null|int|float
             : null;
     }
 
-    public function getStringValue(mixed $value): null|string
-    {
-        return $value instanceof DateTimeInterface
-            ? $value->copy()->setTimezone('UTC')->format(self::FORMAT)
-            : null;
-    }
-
-    public function isIdempotent(): bool
-    {
-        return true;
-    }
-
     public function useHmacVerification(): bool
     {
         return false;

src/DataType/DateTimeImmutableHandler.php

@@ -57,18 +57,6 @@ public function getNumericValue(mixed $value): null|int|float
             : null;
     }
 
-    public function getStringValue(mixed $value): null|string
-    {
-        return $value instanceof DateTimeInterface
-            ? $value->copy()->setTimezone('UTC')->format(self::FORMAT)
-            : null;
-    }
-
-    public function isIdempotent(): bool
-    {
-        return true;
-    }
-
     public function useHmacVerification(): bool
     {
         return false;

src/DataType/FloatHandler.php

@@ -24,9 +24,4 @@ public function getNumericValue(mixed $value): null|int|float
     {
         return $value;
     }
-
-    public function getStringValue(mixed $value): null|string
-    {
-        return (string) $value;
-    }
 }

src/DataType/HandlerInterface.php

@@ -34,8 +34,6 @@ public function serializeValue(mixed $value): string;
 
     public function getNumericValue(mixed $value): null|int|float;
 
-    public function getStringValue(mixed $value): null|string;
-
     /**
      * Convert a serialized string back to its original value.
      *
@@ -45,10 +43,5 @@ public function getStringValue(mixed $value): null|string;
      */
     public function unserializeValue(string $serializedValue): mixed;
 
-    /**
-     * Indicate whether multiple serializations of the same value will produce the same result.
-     */
-    public function isIdempotent(): bool;
-
     public function useHmacVerification(): bool;
 }

src/DataType/IntegerHandler.php

@@ -16,9 +16,4 @@ public function getNumericValue(mixed $value): null|int|float
     {
         return $value;
     }
-
-    public function getStringValue(mixed $value): null|string
-    {
-        return (string) $value;
-    }
 }

src/DataType/ModelCollectionHandler.php

@@ -123,16 +123,6 @@ public function getNumericValue(mixed $value): null|int|float
         return null;
     }
 
-    public function getStringValue(mixed $value): null|string
-    {
-        return null;
-    }
-
-    public function isIdempotent(): bool
-    {
-        return true;
-    }
-
     public function useHmacVerification(): bool
     {
         return false;