-
-
Notifications
You must be signed in to change notification settings - Fork 389
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Migrations always generated for custom type with DBAL 4 #1441
Comments
@greg0ire I am tagging you as you authored some of the relevant changes and I know you have deep insight into removing comments in DBAL 4. I was digging into the code, the issue lies here: So
This function then calls Now check DBAL 3 code: This uses the comment to deduce the proper type and override whatever type is in DB. Without this now, DBAL considers any unknown DB type to be Now the new schema uses the So, since we removed comments in DBAL 4, how can the |
Please upgrade the ORM to 3.8.5: doctrine/dbal#6423 EDIT: I shouldn't answer so hastily |
@greg0ire I am using DBAL 4.0.4 which is based on 3.8.6 that includes the linked fix. ORM is at 3.2.1 and there is no newer version. And the issue is ONLY with DBAL 4 not DBAL 3 |
@michnovka It seems that when Doctrine reads the schema of your database, the
|
@berkut1 this is indeed a mistaken assumption imo, as getSQLDeclaration not being called is a consequence of not having the proper type. The type has to be known in order for that function to be called. And it is called, just for the new schema. The issue is that oldSchema is created based on database ONLY. It ignores PHP code. And thats good, as the php code defines the new schema, if it took the PHP code as a hint of the type, then migration has no point, as it would be the same type always. I honestly see no other way than to somehow store in DB the name of the custom type which was used. Which is exactly what the DB comment was used for. WDYT? |
@michnovka |
@berkut1 how are they useless when even in 3.9.x there is this code: And this is where it fails, this is where in 3.8.x version it assigned properly custom type, but in 4.0.x it assigns StringType (as the code assigning type based on comment is missing) And again, think about what I observed above - the oldSchema is created based on DB data ONLY. not PHP. Only DB. So if DB has no indication about the type, how can it be properly determined for nonstandard types?
This has no sense for migrations. If I had |
DBAL 3.x should have backward compatibility, that why it can works with DC2Types and without. there was added this https://github.com/doctrine/dbal/blob/893417fee2bc5a94a10a2010ae83cab927e21df3/src/Platforms/AbstractPlatform.php#L108 |
With DBAL 3.8.X and And of course it is. Read again where and why it happens, I explained exactly where the issue lies. oldSchema is created based on DB details only. If we dont use comments (either as they are removed in DBAL 4, or if we disable them manually in DBAL 3) then oldSchema will have no idea what type was used before. Thanks for your interest and will to help. |
Here, |
gives
and
gives
UPD: |
@michnovka it's unclear to me whether you know what platform-aware comparison is. In case you don't, since DBAL 3.2.0, platform-aware comparison allows to compare the generated SQL instead of comparing schemas. This means that 2 different schemas could result in an empty diff, if they result in the same SQL being generated. More on this here: https://www.doctrine-project.org/2021/11/26/dbal-3.2.0.html Sorry if you already know this, it's genuinely hard to tell for me. |
@greg0ire yes, I get this. The problem is that when the comparator tries to compare the schemas, it is working with oldSchema and newSchema. oldSchema is This means that when it tries to compare SQL generated between old and new schemas, it calls And this is expected. As if in DB there is no info about which custom type the column is, how can the Now the new schema is another |
Ah I get it now, thanks for explaining it again. I think this means the issue is that the DBAL does not understand |
yes, it all comes to a simple problem - the oldSchema is generated using ONLY DB introspection. And if DB contains no info about which type was used, then it cannot call proper And no, the problem does not simply limit to enum. I can make an example with another custom type which will have the same issue without enums. ALL custom types have this issue in fact, i.e. all custom types with custom |
Here, Doctrine is reading your database. Does it also read the type as a string here? |
This looks related: doctrine/dbal#5308 (comment) |
No, here it fetches it from DB like this:
But this is just one step before And @greg0ire this is where the issue lies: $column = new Column($tableColumn['field'], Type::getType($type), $options); specifically the |
Ah, I understand now. It calls mapping_types:
enum: ancestor_enum
types:
ancestor_enum : 'App\Model\AncestorEnumType' Unfortunately, I can't help with how to do this without Symfony. UPD: |
@greg0ire as a temporary workaround, how can I tell migration to ignore certain columns? Thanks! |
Maybe you can use asset filtering, but I don't think it works at the column level. |
So the issue is this:
And on the new schema, we get correct type for column and call its These do not match. Proposed solution: Why not leverage I dont understand why @greg0ire WDYT? |
Sounds good, feel free to give it a try, and see if anything breaks 👍 |
I tried to go this way and I am afraid this would create tons of unexpected issues. There are specific test cases that ensure that
so for whatever reason this is considered important. Even the comment in the So maybe a solution would be to add comments back so that the type can be deduced properly? |
I don't think that's the direction we want to go, no. Re-reading the cookbook you mentioned I see
It seems like that might have been fixed in doctrine/dbal#5224, which might make solution 1 the way to go here. |
@michnovka As I mentioned in my comment on the instructions, I solved the problem for myself in this way: doctrine:
dbal:
mapping_types:
inet: my_inet
types:
my_inet: { class: 'App\Model\InetType' } I want to clarify, as stated in the instructions at https://www.doctrine-project.org/projects/doctrine-orm/en/3.2/cookbook/mysql-enums.html, it is recommended to do it like this $conn->getDatabasePlatform()->registerDoctrineTypeMapping('enum', 'string'); but this solution works only for DBAL3. For DBAL4, you need to refer not to However, if you haven't tried this solution at all, you can first refer to |
@berkut1 Thanks for the suggestion, but this is not a solution for my case. It works in your case when you have 1:1 mapping between DB type But in my case, every |
Yes, I understand, which is why I initially suggested creating an abstract Enum class and registering it, then inheriting from it for custom types. In other words, you need to make Doctrine understand that it has a base Enum class for comparison, with which it will compare all the descendants of this class. |
has anybody started work on the Verbatim type? I might have some time in end of October/November to look into this. |
I thought I would give it a try, but I didn't yet. And I won't have much time for a while. Feel free to start whenever you want. And notice everybody here when you do, so that we don't duplicate work. |
Linking a couple more existing issues for reference: doctrine/dbal#4470, doctrine/dbal#5306. |
I encounter the same issue with my custom types.
in my migration diff, when it worked fine with It does not give a good experience with the DBAL 4 migration, and it seems like it impacts a lot of developer. I read all the discussion and maybe I miss-understand but I feel like there is currently no solution to avoid this extra sql in every diff. Isn't it ? The comment was really useful for custom type. If there is no known solution, what about re-introducing it back ? At least this could be optional like an option to toggle. This way if comments have bad impact for some user they can disable it. |
As I said, we have had a solution for years and it kept working with dbal 4. I would create a PR, but it's for mysql only 🫤 |
@VincentLanglet Your feeling is correct, in my opinion. I haven’t found any alternatives either. I’ve been begging for merging this PR doctrine/dbal#6444 for months, but they don’t want to for obscure reasons. A good legacy solution would be to follow the approach of the doctrine/annotation bundle and make it optional, but they seem to have missed the point for months. If you’re interested, I created this package - a code modifier that reintroduces the missing lines: https://packagist.org/packages/glitchr/doctrine-dc2type. Please use with caution and inspect the package on a fresh symfony skeleton if you wish, there is nothing too complex or fancy, but it is nothing official. |
Your complaints have been duly noted. Now, can we please get back to discussing solutions? I mean, you can just stay on DBAL 3 until the problem is solved or help working on a proper solution. We will still maintain DBAL 3 for some time. |
Sure, but I'm unsure how to help. I certainly lack of knowledge about how this work. If I debug When the data_type in my DB is
Personally, I could almost fix my issue if I overriden the
method in my customs Type, but it would require to update the MySQLSchemaManager to something like
Then https://www.doctrine-project.org/projects/doctrine-orm/en/latest/cookbook/mysql-enums.html could be updated to implement
(the abstract EnumType could even be provided by doctrine eventually...) Do you think it could a good improvement @greg0ire @derrabus ? (And how avoid a BC break...)
DBAL 4 still have benefits ; I like the evolution done to bigint. |
Should the And @stof
This is not true, we have that, see #1441 (comment) We can just create a But I think we can handle all this inside Then we just make special case in comparator for Am I missing something, or does it sound too easy to you too? |
Does doctrine/dbal#6536 solve the issue for those of you who are using enum columns? |
@michnovka both the schema introspection and the schema comparison are provided by DBAL. So it would definitely be used outside Migrations (and Migrations would not even use it directly but only through the usages in DBAL itself) |
I'm getting Also I tried to change my mapping and/or change my enumType ; in both case the diff is correct. |
| Q | A |------------- | ----------- | Type | feature | Fixed issues | doctrine/migrations#1441 (partly) #### Summary This PR adds an `EnumType` that allows us to introspect and diff tables that make use of MySQL's `ENUM` column type.
Thanks @derrabus this indeed solves the issues with ENUM and migrations. Only thing I noticed was that if my sqlDeclaration defined the type as lowercase Changing my custom type's sql declaration to uppercase ENUM resolved this. |
So, this is your fix then. 🙂 |
@derrabus (or @michnovka ?), could you please clarify how the new I updated my playground to the latest versions of all packages, and I still see The model is declared there, and Doctrine is configured in the simplest manner I know of. Basically I am doing: <?php
use Doctrine\ORM\Mapping as ORM;
enum UserRole: string
{
case Visitor = 'visitor';
case Member = 'member';
case Admin = 'admin';
}
enum UserStatus: string
{
case New = 'new';
case Active = 'active';
case Archived = 'archived';
}
#[ORM\Entity]
class User
{
#[ORM\Id]
#[ORM\Column(type: 'integer')]
public ?int $id = null;
#[ORM\Column]
public UserRole $role = UserRole::Visitor;
#[ORM\Column]
public UserStatus $status = UserStatus::New;
} And that: function createEntityManager(): EntityManager
{
$config = ORMSetup::createAttributeMetadataConfiguration(['my-path-to-my-entity'], true);
$connection = DriverManager::getConnection([
'driverClass' => Driver::class,
'host' => '127.0.0.1',
'dbname' => 'test',
'user' => 'test',
'password' => 'test',
], $config);
$entityManager = new EntityManager($connection, $config);
return $entityManager;
} ...but it still fails to have |
@PowerKiKi still by default doctrine will use #[ORM\Column(type: 'enum', enumType: UserRole::class, values: [...])]
public UserRole $role = UserRole::Visitor; If you do not want to specify values here, and want by default all enum values to be used in DB as well, you have to use a custom type. You can look into https://github.com/Elao/PhpEnums that makes this much easier. You can also look into Elao/PhpEnums#212 where I show on top my implementation which auto-registers enums as types. |
Please read doctrine/orm#11666.
No, don't do that. Custom enum types should be obsolete now.
Not anymore, really. |
@derrabus this is so cool, thanks a lot, I was unaware of this new PR @PowerKiKi if you still want to use MySQL enums by default, you can take advantage of TypedFieldMapper. Here is my solution for the interested:
<?php
declare(strict_types=1);
namespace App\Doctrine\ORM\TypedFieldMapper;
use Doctrine\DBAL\Types\Types;
use Doctrine\ORM\Mapping\TypedFieldMapper;
use ReflectionNamedType;
use ReflectionProperty;
class EnumTypedFieldMapper implements TypedFieldMapper
{
/**
* {@inheritdoc}
*/
public function validateAndComplete(array $mapping, ReflectionProperty $field): array
{
$type = $field->getType();
if (
! isset($mapping['type'])
&& ($type instanceof ReflectionNamedType)
) {
if (!$type->isBuiltin() && enum_exists($type->getName())) {
$mapping['type'] = Types::ENUM;
}
}
return $mapping;
}
}
services:
App\Doctrine\ORM\TypedFieldMapper\EnumTypedFieldMapper:
Doctrine\ORM\Mapping\DefaultTypedFieldMapper:
arguments:
$typedFieldMappings:
Decimal\Decimal: 'decimal'
Carbon\CarbonImmutable: 'datetime_immutable'
Doctrine\ORM\Mapping\ChainTypedFieldMapper:
arguments:
$typedFieldMappers:
- '@App\Doctrine\ORM\TypedFieldMapper\EnumTypedFieldMapper'
- '@Doctrine\ORM\Mapping\DefaultTypedFieldMapper'
doctrine:
orm:
typed_field_mapper: 'Doctrine\ORM\Mapping\ChainTypedFieldMapper' now you can use #[ORM\Column]
public UserRole $role = UserRole::Visitor; And it will make your field an ENUM type in database Note that this requires ORM 3.3, DBAL 4.2 and DoctrineBundle 2.13 |
@derrabus, this is great, thank you ! My playground is now working as expected. Specifying the type of the column is an obvious solution I should have thought about, @michnovka, thanks for the hint for From my point of view, which is only concerned about enum, this issue may be closed as resolved. |
Well, we still have the issue with the Not sure if this should remain open as a "placeholder" for that problem too. A short patch that adds a middleware to the connection setting a custom platform (extended on
|
… and my answer to his comment still applies. |
I still generally have a problem with the dropping of D2Ctype comments. By doing so. Doctrine has lost the ability to have more than one DBAL type having the same column definition. When the comparator tries to detect if anything has changed, it's using the column definition to determine the DBAL type. When there are two DBAL types with the same column definition, it just picks the first one. When there were D2Ctype comments, the comparator could distinguish between the two DBAL types. |
It does? I can't imagine this is the case. It would have to run through all possible parameters, too, to find the type based on a schema. Just imagine it finds a All the migrations need to do is compare what is with what should be. And for that the comments were superfluous. |
I think you're right @uncaught, you would expect the comparator to be generating a dummy schema based on the current state of the entities, and then compare that with the actual schema, but that's not what's happening. @michnovka details how it works in their first comment Notice the call to Edit: lets be more precise about this... the comparator isn't generating anything, it's just comparing what it's being given... but the point is the the |
Bug Report
Summary
I have issue with DBAL 4 and custom type, the migrations keep getting generated again and again. Furthermore, the down migration looks totally bogus. This is possibly related to #1435
I know that DBAL 4 dropped requiresSqlHint (in doctrine/dbal#5107 , afterwards some issues were found and fixed - doctrine/dbal#6257 )
So when I am using custom type, I expect the first migration diff to drop the
DC2Type
comments. However my tables have these fields already dropped and yet the migration is being generated.I then have my entity as
and in
Kernel.php
I set up type mapping insideKernel::process()
:Now I know that the types are assigned correctly, as migrations generate up like this:
but it is generated ALWAYS.
and the
down()
migration looks even weirder:Everything works fine with DBAL 3, which uses SQL comments
The text was updated successfully, but these errors were encountered: