oci8 is leaking memory with long query/CLOB

[mvorisek]

Description

The following code:

<?php

    public function testCharacterTypeFieldLong(): void
    {
        $conn = $this->db->getConnection()->getConnection()->getNativeConnection();
        // var_dump(get_debug_type($conn));
        // prints string(26) "resource (oci8 connection)"

        for ($i = 0; $i < 10; $i++) {
            $sql = 'select CONCAT(CONCAT(TO_CLOB(\'atk__binary__u5f8mzx4vsm8g2c9__7eedd908a7b43ba3b8ec88338437e8bce2087e297a2dc612deb10abe043100adc20e566a3dce26ca6652419d1e992245fdf793148f18f34f746047d8d41c5c704b4ef28e199215f65d714a1dd5d9756146675340cb27cf576b3cfc4498239c1fdfc7137b2c097f28e3acc30f300105b00bbfa2b93aa6b5bde985368932ede5aec10d078103aa11c5b2e17d2a792eddeba887348338e7a4b7ef8ba0bb5f7348f8d794d36f5b4cf463779017db8c1bf06d59fed19a219625fa69553e9ecdc9516542fb97249b20d0ff6c58506443c8cc9f68543f6e5a4dd295d6f95e7249f18d1ada91166276f5e682398635a9eaa1ba8aeea5b6ab10c4020680afc00ce4dc782f7c2be0b3a7b43ba3b8ec88338437e8bce2087e297a2dc612deb10abe043100adc20e566a3dce26ca6652419d1e992245fdf793148f18f34f746047d8d41c5c704b4ef28e199215f65d714a1dd5d9756146675340cb27cf576b3cfc4498239c1fdfc7137b2c097f28e3acc30f300105b00bbfa2b93aa6b5bde985368932ede5aec10d078103aa11c5b2e17d2a792eddeba887348338e7a4b7ef8ba0bb5f7348f8d794d36f5b4cf463779017db8c1bf06d59fed19a219625fa69553e9ecdc9516542fb97249b20d0ff6c58506443c8cc9f68543f6e5a4dd295d6f95e7249f18d1ada9116627\'), TO_CLOB(\'6f5e682398635a9eaa1ba8aeea5b6ab10c4020680afc00ce4dc782f7c2be0b3a7b43ba3b8ec88338437e8bce2087e297a2dc612deb10abe043100adc20e566a3dce26ca6652419d1e992245fdf793148f18f34f746047d8d41c5c704b4ef28e199215f65d714a1dd5d9756146675340cb27cf576b3cfc4498239c1fdfc7137b2c097f28e3acc30f300105b00bbfa2b93aa6b5bde985368932ede5aec10d078103aa11c5b2e17d2a792eddeba887348338e7a4b7ef8ba0bb5f7348f8d794d36f5b4cf463779017db8c1bf06d59fed19a219625fa69553e9ecdc9516542fb97249b20d0ff6c58506443c8cc9f68543f6e5a4dd295d6f95e7249f18d1ada91166276f5e682398635a9eaa1ba8aeea5b6ab10c4020680afc00ce4dc782f7c2be0b3a7b43ba3b8ec88338437e8bce2087e297a2dc612deb10abe043100adc20e566a3dce26ca6652419d1e992245fdf793148f18f34f746047d8d41c5c704b4ef28e199215f65d714a1dd5d9756146675340cb27cf576b3cfc4498239c1fdfc7137b2c097f28e3acc30f300105b00bbfa2b93aa6b5bde985368932ede5aec10d078103aa11c5b2e17d2a792eddeba887348338e7a4b7ef8ba0bb5f7348f8d794d36f5b4cf463779017db8c1bf06d59fed19a219625fa69553e9ecdc9516542fb97249b20d0ff6c58506443c8cc9f68543f6e5a4dd295d\')), CONCAT(TO_CLOB(\'6f95e7249f18d1ada91166276f5e682398635a9eaa1ba8aeea5b6ab10c4020680afc00ce4dc782f7c2be0b3a7b43ba3b8ec88338437e8bce2087e297a2dc612deb10abe043100adc20e566a3dce26ca6652419d1e992245fdf793148f18f34f746047d8d41c5c704b4ef28e199215f65d714a1dd5d9756146675340cb27cf576b3cfc4498239c1fdfc7137b2c097f28e3acc30f300105b00bbfa2b93aa6b5bde985368932ede5aec10d078103aa11c5b2e17d2a792eddeba887348338e7a4b7ef8ba0bb5f7348f8d794d36f5b4cf463779017db8c1bf06d59fed19a219625fa69553e9ecdc9516542fb97249b20d0ff6c58506443c8cc9f68543f6e5a4dd295d6f95e7249f18d1ada91166276f5e682398635a9eaa1ba8aeea5b6ab10c4020680afc00ce4dc782f7c2be0b3a7b43ba3b8ec88338437e8bce2087e297a2dc612deb10abe043100adc20e566a3dce26ca6652419d1e992245fdf793148f18f34f746047d8d41c5c704b4ef28e199215f65d714a1dd5d9756146675340cb27cf576b3cfc4498239c1fdfc7137b2c097f28e3acc30f300105b00bbfa2b93aa6b5bde985368932ede5aec10d078103aa11c5b2e17d2a792eddeba887348338e7a4b7ef8ba0bb5f7348f8d794d36f5b4cf463779017db8c1bf06d59fed19a219625fa69553e9ecdc9516542fb97249b20d0ff6c5850644\'), CONCAT(TO_CLOB(\'3c8cc9f68543f6e5a4dd295d6f95e7249f18d1ada91166276f5e682398635a9eaa1ba8aeea5b6ab10c4020680afc00ce4dc782f7c2be0b3a7b43ba3b8ec88338437e8bce2087e297a2dc612deb10abe043100adc20e566a3dce26ca6652419d1e992245fdf793148f18f34f746047d8d41c5c704b4ef28e199215f65d714a1dd5d9756146675340cb27cf576b3cfc4498239c1fdfc7137b2c097f28e3acc30f300105b00bbfa2b93aa6b5bde985368932ede5aec10d078103aa11c5b2e17d2a792eddeba887348338e7a4b7ef8ba0bb5f7348f8d794d36f5b4cf463779017db8c1bf06d59fed19a219625fa69553e9ecdc9516542fb97249b20d0ff6c58506443c8cc9f68543f6e5a4dd295d6f95e7249f18d1ada91166276f5e682398635a9eaa1ba8aeea5b6ab10c4020680afc00ce4dc782f7c2be0b3a7b43ba3b8ec88338437e8bce2087e297a2dc612deb10abe043100adc20e566a3dce26ca6652419d1e992245fdf793148f18f34f746047d8d41c5c704b4ef28e199215f65d714a1dd5d9756146675340cb27cf576b3cfc4498239c1fdfc7137b2c097f28e3acc30f300105b00bbfa2b93aa6b5bde985368932ede5aec10d078103aa11c5b2e17d2a792eddeba887348338e7a4b7ef8ba0bb5f7348f8d794d36f5b4cf463779017db8c1bf06d59fed19a219625fa69553e9ecdc951654\'), TO_CLOB(\'2fb97249b20d0ff6c58506443c8cc9f68543f6e5a4dd295d6f95e7249f18d1ada91166276f5e682398635a9eaa1ba8aeea5b6ab10c4020680afc00ce4dc782f7c2be0\')))) AS "v" from "DUAL"';
            $expectedStr = 'atk__binary__u5f8mzx4vsm8g2c9__7eedd908a7b43ba3b8ec88338437e8bce2087e297a2dc612deb10abe043100adc20e566a3dce26ca6652419d1e992245fdf793148f18f34f746047d8d41c5c704b4ef28e199215f65d714a1dd5d9756146675340cb27cf576b3cfc4498239c1fdfc7137b2c097f28e3acc30f300105b00bbfa2b93aa6b5bde985368932ede5aec10d078103aa11c5b2e17d2a792eddeba887348338e7a4b7ef8ba0bb5f7348f8d794d36f5b4cf463779017db8c1bf06d59fed19a219625fa69553e9ecdc9516542fb97249b20d0ff6c58506443c8cc9f68543f6e5a4dd295d6f95e7249f18d1ada91166276f5e682398635a9eaa1ba8aeea5b6ab10c4020680afc00ce4dc782f7c2be0b3a7b43ba3b8ec88338437e8bce2087e297a2dc612deb10abe043100adc20e566a3dce26ca6652419d1e992245fdf793148f18f34f746047d8d41c5c704b4ef28e199215f65d714a1dd5d9756146675340cb27cf576b3cfc4498239c1fdfc7137b2c097f28e3acc30f300105b00bbfa2b93aa6b5bde985368932ede5aec10d078103aa11c5b2e17d2a792eddeba887348338e7a4b7ef8ba0bb5f7348f8d794d36f5b4cf463779017db8c1bf06d59fed19a219625fa69553e9ecdc9516542fb97249b20d0ff6c58506443c8cc9f68543f6e5a4dd295d6f95e7249f18d1ada91166276f5e682398635a9eaa1ba8aeea5b6ab10c4020680afc00ce4dc782f7c2be0b3a7b43ba3b8ec88338437e8bce2087e297a2dc612deb10abe043100adc20e566a3dce26ca6652419d1e992245fdf793148f18f34f746047d8d41c5c704b4ef28e199215f65d714a1dd5d9756146675340cb27cf576b3cfc4498239c1fdfc7137b2c097f28e3acc30f300105b00bbfa2b93aa6b5bde985368932ede5aec10d078103aa11c5b2e17d2a792eddeba887348338e7a4b7ef8ba0bb5f7348f8d794d36f5b4cf463779017db8c1bf06d59fed19a219625fa69553e9ecdc9516542fb97249b20d0ff6c58506443c8cc9f68543f6e5a4dd295d6f95e7249f18d1ada91166276f5e682398635a9eaa1ba8aeea5b6ab10c4020680afc00ce4dc782f7c2be0b3a7b43ba3b8ec88338437e8bce2087e297a2dc612deb10abe043100adc20e566a3dce26ca6652419d1e992245fdf793148f18f34f746047d8d41c5c704b4ef28e199215f65d714a1dd5d9756146675340cb27cf576b3cfc4498239c1fdfc7137b2c097f28e3acc30f300105b00bbfa2b93aa6b5bde985368932ede5aec10d078103aa11c5b2e17d2a792eddeba887348338e7a4b7ef8ba0bb5f7348f8d794d36f5b4cf463779017db8c1bf06d59fed19a219625fa69553e9ecdc9516542fb97249b20d0ff6c58506443c8cc9f68543f6e5a4dd295d6f95e7249f18d1ada91166276f5e682398635a9eaa1ba8aeea5b6ab10c4020680afc00ce4dc782f7c2be0b3a7b43ba3b8ec88338437e8bce2087e297a2dc612deb10abe043100adc20e566a3dce26ca6652419d1e992245fdf793148f18f34f746047d8d41c5c704b4ef28e199215f65d714a1dd5d9756146675340cb27cf576b3cfc4498239c1fdfc7137b2c097f28e3acc30f300105b00bbfa2b93aa6b5bde985368932ede5aec10d078103aa11c5b2e17d2a792eddeba887348338e7a4b7ef8ba0bb5f7348f8d794d36f5b4cf463779017db8c1bf06d59fed19a219625fa69553e9ecdc9516542fb97249b20d0ff6c58506443c8cc9f68543f6e5a4dd295d6f95e7249f18d1ada91166276f5e682398635a9eaa1ba8aeea5b6ab10c4020680afc00ce4dc782f7c2be0b3a7b43ba3b8ec88338437e8bce2087e297a2dc612deb10abe043100adc20e566a3dce26ca6652419d1e992245fdf793148f18f34f746047d8d41c5c704b4ef28e199215f65d714a1dd5d9756146675340cb27cf576b3cfc4498239c1fdfc7137b2c097f28e3acc30f300105b00bbfa2b93aa6b5bde985368932ede5aec10d078103aa11c5b2e17d2a792eddeba887348338e7a4b7ef8ba0bb5f7348f8d794d36f5b4cf463779017db8c1bf06d59fed19a219625fa69553e9ecdc9516542fb97249b20d0ff6c58506443c8cc9f68543f6e5a4dd295d6f95e7249f18d1ada91166276f5e682398635a9eaa1ba8aeea5b6ab10c4020680afc00ce4dc782f7c2be0b3a7b43ba3b8ec88338437e8bce2087e297a2dc612deb10abe043100adc20e566a3dce26ca6652419d1e992245fdf793148f18f34f746047d8d41c5c704b4ef28e199215f65d714a1dd5d9756146675340cb27cf576b3cfc4498239c1fdfc7137b2c097f28e3acc30f300105b00bbfa2b93aa6b5bde985368932ede5aec10d078103aa11c5b2e17d2a792eddeba887348338e7a4b7ef8ba0bb5f7348f8d794d36f5b4cf463779017db8c1bf06d59fed19a219625fa69553e9ecdc9516542fb97249b20d0ff6c58506443c8cc9f68543f6e5a4dd295d6f95e7249f18d1ada91166276f5e682398635a9eaa1ba8aeea5b6ab10c4020680afc00ce4dc782f7c2be0b3a7b43ba3b8ec88338437e8bce2087e297a2dc612deb10abe043100adc20e566a3dce26ca6652419d1e992245fdf793148f18f34f746047d8d41c5c704b4ef28e199215f65d714a1dd5d9756146675340cb27cf576b3cfc4498239c1fdfc7137b2c097f28e3acc30f300105b00bbfa2b93aa6b5bde985368932ede5aec10d078103aa11c5b2e17d2a792eddeba887348338e7a4b7ef8ba0bb5f7348f8d794d36f5b4cf463779017db8c1bf06d59fed19a219625fa69553e9ecdc9516542fb97249b20d0ff6c58506443c8cc9f68543f6e5a4dd295d6f95e7249f18d1ada91166276f5e682398635a9eaa1ba8aeea5b6ab10c4020680afc00ce4dc782f7c2be0';

            $stid = oci_parse($conn, $sql);
            oci_execute($stid);
            $row = oci_fetch_array($stid, \OCI_ASSOC);
            $res = $row['v']->load();
            $this->assertSame(strlen($expectedStr), strlen($res));
            $this->assertTrue($expectedStr === $res);

            $prevMem = $i > 4 ? $mem : null;
            gc_collect_cycles(); gc_collect_cycles();
            $mem = memory_get_usage();
            if ($prevMem !== null && $mem !== $prevMem) {
                throw new \Exception('leak detected');
            }
        }
    }

Resulted in this output:

Exception: leak detected

I have verified, the leak is consistent across PHP 7.4 - master and only oci8 ext is affected. With pdo_oci there is no leak for exactly the same query. See https://github.com/atk4/data/runs/7199098438

For smaller query, even with CLOB, there is also no leak.

PHP Version

tested PHP 7.4, 8.0, 8.1 and master

Operating System

tested linux and Windows

[nielsdos]

I tried a while to reproduce this but I can't. I amended your test case linked in this issue to run multiple iterations.
But the memory usage just always stays the same even with multiple operations. Can you still reproduce this?
If you can, how much is the memory usage increase? Knowing the amount of bytes often helps tracking down the leak if automatic tools fail.
I do get a bunch of memory errors from the OCI library itself though, but that's not something we can do much about.

[mvorisek]

Please fork https://github.com/mvorisek/php-src/tree/fix_oci8_mem_leak branch (php/php-src@master...1310df6), it contains php-src test case.

[nielsdos]

There is indeed something weird going on.
I don't know the root cause yet, but from my analysis something strange happens in the 6th iteration.
When the statement fetch is executed the memory usage increases, which is normal. But the strange thing is that in the 6th iteration it increases by 600 bytes while in the previous iterations it increases with 152 bytes. For other iterations that 152 bytes is released again, and it looks like for iteration 6 also only 152 from the 600 bytes are freed somehow.

[nielsdos]

Okay. So the row fetching code calls a callback php_oci_lob_create() and in the 6th iteration (so the 6th time calling that function), the call to zend_hash_index_update_ptr() inside php_oci_lob_create() will cause the connection->descriptors hashtable to grow, which results in the memory increase you're seeing.
From the documentation inside php_oci8_int.h:

HashTable	   *descriptors;				/* descriptors hash, used to flush all the LOBs using this connection on commit */

but I don't get it, why would we need that for fetching instead of inserting ? I'm going to inspect that code more now.

[nielsdos]

Okay I can confirm that not adding the descriptor to that table here gets rid of the leak:

https://github.com/php/php-src/blob/345abce590bf6b7aa2ffd49ba18e2ff479c3b8bd/ext/oci8/oci8_lob.c#L91-L107

It leaks because in the example test code we never commit anything, we just fetch rows. I think we should not add the descriptor to that table if we're just fetching, because otherwise we will never clear that table.
In any case, I did find the root cause, but I don't know enough about OCI8 to fix this myself. So I'm marking this as verified and my comments can be useful for the person fixing this.

[mvorisek]

@nielsdos but why does this leak only with long query/CLOB? With small query/CLOB, there is no leak.

[nielsdos]

@nielsdos but why does this leak only with long query/CLOB? With small query/CLOB, there is no leak.

I don't think I checked the reason for this, or at least I don't remember a reason.

[jorgsowa]

This issue could be move to proper repository as the extension has been decoupled from core.

https://github.com/php/pecl-database-oci8