X Tutup
The Wayback Machine - https://web.archive.org/web/20201019000956/https://github.com/mongodb/mongo-python-driver/pull/496
Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PYTHON-2382 Destroy codec options struct in _cbson._element_to_dict #496

Merged

Conversation

@prashantmital
Copy link
Contributor

@prashantmital prashantmital commented Oct 6, 2020

Tested using a slightly modified version of the user-provided repro script:

import gc
from typing import Any
from pymongo import MongoClient
from bson.codec_options import CodecOptions
from bson.codec_options import TypeCodec, TypeRegistry


# %% Define method to analyse the memory
def memory_report(prefix):
    gc.collect()
    rr = {}

    for obj in gc.get_objects():

        tt = type(obj)
        if ('bson.codec_options.CodecOptions' in str(tt)):
            refs = gc.get_referrers(obj)
            print(tt)
            print(len(refs))
            # if len(refs) == 1:
            #     objgraph.show_refs([obj], filename='refs.png')
            rr[tt] = rr.get(tt, 0) + 1

    print(prefix + ' memory report:')
    for key in rr:
        nn = rr[key]
        print(f' {key}: {nn}')


# %% Create codec

class MyClass:
    pass


class MyCodec(TypeCodec):  # type: ignore

    @property
    def python_type(self) -> Any:
        return MyClass

    def transform_python(self, value: Any) -> Any:
        # print(f'transform_python: {value}')
        return value

    @property
    def bson_type(self) -> Any:
        return int

    def transform_bson(self, value: Any) -> Any:
        # print(f'transform_bson: {value}')
        return value * 2


codec = MyCodec()
type_registry = TypeRegistry([codec])
codec_options = CodecOptions(type_registry=type_registry)

client = MongoClient(replicaset='repl0')
database = client['my_database']
collection = database['my_collection'].with_options(codec_options)
collection.drop()

memory_report('before ')
for ii in range(10):
    collection.insert_one({'a': 2})
    cursor = collection.find({})
    for data in cursor:
        assert data['a'] == 4
memory_report('after ')
@prashantmital prashantmital requested a review from ShaneHarvey Oct 6, 2020
@peendebak
Copy link

@peendebak peendebak commented Oct 6, 2020

@prashantmital Tested on my system and fixes the leak. Thanks!

Copy link
Member

@ShaneHarvey ShaneHarvey left a comment

LGTM. FYI I think this bug also occurs when inflating a RawBSONDocument.

@prashantmital prashantmital merged commit 594b211 into mongodb:master Oct 6, 2020
11 checks passed
11 checks passed
continuous-integration/travis-ci/pr The Travis CI build passed
Details
evergreen patch finished in 13m29.887s
Details
evergreen/tests-python-version-requires-openssl-102-plus-test-encryption__platform~ubuntu-16.04_auth-ssl~noauth-nossl_python-version~3.7_encryption~encryption 15 succeeded, none failed in 12m49.596s
Details
evergreen/tests-python-version-requires-openssl-102-plus-test-encryption__platform~ubuntu-16.04_auth-ssl~noauth-nossl_python-version~3.8_encryption~encryption 15 succeeded, none failed in 13m11.931s
Details
evergreen/tests-python-version-requires-openssl-102-plus-test-encryption__platform~ubuntu-16.04_auth-ssl~noauth-nossl_python-version~3.9_encryption~encryption 15 succeeded, none failed in 12m34.925s
Details
evergreen/tests-python-version-requires-openssl-102-plus-test-ssl__platform~ubuntu-16.04_auth-ssl~auth-ssl_python-version~3.7 21 succeeded, none failed in 12m57.731s
Details
evergreen/tests-python-version-requires-openssl-102-plus-test-ssl__platform~ubuntu-16.04_auth-ssl~auth-ssl_python-version~3.8 21 succeeded, none failed in 13m29.887s
Details
evergreen/tests-python-version-requires-openssl-102-plus-test-ssl__platform~ubuntu-16.04_auth-ssl~auth-ssl_python-version~3.9 21 succeeded, none failed in 13m16.351s
Details
evergreen/tests-python-version-requires-openssl-102-plus-test-ssl__platform~ubuntu-16.04_auth-ssl~noauth-nossl_python-version~3.7 21 succeeded, none failed in 12m56.961s
Details
evergreen/tests-python-version-requires-openssl-102-plus-test-ssl__platform~ubuntu-16.04_auth-ssl~noauth-nossl_python-version~3.8 21 succeeded, none failed in 13m1.237s
Details
evergreen/tests-python-version-requires-openssl-102-plus-test-ssl__platform~ubuntu-16.04_auth-ssl~noauth-nossl_python-version~3.9 21 succeeded, none failed in 13m7.254s
Details
@prashantmital prashantmital deleted the prashantmital:PYTHON-2382/memory-leak-fix branch Oct 6, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

3 participants
You can’t perform that action at this time.
X Tutup