Speed improvements on Record init

[srfwx]

pynetbox version

v7.4.1

NetBox version

v4.1.7

Feature type

Change to existing functionality

Proposed functionality

Hi,

We've built a caching layer to Pynetbox where responses are stored as json inside a Redis cache, then Record objects are reconstructed from there on cache hit.

However I've noticed that the _init_ of a Record object can still take a substantial amount of time due to the various processing steps

As a first step I have:

• simplified the _endpoint_from_url method so that it doesn't need urlsplit leading to a 50% speed improvement of that method
• used marshall dump/load instead of deep_copy for cache initialization
• refactored the _parse_values method to limit the number of instance type checking and number of key/value that need further processing

All in all, according to a few benchmarks I made, the whole Record init shows more than 50% speed improvement. Below is an example when loading 14407 records.

Benchmark using cProfile with Pynetbox v7.4.1

      ncalls  tottime  percall  cumtime  percall filename:lineno(function)
142153/14407    0.829    0.000    8.681    0.001 .venv/lib/python3.10/site-packages/pynetbox/core/response.py:278(__init__)

Benchmark using cProfile with our fork

      ncalls  tottime  percall  cumtime  percall filename:lineno(function)
142153/14407    0.343    0.000    4.037    0.000 .venv/lib/python3.10/site-packages/pynetbox/core/response.py:263(__init__)

Would you be interested in having any/all of these things merged into upstream?

Use case

General speed improvement of the library

External dependencies

None

[srfwx]

Latest benchmark after adding a cache of endpoints on App object and cache of apps on PluginsApp, reaching now a 70% speed improvement.

      ncalls  tottime  percall  cumtime  percall filename:lineno(function)
142153/14407    0.220    0.000    2.763    0.000  .venv/lib/python3.10/site-packages/pynetbox/core/response.py:263(__init__)

[arthanson]

@srfwx can you provide a PR for it so we can review it? It sounds beneficial but hard to say without seeing the exact code.

[srfwx]

Hi,
Yes of course, here is the PR #659
I followed the contribution guidelines and opened an issue first.
Edit: last commit fix the tests

[srfwx]

Further improvement achieved by only computing the Record.endpoint on demand and not during init. (0f9e41b)

      ncalls  tottime  percall  cumtime  percall filename:lineno(function)
142153/14407    0.200    0.000    2.305    0.000 .venv/lib/python3.10/site-packages/pynetbox/core/response.py:262(__init__)

[srfwx]

I've noticed that nested Record are instantiated from scratch each time, although when dealing with a RecordSet you'll likely be encountering the same sub Record many times. Hence temporarily caching and re-using instead of processing again lead to even better performance. My benchmark is now showing a ~90% improvement

      ncalls  tottime  percall  cumtime  percall filename:lineno(function)
16409/14407    0.023    0.000    1.001    0.000 .venv/lib/python3.10/site-packages/pynetbox/core/response.py:261(__init__)

💡 This also gives the benefit that if you call .full_details() on any nested Record, then full details will be available in all other Records from the RecordSet with the same linked object

[srfwx]

Nested dictionaries representing a Choice value are instantiated as nested Record with broken methods as they don't have an URL or endpoint.
With d3b1a8d I've introduced a simplified ValueRecord class for such dictionaries.