SlotClasses
SCons creates a lot of instances of various classes when conducting large software builds. Some of those objects only have few attributes so the overhead that every instance carries accounts for a large amount of memory. Chris AtLee proposed the application of slotted classes in SCons. Jean Brouwers, the author of asizeof, gives a good motivation for using classes with slots.
- Introduced with Python 2.2 * Cannot be weakly referenced (needs Python 2.3)
- Instance attributes to be set must have a slot assigned, otherwise an
AttributeErroris thrown * Might have a__dict__slot to overcome this problem (needs Python 2.3) - Can't be pickled automatically,
__setstate__and__getstate__must be specified - Existing methods can't be overridden
Choosing a class for assigning slots can be eased by evaluating which type of instances take up most of the memory. The following chart was generated with the HtmlStats facility of the Heapmonitor while making an up-to-check of Ardour.
The Executor objects are interesting due to the large number of instances in memory intensive builds. Furthermore, the classes are easily converted to slotted classes due to the small number of attributes. The only needed change concerns method assignments. This problem can be circumvented creating a slot for the method and initializing the slot to the actual method.
| Hello World (Clean Build) | Executor.Executor | Executor.Null |
|---|---|---|
| Instance # | 4 | 4 |
| Original size (average) | 906 Byte | 1006 Byte |
| Original size (min) | 856 Byte | 944 Byte |
| Original size (max) | 1.02 KB | 1.16 KB |
| Slot class size (average) | 676 Byte | 924 Byte |
| Slot class size (min) | 608 Byte | 840 Byte |
| Slot class size (max) | 880 Byte | 1.15 KB |
| Ardour (Up-to-date check) | Executor.Executor | Executor.Null |
|---|---|---|
| Instance # | 1348 | 2135 |
| Original size (average) | 856 Byte | 944 Byte |
| Original size (total) | 1.12 MB (4%) | 1.92 MB (7%) |
| Slot class size (average) | 681 Byte | 840 Byte |
| Slot class size (total) | 897.14 KB (3.08%) | 1.71 MB (6%) |
Depending on the project, Node objects take between 60-90% of the memory observable with the Heapmonitor. This makes them the perfect candidates for memory reduction.
An Attrs instance is stored in each Node object. It serves as a dictionary to store additional information - presumably by Tools. Turning this into a slotted class is easy but only allows a very minor memory reduction. Moreover, Attrs should still be used as a general purpose dictionary so I attached a __dict__ slot.
Every Node object has a BuildInfo and NodeInfo structure attached to it. This takes a considerable amount of space in the Node object. Trying to turn these objects into slotted classes is more involved, though. The pickling must be done manually and the sconsign format differs. It is possible to use existing sconsign databases. However, once the sconsign database was emitted using slot classes, it cannot be used anymore from older versions of SCons.
| Hello World (Clean Build) | Node.FS.Entry | Node.FS.File | SConsign.DB |
|---|---|---|---|
| Instance # | 2 | 3 | 2 |
| Original size (average) | 5.75 KB | 4.81 KB | 1.02 KB |
| Slot class size (average) | 4.99 KB | 4.46 KB | 916 B |
Ardour (Up-to-date check) | Node.FS.Entry | Node.FS.File | SConsign.DB
Instance # | 2238 | 1393 | 69
Original size (average) | 7.72 KB | 5.88 KB | 8.06 KB
Slot class size (average) | 7.26 KB | 5.46 KB | 6.68 KB
The previous results look promising but do not provide an overview about the whole memory consumption and execution time impact.
| x | Ardour up-to-date check memory | Ardour up-to-date check runtime |
|---|---|---|
| Classic | 60.34 MB | 43.7s |
Executor slotted |
58.60 MB | 43.7s |
*Info slotted |
56.94 MB | 45.1s |
| All patches applied | 55.14 MB | 45.2s |
All tests were run on 32bit Linux with Python 2.5.2 and the Heapmonitor Branch of SCons based on version 0.98.5.