Commit | Line | Data |
e98cedbf |
1 | package Devel::Size; |
2 | |
e98cedbf |
3 | use strict; |
9fc9ab86 |
4 | use vars qw($VERSION @ISA @EXPORT @EXPORT_OK %EXPORT_TAGS $warn $dangle); |
e98cedbf |
5 | |
9fc9ab86 |
6 | require 5.008; |
e98cedbf |
7 | require Exporter; |
8 | require DynaLoader; |
9 | |
a6ea0805 |
10 | @ISA = qw(Exporter DynaLoader); |
e98cedbf |
11 | |
9fc9ab86 |
12 | # This allows declaration use Devel::Size ':all'; |
a6ea0805 |
13 | %EXPORT_TAGS = ( 'all' => [ qw( |
9fc9ab86 |
14 | size total_size |
e98cedbf |
15 | ) ] ); |
16 | |
a6ea0805 |
17 | @EXPORT_OK = ( @{ $EXPORT_TAGS{'all'} } ); |
e98cedbf |
18 | |
0430b7f7 |
19 | @EXPORT = qw( ); |
0d46c0bd |
20 | $VERSION = '0.72_50'; |
e98cedbf |
21 | |
22 | bootstrap Devel::Size $VERSION; |
23 | |
78dfb4e7 |
24 | $warn = 1; |
9fc9ab86 |
25 | $dangle = 0; ## Set true to enable warnings about dangling pointers |
ebb2c5b9 |
26 | |
e98cedbf |
27 | 1; |
28 | __END__ |
e98cedbf |
29 | |
6ea94d90 |
30 | =pod |
31 | |
e98cedbf |
32 | =head1 NAME |
33 | |
0bff12d8 |
34 | Devel::Size - Perl extension for finding the memory usage of Perl variables |
e98cedbf |
35 | |
36 | =head1 SYNOPSIS |
37 | |
0bff12d8 |
38 | use Devel::Size qw(size total_size); |
e98cedbf |
39 | |
0bff12d8 |
40 | my $size = size("A string"); |
41 | |
42 | my @foo = (1, 2, 3, 4, 5); |
43 | my $other_size = size(\@foo); |
44 | |
45 | my $foo = {a => [1, 2, 3], |
9fc9ab86 |
46 | b => {a => [1, 3, 4]} |
5c2e1b12 |
47 | }; |
5a83b7cf |
48 | my $total_size = total_size($foo); |
5c2e1b12 |
49 | |
e98cedbf |
50 | =head1 DESCRIPTION |
51 | |
5a83b7cf |
52 | This module figures out the real size of Perl variables in bytes, as |
53 | accurately as possible. |
54 | |
0bff12d8 |
55 | Call functions with a reference to the variable you want the size |
56 | of. If the variable is a plain scalar it returns the size of |
5a83b7cf |
57 | this scalar. If the variable is a hash or an array, use a reference |
0bff12d8 |
58 | when calling. |
59 | |
60 | =head1 FUNCTIONS |
61 | |
62 | =head2 size($ref) |
e98cedbf |
63 | |
5c2e1b12 |
64 | The C<size> function returns the amount of memory the variable |
0bff12d8 |
65 | returns. If the variable is a hash or an array, it only reports |
66 | the amount used by the structure, I<not> the contents. |
67 | |
68 | =head2 total_size($ref) |
5c2e1b12 |
69 | |
0bff12d8 |
70 | The C<total_size> function will traverse the variable and look |
71 | at the sizes of contents. Any references contained in the variable |
72 | will also be followed, so this function can be used to get the |
73 | total size of a multidimensional data structure. At the moment |
74 | there is no way to get the size of an array or a hash and its |
75 | elements without using this function. |
5c2e1b12 |
76 | |
b98fcdb9 |
77 | =head1 EXPORT |
e98cedbf |
78 | |
0bff12d8 |
79 | None but default, but optionally C<size> and C<total_size>. |
e98cedbf |
80 | |
b98fcdb9 |
81 | =head1 UNDERSTANDING MEMORY ALLOCATION |
82 | |
83 | Please note that the following discussion of memory allocation in perl |
84 | is based on the perl 5.8.0 sources. While this is generally |
85 | applicable to all versions of perl, some of the gory details are |
86 | omitted. It also makes some presumptions on how your system memory |
87 | allocator works so, while it will be generally correct, it may not |
88 | exactly reflect your system. (Generally the only issue is the size of |
89 | the constant values we'll talk about, not their existence) |
90 | |
91 | =head2 The C library |
92 | |
9fc9ab86 |
93 | It's important first to understand how your OS and libraries handle |
b98fcdb9 |
94 | memory. When the perl interpreter needs some memory, it asks the C |
95 | runtime library for it, using the C<malloc()> call. C<malloc> has one |
96 | parameter, the size of the memory allocation you want, and returns a |
97 | pointer to that memory. C<malloc> also makes sure that the pointer it |
98 | returns to you is properly aligned. When you're done with the memory |
99 | you hand it back to the library with the C<free()> call. C<free> has |
9fc9ab86 |
100 | one parameter, the pointer that C<malloc> returned. |
101 | There are a couple of interesting ramifications to this. |
b98fcdb9 |
102 | |
103 | Because malloc has to return an aligned pointer, it will round up the |
104 | memory allocation to make sure that the memory it returns is aligned |
105 | right. What that alignment is depends on your CPU, OS, and compiler |
106 | settings, but things are generally aligned to either a 4 or 8 byte |
107 | boundary. That means that if you ask for 1 byte, C<malloc> will |
108 | silently round up to either 4 or 8 bytes, though it doesn't tell the |
109 | program making the request, so the extra memory can't be used. |
110 | |
111 | Since C<free> isn't given the size of the memory chunk you're |
112 | freeing, it has to track it another way. Most libraries do this by |
113 | tacking on a length field just before the memory it hands to your |
114 | program. (It's put before the beginning rather than after the end |
115 | because it's less likely to get mangled by program bugs) This size |
116 | field is the size of your platform integer, Generally either 4 or 8 |
117 | bytes. |
118 | |
119 | So, if you asked for 1 byte, malloc would build something like this: |
120 | |
121 | +------------------+ |
122 | | 4 byte length | |
123 | +------------------+ <----- the pointer malloc returns |
124 | | your 1 byte | |
125 | +------------------+ |
126 | | 3 bytes padding | |
127 | +------------------+ |
128 | |
129 | As you can see, you asked for 1 byte but C<malloc> used 8. If your |
130 | integers were 8 bytes rather than 4, C<malloc> would have used 16 bytes |
131 | to satisfy your 1 byte request. |
132 | |
133 | The C memory allocation system also keeps a list of free memory |
134 | chunks, so it can recycle freed memory. For performance reasons, some |
135 | C memory allocation systems put a limit to the number of free |
136 | segments that are on the free list, or only search through a small |
137 | number of memory chunks waiting to be recycled before just |
138 | allocating more memory from the system. |
139 | |
140 | The memory allocation system tries to keep as few chunks on the free |
141 | list as possible. It does this by trying to notice if there are two |
142 | adjacent chunks of memory on the free list and, if there are, |
143 | coalescing them into a single larger chunk. This works pretty well, |
144 | but there are ways to have a lot of memory on the free list yet still |
145 | not have anything that can be allocated. If a program allocates one |
146 | million eight-byte chunks, for example, then frees every other chunk, |
147 | there will be four million bytes of memory on the free list, but none |
148 | of that memory can be handed out to satisfy a request for 10 |
149 | bytes. This is what's referred to as a fragmented free list, and can |
150 | be one reason why your program could have a lot of free memory yet |
151 | still not be able to allocate more, or have a huge process size and |
152 | still have almost no memory actually allocated to the program running. |
153 | |
154 | =head2 Perl |
155 | |
156 | Perl's memory allocation scheme is a bit convoluted, and more complex |
0430b7f7 |
157 | than can really be addressed here, but there is one common spot where Perl's |
b98fcdb9 |
158 | memory allocation is unintuitive, and that's for hash keys. |
159 | |
160 | When you have a hash, each entry has a structure that points to the |
161 | key and the value for that entry. The value is just a pointer to the |
162 | scalar in the entry, and doesn't take up any special amount of |
163 | memory. The key structure holds the hash value for the key, the key |
164 | length, and the key string. (The entry and key structures are |
165 | separate so perl can potentially share keys across multiple hashes) |
166 | |
167 | The entry structure has three pointers in it, and takes up either 12 |
168 | or 24 bytes, depending on whether you're on a 32 bit or 64 bit |
169 | system. Since these structures are of fixed size, perl can keep a big |
170 | pool of them internally (generally called an arena) so it doesn't |
171 | have to allocate memory for each one. |
172 | |
173 | The key structure, though, is of variable length because the key |
174 | string is of variable length, so perl has to ask the system for a |
175 | memory allocation for each key. The base size of this structure is |
176 | 8 or 16 bytes (once again, depending on whether you're on a 32 bit or |
177 | 64 bit system) plus the string length plus two bytes. |
178 | |
179 | Since this memory has to be allocated from the system there's the |
180 | malloc size-field overhead (4 or 8 bytes) plus the alignment bytes (0 |
181 | to 7, depending on your system and the key length) |
182 | that get added on to the chunk perl requests. If the key is only 1 |
183 | character, and you're on a 32 bit system, the allocation will be 16 |
184 | bytes. If the key is 7 characters then the allocation is 24 bytes on |
185 | a 32 bit system. If you're on a 64 bit system the numbers get even |
186 | larger. |
187 | |
b98fcdb9 |
188 | =head1 DANGERS |
189 | |
c037a281 |
190 | Since version 0.72, Devel::Size uses a new pointer tracking mechanism |
9fc9ab86 |
191 | that consumes far less memory than was previously the case. It does this |
192 | by using a bit vector where 1 bit represents each 4- or 8-byte aligned pointer |
193 | (32- or 64-bit platform dependant) that could exist. Further, it segments |
194 | that bit vector and only allocates each chunk when an address is seen within |
195 | that chunk. By default, the module builds a static table of 8,192 slots of |
196 | 16k chunks which is sufficient to cover the full 4GB virtual address space on |
197 | 32-bit platforms. Or the first 8GB on 64-bit platforms. |
198 | |
199 | Besides saving a lot of memory, this change means that Devel::Size |
200 | runs significantly faster than previous versions. |
201 | |
202 | One caveat of this new mechanism is that on 64-bit platforms with more than 8GB |
203 | of memory a new fatal error may be seen. See the next section. |
b98fcdb9 |
204 | |
5073b933 |
205 | =head1 Messages: texts originating from this module. |
206 | |
207 | =head2 Errors |
208 | |
209 | =over 4 |
210 | |
9fc9ab86 |
211 | =item "Devel::Size: Please rebuild D::S with TRACKING_SLOTS > 8192" |
212 | |
213 | This fatal error may be produced when using Devel::Size on 64-bit platforms |
214 | with more than 8GB of virtual memory. It indicates that a pointer has been |
215 | encountered that is to high for the internal pointer tracking mechanism. |
216 | |
217 | The solution is to rebuild Devel::Size having edited Size.XS to increase |
218 | the value of |
219 | |
220 | #define TRACKING_SLOTS 8192 |
5073b933 |
221 | |
9fc9ab86 |
222 | On 64-bit platforms, Devel::Size requires 1 slot for each 1MB of virtual |
223 | address space. So, for a system with 12GB of memory, this should be set to |
224 | 12GB / 1MB = 12884901888 / 1048576 = 12288 ( 12 * 1024 ). |
225 | |
226 | =item "Devel::Size: Unknown variable type" |
227 | |
228 | The thing (or something contained within it) that you gave to |
5073b933 |
229 | total_size() was unrecognisable as a Perl entity. |
230 | |
231 | =back |
232 | |
233 | =head2 warnings |
234 | |
9fc9ab86 |
235 | These messages warn you that for some types, the sizes calculated may not include |
236 | everything that could be associated with those types. The differences are usually |
5073b933 |
237 | insignificant for most uses of this module. |
238 | |
239 | These may be disabled by setting |
240 | |
9fc9ab86 |
241 | $Devel::Size::warn = 0 |
5073b933 |
242 | |
243 | =over 4 |
244 | |
9fc9ab86 |
245 | =item "Devel::Size: Calculated sizes for CVs are incomplete" |
246 | |
247 | =item "Devel::Size: Calculated sizes for FMs are incomplete" |
5073b933 |
248 | |
9fc9ab86 |
249 | =item "Devel::Size: Calculated sizes for compiled regexes are incompatible, and probably always will be" |
250 | |
251 | =back |
252 | |
c037a281 |
253 | =head2 New warnings since 0.72 |
9fc9ab86 |
254 | |
255 | Devel::Size has always been vulnerable to trapping when traversing Perl's |
256 | internal data structures, if it encounters uninitialised (dangling) pointers. |
257 | |
1a36ac09 |
258 | MSVC provides exception handling able to deal with this possibility, and when |
259 | built with MSVC Devel::Size will now attempt to ignore (or log) them and |
260 | continue. These messages are mainly of interest to Devel::Size and core |
261 | developers, and so are disabled by default. |
9fc9ab86 |
262 | |
263 | They may be enabled by setting |
264 | |
265 | $Devel::Size::dangle = 0 |
266 | |
267 | =over 4 |
268 | |
269 | =item "Devel::Size: Can't determine class of operator OPx_XXXX, assuming BASEOP\n" |
270 | |
271 | =item "Devel::Size: Encountered bad magic at: 0xXXXXXXXX" |
272 | |
273 | =item "Devel::Size: Encountered dangling pointer in opcode at: 0xXXXXXXXX" |
274 | |
275 | =item "Devel::Size: Encountered invalid pointer: 0xXXXXXXXX" |
5073b933 |
276 | |
5073b933 |
277 | =back |
278 | |
e98cedbf |
279 | =head1 BUGS |
280 | |
fea63ffa |
281 | Doesn't currently walk all the bits for code refs, formats, and |
6a9ad7ec |
282 | IO. Those throw a warning, but a minimum size for them is returned. |
e98cedbf |
283 | |
b98fcdb9 |
284 | Devel::Size only counts the memory that perl actually allocates. It |
285 | doesn't count 'dark' memory--memory that is lost due to fragmented free lists, |
286 | allocation alignments, or C library overhead. |
287 | |
e98cedbf |
288 | =head1 AUTHOR |
289 | |
290 | Dan Sugalski dan@sidhe.org |
291 | |
98ecbbc6 |
292 | Small portion taken from the B module as shipped with perl 5.6.2. |
293 | |
9fc9ab86 |
294 | New pointer tracking & exception handling by BrowserUK |
295 | |
0430b7f7 |
296 | Maintained now by Tels <http://bloodgate.com> |
297 | |
98ecbbc6 |
298 | =head1 COPYRIGHT |
299 | |
6ea94d90 |
300 | Copyright (C) 2005 Dan Sugalski, Copyright (C) 2007-2008 Tels |
98ecbbc6 |
301 | |
302 | This module is free software; you can redistribute it and/or modify it |
5a83b7cf |
303 | under the same terms as Perl v5.8.8. |
98ecbbc6 |
304 | |
e98cedbf |
305 | =head1 SEE ALSO |
306 | |
0430b7f7 |
307 | perl(1), L<Devel::Size::Report>. |
e98cedbf |
308 | |
309 | =cut |