memory leak using omapi? RESOLVED

sscdvp at gmail.com sscdvp at gmail.com
Wed Jan 26 09:01:43 UTC 2011


To summarize:

Problem: memory leak in handling OMAPI messages

Description: The problem contains in incorrect arguments to
omapi_signal while processing OMAPI STATUS replies by client side (usually
such replies are sent for error state operations like non-existent object
searches and so on). The problem is observed only on client side.

The patch against dhcp-4.2.0-P2 is attached (it resolves the problem and
some minory fixes).
Regards,
Serghei Samsi
P.S.
Solaris MDB output is attached

2011/1/25 Wilson Yang <wyang at neoninc.org>

>  Hi Serghei,
>
>
>
> You’ve got it!
>
> After wiping out the platform, I am still seeing the issue from this
> particular platform with the first two modifications.
>
> The good news is with _*all*_ three modifications you suggested, it works
> great now. E.g. NO leak on any of the platform(s).
>
>
>
> Good job well done.
>
> Appreciate your great help and timely responses.
>
> It is my pleasure working with you.
>
>
>
> Wilson Yang
>
> 720-836-2444
>
>
>
> *From:* sscdvp at gmail.com [mailto:sscdvp at gmail.com]
> *Sent:* Monday, January 24, 2011 1:16 PM
>  *To:* Wilson Yang
> *Subject:* Re: memory leak using omapi?
>
>
>
>
>
> 2011/1/24 Wilson Yang <wyang at neoninc.org>
>
>
>
> Agree with your analysis, memory leak occurs when non-existent server lease
> is acquired…  and it is all right when objects are presented. I had a
> workaround to minimize clients to be queried.
>
>
>
>
> You are trying to minimize influence of leak by minimizing the
> functionality use? Hmm...
>
>
>  As in my last mail, your previous two modifications DO fix the issue on
> deployment platform per what I am seeing.
>
>
> But valgrind output says "no". It means that previous patches did nothing
> with the initial problem although it fix other minor problems. The situation
> is very clear at the moment, because the leak visualized by valgrind is the
> one, so it can be fixed easy.
>
> As to your source code, you can omit any delay and make endless loop for
> sending traffic and you will convince yourself very soon if the leak is
> presented.
>
> Also I don't think seriously if rebuilding isc dhcp and/or reinstalling
> packages will resolve the problem or will change behavior of code - isc dhcp
> has very minimal environment, and almost all code is self-contained and
> independent.
>
> Can you try the last proposed patch just to be sure if it does the trick?
> With valgrind running output, of course.
>
> Thanks in advance.
>
> Regards, Serghei
>
>
>
> *From:* sscdvp at gmail.com [mailto:sscdvp at gmail.com]
> *Sent:* Monday, January 24, 2011 11:34 AM
>
>
> *To:* Wilson Yang
> *Subject:* Re: memory leak using omapi?
>
>
>
> Hi, Wilson,
> What do you mean by "platforms"? Is it the same OS with the same patch set,
> right? If the answer is "no", we have more opportunities to test the patch
> correctness.)
>
>      The latest patch should fix the cause of the problem you discovered -
> omapi signal with invalid argument. Let me explain shortly - the core of the
> problem - the memory leak occurs when non-existent server object (e.g.
> lease) is acquired via OMAPI, it's all right while querying the objects
> (leases) already presented in server's memory. I suppose it is the case why
> the platforms behavior so differs.
> Please correct me if I was wrong.
> The problem is only observed on the client-side only, and the mentioned
> patch touches the client part of OMAPI only. You code (list_dhclient) should
> be relinked with modified version of libdhcpctl (statically).
>
> I will wait for your response.
>
> Regards, Serghei
>
> 2011/1/24 Wilson Yang <wyang at neoninc.org>
>
> Hi Serghei,
>
>
>
> Not at all.
>
> I tested on three platforms, two for deployment and one for development.
>
> Memory leak issue from deployment platforms seem to be fixed with your
> suggestions last week. E.g. your first two modifications do fix it !
> However, issue from the development platform persists.
>
> I am verifying why differences are seeing among them and keep you posted.
>
> I’ll hold off this change for now if that is OK with you.
>
> Appreciate your great help and timely responses! J
>
>
>
> Wilson Yang
>
> 720-836-2444
>
>
>
> *From:* sscdvp at gmail.com [mailto:sscdvp at gmail.com]
> *Sent:* Sunday, January 23, 2011 11:35 AM
> *To:* Wilson Yang
>
>
> *Subject:* Re: memory leak using omapi?
>
>
>
> Hi, Wilson.
>
> Apologies for not responding last days.
> Can you make the following changes in omapip/messages.c in method
> omapi_message_process_internal() along with the changes posted by me before?
>
> --cut--
>               case OMAPI_OP_STATUS:
>                 /* The return status of a request. */
>                 if (!m)
>                         return ISC_R_UNEXPECTED;
>
>                 /* Get the wait status. */
>                 status = omapi_get_value_str (mo, message -> id_object,
>                                               "result", &tv);
>                 if (status == ISC_R_SUCCESS) {
>                         status = omapi_get_int_value (&wsi, tv -> value);
>                         waitstatus = wsi;
>                         omapi_value_dereference (&tv, MDL);
>                         if (status != ISC_R_SUCCESS)
>                                 waitstatus = ISC_R_UNEXPECTED;
>                 } else
>                         waitstatus = ISC_R_UNEXPECTED;
>
>  /*               status = omapi_get_value_str (mo, message -> id_object,
>                                               "message", &tv);
>                 omapi_signal ((omapi_object_t *)m, "status", waitstatus,
> tv);
>                 if (status == ISC_R_SUCCESS)
>                         omapi_value_dereference (&tv, MDL);*/
>                 omapi_signal ((omapi_object_t *)m,
>                                      "status", waitstatus,
>                                      (omapi_typed_data_t *)0);
>
>                 omapi_message_unregister((omapi_object_t *)m);
>
>                 return ISC_R_SUCCESS;
> --cut--
>
> Please confirm if this patch fixes the problem.
>
> Regards,
> Serghei Samsi
>
> 2011/1/21 Wilson Yang <wyang at neoninc.org>
>
> Hi Serghei,
>
>
>
> valgrind output attached.
>
> Thanks.
>
>
>
> Wilson Yang
>
> 720-836-2444
>
>
>
> *From:* sscdvp at gmail.com [mailto:sscdvp at gmail.com]
> *Sent:* Thursday, January 20, 2011 3:02 PM
> *To:* wyang at neoninc.org
>
>
> *Subject:* Re: memory leak using omapi?
>
>
>
> Hi, Wilson,
>
> Hmm, the problem still persists. According to your trace it seems that
> omapi_protocol_object loses some pointers in omapi_protocol_signal_handler()
> which are not dereferenced in some cases.
>
> Can you add four lines in protocol.c and retest (with valgrind)?
>
> --cut--
> isc_result_t omapi_protocol_destroy (omapi_object_t *h,
>                                      const char *file, int line)
> {
>         omapi_protocol_object_t *p;
>         if (h -> type != omapi_type_protocol)
>                 return ISC_R_INVALIDARG;
>         p = (omapi_protocol_object_t *)h;
>         if (p -> message)
>                 omapi_message_dereference (&p -> message, file, line);
>         if (p -> name)
>                 omapi_data_string_dereference (&p -> name, file, line);
>         if (p -> value)
>                  omapi_typed_data_dereference (&p -> value, file, line);
>
> --cut--
>
> Regards,
> Serghei Samsi
>
> 2011/1/20 Wilson Yang <wyang at neoninc.org>
>
> Hi, Serghei,
>
>
>
> Attached the code I am using and the output of valgrind. The change you
> suggested was made to dhcp-4.1-ESV.
>
> It seems OK per linux “ps” output, but valgrind still complains.
>
> Thanks again for your time and effort and let me know if this is OK or more
> mod needed.
>
>
>
> Wilson Yang
>
> 720-836-2444
>
>
>
> *From:* sscdvp at gmail.com [mailto:sscdvp at gmail.com]
> *Sent:* Thursday, January 20, 2011 5:44 AM
> *To:* dhcp-users at lists.isc.org; wyang at neoninc.org
> *Subject:* Re: memory leak using omapi?
>
>
>
> Hi,
>
> While looking through the file message.c,
>
> I found that in function omapi_message_destroy()
>
> in the following block
>
>
>
> --cut--
>
>         m = (omapi_message_object_t *)h;
>         if (m -> authenticator) {
>                 omapi_typed_data_dereference (&m -> authenticator, file,
> line);
>         }
>         if (!m -> prev && omapi_registered_messages != m)
>                 omapi_message_unregister (h);
> --cut--
>
>
>
> omapi message_unregister() for h argument (m has the value of h!) is never
> processed because it tests the same condition for exiting prematurally
> (return ISC_R_INVALIDARG):
>
>
>
> --cut--
>
> isc_result_t omapi_message_unregister (omapi_object_t *mo)
> {
>         omapi_message_object_t *m;
>         omapi_message_object_t *n;
>
>         if (mo -> type != omapi_type_message)
>                 return ISC_R_INVALIDARG;
>         m = (omapi_message_object_t *)mo;
>
>         /* Not registered? */
>         if (!m -> prev && omapi_registered_messages != m)
>                 return ISC_R_INVALIDARG;
>
> --cut--
>
> I suggest commenting the block in omapi_message_destroy()
>
> like this (leaving omapi_message_unregister() to try it's own test):
>
>
>
> --cut--
>
>         //if (!m -> prev && omapi_registered_messages != m)
>                 omapi_message_unregister (h);
> --cut--
>
>
>
> Can you make this change, recompile & run a valgrind test again?
>
>
>
> Regards,
>
> Serghei Samsi
>
>
>
>
>
> 2011/1/19 Wilson Yang <wyang at neoninc.org>
>
> Hi,
>
>
>
> Found a email thread related to memory leak using omapi. Not sure if this
> has been fixed or a work around is available.
>
> Could you please shed some light on this?
>
> Appreciate your help in advance.
>
>
>
> An issue record, [ISC-Bugs #22784] has been created to address this
> behavior.
>
>
>
> Here is valgrind output while using v3.1-ESV
>
> ==2849== 70 (12 direct, 58 indirect) bytes in 1 blocks are definitely lost
> in loss record 37 of 39
>
> ==2849==    at 0x4022FBA: malloc (vg_replace_malloc.c:236)
>
> ==2849==    by 0x804FABF: dmalloc (alloc.c:80)
>
> ==2849==    by 0x805066C: omapi_value_new (alloc.c:1025)
>
> ==2849==    by 0x8055D10: omapi_generic_set_value (generic.c:160)
>
> ==2849==    by 0x805881A: omapi_message_set_value (message.c:160)
>
> ==2849==    by 0x8056A35: omapi_set_value (support.c:356)
>
> ==2849==    by 0x804CFFF: omapi_protocol_signal_handler (protocol.c:653)
>
> ==2849==    by 0x8052C3D: omapi_connection_signal_handler
> (connection.c:1018)
>
> ==2849==    by 0x805566E: omapi_io_signal_handler (dispatch.c:614)
>
> ==2849==    by 0x8056976: omapi_signal (support.c:290)
>
> ==2849==    by 0x804EBA7: omapi_connection_reader_trace (buffer.c:261)
>
> ==2849==    by 0x804E75F: omapi_connection_reader (buffer.c:137)
>
>
>
> And valgrind output while using v4.1-ESV,
>
> ==5828== 70 (12 direct, 58 indirect) bytes in 2 blocks are definitely lost
> in loss record 39 of 39
>
> ==5828==    at 0x4022FBA: malloc (vg_replace_malloc.c:236)
>
> ==5828==    by 0x804F596: dmalloc (alloc.c:75)
>
> ==5828==    by 0x804F663: omapi_value_new (alloc.c:1021)
>
> ==5828==    by 0x805473D: omapi_generic_set_value (generic.c:157)
>
> ==5828==    by 0x805784A: omapi_message_set_value (message.c:162)
>
> ==5828==    by 0x804D2DA: omapi_protocol_signal_handler (protocol.c:649)
>
> ==5828==    by 0x8055866: omapi_signal (support.c:286)
>
> ==5828==    by 0x804EAC3: omapi_connection_reader_trace (buffer.c:265)
>
> ==5828==    by 0x80536D2: omapi_one_dispatch (dispatch.c:520)
>
> ==5828==    by 0x8053D53: omapi_wait_for_completion (dispatch.c:290)
>
> ==5828==    by 0x804AC39: dhcpctl_wait_for_completion (dhcpctl.c:141)
>
> ==5828==    by 0x8049B7B: main (in /home/user/list_dhclient)
>
>
>
>
>
> Wilson Yang
>
> 720-836-2444
>
>
>
>
> _______________________________________________
> dhcp-users mailing list
> dhcp-users at lists.isc.org
> https://lists.isc.org/mailman/listinfo/dhcp-users
>
>
>
>
>
>
>
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.isc.org/pipermail/dhcp-users/attachments/20110126/5fd68120/attachment.html>
-------------- next part --------------
findleaks:                maximum buffers => 104754
findleaks:                 actual buffers => 104691
findleaks:
findleaks:             potential pointers => 96098
findleaks:                     dismissals => 63709         (66.2%)
findleaks:                         misses => 29700         (30.9%)
findleaks:                           dups => 2617          ( 2.7%)
findleaks:                        follows => 72            ( 0.0%)
findleaks:
findleaks:              elapsed wall time => 3 seconds
findleaks:
CACHE     LEAKED   BUFCTL CALLER
08089710   20923 080b4e98 dmalloc+0x12
08089010   20813 080bd2f0 dmalloc+0x12
08089010     111 080bd278 dmalloc+0x12
08086710   19369 080a9a20 dmalloc+0x12
08086710   22478 080a9840 dmalloc+0x12
08086710   20925 080a9750 dmalloc+0x12
----------------------------------------------------------------------
   Total  104619 buffers, 3515176 bytes

umem_alloc_56 leak: 20923 buffers, 56 bytes each, 1171688 bytes total
            ADDR          BUFADDR        TIMESTAMP           THREAD
                            CACHE          LASTLOG         CONTENTS
         80b4e98          80b2f38    9b01f078ad866                1
                          8089710                0                0
                 libumem.so.1`umem_cache_alloc_debug+0x16c
                 libumem.so.1`umem_cache_alloc+0x15c
                 libumem.so.1`umem_alloc+0x3f
                 libumem.so.1`malloc+0x23
                 dmalloc+0x12
                 omapi_typed_data_new+0xaa
                 omapi_protocol_signal_handler+0x4cf
                 omapi_connection_signal_handler+0x38
                 omapi_io_signal_handler+0x38
                 omapi_signal+0x2e
                 omapi_connection_reader_trace+0x2c9
                 omapi_connection_reader+0x14
                 omapi_one_dispatch+0x6ab
                 omapi_wait_for_completion+0xba
                 dhcpctl_wait_for_completion+0x12

umem_alloc_40 leak: 20813 buffers, 40 bytes each, 832520 bytes total
            ADDR          BUFADDR        TIMESTAMP           THREAD
                            CACHE          LASTLOG         CONTENTS
         80bd2f0          80baf90    9b01f078e831f                1
                          8089010                0                0
                 libumem.so.1`umem_cache_alloc_debug+0x16c
                 libumem.so.1`umem_cache_alloc+0x15c
                 libumem.so.1`umem_alloc+0x3f
                 libumem.so.1`malloc+0x23
                 dmalloc+0x12
                 omapi_typed_data_new+0xaa
                 omapi_protocol_signal_handler+0x4cf
                 omapi_connection_signal_handler+0x38
                 omapi_io_signal_handler+0x38
                 omapi_signal+0x2e
                 omapi_connection_reader_trace+0x2c9
                 omapi_connection_reader+0x14
                 omapi_one_dispatch+0x6ab
                 omapi_wait_for_completion+0xba
                 dhcpctl_wait_for_completion+0x12

umem_alloc_40 leak: 111 buffers, 40 bytes each, 4440 bytes total
            ADDR          BUFADDR        TIMESTAMP           THREAD
                            CACHE          LASTLOG         CONTENTS
         80bd278          80bafc8    9b01f0b0676d7                1
                          8089010                0                0
                 libumem.so.1`umem_cache_alloc_debug+0x16c
                 libumem.so.1`umem_cache_alloc+0xe1
                 libumem.so.1`umem_alloc+0x3f
                 libumem.so.1`malloc+0x23
                 dmalloc+0x12
                 omapi_typed_data_new+0xaa
                 omapi_protocol_signal_handler+0x4cf
                 omapi_connection_signal_handler+0x38
                 omapi_io_signal_handler+0x38
                 omapi_signal+0x2e
                 omapi_connection_reader_trace+0x2c9
                 omapi_connection_reader+0x14
                 omapi_one_dispatch+0x6ab
                 omapi_wait_for_completion+0xba
                 dhcpctl_wait_for_completion+0x12

umem_alloc_24 leak: 19369 buffers, 24 bytes each, 464856 bytes total
            ADDR          BUFADDR        TIMESTAMP           THREAD
                            CACHE          LASTLOG         CONTENTS
         80a9a20          80a5d28    9b01f078ad328                1
                          8086710                0                0
                 libumem.so.1`umem_cache_alloc_debug+0x16c
                 libumem.so.1`umem_cache_alloc+0x15c
                 libumem.so.1`umem_alloc+0x3f
                 libumem.so.1`malloc+0x23
                 dmalloc+0x12
                 omapi_data_string_new+0x28
                 omapi_protocol_signal_handler+0x42f
                 omapi_connection_signal_handler+0x38
                 omapi_io_signal_handler+0x38
                 omapi_signal+0x2e
                 omapi_connection_reader_trace+0x2c9
                 omapi_connection_reader+0x14
                 omapi_one_dispatch+0x6ab
                 omapi_wait_for_completion+0xba
                 dhcpctl_wait_for_completion+0x12

umem_alloc_24 leak: 22478 buffers, 24 bytes each, 539472 bytes total
            ADDR          BUFADDR        TIMESTAMP           THREAD
                            CACHE          LASTLOG         CONTENTS
         80a9840          80a5dc8    9b01f078e7d0c                1
                          8086710                0                0
                 libumem.so.1`umem_cache_alloc_debug+0x16c
                 libumem.so.1`umem_cache_alloc+0xe1
                 libumem.so.1`umem_alloc+0x3f
                 libumem.so.1`malloc+0x23
                 dmalloc+0x12
                 omapi_data_string_new+0x28
                 omapi_protocol_signal_handler+0x42f
                 omapi_connection_signal_handler+0x38
                 omapi_io_signal_handler+0x38
                 omapi_signal+0x2e
                 omapi_connection_reader_trace+0x2c9
                 omapi_connection_reader+0x14
                 omapi_one_dispatch+0x6ab
                 omapi_wait_for_completion+0xba
                 dhcpctl_wait_for_completion+0x12

umem_alloc_24 leak: 20925 buffers, 24 bytes each, 502200 bytes total
            ADDR          BUFADDR        TIMESTAMP           THREAD
                            CACHE          LASTLOG         CONTENTS
         80a9750          80a5e18    9b01f078e8bb2                1
                          8086710                0                0
                 libumem.so.1`umem_cache_alloc_debug+0x16c
                 libumem.so.1`umem_cache_alloc+0xe1
                 libumem.so.1`umem_alloc+0x3f
                 libumem.so.1`malloc+0x23
                 dmalloc+0x12
                 omapi_value_new+0x17
                 omapi_generic_set_value+0x324
                 omapi_message_set_value+0x210
                 omapi_set_value+0x2d
                 omapi_protocol_signal_handler+0x576
                 omapi_connection_signal_handler+0x38
                 omapi_io_signal_handler+0x38
                 omapi_signal+0x2e
                 omapi_connection_reader_trace+0x2c9
                 omapi_connection_reader+0x14

-------------- next part --------------
A non-text attachment was scrubbed...
Name: dhcp-4.2.0-P2-omapi-messages-memleaks.patch
Type: application/octet-stream
Size: 1481 bytes
Desc: not available
URL: <https://lists.isc.org/pipermail/dhcp-users/attachments/20110126/5fd68120/attachment.obj>


More information about the dhcp-users mailing list