Re: Messages sent with PostMessage getting lost

Giganews Newsgroups
Subject: Re: Messages sent with PostMessage getting lost
Posted by:  Martin James (mjames_falc…
Date: Mon, 17 Oct 2005

> I'm writing an Indy app that send emails in parallel from threads.  I
> the threads to report back progress, bytes sent, etc regularly and I had
> done this by accessing strings in the thread from the VCL loop.  Now,
> being wrapped over the knuckles by Martin James (and quite rightly too I
> realise) I am in the process of reporting back progress by PostMessage.

Well, I would not like any advice I might give to be interpreted as
knuckle-rapping <g> .  If I regarded the valuable help I have received in
these groups in such a way, I would have all my fingers worn off by now :)

> Just to satisfy my curiousity I wrote a test program that started a thread
> and furiously assigned to a string from within the thread, and then
> the string from the main loop (via Application.OnIdle).  As expected,
> so often (but not very often), the string as seen by the main loop is not
> what it should be.  Also, and very much less often, I get an AV.
> confirmation that it's not a good idea.

Yes, as expected, occasioally the string writes/reads will get interrupted
and the reader/writer thread will get preempted.

> Next step was to try posting the string back.  I create a record for the
> string with New in the thread and pass it's address as a message parameter
> and post it off.

Should be fine though I prefer to use a class - ideally one class that
performs all ITC for the whole app.  A 'command' enumeration in the class
tells the consumer what an instance contains.

> In the main thread message handler I grab the string, check it, and
> the memory that was used to transport it back.  As expected, it runs
> errors in the received string.
> However...
> Just to really push things, I didn't send off one message in the thread -
> send off 1000 at a time and follow with a sleep so that the main thread
> a chance to handle the messages.  I keep track of messages posted and
> messages delivered and find that over time, I lose some of them.

*Never* seen this happen, except when deliberately trying to provoke the
Windows message queue by posting above the 10000-message limit.  If there
are more than 10000 messages in the queue, another postMessage will fail,
(and return an error result).  In your test app, check the PM return result
to see if it is zero.
SDK for PostMessage:

Return Values

If the function succeeds, the return value is nonzero.
If the function fails, the return value is zero. To get extended error
information, call GetLastError.

> Presumably
> the storage I create to transport the string gets lost also - i.e. I leak
> memory.

Yes - so you should try and ensure that this does not happen :)

> What's going on here?  Can we not assume that all messages posted make it
> the handler?

Message queue communication is absolutely at the core of Windows operation.
If Windows 'lost' *any* messages without any error notification, the OS
would become so unstable, so quickly, that it would probably crash before it
got started.

I suspect that there are few apps that can expect to have 10000 messages
outstanding in the main-thread MQ, so you can probably safely ignore this

In my apps, I do not allow this limit to be reached anyway.  I do this by
inter-thread comms object pooling.  I create, say 500, inter-thread comms
objects at app startup and push them onto a Producer-Consumer queue.  Any
thread that wants one of these objects, (eg. to send a status report to the
main thread for display),  gets one from this pool, rather than creating one
in the 'usual' way.  Once got, the thread fills up the instance with status,
or whatever, data and PostMessages it off.  The main thread message-handler
gets the object, updates whatever VCL components it needs to, then, instead
of freeing the object, it returns it to the pool queue for re-use.

Such an approach has many advantages, one of which is that no more than 500,
(or whatever),  of these objects can be PostMessaged off to the main thread.
If all 500 objects are queued up in the main-thread message-queue, any
secondary thread trying to get another from the, (now empty), P-C pool queue
is blocked until the main thread returns objects to the pool after
processing them.  This auto-regulates the inter-thread comms of the whole
app, preventing memory use runaway and Windows-message-queue overload.

There are some other advantages:

Memory-management calls for creation/destruction of the ITC objects are
eliminated, though at the cost of perhaps some extra code because an object
from the pool will not be initialised.  This may be a disadvantage, or not,
or an advantage, depending on the app - you just have to be aware of it.

Fetching/releasing objects from a  good producer-consumer queue class is
much faster than create/destroy, especially for complex objects.

The use of an ITC pool eliminates any need for an inter-thread PC-queue to
apply any 'queue full' mechanism, where a thread attempting to 'push' an
object gets blocked because the queue size has reached some limit.  There
are only a limited number of objects in the app to start with, so the P-C
queue code does not need a load of extra complexity to apply any limit
itself, eg:

With an Object pool:

var thisObject:ITCObject;
    myPool.getObject(@thisObject,INFINITE);  //gets blocked here if pool
    consumerThreadQueue.push(thisObject);  // this queue needs no limiter

Without an object pool;

var thisObject:ITCObject;
    consumerThreadQueue.push(thisObject);  // this queue needs complex code
    // to apply an object count limit and block this thread if queue full.

Monitoring of the objects in use is much easier with a pool than when using
the memory-manager and/or complex 3-rd-party memory monitors.  A P-C queue
class typically has an accessible 'count' property that can be simple dumped
to a status bar, (or whatever), on a simple 1-sec timer.  This allows the
developer/user to watch the pool level as the app runs.  Leaks from the
pool, for example are easily spotted - if the app is loaded up, then the
load removed, the displayed count should go back to it's initial value as
all the objects get released.  If the count is lower, you have leaked an
object or two.  This pool-monitor timer can be left in deliverables - then
knowledgeable customers ring up and tell you that there is a leak in
scenario 'X', but the app is still working.  The pool level could could also
be logged along with whatever else the app logs.  This might enable you to
spot and fix occasional, intermittent leaks without the customer ever
realising that they had a problem.



In response to

Messages sent with PostMessage getting lost posted by Ross McMillan on Mon, 17 Oct 2005