Indy HTTPS POST is truncating the response at 34076 bytes

Giganews Newsgroups
Subject: Indy HTTPS POST is truncating the response at 34076 bytes
Posted by:  aandea (al…@aandea.com)
Date: Mon, 22 May 2006

There is an error in Indy 10.0.76

The error manifests itself when reading data from an HTTPS POST (using Indy SSL as a client with OpenSSL), when the return data is larger than 34076 (32768 + 1308) and the data is being returned very fast in a single package (this is happening when using an SSL accelerator card like ncipher Ultra).

The return result will be truncated at 34076, the rest of the data is lost and no errors are raised when this data is lost

To walk through this bug we?ll assume that the return response should be 34438 in size (this is what the server is actually returning).

Below is the main processing logic from the Indy HTTP ReadResult function. This function will ignore the EIdConnClosedGracefully exception as these are signaling the end of read data. This error will raise an invalid EIdConnClosedGracefully exception, thus terminating the processing loop faster. Because of this legitimate code, the exception will never reach the user, which believe that it got all the data:

procedure TIdCustomHTTP.ReadResult(AResponse: TIdHTTPResponse);
...
        try
          IOHandler.ReadStream(LS, AResponse.ContentLength);
        except
          on E: EIdConnClosedGracefully do
        end;
...
end;

The above procedure is calling
procedure TIdIOHandler.ReadStream(AStream: TIdStreamVCL; AByteCount: Integer;
to do the reading. The reading is done in a while loop

procedure TIdIOHandler.ReadStream(AStream: TIdStreamVCL; AByteCount: Integer;
  AReadUntilDisconnect: Boolean);
...
    // If data already exists in the buffer, write it out first.
    if FInputBuffer.Size > 0 then begin
      i := Min(FInputBuffer.Size, LWorkCount);
      FInputBuffer.ExtractToStream(AStream, i);
      Dec(LWorkCount, i);
    end;

    while Connected and (LWorkCount > 0) do begin    
      i := Min(LWorkCount, LBufSize);
      //TODO: Improve this - dont like the use of the exception handler
      //DONE -oAPR: Dont use a string, use a memory buffer or better yet the buffer itself.
      try
        try
          SetLength(LBuf, 0); // clear the buffer
          ReadBytes(LBuf, i, False);
        except
        ...
        end;
      finally
        if i > 0 then begin
          AStream.Write(LBuf, i);
          Dec(LWorkCount, i);
        end;
      end;
    end;
...
end;

to be noted that when first executing this code, there already is data in the buffer : 1308 bytes. Thus this data will be first extracted to the result stream.
LWorkCount  just became 34438  - 1308  = 33130
The code will enter the while loop to read data. The buffer size is 32768, so i := Min(LWorkCount, LBufSize) will choose to read 32768 bytes (the size of the buffer). The data will be read and saved to the result stream. On the next test of the while condition, ?while Connected and (LWorkCount > 0) do ? is evaluate. The Connected function will call :

function TIdIOHandlerStack.Connected: Boolean;
begin
  ReadFromSource(False, 0, False);
  Result := inherited Connected;
end;

this in turn is calling the ReadFromSource (the main handler read routine)

function TIdSSLIOHandlerSocketOpenSSL.ReadFromSource(
ARaiseExceptionIfDisconnected: Boolean; ATimeout: Integer;
ARaiseExceptionOnTimeout: Boolean): Integer;
...
    repeat
      if Readable(ATimeout) then begin
        if Assigned(FRecvBuffer) then begin
    ...
              LByteCount := Recv(LBuffer);
    ...
        end else begin
          LByteCount := 0;
    ...
        end;
        FClosedGracefully := LByteCount = 0;
    ...
        CheckForDisconnect(ARaiseExceptionIfDisconnected);
        Result := LByteCount;
      end else begin
    ...
      end;
    until (LByteCount <> 0) or (Connected = False);
end;

There is no more data to read as the previous read loop read and processed 32768 and left 362 bytes in the TIdIOHandler.FInputBuffer. Thus lByteCount will be 0. This is causing the FClosedGracefully to be set.
When CheckForDisconnect is executed (code follows),
procedure TIdIOHandlerStack.CheckForDisconnect(
ARaiseExceptionIfDisconnected: Boolean; AIgnoreBuffer: Boolean);
...
  if ClosedGracefully then begin
    if BindingAllocated then begin
      Close;
      // Call event handlers to inform the user that we were disconnected
      DoStatus(hsDisconnected);
      //DoOnDisconnected;
    end;
    LDisconnected := True;
  end else begin
  ...
  end;
  ...
end;

the ClosedGracefully flag begin set will close the socket connection (by calling TIdIOHandlerSocket.Close; which calls FBinding.CloseSocket;). This will set the TIdSocketHandle.FHandleAllocated flag to false.

Back into the ReadFromSource code now, the ?repeat condition is until (LByteCount <> 0) or (Connected = False)?;. LByteCount begin 0, the computer must now evaluate Connected again. TIdIOHandlerStack.Connected will call  ReadFromSource(False, 0, False) again. This time we will look to what?s happening in if Readable(ATimeout) then begin statement. The Readable function will call (through Binding.Readable) TIdSocketHandle.Readable. In this function :
function TIdSocketHandle.Readable(AMSec: Integer = IdTimeoutDefault): Boolean;

  function CheckIsReadable(AMSec: Integer): Boolean;
  begin
    if HandleAllocated then begin
      Result := Select(AMSec);
    end else begin
      raise EIdConnClosedGracefully.Create(RSConnectionClosedGracefully);
    end;
  end;

begin
  ...
  Result := CheckIsReadable(AMSec);
end;

CheckisReadable will raise an EIdConnClosedGracefully exception because the HandleAllocated is false.
This exeption will cause the flow to exit all the way to ReadREsult procedure which will eat the exception. Thus, the 362 bytes left in TIdIOHandler.FInputBuffer will never be processed. The response return will be missing the tail (the 362 bytes) and the user will never know.

Well, user will almost never know. In our case we are retuning a compressed (Zip) stream, which fails on decompression because of the missing 362 bytes from the tail.

Replies