Thanks to visit codestin.com
Credit goes to github.com

Skip to content
This repository was archived by the owner on Jan 23, 2023. It is now read-only.

SslStream Improvements for Unix Performance #22998

Closed
wants to merge 55 commits into from
Closed

SslStream Improvements for Unix Performance #22998

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
55 commits
Select commit Hold shift + click to select a range
4b4b72c
Added reference to AppSet/AppGet from openssl
Drawaes Aug 2, 2017
4079b92
Added Bio methods to the openssl native shim
Drawaes Aug 2, 2017
1fa274a
Add basic custom bio interop code, write and read still needed
Drawaes Aug 2, 2017
d5fc983
Added BIO.Custom file
Drawaes Aug 2, 2017
7bbcc3d
Added Write Buffer and changed disposable to be public to match inter…
Drawaes Aug 2, 2017
dd8fae2
Added the custom bio into the main code line for SslStream
Drawaes Aug 2, 2017
7c51afe
Add the BIO Control method
Drawaes Aug 2, 2017
7080b69
Removed pending reference from DoHandshake
Drawaes Aug 2, 2017
860bba1
Added custom bio read code
Drawaes Aug 2, 2017
6b19a38
changed read flags
Drawaes Aug 2, 2017
7f7f47a
Added write method
Drawaes Aug 2, 2017
0be51ac
fixed indexing on resize
Drawaes Aug 2, 2017
4694662
updated the output ref on encrypt
Drawaes Aug 3, 2017
5f4e2bf
Added short circuit for the dohandshake to return the shutdown alert
Drawaes Aug 3, 2017
293ea41
Simplify the write bio
Drawaes Aug 5, 2017
d83a091
Changed the write to use a span
Drawaes Aug 5, 2017
05b412b
Typo, intput
Drawaes Aug 5, 2017
56a7e43
Fix span copyto
Drawaes Aug 5, 2017
e6dd27e
Added length update
Drawaes Aug 5, 2017
93e0ad9
Change the pinnable write buffer pool size and the max message size t…
Drawaes Aug 5, 2017
7904f68
Remove some flag settings
Drawaes Aug 5, 2017
bebfc15
Simplify the read bio
Drawaes Aug 5, 2017
6033a15
Changed the write path to allow for multiple writes to the output buffer
Drawaes Aug 6, 2017
f4c0005
Resetting the pinned buffers to the original size
Drawaes Aug 6, 2017
b4d2301
Changed pinnable buffers on write to use one for the write then retur…
Drawaes Aug 6, 2017
f4dbc39
Reset Sln Changes before PR
Drawaes Aug 6, 2017
0e78e71
Added the true flag to the new methods
Drawaes Aug 6, 2017
3014c47
Change from Macros that aren't in older OpenSSL to the underlying fun…
Drawaes Aug 6, 2017
a47874f
Removed var voliations in files edited (there was one in an loop i di…
Drawaes Aug 7, 2017
fb87ba4
Added named parameters to increase readability
Drawaes Aug 7, 2017
dcbed7e
Removed raw set flags methods and provided shim methods for the two f…
Drawaes Aug 8, 2017
3ec53c6
React to shim changes in the managed layer
Drawaes Aug 8, 2017
3c1d3bd
React to shim changes in the managed layer
Drawaes Aug 8, 2017
f6dcc55
Fix the flags change to be the correct direction
Drawaes Aug 8, 2017
0a27d81
Cleaned up spacing and formatting and the interop code
Drawaes Aug 8, 2017
bc46993
Reacting to Review
Drawaes Aug 9, 2017
5ba0c75
Fixing file bit mode to remove execute caused by windows file copy
Drawaes Aug 9, 2017
a049480
More bit mode changes
Drawaes Aug 9, 2017
fc6f327
Fix style using clang-format and moved statics and typedefs to the top
Drawaes Aug 18, 2017
8fd60cc
Reacting to review changes
Drawaes Aug 18, 2017
f51c9e8
Changed Take Bytes
Drawaes Aug 18, 2017
03da084
nit var
Drawaes Aug 18, 2017
172a1de
nit sln changes
Drawaes Aug 18, 2017
8249c15
removed initialize for static constructor, nit readonly
Drawaes Aug 18, 2017
5e97bf7
removed initialize for static constructor, nit readonly
Drawaes Aug 18, 2017
f8e86a6
Changed if to be more concise
Drawaes Aug 18, 2017
9a800a3
Removed finalizers, added assert removed usless is allocated test. ad…
Drawaes Aug 18, 2017
90eb0f7
wrong space removed
Drawaes Aug 18, 2017
3f35c68
Changed span to bye[], changed continuation to run sync and to propag…
Drawaes Aug 18, 2017
155e8e9
Fixed compile error
Drawaes Aug 18, 2017
e3f1577
Reverted Continutation
Drawaes Aug 19, 2017
c67a9e2
Fixed bug around write being triggered during decrypt. Now correctly …
Drawaes Aug 20, 2017
a1ea5b5
PinnableBufferCache -> ArrayPool
benaadams Aug 20, 2017
354a992
Merge pull request #3 from benaadams/arraypool
Drawaes Aug 20, 2017
4a1734b
Fixed issue with the maxsize calculation and removed pinnable tests
Drawaes Aug 20, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Changed span to bye[], changed continuation to run sync and to propag… 
…ate errors
  • Loading branch information
@Drawaes
Drawaes committed Aug 18, 2017
commit 3f35c6875165df2528c15e285358016890920404
3 changes: 2 additions & 1 deletion src/Common/src/Interop/Unix/System.Security.Cryptography.Native/Interop.Ssl.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -391,8 +391,9 @@ public int Write(Span<byte> input)
}
else if (_byteArray.Length - _bytesWritten < input.Length)
{
var oldSpan = new Span<byte>(_byteArray);
byte[] oldArray = _byteArray;
_byteArray = new byte[input.Length + _bytesWritten];
Buffer.BlockCopy(oldArray, 0, _byteArray, 0, _bytesWritten);
oldSpan.CopyTo(_byteArray);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How is this going to compile? :)

}
}
Expand Down
30 changes: 19 additions & 11 deletions src/System.Net.Security/src/System/Net/Security/SslStreamInternal.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,11 @@ internal SslStreamInternal(SslState sslState)

_decryptedBytesOffset = 0;
_decryptedBytesCount = 0;
_freeBufferAction = (_, buffer) => FreeBuffer((byte[])buffer);
_freeBufferAction = (task, buffer) =>
{
FreeBuffer((byte[])buffer);
task.GetAwaiter().GetResult(); // propagate any exception
};
}

// If we have a read buffer from the pinnable cache, return it.
Expand Down Expand Up @@ -415,14 +419,15 @@ private void StartWriting(byte[] buffer, int offset, int count, AsyncProtocolReq
{
// Prepare for the next request.
asyncRequest.SetNextRequest(buffer, offset + chunkBytes, count - chunkBytes, s_resumeAsyncWriteCallback);
Task t = _sslState.InnerStream.WriteAsync(outBuffer, 0, encryptedBytes)
.ContinueWith(_freeBufferAction, outBuffer);
Task t = _sslState.InnerStream.WriteAsync(outBuffer, 0, encryptedBytes);
if (t.IsCompleted)
{
FreeBuffer(outBuffer);
t.GetAwaiter().GetResult();
}
else
{
t = t.ContinueWith(_freeBufferAction, outBuffer, CancellationToken.None, TaskContinuationOptions.ExecuteSynchronously, TaskScheduler.Default);
IAsyncResult ar = TaskToApm.Begin(t, s_writeCallback, asyncRequest);
if (!ar.CompletedSynchronously)
{
Expand All @@ -434,8 +439,14 @@ private void StartWriting(byte[] buffer, int offset, int count, AsyncProtocolReq
}
else
{
_sslState.InnerStream.Write(outBuffer, 0, encryptedBytes);
FreeBuffer(outBuffer);
try
{
_sslState.InnerStream.Write(outBuffer, 0, encryptedBytes);
}
finally
{
FreeBuffer(outBuffer);
}
}

offset += chunkBytes;
Expand All @@ -447,11 +458,7 @@ private void StartWriting(byte[] buffer, int offset, int count, AsyncProtocolReq
} while (count != 0);

}

if (asyncRequest != null)
{
asyncRequest.CompleteUser();
}
asyncRequest?.CompleteUser();
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@stephentoub I have narrowed the issue down. It seems that it is nothing to do with the continuation/free up above. The issue is around calling asyncRequest.CompleteUser() on the current thread. So ~180k rps with the code as it stands and 256 connections (24 core/48 thread server)

Task.Run(() => asyncRequest?.CompleteUser());
It jumps to > 900k rps

This is only effecting Unix, to me it seems like a contention issue because the RPS scales up to ~10 connections and then slowly goes down from there unless you put in a Task.Run().

Looking at it, could it be something in the lazy result?

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CompleteUser is basically just calling the user callback. There shouldn't be any contention in the CompleteUser call itself (though of course, there could be).

The main difference here is likely something in the user code. To test this, you could add Task.Run to the user code so that the user callback will be enqueued, instead of run immediately, and see if that makes any difference.

Which test are you seeing this with? I'm not sure quite what the user's callback will be doing in this case.

}

private void FreeBuffer(byte[] buffer)
Copy link
Member

@stephentoub stephentoub Aug 18, 2017 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you make this static? Then the _freeBufferAction delegate won't need to allocate a delegate per stream instance, and _freeBufferAction could be made a static s_freeBufferAction.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

see @benaadams and my comments above, if you want to lose the this.GetHashCode() in the logging then sure I am good with that.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So do we want to lose the gethashcode for the logging, does seem like a waste to have for just that?

Expand Down Expand Up @@ -814,7 +821,8 @@ private int ProcessFrameBody(int readBytes, byte[] buffer, int offset, int count
private int ProcessReadErrorCode(SecurityStatusPal status, AsyncProtocolRequest asyncRequest, byte[] extraBuffer)
{
ProtocolToken message = new ProtocolToken(null, status);
if (NetEventSource.IsEnabled) NetEventSource.Info(null, $"***Processing an error Status = {message.Status}");
if (NetEventSource.IsEnabled)
NetEventSource.Info(null, $"***Processing an error Status = {message.Status}");

if (message.Renegotiate)
{
Expand Down