Thanks to visit codestin.com
Credit goes to github.com

Skip to content
This repository was archived by the owner on Aug 2, 2023. It is now read-only.

Api brotli changes #1621

Merged
merged 26 commits into from
Jun 27, 2017
Merged

Api brotli changes #1621

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
26 commits
Select commit Hold shift + click to select a range
f2da3ba
CaterburyPerf
Vedin Jun 16, 2017
da92380
flush/compress
Vedin Jun 19, 2017
790191c
huge changes
Vedin Jun 21, 2017
f3b3e58
space
Vedin Jun 21, 2017
c87498f
small
Vedin Jun 21, 2017
e152fd1
flush/compress
Vedin Jun 19, 2017
d06e4b4
huge changes
Vedin Jun 21, 2017
fb317f5
space
Vedin Jun 21, 2017
8fef201
small
Vedin Jun 21, 2017
d31a1c6
Merge branch 'APIBrotliChanges2' of https://github.com/Vedin/corefxla…
Vedin Jun 21, 2017
78a1a75
resolve merge conflicts
Vedin Jun 21, 2017
d9b7587
unsaved chagnes
Vedin Jun 21, 2017
5a6f037
test fix depends on APIchanges
Vedin Jun 21, 2017
fb51724
resolve issues
Vedin Jun 21, 2017
1972d8b
Less alocation, change State
Vedin Jun 22, 2017
6739e69
issue
Vedin Jun 22, 2017
59f3981
remove unnecessary
Vedin Jun 22, 2017
959a119
resolve
Vedin Jun 22, 2017
7f8b3d4
clean up changes, change state
Vedin Jun 23, 2017
d4ddd1f
merge conflicts
Vedin Jun 23, 2017
38ef5ec
resolve
Vedin Jun 23, 2017
649f42c
guidelines
Vedin Jun 23, 2017
516f4c6
fix bug
Vedin Jun 25, 2017
e1aa814
resolve issues
Vedin Jun 26, 2017
1120e4f
change access
Jun 26, 2017
2f71a9b
small issues
Vedin Jun 26, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
resolve issues
  • Loading branch information
@Vedin
Vedin committed Jun 21, 2017
commit fb51724e23d7c5a8bb1c9196087ad07afc93589d
17 changes: 6 additions & 11 deletions src/System.IO.Compression.Brotli/System/IO/Compression/Brotli.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@ public void InitializeEncoder()

public void SetQuality(uint quality)
{
if (quality < MinQuality || quality > MaxQuality)
if (quality > MaxQuality)
{
throw new ArgumentException(BrotliEx.WrongQuality);
}
Expand All @@ -64,7 +64,7 @@ public void SetQuality()

public void SetWindow(uint window)
{
if (window < MinWindowBits || window > MaxWindowBits)
if (window - MinWindowBits > MaxWindowBits - MinWindowBits)
{
throw new ArgumentException(BrotliEx.WrongWindowSize);
}
Expand Down Expand Up @@ -103,9 +103,9 @@ private static TransformationStatus GetTransformationStatusFromBrotliDecoderResu
return TransformationStatus.InvalidData;
}

public static TransformationStatus FlushEncoder(Span<byte> source, Span<byte> destination, out int bytesConsumed, out int bytesWritten, ref State state, bool is_finished = true)
public static TransformationStatus FlushEncoder(Span<byte> source, Span<byte> destination, out int bytesConsumed, out int bytesWritten, ref State state, bool isFinished = true)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why change the input source to Span instead of ReadOnlySpan?

Copy link
Contributor Author

@Vedin Vedin Jun 26, 2017 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is different methods, but yes their should be ReadOnlySpan also

{
BrotliEncoderOperation operation = is_finished ? BrotliEncoderOperation.Finish : BrotliEncoderOperation.Flush;
BrotliEncoderOperation operation = isFinished ? BrotliEncoderOperation.Finish : BrotliEncoderOperation.Flush;
bytesConsumed = source.Length;
bytesWritten = destination.Length;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These should not be set to source.Length and destination.Length in the beginning.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

really it should be because we send this variables to the native library (where they means array size)
(see written and consumed)

if (state.BrotliNativeState == IntPtr.Zero) return TransformationStatus.InvalidData;
Expand Down Expand Up @@ -171,7 +171,6 @@ public static TransformationStatus Compress(ReadOnlySpan<byte> source, Span<byte
public static TransformationStatus Decompress(ReadOnlySpan<byte> source, Span<byte> destination, out int bytesConsumed, out int bytesWritten, ref State state)
{
bool endOfStream = false;
bool errorDetected = false;
bytesConsumed = source.Length;
bytesWritten = destination.Length;
BrotliDecoderResult LastDecoderResult = BrotliDecoderResult.NeedsMoreInput;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is NeedsMoreInput the starting state of LastDecoderResult?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Because at start we haven't send anything for decompress, so we need more input

Expand Down Expand Up @@ -202,17 +201,13 @@ public static TransformationStatus Decompress(ReadOnlySpan<byte> source, Span<by
endOfStream = true;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this bool necessary? The other two conditions return, so reaching here automatically implies endOfStream is true. No?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

endOfStream was deleted

}

if (endOfStream && !BrotliNative.BrotliDecoderIsFinished(state.BrotliNativeState))
{
errorDetected = true;
}
if (LastDecoderResult == BrotliDecoderResult.Error || errorDetected)
if (LastDecoderResult == BrotliDecoderResult.Error || endOfStream && !BrotliNative.BrotliDecoderIsFinished(state.BrotliNativeState))
{
var error = BrotliNative.BrotliDecoderGetErrorCode(state.BrotliNativeState);
var text = BrotliNative.BrotliDecoderErrorString(error);
throw new System.IO.IOException(text + BrotliEx.unableDecode);
}
if (endOfStream && !BrotliNative.BrotliDecoderIsFinished(state.BrotliNativeState) && LastDecoderResult == BrotliDecoderResult.NeedsMoreInput)
if (LastDecoderResult == BrotliDecoderResult.NeedsMoreInput)
{
throw new System.IO.IOException(BrotliEx.FinishDecompress);
}
Expand Down
23 changes: 14 additions & 9 deletions src/System.IO.Compression.Brotli/System/IO/Compression/BrotliStream.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,10 @@ public partial class BrotliStream : Stream

public BrotliStream(Stream baseStream, CompressionMode mode, bool leaveOpen, int bufferSize, CompressionLevel quality) : this(baseStream, mode, leaveOpen, bufferSize)
{
if (_mode == CompressionMode.Decompress) throw new System.IO.IOException(BrotliEx.QualityAndWinSize);
if (_mode == CompressionMode.Decompress)
{
throw new System.IO.IOException(BrotliEx.QualityAndWinSize);
}
else
{
_state.SetQuality((uint)Brotli.GetQualityFromCompressionLevel(quality));
Expand Down Expand Up @@ -238,7 +241,8 @@ public override int Read(byte[] buffer, int offset, int count)
{
throw new TimeoutException(BrotliEx.TimeoutRead);
}
while (true) {
while (true)
{
if (transformationResult == TransformationStatus.NeedMoreSourceData)
{
_availableInput = _stream.Read(_nextInput, 0, (int)_bufferSize);
Expand All @@ -252,15 +256,16 @@ public override int Read(byte[] buffer, int offset, int count)
{
BufferStream.Write(_bufferOutput, 0, _availableOutput);
bytesRead += _availableOutput;
} else
}
else
{
endOfStream = true;
break;
}
transformationResult = Brotli.Decompress(_nextInput, _bufferOutput, out _availableInput, out _availableOutput, ref _state);
if (bytesRead >= count) break;
}
if (transformationResult==TransformationStatus.Done)
if (transformationResult == TransformationStatus.Done)
{
bytesRead += _availableOutput;
BufferStream.Write(_bufferOutput, 0, _availableOutput);
Expand Down Expand Up @@ -310,18 +315,18 @@ public override void Write(byte[] buffer, int offset, int count)
throw new TimeoutException(BrotliEx.TimeoutWrite);
}
copyLen = bytesRemain > _bufferSize ? _bufferSize : bytesRemain;
byte[] _bufferInput = new byte[copyLen];
Array.Copy(buffer, currentOffset, _bufferInput, 0, copyLen);
byte[] bufferInput = new byte[copyLen];
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is this extra buffer allocation and copy necessary? If you are going to need an extra buffer on each write call, you might want to rent a buffer from a pool, have your own buffer that you re-use, or use a stack buffer (if it isn't too large).

Copy link
Contributor Author

@Vedin Vedin Jun 22, 2017 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not necessary It is temporary. What the problem is that signature of Compress takes out parameter (consumed) and input is exactly as source size. I can change consumed to ref parameter, but it is inconsistent with other Encode/Decode methods, which using Span.

May be I can fix it using Span.Slice and just cut the buffer instead of copy it to _bufferInput?

Copy link
Contributor Author

@Vedin Vedin Jun 22, 2017 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@shiftylogic Is something like this ok?

Span<byte> bufferInput = new Span<byte>(buffer);
bufferInput.Slice(0, copyLen);
transformationResult = Brotli.Compress(bufferInput, _bufferOutput, out _availableInput, out _availableOutput, ref _state);

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is this extra array and copy needed? Why can't you use the input directly for the compress call?

Array.Copy(buffer, currentOffset, bufferInput, 0, copyLen);
bytesRemain -= copyLen;
currentOffset += copyLen;
_availableInput = copyLen;
_availableOutput = _bufferSize;
TransformationStatus transformationResult = TransformationStatus.DestinationTooSmall;
while (transformationResult == TransformationStatus.DestinationTooSmall)
{
transformationResult = Brotli.Compress(_bufferInput, _bufferOutput, out _availableInput, out _availableOutput, ref _state);
if (transformationResult==TransformationStatus.InvalidData) throw new System.IO.IOException(BrotliEx.unableEncode);
if (transformationResult==TransformationStatus.DestinationTooSmall) _stream.Write(_bufferOutput, 0, _availableOutput);
transformationResult = Brotli.Compress(bufferInput, _bufferOutput, out _availableInput, out _availableOutput, ref _state);
if (transformationResult == TransformationStatus.InvalidData) throw new System.IO.IOException(BrotliEx.unableEncode);
if (transformationResult == TransformationStatus.DestinationTooSmall) _stream.Write(_bufferOutput, 0, _availableOutput);
}
}
}
Expand Down