util-logging: Pick up log file size #177

malikvivek · 2016-10-05T21:12:37Z

Problem

FileHandler sets 'bytesWrittenToFile' as 0 in 'openLog()'.
This forces log file size to 0 every-time application is restarted, even if it exists.

Solution

Set 'bytesWrittenToFile' as file size in openStream(), if log file exists.
Else set it to 0. Remove 'bytesWrittenToFile' from 'openLog()'

Result

As a result of this merge, the FileHandler will rotate log files with log policy based on Max file size when the limit is reached and prevent the files from growing too large.

ISSUE: #176

Problem FileHandler sets 'bytesWrittenToFile' as 0 in 'openLog()'. This forces log file size to 0 every-time application is restarted, even if it exists. Solution Set 'bytesWrittenToFile' as file size in openStream(), if log file exists. Else set it to 0. Remove 'bytesWrittenToFile' from 'openLog()' ISSUE: twitter#176

codecov-io · 2016-10-05T21:34:30Z

Current coverage is 66.52% (diff: 100%)

Merging #177 into develop will decrease coverage by 0.01%

@@            develop       #177   diff @@
==========================================
  Files           196        196          
  Lines          8473       8474     +1   
  Methods        7691       7681    -10   
  Messages          0          0          
  Branches        732        743    +11   
==========================================
- Hits           5638       5637     -1   
- Misses         2835       2837     +2   
  Partials          0          0

Powered by Codecov. Last update e684872...0dc9b89

taylorleese · 2016-10-05T21:28:10Z

util-logging/src/test/scala/com/twitter/logging/FileHandlerTest.scala

+      * The test succeeds if the roll over takes place as desired else fails.
+      */
+    withTempFolder {
+      1 to 5 foreach{testRun =>


style: (1 to 5).foreach { testRun =>

taylorleese · 2016-10-05T21:28:40Z

util-logging/src/test/scala/com/twitter/logging/FileHandlerTest.scala

+            }
+          }
+          val files = listLogFiles(folderName+"/LogFileDir")
+          files.foreach{f: File =>


style: files.foreach { f: File =>

Will add white spaces as per style and push again.

taylorleese · 2016-10-05T21:29:02Z

util-logging/src/test/scala/com/twitter/logging/FileHandlerTest.scala

+            formatter = BareFormatter
+          ).apply()
+
+          for (a <- 1 to 10000){


style: for (a <- 1 to 10000) {

taylorleese · 2016-10-05T21:29:15Z

util-logging/src/test/scala/com/twitter/logging/FileHandlerTest.scala

+          val files = listLogFiles(folderName+"/LogFileDir")
+          files.foreach{f: File =>
+            val len = f.length().bytes
+            if (len > fileSizeInMegaBytes.megabytes){


style: if (len > fileSizeInMegaBytes.megabytes) {

taylorleese · 2016-10-05T21:51:44Z

util-logging/src/main/scala/com/twitter/logging/FileHandler.scala

+    val file = new File(filename)
+    val dir = file.getParentFile
    if ((dir ne null) && !dir.exists) dir.mkdirs
+    bytesWrittenToFile = if (file.exists()) file.length() else 0


It's a bit awkward this is setting bytesWrittenToFile outside an immediate synchronized block which seems like it would be safer. It does appear it's wrapped in a synchronized block by callers though.

The bytesWrittenToFile is set by openStream() which is called inside a synchronized block under openLog(). That preserves it's synchronized lock.

The purpose of removing it from openLog and adding it to openStream is to make sure that as and when the log file opened up or created, the file size is set at that moment.

Yep, I understand both of those points and I still think it's awkward. I'll drop this if nobody else has concerns though.

ISSUE: twitter#176

taylorleese · 2016-10-06T00:09:19Z

util-logging/src/test/scala/com/twitter/logging/FileHandlerTest.scala

+      * The test succeeds if the roll over takes place as desired else fails.
+      */
+    withTempFolder {
+      (1 to 5) foreach { testRun =>


style: (1 to 5) foreach => (1 to 5).foreach

Sorry! My bad. Pushing the update.

taylorleese · 2016-10-06T00:11:16Z

util-logging/src/test/scala/com/twitter/logging/FileHandlerTest.scala

+          val files = listLogFiles(folderName+"/LogFileDir")
+          files.foreach { f: File =>
+            val len = f.length().bytes
+            if (len > fileSizeInMegaBytes.megabytes) {


Is this comparing bytes to megabytes?

No it is not. It is comparing bytes (return type of Storage unit). fileSizeinMegaBytes is a Long type and to convert it into storage type, .megabytes is added. This just multiplies the constant Long and gives the storage size and type in bytes equivalent to megabytes

Making len = f.lenth().megabytes will not set as current size during execution in megabytes but will multiply it with (1024 * 1024), thereby setting the file size as very large which will cause the test to fail and is not the right functionality.

println(s"File size is ${len} and the set limit is ${fileSizeInMegaBytes.megabytes}")

Adding this print statement in test, just before the 'if' clause will tell us that it compares the byte sizes each time.

The output that it generates:

Testing started at 9:07 AM ... File size is 300000.bytes and the set limit is 1048576.bytes File size is 600000.bytes and the set limit is 1048576.bytes File size is 900000.bytes and the set limit is 1048576.bytes File size is 1048560.bytes and the set limit is 1048576.bytes File size is 151440.bytes and the set limit is 1048576.bytes File size is 1048560.bytes and the set limit is 1048576.bytes File size is 451440.bytes and the set limit is 1048576.bytes Process finished with exit code 0

Ah, didn't realize this was comparing StorageUnit types at first glance. Thx.

mosesn · 2016-10-06T22:57:15Z

util-logging/src/test/scala/com/twitter/logging/FileHandlerTest.scala

+          val fileSizeInMegaBytes: Long = 1
+          val record1 = new javalog.LogRecord(logLevel, "Sending bytes to fill up file")
+          val handler = FileHandler(
+            filename = folderName+"/LogFileDir/testFileSize.log",


nit: s/folderName+"/LogFileDir/testFileSize.log"/folderName + "/LogFileDir/testFileSize.log"

mosesn · 2016-10-06T22:58:42Z

util-logging/src/test/scala/com/twitter/logging/FileHandlerTest.scala

+      */
+    withTempFolder {
+      (1 to 5).foreach { testRun =>
+        s" should succeed to rollover at set limit when test is restarted, execution ${testRun}" in {


should is implicit here.

s/ should succeed to rollover/rollover

mosesn · 2016-10-06T22:59:12Z

util-logging/src/test/scala/com/twitter/logging/FileHandlerTest.scala

+    /**
+      * Execute this test individually to affirm that the roll policy based upon the size works
+      * even after the test case is executed couple of times.
+      * This test mimics the scenario that, if log file exists, FileHandler should pick up it's size


I don't love this testing style. how about instead having two Filehandlers in one test?

Well, I initially thought of executing this test individually. I think I forgot to remove the comments.
It'll not be executed in that manner and WILL be executed every-time, just like other tests. This is definitely NOT a test that has to be executed individually, but every-time.

@malikvivek that's not the part of it that bothers me–it's that we execute the test multiple times so that subsequent runs of the test will be affected by previous runs. we should instead have all of our tests be isolated from one another.

Oh! Okay. Got you. I've made the changes and instead of (1 to 5).foreach, using two handlers. I'll make the code more Scalastyle and push changes asap. Thanks

malikvivek · 2016-10-12T18:08:39Z

Can I get an update on this please?

mosesn · 2016-10-12T18:55:20Z

LGTM, I'll work on getting these merged in.

malikvivek · 2016-10-12T18:56:40Z

Thanks a lot! Appreciate it!

mosesn · 2016-10-13T22:14:56Z

This was merged internally, thanks for the PR!

ryanoneill · 2016-10-17T20:50:21Z

Hi @malikvivek, thank you for your contribution. This has been pushed back out to GitHub in bdc7809

Going to now close this pull request.

taylorleese reviewed Oct 5, 2016

View reviewed changes

Rectify Scala Style in FileHandlerTest

6e0ebb1

ISSUE: twitter#176

taylorleese reviewed Oct 6, 2016

View reviewed changes

Vivek Malik added 2 commits October 6, 2016 10:16

Rectify Scala style in FileHandlerTest, patch 2.

b01f676

Set len as bytes Storage type.

23365b5

mosesn suggested changes Oct 6, 2016

View reviewed changes

Vivek Malik added 2 commits October 10, 2016 09:57

Remove test iteration. Add multiple file handlers

f1958a3

Remove test iteration. Add multiple file handlers, patch 2

0dc9b89

mosesn approved these changes Oct 12, 2016

View reviewed changes

mosesn added the Ship It label Oct 12, 2016

mosesn added the Merged Internally label Oct 13, 2016

ryanoneill closed this Oct 17, 2016

util-logging: Pick up log file size #177

util-logging: Pick up log file size #177

Uh oh!

Conversation

malikvivek commented Oct 5, 2016

Uh oh!

codecov-io commented Oct 5, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Current coverage is 66.52% (diff: 100%)

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

malikvivek commented Oct 12, 2016

Uh oh!

mosesn commented Oct 12, 2016

Uh oh!

malikvivek commented Oct 12, 2016

Uh oh!

mosesn commented Oct 13, 2016

Uh oh!

ryanoneill commented Oct 17, 2016

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

5 participants

codecov-io commented Oct 5, 2016 •

edited

Loading