Fix crash caused by race in the async process. (tensorflow#5250)

reedwm · Taylor Robie · commit 5856878daad3 · 2018-09-05T10:47:37.000-07:00
When constructing the evaluation records, data_async_generation.py would copy the records into the final directory. The main process would wait until the eval records existed. However, the main process would sometimes read the eval records before they were fully copied, causing a DataLossError.
diff --git a/official/recommendation/data_async_generation.py b/official/recommendation/data_async_generation.py
@@ -329,8 +329,7 @@ def _construct_eval_record(cache_paths, eval_batch_size):
           items=items[i, :]
       )
       writer.write(batch_bytes)
-  tf.gfile.Copy(intermediate_fpath, dest_fpath)
-  tf.gfile.Remove(intermediate_fpath)
+  tf.gfile.Rename(intermediate_fpath, dest_fpath)
   log_msg("Eval TFRecords file successfully constructed.")
 
 

Original file line number	Diff line number	Diff line change
`@@ -329,8 +329,7 @@ def _construct_eval_record(cache_paths, eval_batch_size):`
`329`	`329`	`items=items[i, :]`
`330`	`330`	`)`
`331`	`331`	`writer.write(batch_bytes)`
`332`		`- tf.gfile.Copy(intermediate_fpath, dest_fpath)`
`333`		`- tf.gfile.Remove(intermediate_fpath)`
	`332`	`+ tf.gfile.Rename(intermediate_fpath, dest_fpath)`
`334`	`333`	`log_msg("Eval TFRecords file successfully constructed.")`
`335`	`334`
`336`	`335`