Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Commit 1d74db7

Browse files
bd1976brisbd1976bris
authored
[DTLTO] Make temporary file handling consistent (#176807)
DTLTO emits temporary files to allow distribution of archive member inputs. It also emits temporary files from the ThinLTO backend, such as the index files needed for each distributed ThinLTO backend compilation. This change brings archive member temporary files into line with those produced by the ThinLTO backend. They are now emitted in the same location, warnings are emitted if they cannot be deleted, and they are cleaned up on abnormal exit (e.g. Ctrl-C). All temporary files are preserved when --save-temps is specified. The existing signal-handling test has been extended to cover the full set of DTLTO temporary files, and a new test has been added to exercise temporary file handling in normal operation. Additionally, a minimal test has been added to show the COFF behaviour. SIE Internal tracker: TOOLCHAIN-21022
1 parent 29fd868 commit 1d74db7

10 files changed

Lines changed: 365 additions & 68 deletions

File tree

‎cross-project-tests/dtlto/link-savetemps.test‎

Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,44 @@
1+
REQUIRES: lld-link
2+
3+
# Test that DTLTO temporary files are "best-effort" cleaned up unless
4+
# --save-temps is specified. We use archives in this test as the handling for
5+
# archives requires a superset of the temporary files used for object inputs.
6+
7+
RUN: rm -rf %t && split-file %s %t && cd %t
8+
9+
RUN: %clang --target=x86_64-pc-windows-msvc -O2 t1.c -flto=thin -c
10+
11+
RUN: lld-link /lib /out:t.lib t1.o
12+
13+
DEFINE: %{tdir} = dummy-to-make-lit-work
14+
DEFINE: %{dtlto} = mkdir %{tdir} && \
15+
DEFINE: lld-link /subsystem:console /machine:x64 /out:%{tdir}/my.exe \
16+
DEFINE: /wholearchive:t.lib \
17+
DEFINE: -thinlto-distributor:%python \
18+
DEFINE: -thinlto-distributor-arg:%llvm_src_root/utils/dtlto/local.py \
19+
DEFINE: -thinlto-remote-compiler:%clang
20+
21+
# Check that temporary files are removed normally.
22+
REDEFINE: %{tdir} = empty
23+
RUN: %{dtlto}
24+
RUN: ls %{tdir} | sort | FileCheck %s --check-prefixes=BOOKEND,ELF
25+
26+
# Check that --save-temps preserves temporary files.
27+
REDEFINE: %{tdir} = savetemps
28+
RUN: %{dtlto} /lldsavetemps
29+
RUN: ls %{tdir} | sort | FileCheck %s --check-prefixes=BOOKEND,TEMPS,ELF,OTHER
30+
31+
# No files are expected before.
32+
BOOKEND-NOT: {{.}}
33+
TEMPS: {{^}}my.[[#PID:]].dist-file.json{{$}}
34+
ELF: {{^}}my.exe{{$}}
35+
OTHER: {{^}}my.exe.resolution.txt{{$}}
36+
# Filename composition: <archive><member><member offset>.<task>.<pid>.<task>.<pid>.native.o.
37+
TEMPS: {{^}}t.libt1.o[[#T1_OFFSET:]].1.[[#%X,HEXPID:]].1.[[#PID]].native.o{{$}}
38+
TEMPS: {{^}}t.libt1.o[[#T1_OFFSET]].1.[[#%X,HEXPID]].1.[[#PID]].native.o.thinlto.bc{{$}}
39+
TEMPS: {{^}}t.libt1.o[[#T1_OFFSET]].1.[[#%X,HEXPID]].o{{$}}
40+
# No files are expected after.
41+
BOOKEND-NOT: {{.}}
42+
43+
#--- t1.c
44+
__attribute__((retain)) int mainCRTStartup() { return 0; }

‎cross-project-tests/dtlto/local_codegen_and_wait.py‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
"""
2+
This simple distributor performs code generation locally, creates the
3+
"send-signal1" file, and then waits for the "send-signal2" file to appear
4+
before exiting. It is intended to be used in tandem with test_temps.py.
5+
Please see test_temps.py for more information.
6+
"""
7+
8+
import json, subprocess, sys, time, os, pathlib
9+
10+
# Load the DTLTO information from the input JSON file.
11+
data = json.loads(pathlib.Path(sys.argv[-1]).read_bytes())
12+
13+
# Iterate over the jobs and execute the codegen tool.
14+
for job in data["jobs"]:
15+
subprocess.check_call(data["common"]["args"] + job["args"])
16+
17+
pathlib.Path("send-signal1").touch()
18+
19+
while not os.path.exists("send-signal2"):
20+
time.sleep(0.05)

‎cross-project-tests/dtlto/savetemps-lock.test‎

Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,44 @@
1+
# This test relies on locking files which is difficult to do in a way the keeps
2+
# a test robust on Linux, so it is restricted to Windows.
3+
REQUIRES: ld.lld,system-windows
4+
5+
# Test that a warning is emitted for each DTLTO temporary file that cannot be
6+
# removed. This test uses archives because archive handling exercises a superset
7+
# of the temporary files used for object inputs.
8+
#
9+
# This scenario is logically related to the cases in savetemps.test; however, it
10+
# is placed here to maintain coverage, as this behavior can only be tested
11+
# effectively on Windows.
12+
13+
RUN: rm -rf %t && split-file %s %t && cd %t
14+
15+
RUN: %clang --target=x86_64-linux-gnu -O2 t1.c t2.c -flto=thin -c
16+
17+
RUN: llvm-ar rcs t.a t1.o t2.o
18+
19+
# Check that a warning is reported for each temporary file that cannot be
20+
# removed. Note that the use of the name "locked" for the output directory
21+
# triggers special behaviour in test_temps.py.
22+
RUN: mkdir locked && %python %S/test_temps.py locked lock \
23+
RUN: %clang --target=x86_64-linux-gnu -nostdlib -O2 -flto=thin \
24+
RUN: -fuse-ld=lld -Wl,--whole-archive t.a -o locked/t.elf -shared \
25+
RUN: -fthinlto-distributor=%python \
26+
RUN: -Xthinlto-distributor=%S/local_codegen_and_wait.py 2>&1 \
27+
RUN: | FileCheck %s --implicit-check-not=warning
28+
29+
# Sanity check for the expected test_temps.py behaviour.
30+
CHECK-DAG: Lock any files in the output directory.
31+
CHECK-DAG: warning: could not remove the file 'locked{{/|\\}}t.[[#PID:]].dist-file.json': {{.*}}
32+
# Filename composition: <archive>(<member> at <offset>).<task>.<pid>.<task>.<pid>.native.o.
33+
CHECK-DAG: warning: could not remove the file 'locked{{/|\\}}t.a(t1.o at [[#T1_OFFSET:]]).1.[[#%X,HEXPID:]].1.[[#PID]].native.o': {{.*}}
34+
CHECK-DAG: warning: could not remove the file 'locked{{/|\\}}t.a(t1.o at [[#T1_OFFSET]]).1.[[#%X,HEXPID]].1.[[#PID]].native.o.thinlto.bc': {{.*}}
35+
CHECK-DAG: warning: could not remove the file 'locked{{/|\\}}t.a(t2.o at [[#T2_OFFSET:]]).2.[[#%X,HEXPID]].2.[[#PID]].native.o': {{.*}}
36+
CHECK-DAG: warning: could not remove the file 'locked{{/|\\}}t.a(t2.o at [[#T2_OFFSET]]).2.[[#%X,HEXPID]].2.[[#PID]].native.o.thinlto.bc': {{.*}}
37+
CHECK-DAG: warning: could not remove temporary DTLTO input file 'locked{{/|\\}}t.a(t1.o at [[#T1_OFFSET]]).1.[[#%X,HEXPID]].o': {{.*}}
38+
CHECK-DAG: warning: could not remove temporary DTLTO input file 'locked{{/|\\}}t.a(t2.o at [[#T2_OFFSET]]).2.[[#%X,HEXPID]].o': {{.*}}
39+
40+
#--- t1.c
41+
__attribute__((retain)) int t1(int x) { return x; }
42+
43+
#--- t2.c
44+
__attribute__((retain)) int t2(int x) { return x; }

‎cross-project-tests/dtlto/savetemps.test‎

Lines changed: 70 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,70 @@
1+
REQUIRES: ld.lld
2+
3+
# Test that DTLTO temporary files are "best-effort" cleaned up unless
4+
# --save-temps is specified. We use archives in this test as the handling for
5+
# archives requires a superset of the temporary files used for object inputs.
6+
7+
RUN: rm -rf %t && split-file %s %t && cd %t
8+
9+
RUN: %clang --target=x86_64-linux-gnu -O2 t1.c t2.c -flto=thin -c
10+
11+
RUN: llvm-ar rcs t.a t1.o t2.o
12+
13+
DEFINE: %{tdir} = dummy-to-make-lit-work
14+
DEFINE: %{action} = dummy-to-make-lit-work
15+
DEFINE: %{test-temps-dtlto} = \
16+
DEFINE: mkdir %{tdir} && %python %S/test_temps.py %{tdir} %{action} \
17+
DEFINE: %clang --target=x86_64-linux-gnu -nostdlib -O2 -flto=thin \
18+
DEFINE: -fuse-ld=lld -Wl,--whole-archive t.a -o %{tdir}/t.elf -shared \
19+
DEFINE: -fthinlto-distributor=%python \
20+
DEFINE: -Xthinlto-distributor=%S/local_codegen_and_wait.py
21+
22+
# Check that all temporary files are removed in normal operation.
23+
REDEFINE: %{tdir} = empty
24+
REDEFINE: %{action} = none
25+
RUN: %{test-temps-dtlto}
26+
RUN: ls %{tdir} | sort | FileCheck %s --check-prefixes=BOOKEND,ELF
27+
28+
# Check that --save-temps preserves temporary files.
29+
REDEFINE: %{tdir} = savetemps
30+
REDEFINE: %{action} = none
31+
RUN: %{test-temps-dtlto} -Wl,--save-temps
32+
RUN: ls %{tdir} | sort | FileCheck %s --check-prefixes=BOOKEND,TEMPS,INDEX,ELF
33+
34+
# Check that --thinlto-emit-index-files preserves the index files.
35+
REDEFINE: %{tdir} = index
36+
REDEFINE: %{action} = none
37+
RUN: %{test-temps-dtlto} -Wl,--thinlto-emit-index-files
38+
RUN: ls %{tdir} | sort | FileCheck %s --check-prefixes=BOOKEND,INDEX,ELF
39+
40+
# No files are expected before.
41+
BOOKEND-NOT: {{.}}
42+
TEMPS: {{^}}t.[[#PID:]].dist-file.json{{$}}
43+
# Filename composition: <archive>(<member> at <offset>).<task>.<pid>.<task>.<pid>.native.o.
44+
TEMPS: {{^}}t.a(t1.o at [[#T1_OFFSET:]]).1.[[#%X,HEXPID:]].1.[[#PID]].native.o{{$}}
45+
INDEX: {{^}}t.a(t1.o at [[#T1_OFFSET:]]).1.[[#%X,HEXPID:]].1.[[#PID:]].native.o.thinlto.bc{{$}}
46+
TEMPS: {{^}}t.a(t1.o at [[#T1_OFFSET]]).1.[[#%X,HEXPID]].o{{$}}
47+
TEMPS: {{^}}t.a(t2.o at [[#T2_OFFSET:]]).2.[[#%X,HEXPID]].2.[[#PID]].native.o{{$}}
48+
INDEX: {{^}}t.a(t2.o at [[#T2_OFFSET:]]).2.[[#%X,HEXPID]].2.[[#PID]].native.o.thinlto.bc{{$}}
49+
TEMPS: {{^}}t.a(t2.o at [[#T2_OFFSET]]).2.[[#%X,HEXPID]].o{{$}}
50+
ELF: {{^}}t.elf{{$}}
51+
TEMPS: {{^}}t.elf.resolution.txt{{$}}
52+
# No files are expected after.
53+
BOOKEND-NOT: {{.}}
54+
55+
# Check that no warnings are produced if temporary files are missing. Note that
56+
# the use of the name "removed" for the output directory triggers special
57+
# behaviour in test_temps.py.
58+
REDEFINE: %{tdir} = removed
59+
REDEFINE: %{action} = remove
60+
RUN: %{test-temps-dtlto} 2>&1 | FileCheck %s --check-prefix=NOWARN --allow-empty
61+
RUN: ls %{tdir} | sort | FileCheck %s --check-prefixes=BOOKEND,ELF
62+
# Sanity check for the expected test_temps.py behaviour.
63+
NOWARN: Remove non-essential files in the output directory.
64+
NOWARN-NOT: warning
65+
66+
#--- t1.c
67+
__attribute__((retain)) int t1(int x) { return x; }
68+
69+
#--- t2.c
70+
__attribute__((retain)) int t2(int x) { return x; }

‎cross-project-tests/dtlto/signal.test‎

Lines changed: 19 additions & 55 deletions
Original file line numberDiff line numberDiff line change
@@ -1,19 +1,22 @@
11
REQUIRES: ld.lld
22

33
# Test that if a link is terminated by a signal (or the equivalent on
4-
# Windows), e.g. CTRL-C, DTLTO temporary files are cleaned up.
4+
# Windows), e.g. CTRL-C, DTLTO temporary files are cleaned up. We use
5+
# archives in this test as the handling for archives requires a superset
6+
# of the temporary files used for object inputs.
57

68
RUN: rm -rf %t && split-file %s %t && cd %t
79

810
RUN: %clang --target=x86_64-linux-gnu -O2 t1.c t2.c -flto=thin -c
911

12+
RUN: llvm-ar rcs t.a t1.o t2.o
13+
1014
DEFINE: %{tdir} = dummy-to-make-lit-work
11-
DEFINE: %{kill-dtlto} = rm -f send-signal && mkdir %{tdir} && \
12-
DEFINE: %python killer.py \
13-
DEFINE: %clang --target=x86_64-linux-gnu -nostdlib -O2 -flto=thin \
14-
DEFINE: -fuse-ld=lld -Wl,--whole-archive t1.o t2.o -o %{tdir}/t.elf -shared \
15-
DEFINE: -fthinlto-distributor=%python \
16-
DEFINE: -Xthinlto-distributor=local_codegen_and_wait.py
15+
DEFINE: %{kill-dtlto} = mkdir %{tdir} && %python %S/test_temps.py %{tdir} kill \
16+
DEFINE: %clang --target=x86_64-linux-gnu -nostdlib -O2 -flto=thin \
17+
DEFINE: -fuse-ld=lld -Wl,--whole-archive t.a -o %{tdir}/t.elf -shared \
18+
DEFINE: -fthinlto-distributor=%python \
19+
DEFINE: -Xthinlto-distributor=%S/local_codegen_and_wait.py
1720

1821
# Check that all temporary files are removed if the process is aborted.
1922
REDEFINE: %{tdir} = empty
@@ -25,7 +28,7 @@ EMPTY-NOT: {{.}}
2528
# Check that --save-temps preserves temporary files if the process is aborted.
2629
REDEFINE: %{tdir} = savetemps
2730
RUN: %{kill-dtlto} -Wl,--save-temps
28-
RUN: ls %{tdir} | sort | FileCheck %s --check-prefixes=BOOKEND,TEMPS,INDEX
31+
RUN: ls %{tdir} | sort | FileCheck %s --check-prefixes=BOOKEND,TEMPS,INDEX,OTHER
2932

3033
# Check that --thinlto-emit-index-files preserves the index files if the process
3134
# is aborted.
@@ -36,11 +39,14 @@ RUN: ls %{tdir} | sort | FileCheck %s --check-prefixes=BOOKEND,INDEX
3639
# No files are expected before.
3740
BOOKEND-NOT: {{.}}
3841
TEMPS: {{^}}t.[[#PID:]].dist-file.json{{$}}
39-
TEMPS: {{^}}t.elf.{{.+$}}
40-
TEMPS: {{^}}t1.1.[[#PID]].native.o{{$}}
41-
INDEX: {{^}}t1.1.[[#PID:]].native.o.thinlto.bc{{$}}
42-
TEMPS: {{^}}t2.2.[[#PID]].native.o{{$}}
43-
INDEX: {{^}}t2.2.[[#PID]].native.o.thinlto.bc{{$}}
42+
# Filename composition: <archive>(<member> at <offset>).<task>.<pid>.<task>.<pid>.native.o.
43+
TEMPS: {{^}}t.a(t1.o at [[#T1_OFFSET:]]).1.[[#%X,HEXPID:]].1.[[#PID]].native.o{{$}}
44+
INDEX: {{^}}t.a(t1.o at [[#T1_OFFSET:]]).1.[[#%X,HEXPID:]].1.[[#PID:]].native.o.thinlto.bc{{$}}
45+
TEMPS: {{^}}t.a(t1.o at [[#T1_OFFSET]]).1.[[#%X,HEXPID]].o{{$}}
46+
TEMPS: {{^}}t.a(t2.o at [[#T2_OFFSET:]]).2.[[#%X,HEXPID]].2.[[#PID]].native.o{{$}}
47+
INDEX: {{^}}t.a(t2.o at [[#T2_OFFSET:]]).2.[[#%X,HEXPID]].2.[[#PID]].native.o.thinlto.bc{{$}}
48+
TEMPS: {{^}}t.a(t2.o at [[#T2_OFFSET]]).2.[[#%X,HEXPID]].o{{$}}
49+
OTHER: {{^}}t.elf.resolution.txt{{$}}
4450
# No files are expected after.
4551
BOOKEND-NOT: {{.}}
4652

@@ -49,45 +55,3 @@ __attribute__((retain)) int t1(int x) { return x; }
4955

5056
#--- t2.c
5157
__attribute__((retain)) int t2(int x) { return x; }
52-
53-
#--- local_codegen_and_wait.py
54-
"""Perform codegen locally, create "send-signal" file and wait."""
55-
from pathlib import Path
56-
import json, subprocess, sys, time
57-
58-
# Load the DTLTO information from the input JSON file.
59-
data = json.loads(Path(sys.argv[-1]).read_bytes())
60-
61-
# Iterate over the jobs and execute the codegen tool.
62-
for job in data["jobs"]:
63-
subprocess.check_call(data["common"]["args"] + job["args"])
64-
Path("send-signal").touch()
65-
while True:
66-
time.sleep(1)
67-
68-
#--- killer.py
69-
"""Run command, wait for "send-signal" file to exist, and then send a
70-
termination signal."""
71-
import os, sys, time, signal, subprocess
72-
73-
if os.name == "nt":
74-
# CREATE_NEW_PROCESS_GROUP is used so that p.send_signal(CTRL_BREAK_EVENT)
75-
# does not get sent to the LIT processes that are running the test.
76-
kwargs = {"creationflags": subprocess.CREATE_NEW_PROCESS_GROUP}
77-
else:
78-
# Makes the child a process-group leader so os.killpg(p.pid, SIGINT) works.
79-
kwargs = {"start_new_session": True}
80-
81-
p = subprocess.Popen(sys.argv[1:], **kwargs)
82-
83-
while not os.path.exists("send-signal"):
84-
time.sleep(0.05)
85-
86-
if os.name == "nt":
87-
# Note that CTRL_C_EVENT does not appear to work for clang.
88-
p.send_signal(signal.CTRL_BREAK_EVENT)
89-
else:
90-
os.killpg(p.pid, signal.SIGINT)
91-
p.wait()
92-
93-
sys.exit(0 if p.returncode != 0 else 1)

0 commit comments

Comments
 (0)