CPL-6984: Dockerized compilations #38

OliLay · 2022-07-18T13:14:17Z

This PR adds support for remote compilations inside a docker container specified by the client.
E2E tests for the docker compilation are executed by the CI only in the main branch due to longer runtime. (similar to schroot)

Instructions on how to use the feature can be found in the README.

Furthermore, the .pylintrc file is updated according to the latest available version from Google, and E2E tests were refactored a bit to reduce code duplication that would else have been added by introducing this feature.
During testing I noticed that reading the server logs is quite hard for clients simultaneously compiling. Therefore, I changed the logging on the server slightly (1-liner) so that one can match every log line to a certain client by including the thread name, which is different for every thread (and therefore client) that the server starts. This way we can analyze logs much better, especially for simultaneous requests.

spirsch

Mostly minor things, so already approving. 👍

spirsch · 2022-07-20T06:58:52Z

README.md

    compiler=g++
    timeout=60
    compression=lzo
    profile=jammy


Suggested change

profile=jammy

schroot_profile=jammy

spirsch · 2022-07-20T09:56:56Z

README.md

  $ sudo schroot -c jammy -- apt -y install build-essential
  ```
- Execute *schrooted* compilations by specifying `--profile=jammy` via the CLI or in the `client.conf` file
+- Execute *schrooted* compilations by specifying `--schroot-profile=jammy` via the CLI or in the `client.conf` file


A change I apparently missed in my PR a while ago:

Suggested change

- Execute *schrooted* compilations by specifying `--schroot-profile=jammy` via the CLI or in the `client.conf` file

- Execute *schrooted* compilations by specifying `--schroot-profile=jammy` via the CLI or in the `homcc.conf` file

spirsch · 2022-07-20T10:10:07Z

homcc/common/logging.py

+            return f"""[{level_format}%(levelname)s{self.RESET}] %(asctime)s - %(threadName)s
+             - %(pathname)s:%(lineno)d:\n%(message)s"""


Good idea to add thread information as well! 👍

spirsch · 2022-07-20T10:21:25Z

homcc/server/main.py

+    # schroot profiles
+    schroot_profiles: List[str] = load_schroot_profiles()


Just a mention that I created CPL-7103 ticket to improve our current schroot profile loading (and parsing) in the future.

Good idea, thanks for creating the ticket

spirsch · 2022-07-20T10:31:47Z

homcc/server/server.py

+    def check_schroot_profile_argument(self, schroot_profile: Optional[str]) -> bool:
+        """Checks whether the specified schroot profile requested by the client can be used.
+        It can not be used if the schroot profile with the given name is not set up on the server."""
+        if schroot_profile is not None:


Suggested change

if schroot_profile is not None:

if schroot_profile is None:

return False

to reduce indention level like you did in check_docker_container_argument.

spirsch · 2022-07-20T10:32:20Z

homcc/server/server.py

+
+        return True
+
+    def check_docker_container_argument(self, docker_container: Optional[str]):


Suggested change

def check_docker_container_argument(self, docker_container: Optional[str]):

def check_docker_container_argument(self, docker_container: Optional[str]) -> bool:

spirsch · 2022-07-20T10:37:19Z

tests/conftest.py


 def pytest_addoption(parser: pytest.Parser):
    parser.addoption(
        "--runschroot", action="store", type=str, metavar="PROFILE", help="run e2e schroot tests with specified PROFILE"


Only for internal consistency:

Suggested change

"--runschroot", action="store", type=str, metavar="PROFILE", help="run e2e schroot tests with specified PROFILE"

"--runschroot", action="store", type=str, metavar="SCHROOT_PROFILE", help="run e2e schroot tests with specified PROFILE"

and in the marker documentation above and usage below (reason="specify --runschroot=SCHROOT_PROFILE)

spirsch · 2022-07-20T10:47:16Z

tests/server/docker_test.py

+    class TestIsValidContainer:
+        """Tests the method that checks if a docker container is a valid (existing and running) container."""
+


Suggested change

class TestIsValidContainer:

"""Tests the method that checks if a docker container is a valid (existing and running) container."""

can be removed.

spirsch · 2022-07-20T10:55:25Z

tests/e2e/e2e_test.py

+                "--no-schroot-profile" if self.schroot_profile is None else f"--schroot-profile={self.schroot_profile}",
+                "--no-docker-container"
+                if self.docker_container is None
+                else f"--docker-container={self.docker_container}",


We should probably extract these lines (like compression_arg above) to improve visual clarity.

Good hint, this makes things more readable

spirsch · 2022-07-20T11:05:01Z

tests/e2e/e2e_test.py

    def cpp_end_to_end_no_linking(
        self,
-        compiler: str,
-        unused_tcp_port: int,
-        compression: Compression = NoCompression(),
-        profile: Optional[str] = None,
+        basic_arguments: BasicClientArguments,
    ):


Minor formatting to a single line here (and for cpp_end_to_end_preprocessor_side_effects).

def cpp_end_to_end_no_linking(self, basic_arguments: BasicClientArguments):

OliLay added 20 commits July 12, 2022 16:52

initial docker implementation

be011af

Merge branch 'main' into CPL-6984-Dockerized-compilations

6971897

linting, updated .pylintrc, tests

9fea260

refactored e2e test arguments to cope with docker_container arg

afa3dff

Merge branch 'main' into CPL-6984-Dockerized-compilations

7a04bc3

docker unit & e2e test

e835909

separated CI steps

b669068

linting

e33eca0

remove -it flag for CI docker exec

9afeaa0

detach docker run on CI

a1e9ef0

try -dit flag

751ced7

use sudo for apt inside container

9b66c25

try without sudo again

18032a4

use gcc as docker image

8bc113e

add --verbose to e2e client

ab6b006

print stdout upon error, use -fPIC for docker e2e

acddb60

linting and wrong profile arg

70dbce5

fix docu mistakes

6a1f295

more tests, README

26a4174

normalize CI again

d95be92

OliLay marked this pull request as ready for review July 19, 2022 13:54

OliLay requested review from spirsch and wmann-celonis July 19, 2022 13:54

OliLay added 3 commits July 19, 2022 16:39

minor change to README

60ddf83

log thread name on server side (so log lines can be matched to a client)

2843b4e

fixed pylint

9f57147

wmann-celonis approved these changes Jul 20, 2022

View reviewed changes

spirsch approved these changes Jul 20, 2022

View reviewed changes

PR feedback

603b88c

OliLay merged commit 00f0dd3 into main Jul 20, 2022

OliLay deleted the CPL-6984-Dockerized-compilations branch July 20, 2022 12:20

	- Execute schrooted compilations by specifying `--schroot-profile=jammy` via the CLI or in the `client.conf` file
	- Execute schrooted compilations by specifying `--schroot-profile=jammy` via the CLI or in the `homcc.conf` file

		return f"""[{level_format}%(levelname)s{self.RESET}] %(asctime)s - %(threadName)s
		- %(pathname)s:%(lineno)d:\n%(message)s"""

		# schroot profiles
		schroot_profiles: List[str] = load_schroot_profiles()

	if schroot_profile is not None:
	if schroot_profile is None:
	return False


		return True

		def check_docker_container_argument(self, docker_container: Optional[str]):

	"--runschroot", action="store", type=str, metavar="PROFILE", help="run e2e schroot tests with specified PROFILE"
	"--runschroot", action="store", type=str, metavar="SCHROOT_PROFILE", help="run e2e schroot tests with specified PROFILE"

		class TestIsValidContainer:
		"""Tests the method that checks if a docker container is a valid (existing and running) container."""

CPL-6984: Dockerized compilations #38

CPL-6984: Dockerized compilations #38

Uh oh!

Conversation

OliLay commented Jul 18, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

spirsch left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

4 participants

OliLay commented Jul 18, 2022 •

edited

Loading