Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Commit 6046635

Browse files
ksy36ksy36
authored
Fixes mozilla#163 - Add a job for ML classification of broken site reports using bugbug (mozilla#164)
1 parent 75c96b1 commit 6046635

14 files changed

Lines changed: 450 additions & 0 deletions

File tree

‎.circleci/config.yml‎

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -77,6 +77,22 @@ jobs:
7777
name: Build Docker image
7878
command: docker build -t app:build jobs/bq2sftp/
7979

80+
build-job-broken-site-report-ml:
81+
docker:
82+
- image: << pipeline.parameters.git-image >>
83+
steps:
84+
- checkout
85+
- compare-branch:
86+
pattern: ^jobs/broken-site-report-ml/
87+
- setup_remote_docker:
88+
version: << pipeline.parameters.docker-version >>
89+
- run:
90+
name: Build Docker image
91+
command: docker build -t app:build jobs/broken-site-report-ml/
92+
- run:
93+
name: Test Code
94+
command: docker run app:build pytest --flake8 --black
95+
8096
build-job-client-regeneration:
8197
docker:
8298
- image: << pipeline.parameters.git-image >>
@@ -332,6 +348,20 @@ workflows:
332348
only: main
333349

334350

351+
job-broken-site-report-ml:
352+
jobs:
353+
- build-job-broken-site-report-ml
354+
- gcp-gcr/build-and-push-image:
355+
context: data-eng-airflow-gcr
356+
docker-context: jobs/broken-site-report-ml/
357+
path: jobs/broken-site-report-ml/
358+
image: broken-site-report-ml_docker_etl
359+
requires:
360+
- build-job-broken-site-report-ml
361+
filters:
362+
branches:
363+
only: main
364+
335365
job-client-regeneration:
336366
jobs:
337367
- build-job-client-regeneration

‎jobs/broken-site-report-ml/.dockerignore‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
.ci_job.yaml
2+
.ci_workflow.yaml
3+
.DS_Store
4+
*.pyc
5+
.pytest_cache/
6+
__pycache__/
7+
venv/

‎jobs/broken-site-report-ml/.flake8‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
[flake8]
2+
max-line-length = 88

‎jobs/broken-site-report-ml/.gitignore‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
.DS_Store
2+
*.pyc
3+
__pycache__/
4+
venv/

‎jobs/broken-site-report-ml/Dockerfile‎

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
FROM python:3.8
2+
MAINTAINER REPLACE ME <[email protected]>
3+
4+
# https://github.com/mozilla-services/Dockerflow/blob/master/docs/building-container.md
5+
ARG USER_ID="10001"
6+
ARG GROUP_ID="app"
7+
ARG HOME="/app"
8+
9+
ENV HOME=${HOME}
10+
RUN groupadd --gid ${USER_ID} ${GROUP_ID} && \
11+
useradd --create-home --uid ${USER_ID} --gid ${GROUP_ID} --home-dir ${HOME} ${GROUP_ID}
12+
13+
WORKDIR ${HOME}
14+
15+
RUN pip install --upgrade pip
16+
17+
COPY requirements.txt requirements.txt
18+
RUN pip install -r requirements.txt
19+
20+
COPY . .
21+
22+
RUN pip install .
23+
24+
# Drop root and change ownership of the application folder to the user
25+
RUN chown -R ${USER_ID}:${GROUP_ID} ${HOME}
26+
USER ${USER_ID}

‎jobs/broken-site-report-ml/README.md‎

Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,45 @@
1+
# Python Template Job
2+
3+
This is an example of a dockerized Python job.
4+
5+
## Usage
6+
7+
This script is intended to be run in a docker container.
8+
Build the docker image with:
9+
10+
```sh
11+
docker build -t python-template-job .
12+
```
13+
14+
To run locally, install dependencies with:
15+
16+
```sh
17+
pip install -r requirements.txt
18+
```
19+
20+
Run the script with
21+
22+
```sh
23+
python3 -m python_template_job.main
24+
```
25+
26+
## Development
27+
28+
Run tests with:
29+
30+
```sh
31+
pytest
32+
```
33+
34+
`flake8` and `black` are included for code linting and formatting:
35+
36+
```sh
37+
pytest --black --flake8
38+
```
39+
40+
or
41+
42+
```sh
43+
flake8 python_template_job/ tests/
44+
black --diff python_template_job/ tests/
45+
```

‎jobs/broken-site-report-ml/broken_site_report_ml/main.py‎

Lines changed: 202 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,202 @@
1+
import click
2+
import datetime
3+
import logging
4+
import requests
5+
import time
6+
7+
from google.cloud import bigquery
8+
9+
BUGBUG_HTTP_SERVER = "https://bugbug.herokuapp.com"
10+
CLASSIFICATION_LABELS = {0: "valid", 1: "invalid"}
11+
12+
13+
def classification_http_request(url, reports):
14+
reports_list = list(reports.values())
15+
response = requests.post(
16+
url, headers={"X-Api-Key": "docker-etl"}, json={"reports": reports_list}
17+
)
18+
19+
response.raise_for_status()
20+
21+
return response.json()
22+
23+
24+
def get_reports_classification(model, reports, retry_count=21, retry_sleep=10):
25+
"""Get the classification for a list of reports.
26+
27+
Args:
28+
model: The model to use for the classification.
29+
reports: The dict containing reports to classify with uuid used as keys.
30+
retry_count: The number of times to retry the request.
31+
retry_sleep: The number of seconds to sleep between retries.
32+
33+
Returns:
34+
A dictionary with the uuids as keys and classification results as values.
35+
"""
36+
if len(reports) == 0:
37+
return {}
38+
39+
url = f"{BUGBUG_HTTP_SERVER}/{model}/predict/broken_site_report/batch"
40+
41+
json_response = {}
42+
43+
for _ in range(retry_count):
44+
response = classification_http_request(url, reports)
45+
46+
# Check which reports are ready
47+
for uuid, data in response["reports"].items():
48+
if not data.get("ready", True):
49+
continue
50+
51+
# The report is ready, add it to the json_response and pop it
52+
# up from the current batch
53+
reports.pop(uuid, None)
54+
json_response[uuid] = data
55+
56+
if len(reports) == 0:
57+
break
58+
else:
59+
time.sleep(retry_sleep)
60+
61+
else:
62+
total_sleep = retry_count * retry_sleep
63+
msg = f"Couldn't get {len(reports)} report classifications in {total_sleep} seconds, aborting" # noqa
64+
logging.error(msg)
65+
raise Exception(msg)
66+
67+
return json_response
68+
69+
70+
def add_classification_results(client, bq_dataset_id, results):
71+
res = []
72+
for uuid, result in results.items():
73+
bq_result = {
74+
"report_uuid": uuid,
75+
"label": CLASSIFICATION_LABELS[result["class"]],
76+
"created_at": datetime.datetime.utcnow().isoformat(),
77+
"probability": result["prob"][result["class"]],
78+
"is_ml": True,
79+
}
80+
res.append(bq_result)
81+
82+
job_config = bigquery.LoadJobConfig(
83+
source_format=bigquery.SourceFormat.NEWLINE_DELIMITED_JSON,
84+
schema=[
85+
bigquery.SchemaField("report_uuid", "STRING", mode="REQUIRED"),
86+
bigquery.SchemaField("label", "STRING", mode="REQUIRED"),
87+
bigquery.SchemaField("created_at", "DATETIME", mode="REQUIRED"),
88+
bigquery.SchemaField("probability", "FLOAT"),
89+
bigquery.SchemaField("is_ml", "BOOLEAN", mode="REQUIRED"),
90+
],
91+
write_disposition="WRITE_APPEND",
92+
)
93+
94+
labels_table = f"{bq_dataset_id}.labels"
95+
96+
job = client.load_table_from_json(
97+
res,
98+
labels_table,
99+
job_config=job_config,
100+
)
101+
102+
logging.info("Writing to `labels` table")
103+
104+
try:
105+
job.result()
106+
except Exception as e:
107+
print(f"ERROR: {e}")
108+
if job.errors:
109+
for error in job.errors:
110+
logging.error(error)
111+
112+
table = client.get_table(labels_table)
113+
logging.info(f"Loaded {len(res)} rows into {table}")
114+
115+
116+
def record_classification_run(client, bq_dataset_id, is_ok, count):
117+
rows_to_insert = [
118+
{
119+
"run_at": datetime.datetime.utcnow().isoformat(),
120+
"is_ok": is_ok,
121+
"report_count": count,
122+
},
123+
]
124+
bugbug_runs_table = f"{bq_dataset_id}.bugbug_classification_runs"
125+
errors = client.insert_rows_json(bugbug_runs_table, rows_to_insert)
126+
if errors:
127+
logging.error(errors)
128+
else:
129+
logging.info("Last classification run recorded")
130+
131+
132+
def get_last_classification_datetime(client, bq_dataset_id):
133+
query = f"""
134+
SELECT MAX(run_at) AS last_run_at
135+
FROM `{bq_dataset_id}.bugbug_classification_runs`
136+
WHERE is_ok = TRUE
137+
"""
138+
res = client.query(query).result()
139+
row = list(res)[0]
140+
last_run_time = (
141+
row["last_run_at"] if row["last_run_at"] is not None else "2023-11-20T00:00:00"
142+
)
143+
return last_run_time
144+
145+
146+
def get_reports_since_last_run(client, last_run_time):
147+
query = f"""
148+
SELECT
149+
uuid,
150+
comments as body,
151+
url as title
152+
FROM `moz-fx-data-shared-prod.org_mozilla_broken_site_report.user_reports`
153+
WHERE comments != "" AND reported_at > "{last_run_time}"
154+
ORDER BY reported_at
155+
"""
156+
query_job = client.query(query)
157+
return list(query_job.result())
158+
159+
160+
@click.command()
161+
@click.option("--bq_project_id", help="BigQuery project id", required=True)
162+
@click.option("--bq_dataset_id", help="BigQuery dataset id", required=True)
163+
def main(bq_project_id, bq_dataset_id):
164+
client = bigquery.Client(project=bq_project_id)
165+
166+
# Get datetime of the last classification run
167+
last_run_time = get_last_classification_datetime(client, bq_dataset_id)
168+
169+
# Only get reports that were filed since last classification run
170+
# and have non-empty descriptions
171+
rows = get_reports_since_last_run(client, last_run_time)
172+
173+
if not rows:
174+
logging.info(
175+
f"No new reports with filled descriptions were found since {last_run_time}"
176+
)
177+
return
178+
179+
objects_dict = {
180+
row["uuid"]: {field: value for field, value in row.items()} for row in rows
181+
}
182+
183+
is_ok = True
184+
result_count = 0
185+
try:
186+
logging.info("Getting classification results from bugbug.")
187+
result = get_reports_classification("invalidcompatibilityreport", objects_dict)
188+
if result:
189+
result_count = len(result)
190+
logging.info("Saving classification results to BQ.")
191+
add_classification_results(client, bq_dataset_id, result)
192+
193+
except Exception as e:
194+
logging.error(e)
195+
is_ok = False
196+
197+
record_classification_run(client, bq_dataset_id, is_ok, result_count)
198+
199+
200+
if __name__ == "__main__":
201+
logging.getLogger().setLevel(logging.INFO)
202+
main()

‎jobs/broken-site-report-ml/ci_job.yaml‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
build-job-broken-site-report-ml:
2+
docker:
3+
- image: << pipeline.parameters.git-image >>
4+
steps:
5+
- checkout
6+
- compare-branch:
7+
pattern: ^jobs/broken-site-report-ml/
8+
- setup_remote_docker:
9+
version: << pipeline.parameters.docker-version >>
10+
- run:
11+
name: Build Docker image
12+
command: docker build -t app:build jobs/broken-site-report-ml/
13+
- run:
14+
name: Test Code
15+
command: docker run app:build pytest --flake8 --black

‎jobs/broken-site-report-ml/ci_workflow.yaml‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
job-broken-site-report-ml:
2+
jobs:
3+
- build-job-broken-site-report-ml
4+
- gcp-gcr/build-and-push-image:
5+
context: data-eng-airflow-gcr
6+
docker-context: jobs/broken-site-report-ml/
7+
path: jobs/broken-site-report-ml/
8+
image: broken-site-report-ml_docker_etl
9+
requires:
10+
- build-job-broken-site-report-ml
11+
filters:
12+
branches:
13+
only: main

‎jobs/broken-site-report-ml/pytest.ini‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
[pytest]
2+
testpaths =
3+
tests

0 commit comments

Comments
 (0)