From c4b4c2a78a9fc0c532c58504e8cb5441224ff1d9 Mon Sep 17 00:00:00 2001 From: Zhiqiang Liu Date: Mon, 13 Apr 2020 13:12:10 +0800 Subject: [PATCH 01/18] buffer: remove useless comment and WB_REASON_FREE_MORE_MEM, reason. free_more_memory func has been completely removed in commit bc48f001de12 ("buffer: eliminate the need to call free_more_memory() in __getblk_slow()") So comment and `WB_REASON_FREE_MORE_MEM` reason about free_more_memory are no longer needed. Fixes: bc48f001de12 ("buffer: eliminate the need to call free_more_memory() in __getblk_slow()") Reviewed-by: Jan Kara Signed-off-by: Zhiqiang Liu Signed-off-by: Jens Axboe --- fs/buffer.c | 2 +- include/linux/backing-dev-defs.h | 1 - include/trace/events/writeback.h | 1 - 3 files changed, 1 insertion(+), 3 deletions(-) diff --git a/fs/buffer.c b/fs/buffer.c index f73276d746bbfc..763de99508deeb 100644 --- a/fs/buffer.c +++ b/fs/buffer.c @@ -967,7 +967,7 @@ grow_dev_page(struct block_device *bdev, sector_t block, struct page *page; struct buffer_head *bh; sector_t end_block; - int ret = 0; /* Will call free_more_memory() */ + int ret = 0; gfp_t gfp_mask; gfp_mask = mapping_gfp_constraint(inode->i_mapping, ~__GFP_FS) | gfp; diff --git a/include/linux/backing-dev-defs.h b/include/linux/backing-dev-defs.h index 4fc87dee005ab1..ee577a83cfe67a 100644 --- a/include/linux/backing-dev-defs.h +++ b/include/linux/backing-dev-defs.h @@ -54,7 +54,6 @@ enum wb_reason { WB_REASON_SYNC, WB_REASON_PERIODIC, WB_REASON_LAPTOP_TIMER, - WB_REASON_FREE_MORE_MEM, WB_REASON_FS_FREE_SPACE, /* * There is no bdi forker thread any more and works are done diff --git a/include/trace/events/writeback.h b/include/trace/events/writeback.h index d94def25e4dc94..85a33bea76f1a7 100644 --- a/include/trace/events/writeback.h +++ b/include/trace/events/writeback.h @@ -36,7 +36,6 @@ EM( WB_REASON_SYNC, "sync") \ EM( WB_REASON_PERIODIC, "periodic") \ EM( WB_REASON_LAPTOP_TIMER, "laptop_timer") \ - EM( WB_REASON_FREE_MORE_MEM, "free_more_memory") \ EM( WB_REASON_FS_FREE_SPACE, "fs_free_space") \ EMe(WB_REASON_FORKER_THREAD, "forker_thread") From 44575a67314b3768d4415252271e8f60c5c70118 Mon Sep 17 00:00:00 2001 From: Xiaoguang Wang Date: Sun, 19 Apr 2020 10:06:55 +0800 Subject: [PATCH 02/18] io_uring: only restore req->work for req that needs do completion When testing io_uring IORING_FEAT_FAST_POLL feature, I got below panic: BUG: kernel NULL pointer dereference, address: 0000000000000030 PGD 0 P4D 0 Oops: 0000 [#1] SMP PTI CPU: 5 PID: 2154 Comm: io_uring_echo_s Not tainted 5.6.0+ #359 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.11.1-0-g0551a4be2c-prebuilt.qemu-project.org 04/01/2014 RIP: 0010:io_wq_submit_work+0xf/0xa0 Code: ff ff ff be 02 00 00 00 e8 ae c9 19 00 e9 58 ff ff ff 66 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 41 54 49 89 fc 55 53 48 8b 2f <8b> 45 30 48 8d 9d 48 ff ff ff 25 01 01 00 00 83 f8 01 75 07 eb 2a RSP: 0018:ffffbef543e93d58 EFLAGS: 00010286 RAX: ffffffff84364f50 RBX: ffffa3eb50f046b8 RCX: 0000000000000000 RDX: ffffa3eb0efc1840 RSI: 0000000000000006 RDI: ffffa3eb50f046b8 RBP: 0000000000000000 R08: 00000000fffd070d R09: 0000000000000000 R10: 0000000000000000 R11: 0000000000000000 R12: ffffa3eb50f046b8 R13: ffffa3eb0efc2088 R14: ffffffff85b69be0 R15: ffffa3eb0effa4b8 FS: 00007fe9f69cc4c0(0000) GS:ffffa3eb5ef40000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000000000000030 CR3: 0000000020410000 CR4: 00000000000006e0 Call Trace: task_work_run+0x6d/0xa0 do_exit+0x39a/0xb80 ? get_signal+0xfe/0xbc0 do_group_exit+0x47/0xb0 get_signal+0x14b/0xbc0 ? __x64_sys_io_uring_enter+0x1b7/0x450 do_signal+0x2c/0x260 ? __x64_sys_io_uring_enter+0x228/0x450 exit_to_usermode_loop+0x87/0xf0 do_syscall_64+0x209/0x230 entry_SYSCALL_64_after_hwframe+0x49/0xb3 RIP: 0033:0x7fe9f64f8df9 Code: Bad RIP value. task_work_run calls io_wq_submit_work unexpectedly, it's obvious that struct callback_head's func member has been changed. After looking into codes, I found this issue is still due to the union definition: union { /* * Only commands that never go async can use the below fields, * obviously. Right now only IORING_OP_POLL_ADD uses them, and * async armed poll handlers for regular commands. The latter * restore the work, if needed. */ struct { struct callback_head task_work; struct hlist_node hash_node; struct async_poll *apoll; }; struct io_wq_work work; }; When task_work_run has multiple work to execute, the work that calls io_poll_remove_all() will do req->work restore for non-poll request always, but indeed if a non-poll request has been added to a new callback_head, subsequent callback will call io_async_task_func() to handle this request, that means we should not do the restore work for such non-poll request. Meanwhile in io_async_task_func(), we should drop submit ref when req has been canceled. Fix both issues. Fixes: b1f573bd15fd ("io_uring: restore req->work when canceling poll request") Signed-off-by: Xiaoguang Wang Use io_double_put_req() Signed-off-by: Jens Axboe --- fs/io_uring.c | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/fs/io_uring.c b/fs/io_uring.c index 381d50becd04a1..c687f57fb65157 100644 --- a/fs/io_uring.c +++ b/fs/io_uring.c @@ -4200,17 +4200,17 @@ static void io_async_task_func(struct callback_head *cb) spin_unlock_irq(&ctx->completion_lock); + /* restore ->work in case we need to retry again */ + memcpy(&req->work, &apoll->work, sizeof(req->work)); + if (canceled) { kfree(apoll); io_cqring_ev_posted(ctx); req_set_fail_links(req); - io_put_req(req); + io_double_put_req(req); return; } - /* restore ->work in case we need to retry again */ - memcpy(&req->work, &apoll->work, sizeof(req->work)); - __set_current_state(TASK_RUNNING); mutex_lock(&ctx->uring_lock); __io_queue_sqe(req, NULL); @@ -4369,7 +4369,7 @@ static bool io_poll_remove_one(struct io_kiocb *req) hash_del(&req->hash_node); - if (apoll) { + if (do_complete && apoll) { /* * restore ->work because we need to call io_req_work_drop_env. */ From 1c5e1cdd7c1f34e6d7f22f9b819a307e435e780e Mon Sep 17 00:00:00 2001 From: Hans de Goede Date: Mon, 13 Apr 2020 15:09:49 +0200 Subject: [PATCH 03/18] ACPI/PCI: pci_link: use extended_irq union member when setting ext-irq shareable The case ACPI_RESOURCE_TYPE_EXTENDED_IRQ inside acpi_pci_link_set() is correctly using resource->res.data.extended_irq.foo for most settings, but for the shareable setting it so far has accidentally been using resource->res.data.irq.shareable instead of resource->res.data.extended_irq.shareable. Note that the old code happens to also work because the shareable field offset is the same for both the acpi_resource_irq and acpi_resource_extended_irq structs. Acked-by: Bjorn Helgaas Signed-off-by: Hans de Goede Signed-off-by: Rafael J. Wysocki --- drivers/acpi/pci_link.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/drivers/acpi/pci_link.c b/drivers/acpi/pci_link.c index 00a6da2121be30..ed3d2182cf2c0a 100644 --- a/drivers/acpi/pci_link.c +++ b/drivers/acpi/pci_link.c @@ -322,10 +322,10 @@ static int acpi_pci_link_set(struct acpi_pci_link *link, int irq) resource->res.data.extended_irq.polarity = link->irq.polarity; if (link->irq.triggering == ACPI_EDGE_SENSITIVE) - resource->res.data.irq.shareable = + resource->res.data.extended_irq.shareable = ACPI_EXCLUSIVE; else - resource->res.data.irq.shareable = ACPI_SHARED; + resource->res.data.extended_irq.shareable = ACPI_SHARED; resource->res.data.extended_irq.interrupt_count = 1; resource->res.data.extended_irq.interrupts[0] = irq; /* ignore resource_source, it's optional */ From 2c9a583be132d1be1ba54f3433b4d817f15c5464 Mon Sep 17 00:00:00 2001 From: Todd Brandt Date: Wed, 8 Apr 2020 10:58:19 -0700 Subject: [PATCH 04/18] pm-graph v5.6 sleepgraph: - force usage of python3 instead of using system default - fix bugzilla 204773 (https://bugzilla.kernel.org/show_bug.cgi?id=204773) - fix issue of platform info not being reset in -multi (logs fill up) - change -ftop call to "pm_suspend", this is one level below state_store - add -wificheck command to read out the current wifi device details - change -wifi behavior to poll /proc/net/wireless for wifi connect - add wifi reconnect time to timeline, include time in summary column - add "fail on wifi_resume" to timeline and summary when wifi fails - add a set of commands to collect data before/after suspend in the log - add "-cmdinfo" command which prints out all the data collected - check for cmd info tools at start, print found/missing in green/red - fix kernel suspend time calculation: tool used to look for start of pm_suspend_console, but the order has changed. latest kernel starts with ksys_sync, use this instead - include time spent in mem/disk in the header (same as freeze/standby) - ignore turbostat 32-bit capability warnings - print to result.txt when -skiphtml is used, just say result: pass - don't exit on SIGTSTP, it's a ctrl-Z and the tool may come back - -multi argument supports duration as well as count: hours, minutes, seconds - update the -multi status output to be more informative - -maxfail sets maximum consecutive fails before a -multi run is aborted - in -summary, ignore dmesg/ftrace/html files that are 0 size bootgraph: - force usage of python3 instead of using system default README: - add endurance testing instructions Makefile: - remove pycache on uninstall Signed-off-by: Todd Brandt Signed-off-by: Rafael J. Wysocki --- tools/power/pm-graph/Makefile | 4 + tools/power/pm-graph/README | 124 ++++- tools/power/pm-graph/bootgraph.py | 2 +- tools/power/pm-graph/sleepgraph.8 | 65 ++- tools/power/pm-graph/sleepgraph.py | 795 ++++++++++++++++------------- 5 files changed, 619 insertions(+), 371 deletions(-) diff --git a/tools/power/pm-graph/Makefile b/tools/power/pm-graph/Makefile index 845541544570a3..b5310832c19c64 100644 --- a/tools/power/pm-graph/Makefile +++ b/tools/power/pm-graph/Makefile @@ -41,6 +41,10 @@ uninstall : if [ -d $(DESTDIR)$(PREFIX)/lib/pm-graph/config ] ; then \ rmdir $(DESTDIR)$(PREFIX)/lib/pm-graph/config; \ fi; + rm -f $(DESTDIR)$(PREFIX)/lib/pm-graph/__pycache__/* + if [ -d $(DESTDIR)$(PREFIX)/lib/pm-graph/__pycache__ ] ; then \ + rmdir $(DESTDIR)$(PREFIX)/lib/pm-graph/__pycache__; \ + fi; rm -f $(DESTDIR)$(PREFIX)/lib/pm-graph/* if [ -d $(DESTDIR)$(PREFIX)/lib/pm-graph ] ; then \ rmdir $(DESTDIR)$(PREFIX)/lib/pm-graph; \ diff --git a/tools/power/pm-graph/README b/tools/power/pm-graph/README index 96259f6e57152b..afe6beb40ad91b 100644 --- a/tools/power/pm-graph/README +++ b/tools/power/pm-graph/README @@ -1,7 +1,12 @@ - p m - g r a p h + _ + _ __ _ __ ___ __ _ _ __ __ _ _ __ | |__ + | '_ \| '_ ` _ \ _____ / _` | '__/ _` | '_ \| '_ \ + | |_) | | | | | |_____| (_| | | | (_| | |_) | | | | + | .__/|_| |_| |_| \__, |_| \__,_| .__/|_| |_| + |_| |___/ |_| pm-graph: suspend/resume/boot timing analysis tools - Version: 5.5 + Version: 5.6 Author: Todd Brandt Home Page: https://01.org/pm-graph @@ -18,10 +23,6 @@ - upstream version in git: https://github.com/intel/pm-graph/ - Requirements: - - runs with python2 or python3, choice is made by /usr/bin/python link - - python2 now requires python-configparser be installed - Table of Contents - Overview - Setup @@ -29,6 +30,8 @@ - Basic Usage - Dev Mode Usage - Proc Mode Usage + - Endurance Testing + - Usage Examples - Configuration Files - Usage Examples - Config File Options @@ -54,15 +57,18 @@ | SETUP | ------------------------------------------------------------------ - These packages are required to execute the scripts + Package Requirements + - runs with python2 or python3, choice is made by /usr/bin/python link - python - - python-requests + - python-configparser (for python2 sleepgraph) + - python-requests (for googlesheet.py) + - linux-tools-common (for turbostat usage in sleepgraph) Ubuntu: - sudo apt-get install python python-requests + sudo apt-get install python python-configparser python-requests linux-tools-common Fedora: - sudo dnf install python python-requests + sudo dnf install python python-configparser python-requests linux-tools-common The tools can most easily be installed via git clone and make install @@ -190,6 +196,104 @@ _______________ %> sudo ./sleepgraph.py -config config/suspend-proc.cfg +------------------------------------------------------------------ +| ENDURANCE TESTING | +------------------------------------------------------------------ + + The best way to gauge the health of a system is to run a series of + suspend/resumes over an extended period and analyze the behavior. This can be + accomplished with sleepgraph's -multi argument. You specify two numbers: the + number of tests to run OR the duration in days, hours, or minutes, and the + delay in seconds between them. For instance, -multi 20 5: execute 20 tests with + a 5 second delay between each, or -multi 24h 0: execute tests over a 24 hour + period with no delay between tests. You can include any other options you like + to generate the data you want. It's most useful to collect dev mode timelines + as the kprobes don't alter the performance much and you get more insight. + + On completion, the output folder contains a series of folders for the + individual test data and a set of summary pages in the root. The summary.html + file is a tabular list of the tests with relevant info and links. The + summary-issue.html and summary-devices.html files include data taken from + all tests on kernel issues and device performance. The folder looks like this: + + suspend-xN-{date}-{time}: + summary.html + summary-issues.html + summary-devices.html + suspend-{date}-{time} (1) + suspend-{date}-{time} (2) + ... + + These are the relevant arguments to use for testing: + + -m mode + Mode to initiate for suspend e.g. mem, freeze, standby (default: mem). + + -rtcwake t + Use rtcwake to autoresume after t seconds (default: 15). + + -gzip (optional) + Gzip the trace and dmesg logs to save space. The tool can also read in + gzipped logs for processing. This reduces the multitest folder size. + + -dev (optional) + Add kernel source calls and threads to the timeline (default: disabled). + + -multi n d + Execute n consecutive tests at d seconds intervals. The outputs will be + created in a new subdirectory: suspend-xN-{date}-{time}. When the multitest + run is done, the -summary command is called automatically to create summary + html files for all the data (unless you use -skiphtml). -skiphtml will + speed up the testing by not creating timelines or summary html files. You + can then run the tool again at a later time with -summary and -genhtml to + create the timelines. + + -skiphtml (optional) + Run the test and capture the trace logs, but skip the timeline and summary + html generation. This can greatly speed up overall testing. You can then + copy the data to a faster host machine and run -summary -genhtml to + generate the timelines and summary. + + These are the relevant commands to use after testing is complete: + + -summary indir + Generate or regenerate the summary for a -multi test run. Creates three + files: summary.html, summary-issues.html, and summary-devices.html in the + current folder. summary.html is a table of tests with relevant info sorted + by kernel/host/mode, and links to the test html files. summary-issues.html + is a list of kernel issues found in dmesg from all the tests. + summary-devices.html is a list of devices and times from all the tests. + + -genhtml + Used with -summary to regenerate any missing html timelines from their + dmesg and ftrace logs. This will require a significant amount of time if + there are thousands of tests. + +Usage Examples +_______________ + + A multitest is initiated like this: + + %> sudo ./sleepgraph.py -m mem -rtcwake 10 -dev -gzip -multi 2000 0 + + or you can skip timeline generation in order to speed things up + + %> sudo ./sleepgraph.py -m mem -rtcwake 10 -dev -gzip -multi 2000 0 -skiphtml + + The tool will produce an output folder with all the test subfolders inside. + Each test subfolder contains the dmesg/ftrace logs and/or the html timeline + depending on whether you used the -skiphtml option. The root folder contains + the summary.html files. + + The summary for an existing multitest is generated like this: + + %> cd suspend-x2000-{date}-{time} + %> sleepgraph.py -summary . + + or if you need to generate the html timelines you can use -genhtml + + %> cd suspend-xN-{date}-{time} + %> sleepgraph.py -summary . -genhtml ------------------------------------------------------------------ | CONFIGURATION FILES | diff --git a/tools/power/pm-graph/bootgraph.py b/tools/power/pm-graph/bootgraph.py index d3b99a1e92d6f2..2823cd3122f7bb 100755 --- a/tools/power/pm-graph/bootgraph.py +++ b/tools/power/pm-graph/bootgraph.py @@ -1,4 +1,4 @@ -#!/usr/bin/python +#!/usr/bin/env python3 # SPDX-License-Identifier: GPL-2.0-only # # Tool for analyzing boot timing diff --git a/tools/power/pm-graph/sleepgraph.8 b/tools/power/pm-graph/sleepgraph.8 index 43aee64316df06..5126271de98a6a 100644 --- a/tools/power/pm-graph/sleepgraph.8 +++ b/tools/power/pm-graph/sleepgraph.8 @@ -74,8 +74,10 @@ after the test is complete. Switch the display to the requested mode for the test using the xset command. This helps maintain the consistency of test data for better comparison. .TP -\fB-skiphtml\fR -Run the test and capture the trace logs, but skip the timeline generation. +\fB-wifi\fR +If a wifi connection is available, check that it reconnects after resume. Include +the reconnect time in the total resume time calculation and treat wifi timeouts +as resume failures. .SS "advanced" .TP @@ -117,8 +119,24 @@ Include \fIt\fR ms delay before 1st suspend (default: 0 ms). Include \fIt\fR ms delay after last resume (default: 0 ms). .TP \fB-multi \fIn d\fR -Execute \fIn\fR consecutive tests at \fId\fR seconds intervals. The outputs will -be created in a new subdirectory with a summary page: suspend-xN-{date}-{time}. +Used for endurance testing. If \fIn\fR is entirely numeric, it's treated as a count: +Execute \fIn\fR consecutive tests at \fId\fR second intervals. +If \fIn\fR is an integer followed by a "d", "h", or "m", it's treated as a duration: +Execute tests continuously over \fIn\fR days, hours, or minutes at \fId\fR second intervals. +The outputs will be created in a new subdirectory, for count: suspend-{date}-{time}-xN, +for duration: suspend-{date}-{time}-Nm. When the multitest run is done, the \fI-summary\fR +command is called automatically to create summary html files for all the data (unless you +use \fI-skiphtml\fR). \fI-skiphtml\fR will speed up the testing by not creating timelines +or summary html files. You can then run the tool again at a later time with \fI-summary\fR +and \fI-genhtml\fR to create the timelines. +.TP +\fB-maxfail \fIn\fR +Abort a -multi run after \fIn\fR consecutive fails. 0 means never abort (default = 0). +.TP +\fB-skiphtml\fR +Run the test and capture the trace logs, but skip the timeline generation. +You can generate the html timelines later with \fI-dmesg\fR & \fI-ftrace\fR, or +by running \fI-summary\fR and \fI-genhtml\fR. .SS "ftrace debug" .TP @@ -173,11 +191,20 @@ Set trace buffer size to N kilo-bytes (default: all of free memory up to 3GB) .SH COMMANDS .TP \fB-summary \fIindir\fR -Create a summary page of all tests in \fIindir\fR. Creates summary.html -in the current folder. The output page is a table of tests with -suspend and resume values sorted by suspend mode, host, and kernel. -Includes test averages by mode and links to the test html files. -Use -genhtml to include tests with missing html. +Create a set of summary pages for all tests in \fIindir\fR recursively. +Creates summary.html, summary-issues.html, and summary-devices.html in the current folder. +summary.html is a table of tests with relevant info sorted by kernel/host/mode, +and links to the test html files. It identifies the minimum, maximum, and median +suspend and resume times for you with highlights and links in the header. +summary-issues.html is a list of kernel issues found in dmesg from all the tests. +summary-devices.html is a list of devices and times from all the tests. + +Use \fI-genhtml\fR to regenerate any tests with missing html. +.TP +\fB-genhtml\fR +Used with \fI-summary\fR to regenerate any missing html timelines from their +dmesg and ftrace logs. This will require a significant amount of time if there +are thousands of tests. .TP \fB-modes\fR List available suspend modes. @@ -189,10 +216,7 @@ with any options you intend to use to see if they will work. \fB-fpdt\fR Print out the contents of the ACPI Firmware Performance Data Table. .TP -\fB-battery\fR -Print out battery status and current charge. -.TP -\fB-wifi\fR +\fB-wificheck\fR Print out wifi status and connection details. .TP \fB-xon/-xoff/-xstandby/-xsuspend\fR @@ -208,6 +232,9 @@ Print out system info extracted from BIOS. Reads /dev/mem directly instead of go \fB-devinfo\fR Print out the pm settings of all devices which support runtime suspend. .TP +\fB-cmdinfo\fR +Print out all the platform data collected from the system that makes it into the logs. +.TP \fB-flist\fR Print the list of ftrace functions currently being captured. Functions that are not available as symbols in the current kernel are shown in red. @@ -272,14 +299,20 @@ Run two suspends back to back, include a 500ms delay before, after, and in betwe .IP \f(CW$ sudo sleepgraph -m mem -rtcwake 15 -x2 -predelay 500 -x2delay 500 -postdelay 500\fR .PP +Execute a suspend using a custom command. +.IP +\f(CW$ sudo sleepgraph -cmd "echo mem > /sys/power/state" -rtcwake 15\fR +.PP + +.SS "endurance testing using -multi" +.PP Do a batch run of 10 freezes with 30 seconds delay between runs. .IP \f(CW$ sudo sleepgraph -m freeze -rtcwake 15 -multi 10 30\fR .PP -Execute a suspend using a custom command. +Do a batch run of freezes for 24 hours. .IP -\f(CW$ sudo sleepgraph -cmd "echo mem > /sys/power/state" -rtcwake 15\fR -.PP +\f(CW$ sudo sleepgraph -m freeze -rtcwake 15 -multi 24h 0\fR .SS "adding callgraph data" Add device callgraphs. Limit the trace depth and only show callgraphs 10ms or larger. diff --git a/tools/power/pm-graph/sleepgraph.py b/tools/power/pm-graph/sleepgraph.py index f7d1c1f62f860b..9b0404d107686a 100755 --- a/tools/power/pm-graph/sleepgraph.py +++ b/tools/power/pm-graph/sleepgraph.py @@ -1,4 +1,4 @@ -#!/usr/bin/python +#!/usr/bin/env python3 # SPDX-License-Identifier: GPL-2.0-only # # Tool for analyzing suspend/resume timing @@ -58,7 +58,7 @@ import platform import signal import codecs -from datetime import datetime +from datetime import datetime, timedelta import struct import configparser import gzip @@ -81,12 +81,13 @@ def ascii(text): # store system values and test parameters class SystemValues: title = 'SleepGraph' - version = '5.5' + version = '5.6' ansi = False rs = 0 display = '' gzip = False sync = False + wifi = False verbose = False testlog = True dmesglog = True @@ -97,7 +98,8 @@ class SystemValues: cgphase = '' cgtest = -1 cgskip = '' - multitest = {'run': False, 'count': 0, 'delay': 0} + maxfail = 0 + multitest = {'run': False, 'count': 1000000, 'delay': 0} max_graph_depth = 0 callloopmaxgap = 0.0001 callloopmaxlen = 0.005 @@ -148,7 +150,7 @@ class SystemValues: x2delay = 0 skiphtml = False usecallgraph = False - ftopfunc = 'suspend_devices_and_enter' + ftopfunc = 'pm_suspend' ftop = False usetraceevents = False usetracemarkers = True @@ -164,6 +166,8 @@ class SystemValues: predelay = 0 postdelay = 0 pmdebug = '' + tmstart = 'SUSPEND START %Y%m%d-%H:%M:%S.%f' + tmend = 'RESUME COMPLETE %Y%m%d-%H:%M:%S.%f' tracefuncs = { 'sys_sync': {}, 'ksys_sync': {}, @@ -183,9 +187,11 @@ class SystemValues: 'acpi_s2idle_sync': {}, 'acpi_s2idle_begin': {}, 'acpi_s2idle_prepare': {}, + 'acpi_s2idle_prepare_late': {}, 'acpi_s2idle_wake': {}, 'acpi_s2idle_wakeup': {}, 'acpi_s2idle_restore': {}, + 'acpi_s2idle_restore_early': {}, 'hibernate_preallocate_memory': {}, 'create_basic_memory_bitmaps': {}, 'swsusp_write': {}, @@ -270,12 +276,23 @@ class SystemValues: 'intel_opregion_init': {}, 'intel_fbdev_set_suspend': {}, } + infocmds = [ + [0, 'kparams', 'cat', '/proc/cmdline'], + [0, 'mcelog', 'mcelog'], + [0, 'pcidevices', 'lspci', '-tv'], + [0, 'usbdevices', 'lsusb', '-t'], + [1, 'interrupts', 'cat', '/proc/interrupts'], + [1, 'wakeups', 'cat', '/sys/kernel/debug/wakeup_sources'], + [2, 'gpecounts', 'sh', '-c', 'grep -v invalid /sys/firmware/acpi/interrupts/*'], + [2, 'suspendstats', 'sh', '-c', 'grep -v invalid /sys/power/suspend_stats/*'], + [2, 'cpuidle', 'sh', '-c', 'grep -v invalid /sys/devices/system/cpu/cpu*/cpuidle/state*/s2idle/*'], + [2, 'battery', 'sh', '-c', 'grep -v invalid /sys/class/power_supply/*/*'], + ] cgblacklist = [] kprobes = dict() timeformat = '%.3f' cmdline = '%s %s' % \ (os.path.basename(sys.argv[0]), ' '.join(sys.argv[1:])) - kparams = '' sudouser = '' def __init__(self): self.archargs = 'args_'+platform.machine() @@ -295,6 +312,9 @@ def __init__(self): if os.getuid() == 0 and 'SUDO_USER' in os.environ and \ os.environ['SUDO_USER']: self.sudouser = os.environ['SUDO_USER'] + def resetlog(self): + self.logmsg = '' + self.platinfo = [] def vprint(self, msg): self.logmsg += msg+'\n' if self.verbose or msg.startswith('WARNING:'): @@ -304,11 +324,11 @@ def signalHandler(self, signum, frame): return signame = self.signames[signum] if signum in self.signames else 'UNKNOWN' msg = 'Signal %s caused a tool exit, line %d' % (signame, frame.f_lineno) - sysvals.outputResult({'error':msg}) + self.outputResult({'error':msg}) sys.exit(3) def signalHandlerInit(self): capture = ['BUS', 'SYS', 'XCPU', 'XFSZ', 'PWR', 'HUP', 'INT', 'QUIT', - 'ILL', 'ABRT', 'FPE', 'SEGV', 'TERM', 'TSTP'] + 'ILL', 'ABRT', 'FPE', 'SEGV', 'TERM'] self.signames = dict() for i in capture: s = 'SIG'+i @@ -336,6 +356,8 @@ def rootUser(self, fatal=False): self.outputResult({'error':msg}) sys.exit(1) return False + def usable(self, file): + return (os.path.exists(file) and os.path.getsize(file) > 0) def getExec(self, cmd): try: fp = Popen(['which', cmd], stdout=PIPE, stderr=PIPE).stdout @@ -389,12 +411,6 @@ def systemInfo(self, info): r = info['bios-release-date'] if 'bios-release-date' in info else '' self.sysstamp = '# sysinfo | man:%s | plat:%s | cpu:%s | bios:%s | biosdate:%s | numcpu:%d | memsz:%d | memfr:%d' % \ (m, p, c, b, r, self.cpucount, self.memtotal, self.memfree) - try: - kcmd = open('/proc/cmdline', 'r').read().strip() - except: - kcmd = '' - if kcmd: - self.sysstamp += '\n# kparams | %s' % kcmd def printSystemInfo(self, fatal=False): self.rootCheck(True) out = dmidecode(self.mempath, fatal) @@ -441,6 +457,7 @@ def initTestOutput(self, name): self.testdir+'/'+self.prefix+'_'+self.suspendmode+'.html' if not os.path.isdir(self.testdir): os.makedirs(self.testdir) + self.sudoUserchown(self.testdir) def getValueList(self, value): out = [] for i in value.split(','): @@ -486,7 +503,7 @@ def initdmesg(self): fp.close() self.dmesgstart = float(ktime) def getdmesg(self, testdata): - op = self.writeDatafileHeader(sysvals.dmesgfile, testdata) + op = self.writeDatafileHeader(self.dmesgfile, testdata) # store all new dmesg lines since initdmesg was called fp = Popen('dmesg', stdout=PIPE).stdout for line in fp: @@ -716,9 +733,10 @@ def isCallgraphFunc(self, name): if name == f: return True return False - def initFtrace(self): - self.printSystemInfo(False) - pprint('INITIALIZING FTRACE...') + def initFtrace(self, quiet=False): + if not quiet: + sysvals.printSystemInfo(False) + pprint('INITIALIZING FTRACE...') # turn trace off self.fsetVal('0', 'tracing_on') self.cleanupFtrace() @@ -746,7 +764,7 @@ def initFtrace(self): if tgtsize < 65536: tgtsize = int(self.fgetVal('buffer_size_kb')) * cpus break - pprint('Setting trace buffers to %d kB (%d kB per cpu)' % (tgtsize, tgtsize/cpus)) + self.vprint('Setting trace buffers to %d kB (%d kB per cpu)' % (tgtsize, tgtsize/cpus)) # initialize the callgraph trace if(self.usecallgraph): # set trace type @@ -782,7 +800,8 @@ def initFtrace(self): if self.usedevsrc: for name in self.dev_tracefuncs: self.defaultKprobe(name, self.dev_tracefuncs[name]) - pprint('INITIALIZING KPROBES...') + if not quiet: + pprint('INITIALIZING KPROBES...') self.addKprobes(self.verbose) if(self.usetraceevents): # turn trace events on @@ -827,21 +846,10 @@ def writeDatafileHeader(self, filename, testdata): fw = test['fw'] if(fw): fp.write('# fwsuspend %u fwresume %u\n' % (fw[0], fw[1])) - if 'mcelog' in test: - fp.write('# mcelog %s\n' % test['mcelog']) if 'turbo' in test: fp.write('# turbostat %s\n' % test['turbo']) - if 'bat' in test: - (a1, c1), (a2, c2) = test['bat'] - fp.write('# battery %s %d %s %d\n' % (a1, c1, a2, c2)) if 'wifi' in test: - wstr = [] - for wifi in test['wifi']: - tmp = [] - for key in sorted(wifi): - tmp.append('%s:%s' % (key, wifi[key])) - wstr.append('|'.join(tmp)) - fp.write('# wifi %s\n' % (','.join(wstr))) + fp.write('# wifi %s\n' % test['wifi']) if test['error'] or len(testdata) > 1: fp.write('# enter_sleep_error %s\n' % test['error']) return fp @@ -901,23 +909,7 @@ def b64unzip(self, data): def b64zip(self, data): out = base64.b64encode(codecs.encode(data.encode(), 'zlib')).decode() return out - def mcelog(self, clear=False): - cmd = self.getExec('mcelog') - if not cmd: - return '' - if clear: - call(cmd+' > /dev/null 2>&1', shell=True) - return '' - try: - fp = Popen([cmd], stdout=PIPE, stderr=PIPE).stdout - out = ascii(fp.read()).strip() - fp.close() - except: - return '' - if not out: - return '' - return self.b64zip(out) - def platforminfo(self): + def platforminfo(self, cmdafter): # add platform info on to a completed ftrace file if not os.path.exists(self.ftracefile): return False @@ -1001,31 +993,76 @@ def platforminfo(self): footer += '# platform-devinfo: %s\n' % self.b64zip(out) # add a line for each of these commands with their outputs - cmds = [ - ['pcidevices', 'lspci', '-tv'], - ['interrupts', 'cat', '/proc/interrupts'], - ['gpecounts', 'sh', '-c', 'grep -v invalid /sys/firmware/acpi/interrupts/gpe*'], - ] - for cargs in cmds: - name = cargs[0] - cmdline = ' '.join(cargs[1:]) - cmdpath = self.getExec(cargs[1]) - if not cmdpath: + for name, cmdline, info in cmdafter: + footer += '# platform-%s: %s | %s\n' % (name, cmdline, self.b64zip(info)) + + with self.openlog(self.ftracefile, 'a') as fp: + fp.write(footer) + return True + def commonPrefix(self, list): + if len(list) < 2: + return '' + prefix = list[0] + for s in list[1:]: + while s[:len(prefix)] != prefix and prefix: + prefix = prefix[:len(prefix)-1] + if not prefix: + break + if '/' in prefix and prefix[-1] != '/': + prefix = prefix[0:prefix.rfind('/')+1] + return prefix + def dictify(self, text, format): + out = dict() + header = True if format == 1 else False + delim = ' ' if format == 1 else ':' + for line in text.split('\n'): + if header: + header, out['@'] = False, line + continue + line = line.strip() + if delim in line: + data = line.split(delim, 1) + num = re.search(r'[\d]+', data[1]) + if format == 2 and num: + out[data[0].strip()] = num.group() + else: + out[data[0].strip()] = data[1] + return out + def cmdinfo(self, begin, debug=False): + out = [] + if begin: + self.cmd1 = dict() + for cargs in self.infocmds: + delta, name = cargs[0], cargs[1] + cmdline, cmdpath = ' '.join(cargs[2:]), self.getExec(cargs[2]) + if not cmdpath or (begin and not delta): continue - cmd = [cmdpath] + cargs[2:] try: - fp = Popen(cmd, stdout=PIPE, stderr=PIPE).stdout + fp = Popen([cmdpath]+cargs[3:], stdout=PIPE, stderr=PIPE).stdout info = ascii(fp.read()).strip() fp.close() except: continue - if not info: - continue - footer += '# platform-%s: %s | %s\n' % (name, cmdline, self.b64zip(info)) - - with self.openlog(self.ftracefile, 'a') as fp: - fp.write(footer) - return True + if not debug and begin: + self.cmd1[name] = self.dictify(info, delta) + elif not debug and delta and name in self.cmd1: + before, after = self.cmd1[name], self.dictify(info, delta) + dinfo = ('\t%s\n' % before['@']) if '@' in before else '' + prefix = self.commonPrefix(list(before.keys())) + for key in sorted(before): + if key in after and before[key] != after[key]: + title = key.replace(prefix, '') + if delta == 2: + dinfo += '\t%s : %s -> %s\n' % \ + (title, before[key].strip(), after[key].strip()) + else: + dinfo += '%10s (start) : %s\n%10s (after) : %s\n' % \ + (title, before[key], title, after[key]) + dinfo = '\tnothing changed' if not dinfo else dinfo.rstrip() + out.append((name, cmdline, dinfo)) + else: + out.append((name, cmdline, '\tnothing' if not info else info)) + return out def haveTurbostat(self): if not self.tstat: return False @@ -1035,8 +1072,8 @@ def haveTurbostat(self): fp = Popen([cmd, '-v'], stdout=PIPE, stderr=PIPE).stderr out = ascii(fp.read()).strip() fp.close() - if re.match('turbostat version [0-9\.]* .*', out): - sysvals.vprint(out) + if re.match('turbostat version .*', out): + self.vprint(out) return True return False def turbostat(self): @@ -1056,11 +1093,11 @@ def turbostat(self): fp.close() if not keyline or not valline or len(keyline) != len(valline): errmsg = 'unrecognized turbostat output:\n'+rawout.strip() - sysvals.vprint(errmsg) - if not sysvals.verbose: + self.vprint(errmsg) + if not self.verbose: pprint(errmsg) return '' - if sysvals.verbose: + if self.verbose: pprint(rawout.strip()) out = [] for key in keyline: @@ -1068,30 +1105,36 @@ def turbostat(self): val = valline[idx] out.append('%s=%s' % (key, val)) return '|'.join(out) - def checkWifi(self): - out = dict() - iwcmd, ifcmd = self.getExec('iwconfig'), self.getExec('ifconfig') - if not iwcmd or not ifcmd: - return out - fp = Popen(iwcmd, stdout=PIPE, stderr=PIPE).stdout - for line in fp: - m = re.match('(?P\S*) .* ESSID:(?P\S*)', ascii(line)) - if not m: + def wifiDetails(self, dev): + try: + info = open('/sys/class/net/%s/device/uevent' % dev, 'r').read().strip() + except: + return dev + vals = [dev] + for prop in info.split('\n'): + if prop.startswith('DRIVER=') or prop.startswith('PCI_ID='): + vals.append(prop.split('=')[-1]) + return ':'.join(vals) + def checkWifi(self, dev=''): + try: + w = open('/proc/net/wireless', 'r').read().strip() + except: + return '' + for line in reversed(w.split('\n')): + m = re.match(' *(?P.*): (?P[0-9a-f]*) .*', w.split('\n')[-1]) + if not m or (dev and dev != m.group('dev')): continue - out['device'] = m.group('dev') - if '"' in m.group('ess'): - out['essid'] = m.group('ess').strip('"') - break - fp.close() - if 'device' in out: - fp = Popen([ifcmd, out['device']], stdout=PIPE, stderr=PIPE).stdout - for line in fp: - m = re.match('.* inet (?P[0-9\.]*)', ascii(line)) - if m: - out['ip'] = m.group('ip') - break - fp.close() - return out + return m.group('dev') + return '' + def pollWifi(self, dev, timeout=60): + start = time.time() + while (time.time() - start) < timeout: + w = self.checkWifi(dev) + if w: + return '%s reconnected %.2f' % \ + (self.wifiDetails(dev), max(0, time.time() - start)) + time.sleep(0.01) + return '%s timeout %d' % (self.wifiDetails(dev), timeout) def errorSummary(self, errinfo, msg): found = False for entry in errinfo: @@ -1113,8 +1156,9 @@ def errorSummary(self, errinfo, msg): arr[j] = arr[j]\ .replace('\\', '\\\\').replace(']', '\]').replace('[', '\[')\ .replace('.', '\.').replace('+', '\+').replace('*', '\*')\ - .replace('(', '\(').replace(')', '\)') - mstr = ' '.join(arr) + .replace('(', '\(').replace(')', '\)').replace('}', '\}')\ + .replace('{', '\{') + mstr = ' *'.join(arr) entry = { 'line': msg, 'match': mstr, @@ -1122,6 +1166,44 @@ def errorSummary(self, errinfo, msg): 'urls': {self.hostname: [self.htmlfile]} } errinfo.append(entry) + def multistat(self, start, idx, finish): + if 'time' in self.multitest: + id = '%d Duration=%dmin' % (idx+1, self.multitest['time']) + else: + id = '%d/%d' % (idx+1, self.multitest['count']) + t = time.time() + if 'start' not in self.multitest: + self.multitest['start'] = self.multitest['last'] = t + self.multitest['total'] = 0.0 + pprint('TEST (%s) START' % id) + return + dt = t - self.multitest['last'] + if not start: + if idx == 0 and self.multitest['delay'] > 0: + self.multitest['total'] += self.multitest['delay'] + pprint('TEST (%s) COMPLETE -- Duration %.1fs' % (id, dt)) + return + self.multitest['total'] += dt + self.multitest['last'] = t + avg = self.multitest['total'] / idx + if 'time' in self.multitest: + left = finish - datetime.now() + left -= timedelta(microseconds=left.microseconds) + else: + left = timedelta(seconds=((self.multitest['count'] - idx) * int(avg))) + pprint('TEST (%s) START - Avg Duration %.1fs, Time left %s' % \ + (id, avg, str(left))) + def multiinit(self, c, d): + sz, unit = 'count', 'm' + if c.endswith('d') or c.endswith('h') or c.endswith('m'): + sz, unit, c = 'time', c[-1], c[:-1] + self.multitest['run'] = True + self.multitest[sz] = getArgInt('multi: n d (exec count)', c, 1, 1000000, False) + self.multitest['delay'] = getArgInt('multi: n d (delay between tests)', d, 0, 3600, False) + if unit == 'd': + self.multitest[sz] *= 1440 + elif unit == 'h': + self.multitest[sz] *= 60 sysvals = SystemValues() switchvalues = ['enable', 'disable', 'on', 'off', 'true', 'false', '1', '0'] @@ -1210,25 +1292,30 @@ class Data: 'resume_complete': {'order': 9, 'color': '#FFFFCC'}, } errlist = { - 'HWERROR' : '.*\[ *Hardware Error *\].*', - 'FWBUG' : '.*\[ *Firmware Bug *\].*', - 'BUG' : '.*BUG.*', - 'ERROR' : '.*ERROR.*', - 'WARNING' : '.*WARNING.*', - 'IRQ' : '.*genirq: .*', - 'TASKFAIL': '.*Freezing of tasks *.*', - 'ACPI' : '.*ACPI *(?P[A-Za-z]*) *Error[: ].*', - 'DEVFAIL' : '.* failed to (?P[a-z]*) async: .*', - 'DISKFULL': '.*No space left on device.*', - 'USBERR' : '.*usb .*device .*, error [0-9-]*', - 'ATAERR' : ' *ata[0-9\.]*: .*failed.*', - 'MEIERR' : ' *mei.*: .*failed.*', - 'TPMERR' : '(?i) *tpm *tpm[0-9]*: .*error.*', + 'HWERROR' : r'.*\[ *Hardware Error *\].*', + 'FWBUG' : r'.*\[ *Firmware Bug *\].*', + 'BUG' : r'(?i).*\bBUG\b.*', + 'ERROR' : r'(?i).*\bERROR\b.*', + 'WARNING' : r'(?i).*\bWARNING\b.*', + 'FAULT' : r'(?i).*\bFAULT\b.*', + 'FAIL' : r'(?i).*\bFAILED\b.*', + 'INVALID' : r'(?i).*\bINVALID\b.*', + 'CRASH' : r'(?i).*\bCRASHED\b.*', + 'IRQ' : r'.*\bgenirq: .*', + 'TASKFAIL': r'.*Freezing of tasks *.*', + 'ACPI' : r'.*\bACPI *(?P[A-Za-z]*) *Error[: ].*', + 'DISKFULL': r'.*\bNo space left on device.*', + 'USBERR' : r'.*usb .*device .*, error [0-9-]*', + 'ATAERR' : r' *ata[0-9\.]*: .*failed.*', + 'MEIERR' : r' *mei.*: .*failed.*', + 'TPMERR' : r'(?i) *tpm *tpm[0-9]*: .*error.*', } def __init__(self, num): idchar = 'abcdefghij' self.start = 0.0 # test start self.end = 0.0 # test end + self.hwstart = 0 # rtc test start + self.hwend = 0 # rtc test end self.tSuspended = 0.0 # low-level suspend start self.tResumed = 0.0 # low-level resume start self.tKernSus = 0.0 # kernel level suspend start @@ -1240,10 +1327,8 @@ def __init__(self, num): self.stamp = 0 self.outfile = '' self.kerror = False - self.battery = 0 - self.wifi = 0 + self.wifi = dict() self.turbostat = 0 - self.mcelog = 0 self.enterfail = '' self.currphase = '' self.pstl = dict() # process timeline @@ -1308,6 +1393,8 @@ def extractErrorInfo(self): continue dir = 'suspend' if t < self.tSuspended else 'resume' msg = m.group('msg') + if re.match('capability: warning: .*', msg): + continue for err in self.errlist: if re.match(self.errlist[err], msg): list.append((msg, err, dir, t, i, i)) @@ -1316,17 +1403,26 @@ def extractErrorInfo(self): msglist = [] for msg, type, dir, t, idx1, idx2 in list: msglist.append(msg) - sysvals.vprint('kernel %s found in %s at %f' % (type, dir, t)) self.errorinfo[dir].append((type, t, idx1, idx2)) if self.kerror: sysvals.dmesglog = True if len(self.dmesgtext) < 1 and sysvals.dmesgfile: lf.close() return msglist - def setStart(self, time): + def setStart(self, time, msg=''): self.start = time - def setEnd(self, time): + if msg: + try: + self.hwstart = datetime.strptime(msg, sysvals.tmstart) + except: + self.hwstart = 0 + def setEnd(self, time, msg=''): self.end = time + if msg: + try: + self.hwend = datetime.strptime(msg, sysvals.tmend) + except: + self.hwend = 0 def isTraceEventOutsideDeviceCalls(self, pid, time): for phase in self.sortedPhases(): list = self.dmesg[phase]['list'] @@ -1546,6 +1642,14 @@ def trimFreezeTime(self, tZero): self.trimTime(tS, tL, left) self.tLow.append('%.0f'%(tL*1000)) lp = phase + def getMemTime(self): + if not self.hwstart or not self.hwend: + return + stime = (self.tSuspended - self.start) * 1000000 + rtime = (self.end - self.tResumed) * 1000000 + hws = self.hwstart + timedelta(microseconds=stime) + hwr = self.hwend - timedelta(microseconds=rtime) + self.tLow.append('%.0f'%((hwr - hws).total_seconds() * 1000)) def getTimeValues(self): sktime = (self.tSuspended - self.tKernSus) * 1000 rktime = (self.tKernRes - self.tResumed) * 1000 @@ -1883,9 +1987,9 @@ def createProcessUsageEvents(self): c = self.addProcessUsageEvent(ps, tres) if c > 0: sysvals.vprint('%25s (res): %d' % (ps, c)) - def handleEndMarker(self, time): + def handleEndMarker(self, time, msg=''): dm = self.dmesg - self.setEnd(time) + self.setEnd(time, msg) self.initDevicegroups() # give suspend_prepare an end if needed if 'suspend_prepare' in dm and dm['suspend_prepare']['end'] < 0: @@ -2071,7 +2175,7 @@ def startMarker(self): if not self.fevent: return False if sysvals.usetracemarkers: - if(self.name == 'SUSPEND START'): + if(self.name.startswith('SUSPEND START')): return True return False else: @@ -2084,7 +2188,7 @@ def endMarker(self): if not self.fevent: return False if sysvals.usetracemarkers: - if(self.name == 'RESUME COMPLETE'): + if(self.name.startswith('RESUME COMPLETE')): return True return False else: @@ -2444,7 +2548,7 @@ def __init__(self, rowheight, scaleheight): def createHeader(self, sv, stamp): if(not stamp['time']): return - self.html += '
%s v%s
' \ + self.html += '
%s v%s
' \ % (sv.title, sv.version) if sv.logmsg and sv.testlog: self.html += '' @@ -2670,14 +2774,11 @@ class TestProps: stampfmt = '# [a-z]*-(?P[0-9]{2})(?P[0-9]{2})(?P[0-9]{2})-'+\ '(?P[0-9]{2})(?P[0-9]{2})(?P[0-9]{2})'+\ ' (?P.*) (?P.*) (?P.*)$' - batteryfmt = '^# battery (?P\w*) (?P\d*) (?P\w*) (?P\d*)' - wififmt = '^# wifi (?P.*)' + wififmt = '^# wifi *(?P\S*) *(?P\S*) *(?P[0-9\.]+).*' tstatfmt = '^# turbostat (?P\S*)' - mcelogfmt = '^# mcelog (?P\S*)' testerrfmt = '^# enter_sleep_error (?P.*)' sysinfofmt = '^# sysinfo .*' cmdlinefmt = '^# command \| (?P.*)' - kparamsfmt = '^# kparams \| (?P.*)' devpropfmt = '# Device Properties: .*' pinfofmt = '# platform-(?P[a-z,A-Z,0-9]*): (?P.*)' tracertypefmt = '# tracer: (?P.*)' @@ -2695,11 +2796,8 @@ def __init__(self): self.stamp = '' self.sysinfo = '' self.cmdline = '' - self.kparams = '' self.testerror = [] - self.mcelog = [] self.turbostat = [] - self.battery = [] self.wifi = [] self.fwdata = [] self.ftrace_line_fmt = self.ftrace_line_fmt_nop @@ -2721,21 +2819,12 @@ def stampInfo(self, line): elif re.match(self.sysinfofmt, line): self.sysinfo = line return True - elif re.match(self.kparamsfmt, line): - self.kparams = line - return True elif re.match(self.cmdlinefmt, line): self.cmdline = line return True - elif re.match(self.mcelogfmt, line): - self.mcelog.append(line) - return True elif re.match(self.tstatfmt, line): self.turbostat.append(line) return True - elif re.match(self.batteryfmt, line): - self.battery.append(line) - return True elif re.match(self.wififmt, line): self.wifi.append(line) return True @@ -2749,6 +2838,8 @@ def stampInfo(self, line): def parseStamp(self, data, sv): # global test data m = re.match(self.stampfmt, self.stamp) + if not self.stamp or not m: + doError('data does not include the expected stamp') data.stamp = {'time': '', 'host': '', 'mode': ''} dt = datetime(int(m.group('y'))+2000, int(m.group('m')), int(m.group('d')), int(m.group('H')), int(m.group('M')), @@ -2780,10 +2871,6 @@ def parseStamp(self, data, sv): m = re.match(self.cmdlinefmt, self.cmdline) if m: sv.cmdline = m.group('cmd') - if self.kparams: - m = re.match(self.kparamsfmt, self.kparams) - if m: - sv.kparams = m.group('kp') if not sv.stamp: sv.stamp = data.stamp # firmware data @@ -2793,26 +2880,18 @@ def parseStamp(self, data, sv): data.fwSuspend, data.fwResume = int(m.group('s')), int(m.group('r')) if(data.fwSuspend > 0 or data.fwResume > 0): data.fwValid = True - # mcelog data - if len(self.mcelog) > data.testnumber: - m = re.match(self.mcelogfmt, self.mcelog[data.testnumber]) - if m: - data.mcelog = sv.b64unzip(m.group('m')) # turbostat data if len(self.turbostat) > data.testnumber: m = re.match(self.tstatfmt, self.turbostat[data.testnumber]) if m: data.turbostat = m.group('t') - # battery data - if len(self.battery) > data.testnumber: - m = re.match(self.batteryfmt, self.battery[data.testnumber]) - if m: - data.battery = m.groups() # wifi data if len(self.wifi) > data.testnumber: m = re.match(self.wififmt, self.wifi[data.testnumber]) if m: - data.wifi = m.group('w') + data.wifi = {'dev': m.group('d'), 'stat': m.group('s'), + 'time': float(m.group('t'))} + data.stamp['wifi'] = m.group('d') # sleep mode enter errors if len(self.testerror) > data.testnumber: m = re.match(self.testerrfmt, self.testerror[data.testnumber]) @@ -3012,13 +3091,13 @@ def appendIncompleteTraceLog(testruns): if(t.startMarker()): data = testrun[testidx].data tp.parseStamp(data, sysvals) - data.setStart(t.time) + data.setStart(t.time, t.name) continue if(not data): continue # find the end of resume if(t.endMarker()): - data.setEnd(t.time) + data.setEnd(t.time, t.name) testidx += 1 if(testidx >= testcnt): break @@ -3081,7 +3160,7 @@ def parseTraceLog(live=False): doError('%s does not exist' % sysvals.ftracefile) if not live: sysvals.setupAllKprobes() - ksuscalls = ['pm_prepare_console'] + ksuscalls = ['ksys_sync', 'pm_prepare_console'] krescalls = ['pm_restore_console'] tracewatch = ['irq_wakeup'] if sysvals.usekprobes: @@ -3094,7 +3173,7 @@ def parseTraceLog(live=False): testruns = [] testdata = [] testrun = 0 - data = 0 + data, limbo = 0, True tf = sysvals.openlog(sysvals.ftracefile, 'r') phase = 'suspend_prepare' for line in tf: @@ -3141,16 +3220,16 @@ def parseTraceLog(live=False): continue # find the start of suspend if(t.startMarker()): - data = Data(len(testdata)) + data, limbo = Data(len(testdata)), False testdata.append(data) testrun = TestRun(data) testruns.append(testrun) tp.parseStamp(data, sysvals) - data.setStart(t.time) + data.setStart(t.time, t.name) data.first_suspend_prepare = True phase = data.setPhase('suspend_prepare', t.time, True) continue - if(not data): + if(not data or limbo): continue # process cpu exec line if t.type == 'tracing_mark_write': @@ -3167,14 +3246,16 @@ def parseTraceLog(live=False): continue # find the end of resume if(t.endMarker()): - data.handleEndMarker(t.time) + if data.tKernRes == 0: + data.tKernRes = t.time + data.handleEndMarker(t.time, t.name) if(not sysvals.usetracemarkers): # no trace markers? then quit and be sure to finish recording # the event we used to trigger resume end if('thaw_processes' in testrun.ttemp and len(testrun.ttemp['thaw_processes']) > 0): # if an entry exists, assume this is its end testrun.ttemp['thaw_processes'][-1]['end'] = t.time - break + limbo = True continue # trace event processing if(t.fevent): @@ -3197,7 +3278,7 @@ def parseTraceLog(live=False): # -- phase changes -- # start of kernel suspend if(re.match('suspend_enter\[.*', t.name)): - if(isbegin): + if(isbegin and data.tKernSus == 0): data.tKernSus = t.time continue # suspend_prepare start @@ -3225,7 +3306,7 @@ def parseTraceLog(live=False): elif(re.match('machine_suspend\[.*', t.name)): if(isbegin): lp = data.lastPhase() - if lp == 'resume_machine': + if lp.startswith('resume_machine'): data.dmesg[lp]['end'] = t.time phase = data.setPhase('suspend_machine', data.dmesg[lp]['end'], True) data.setPhase(phase, t.time, False) @@ -3320,7 +3401,8 @@ def parseTraceLog(live=False): 'proc': m_proc, }) # start of kernel resume - if(phase == 'suspend_prepare' and kprobename in ksuscalls): + if(data.tKernSus == 0 and phase == 'suspend_prepare' \ + and kprobename in ksuscalls): data.tKernSus = t.time elif(t.freturn): if(key not in tp.ktemp) or len(tp.ktemp[key]) < 1: @@ -3355,7 +3437,7 @@ def parseTraceLog(live=False): sysvals.vprint('WARNING: ftrace start marker is missing') if data and not data.devicegroups: sysvals.vprint('WARNING: ftrace end marker is missing') - data.handleEndMarker(t.time) + data.handleEndMarker(t.time, t.name) if sysvals.suspendmode == 'command': for test in testruns: @@ -3476,6 +3558,10 @@ def parseTraceLog(live=False): data.fwValid = False sysvals.vprint('WARNING: phase "%s" is missing!' % p) lp = p + if not terr and 'dev' in data.wifi and data.wifi['stat'] == 'timeout': + terr = '%s%s failed in wifi_resume (%s %.0fs timeout)' % \ + (sysvals.suspendmode, tn, data.wifi['dev'], data.wifi['time']) + error.append(terr) if not terr and data.enterfail: pprint('test%s FAILED: enter %s failed with %s' % (tn, sysvals.suspendmode, data.enterfail)) terr = 'test%s failed to enter %s mode' % (tn, sysvals.suspendmode) @@ -3933,7 +4019,7 @@ def createHTMLSummarySimple(testruns, htmlfile, title): tAvg, tMin, tMax, tMed = [0.0, 0.0], [0.0, 0.0], [0.0, 0.0], [dict(), dict()] iMin, iMed, iMax = [0, 0], [0, 0], [0, 0] num = 0 - useturbo = False + useturbo = usewifi = False lastmode = '' cnt = dict() for data in sorted(testruns, key=lambda v:(v['mode'], v['host'], v['kernel'], v['time'])): @@ -3952,17 +4038,17 @@ def createHTMLSummarySimple(testruns, htmlfile, title): tAvg, tMin, tMax, tMed = [0.0, 0.0], [0.0, 0.0], [0.0, 0.0], [dict(), dict()] iMin, iMed, iMax = [0, 0], [0, 0], [0, 0] num = 0 - pkgpc10 = syslpi = '' + pkgpc10 = syslpi = wifi = '' if 'pkgpc10' in data and 'syslpi' in data: - pkgpc10 = data['pkgpc10'] - syslpi = data['syslpi'] - useturbo = True + pkgpc10, syslpi, useturbo = data['pkgpc10'], data['syslpi'], True + if 'wifi' in data: + wifi, usewifi = data['wifi'], True res = data['result'] tVal = [float(data['suspend']), float(data['resume'])] list[mode]['data'].append([data['host'], data['kernel'], data['time'], tVal[0], tVal[1], data['url'], res, data['issues'], data['sus_worst'], data['sus_worsttime'], - data['res_worst'], data['res_worsttime'], pkgpc10, syslpi]) + data['res_worst'], data['res_worsttime'], pkgpc10, syslpi, wifi]) idx = len(list[mode]['data']) - 1 if res.startswith('fail in'): res = 'fail' @@ -4002,7 +4088,12 @@ def createHTMLSummarySimple(testruns, htmlfile, title): td = '\t{0}\n' tdh = '\t{0}\n' tdlink = '\thtml\n' - colspan = '14' if useturbo else '12' + cols = 12 + if useturbo: + cols += 2 + if usewifi: + cols += 1 + colspan = '%d' % cols # table header html += '\n\n' + th.format('#') +\ @@ -4013,6 +4104,8 @@ def createHTMLSummarySimple(testruns, htmlfile, title): th.format('Worst Resume Device') + th.format('RD Time') if useturbo: html += th.format('PkgPC10') + th.format('SysLPI') + if usewifi: + html += th.format('Wifi') html += th.format('Detail')+'\n' # export list into html head = ''+\ @@ -4076,6 +4169,8 @@ def createHTMLSummarySimple(testruns, htmlfile, title): if useturbo: html += td.format(d[12]) # pkg_pc10 html += td.format(d[13]) # syslpi + if usewifi: + html += td.format(d[14]) # wifi html += tdlink.format(d[5]) if d[5] else td.format('') # url html += '\n' num += 1 @@ -4224,6 +4319,8 @@ def createHTML(testruns, testfail): kerror = True if(sysvals.suspendmode in ['freeze', 'standby']): data.trimFreezeTime(testruns[-1].tSuspended) + else: + data.getMemTime() # html function templates html_error = '
{2}→
\n' @@ -4242,13 +4339,10 @@ def createHTML(testruns, testfail): ''\ ''\ '\n
{0}{1}
Execution Time: {0} msCommand: {1}
\n' - html_timegroups = '\n'\ - ''\ - ''\ - ''\ - ''\ - '\n
{4}Kernel Suspend: {0} ms{4}Firmware Suspend: {1} ms{4}Firmware Resume: {2} ms{4}Kernel Resume: {3} ms
\n' html_fail = '
{0}
\n' + html_kdesc = '{0}Kernel {2}: {1} ms' + html_fwdesc = '{0}Firmware {2}: {1} ms' + html_wifdesc = '{0}Wifi Resume: {1}' # html format variables scaleH = 20 @@ -4264,13 +4358,10 @@ def createHTML(testruns, testfail): # Generate the header for this timeline for data in testruns: tTotal = data.end - data.start - sktime, rktime = data.getTimeValues() if(tTotal == 0): doError('No timeline data') - if(len(data.tLow) > 0): - low_time = '+'.join(data.tLow) if sysvals.suspendmode == 'command': - run_time = '%.0f'%((data.end-data.start)*1000) + run_time = '%.0f' % (tTotal * 1000) if sysvals.testcommand: testdesc = sysvals.testcommand else: @@ -4279,43 +4370,55 @@ def createHTML(testruns, testfail): testdesc = ordinal(data.testnumber+1)+' '+testdesc thtml = html_timetotal3.format(run_time, testdesc) devtl.html += thtml - elif data.fwValid: - suspend_time = '%.0f'%(sktime + (data.fwSuspend/1000000.0)) - resume_time = '%.0f'%(rktime + (data.fwResume/1000000.0)) - testdesc1 = 'Total' - testdesc2 = '' - stitle = 'time from kernel enter_state(%s) to low-power mode [kernel & firmware time]' % sysvals.suspendmode - rtitle = 'time from low-power mode to return from kernel enter_state(%s) [firmware & kernel time]' % sysvals.suspendmode - if(len(testruns) > 1): - testdesc1 = testdesc2 = ordinal(data.testnumber+1) - testdesc2 += ' ' - if(len(data.tLow) == 0): - thtml = html_timetotal.format(suspend_time, \ - resume_time, testdesc1, stitle, rtitle) - else: - thtml = html_timetotal2.format(suspend_time, low_time, \ - resume_time, testdesc1, stitle, rtitle) - devtl.html += thtml + continue + # typical full suspend/resume header + stot, rtot = sktime, rktime = data.getTimeValues() + ssrc, rsrc, testdesc, testdesc2 = ['kernel'], ['kernel'], 'Kernel', '' + if data.fwValid: + stot += (data.fwSuspend/1000000.0) + rtot += (data.fwResume/1000000.0) + ssrc.append('firmware') + rsrc.append('firmware') + testdesc = 'Total' + if 'time' in data.wifi and data.wifi['stat'] != 'timeout': + rtot += data.end - data.tKernRes + (data.wifi['time'] * 1000.0) + rsrc.append('wifi') + testdesc = 'Total' + suspend_time, resume_time = '%.3f' % stot, '%.3f' % rtot + stitle = 'time from kernel suspend start to %s mode [%s time]' % \ + (sysvals.suspendmode, ' & '.join(ssrc)) + rtitle = 'time from %s mode to kernel resume complete [%s time]' % \ + (sysvals.suspendmode, ' & '.join(rsrc)) + if(len(testruns) > 1): + testdesc = testdesc2 = ordinal(data.testnumber+1) + testdesc2 += ' ' + if(len(data.tLow) == 0): + thtml = html_timetotal.format(suspend_time, \ + resume_time, testdesc, stitle, rtitle) + else: + low_time = '+'.join(data.tLow) + thtml = html_timetotal2.format(suspend_time, low_time, \ + resume_time, testdesc, stitle, rtitle) + devtl.html += thtml + if not data.fwValid and 'dev' not in data.wifi: + continue + # extra detail when the times come from multiple sources + thtml = '\n' + thtml += html_kdesc.format(testdesc2, '%.3f'%sktime, 'Suspend', 'green') + if data.fwValid: sftime = '%.3f'%(data.fwSuspend / 1000000.0) rftime = '%.3f'%(data.fwResume / 1000000.0) - devtl.html += html_timegroups.format('%.3f'%sktime, \ - sftime, rftime, '%.3f'%rktime, testdesc2, sysvals.suspendmode) - else: - suspend_time = '%.3f' % sktime - resume_time = '%.3f' % rktime - testdesc = 'Kernel' - stitle = 'time from kernel enter_state(%s) to firmware mode [kernel time only]' % sysvals.suspendmode - rtitle = 'time from firmware mode to return from kernel enter_state(%s) [kernel time only]' % sysvals.suspendmode - if(len(testruns) > 1): - testdesc = ordinal(data.testnumber+1)+' '+testdesc - if(len(data.tLow) == 0): - thtml = html_timetotal.format(suspend_time, \ - resume_time, testdesc, stitle, rtitle) + thtml += html_fwdesc.format(testdesc2, sftime, 'Suspend', 'green') + thtml += html_fwdesc.format(testdesc2, rftime, 'Resume', 'yellow') + thtml += html_kdesc.format(testdesc2, '%.3f'%rktime, 'Resume', 'yellow') + if 'time' in data.wifi: + if data.wifi['stat'] != 'timeout': + wtime = '%.0f ms'%(data.end - data.tKernRes + (data.wifi['time'] * 1000.0)) else: - thtml = html_timetotal2.format(suspend_time, low_time, \ - resume_time, testdesc, stitle, rtitle) - devtl.html += thtml - + wtime = 'TIMEOUT' + thtml += html_wifdesc.format(testdesc2, wtime, data.wifi['dev']) + thtml += '\n
\n' + devtl.html += thtml if testfail: devtl.html += html_fail.format(testfail) @@ -5082,28 +5185,32 @@ def setRuntimeSuspend(before=True): # Description: # Execute system suspend through the sysfs interface, then copy the output # dmesg and ftrace files to the test output directory. -def executeSuspend(): +def executeSuspend(quiet=False): pm = ProcessMonitor() tp = sysvals.tpath - wifi = sysvals.checkWifi() + if sysvals.wifi: + wifi = sysvals.checkWifi() testdata = [] - battery = True if getBattery() else False # run these commands to prepare the system for suspend if sysvals.display: - pprint('SET DISPLAY TO %s' % sysvals.display.upper()) + if not quiet: + pprint('SET DISPLAY TO %s' % sysvals.display.upper()) displayControl(sysvals.display) time.sleep(1) if sysvals.sync: - pprint('SYNCING FILESYSTEMS') + if not quiet: + pprint('SYNCING FILESYSTEMS') call('sync', shell=True) # mark the start point in the kernel ring buffer just as we start sysvals.initdmesg() # start ftrace if(sysvals.usecallgraph or sysvals.usetraceevents): - pprint('START TRACING') + if not quiet: + pprint('START TRACING') sysvals.fsetVal('1', 'tracing_on') if sysvals.useprocmon: pm.start() + sysvals.cmdinfo(True) # execute however many s/r runs requested for count in range(1,sysvals.execcount+1): # x2delay in between test runs @@ -5119,15 +5226,14 @@ def executeSuspend(): pprint('SUSPEND START') else: pprint('SUSPEND START (press a key to resume)') - sysvals.mcelog(True) - bat1 = getBattery() if battery else False # set rtcwake if(sysvals.rtcwake): - pprint('will issue an rtcwake in %d seconds' % sysvals.rtcwaketime) + if not quiet: + pprint('will issue an rtcwake in %d seconds' % sysvals.rtcwaketime) sysvals.rtcWakeAlarmOn() # start of suspend trace marker if(sysvals.usecallgraph or sysvals.usetraceevents): - sysvals.fsetVal('SUSPEND START', 'trace_marker') + sysvals.fsetVal(datetime.now().strftime(sysvals.tmstart), 'trace_marker') # predelay delay if(count == 1 and sysvals.predelay > 0): sysvals.fsetVal('WAIT %d' % sysvals.predelay, 'trace_marker') @@ -5174,37 +5280,33 @@ def executeSuspend(): # return from suspend pprint('RESUME COMPLETE') if(sysvals.usecallgraph or sysvals.usetraceevents): - sysvals.fsetVal('RESUME COMPLETE', 'trace_marker') + sysvals.fsetVal(datetime.now().strftime(sysvals.tmend), 'trace_marker') + if sysvals.wifi and wifi: + tdata['wifi'] = sysvals.pollWifi(wifi) if(sysvals.suspendmode == 'mem' or sysvals.suspendmode == 'command'): tdata['fw'] = getFPDT(False) - mcelog = sysvals.mcelog() - if mcelog: - tdata['mcelog'] = mcelog - bat2 = getBattery() if battery else False - if battery and bat1 and bat2: - tdata['bat'] = (bat1, bat2) - if 'device' in wifi and 'ip' in wifi: - tdata['wifi'] = (wifi, sysvals.checkWifi()) testdata.append(tdata) + cmdafter = sysvals.cmdinfo(False) # stop ftrace if(sysvals.usecallgraph or sysvals.usetraceevents): if sysvals.useprocmon: pm.stop() sysvals.fsetVal('0', 'tracing_on') # grab a copy of the dmesg output - pprint('CAPTURING DMESG') + if not quiet: + pprint('CAPTURING DMESG') sysvals.getdmesg(testdata) # grab a copy of the ftrace output if(sysvals.usecallgraph or sysvals.usetraceevents): - pprint('CAPTURING TRACE') + if not quiet: + pprint('CAPTURING TRACE') op = sysvals.writeDatafileHeader(sysvals.ftracefile, testdata) fp = open(tp+'trace', 'r') for line in fp: op.write(line) op.close() sysvals.fsetVal('', 'trace') - sysvals.platforminfo() - return testdata + sysvals.platforminfo(cmdafter) def readFile(file): if os.path.islink(file): @@ -5447,25 +5549,6 @@ def dmidecode(mempath, fatal=False): count += 1 return out -def getBattery(): - p, charge, bat = '/sys/class/power_supply', 0, {} - if not os.path.exists(p): - return False - for d in os.listdir(p): - type = sysvals.getVal(os.path.join(p, d, 'type')).strip().lower() - if type != 'battery': - continue - for v in ['status', 'energy_now', 'capacity_now']: - bat[v] = sysvals.getVal(os.path.join(p, d, v)).strip().lower() - break - if 'status' not in bat: - return False - ac = False if 'discharging' in bat['status'] else True - for v in ['energy_now', 'capacity_now']: - if v in bat and bat[v]: - charge = int(bat[v]) - return (ac, charge) - def displayControl(cmd): xset, ret = 'timeout 10 xset -d :0.0 {0}', 0 if sysvals.sudouser: @@ -5715,6 +5798,17 @@ def statusCheck(probecheck=False): status = 'rtcwake is not properly supported' pprint(' is rtcwake supported: %s' % res) + # check info commands + pprint(' optional commands this tool may use for info:') + no = sysvals.colorText('MISSING') + yes = sysvals.colorText('FOUND', 32) + for c in ['turbostat', 'mcelog', 'lspci', 'lsusb']: + if c == 'turbostat': + res = yes if sysvals.haveTurbostat() else no + else: + res = yes if sysvals.getExec(c) else no + pprint(' %s: %s' % (c, res)) + if not probecheck: return status @@ -5780,8 +5874,9 @@ def getArgFloat(name, args, min, max, main=True): doError(name+': value should be between %f and %f' % (min, max), True) return val -def processData(live=False): - pprint('PROCESSING DATA') +def processData(live=False, quiet=False): + if not quiet: + pprint('PROCESSING DATA') sysvals.vprint('usetraceevents=%s, usetracemarkers=%s, usekprobes=%s' % \ (sysvals.usetraceevents, sysvals.usetracemarkers, sysvals.usekprobes)) error = '' @@ -5796,20 +5891,17 @@ def processData(live=False): parseKernelLog(data) if(sysvals.ftracefile and (sysvals.usecallgraph or sysvals.usetraceevents)): appendIncompleteTraceLog(testruns) + if not sysvals.stamp: + pprint('ERROR: data does not include the expected stamp') + return (testruns, {'error': 'timeline generation failed'}) shown = ['bios', 'biosdate', 'cpu', 'host', 'kernel', 'man', 'memfr', - 'memsz', 'mode', 'numcpu', 'plat', 'time'] + 'memsz', 'mode', 'numcpu', 'plat', 'time', 'wifi'] sysvals.vprint('System Info:') for key in sorted(sysvals.stamp): if key in shown: sysvals.vprint(' %-8s : %s' % (key.upper(), sysvals.stamp[key])) - if sysvals.kparams: - sysvals.vprint('Kparams:\n %s' % sysvals.kparams) sysvals.vprint('Command:\n %s' % sysvals.cmdline) for data in testruns: - if data.mcelog: - sysvals.vprint('MCELOG Data:') - for line in data.mcelog.split('\n'): - sysvals.vprint(' %s' % line) if data.turbostat: idx, s = 0, 'Turbostat:\n ' for val in data.turbostat.split('|'): @@ -5819,23 +5911,13 @@ def processData(live=False): s += '\n ' s += val + ' ' sysvals.vprint(s) - if data.battery: - a1, c1, a2, c2 = data.battery - s = 'Battery:\n Before - AC: %s, Charge: %d\n After - AC: %s, Charge: %d' % \ - (a1, int(c1), a2, int(c2)) - sysvals.vprint(s) - if data.wifi: - w = data.wifi.replace('|', ' ').split(',') - s = 'Wifi:\n Before %s\n After %s' % \ - (w[0], w[1]) - sysvals.vprint(s) data.printDetails() - if len(sysvals.platinfo) > 0: - sysvals.vprint('\nPlatform Info:') - for info in sysvals.platinfo: - sysvals.vprint(info[0]+' - '+info[1]) - sysvals.vprint(info[2]) - sysvals.vprint('') + if len(sysvals.platinfo) > 0: + sysvals.vprint('\nPlatform Info:') + for info in sysvals.platinfo: + sysvals.vprint('[%s - %s]' % (info[0], info[1])) + sysvals.vprint(info[2]) + sysvals.vprint('') if sysvals.cgdump: for data in testruns: data.debugPrint() @@ -5845,7 +5927,8 @@ def processData(live=False): return (testruns, {'error': 'timeline generation failed'}) sysvals.vprint('Creating the html timeline (%s)...' % sysvals.htmlfile) createHTML(testruns, error) - pprint('DONE') + if not quiet: + pprint('DONE') data = testruns[0] stamp = data.stamp stamp['suspend'], stamp['resume'] = data.getTimeValues() @@ -5872,31 +5955,28 @@ def rerunTest(htmlfile=''): doError('a directory already exists with this name: %s' % sysvals.htmlfile) elif not os.access(sysvals.htmlfile, os.W_OK): doError('missing permission to write to %s' % sysvals.htmlfile) - testruns, stamp = processData(False) - sysvals.logmsg = '' + testruns, stamp = processData() + sysvals.resetlog() return stamp # Function: runTest # Description: # execute a suspend/resume, gather the logs, and generate the output -def runTest(n=0): +def runTest(n=0, quiet=False): # prepare for the test - sysvals.initFtrace() + sysvals.initFtrace(quiet) sysvals.initTestOutput('suspend') # execute the test - testdata = executeSuspend() + executeSuspend(quiet) sysvals.cleanupFtrace() if sysvals.skiphtml: + sysvals.outputResult({}, n) sysvals.sudoUserchown(sysvals.testdir) return - if not testdata[0]['error']: - testruns, stamp = processData(True) - for data in testruns: - del data - else: - stamp = testdata[0] - + testruns, stamp = processData(True, quiet) + for data in testruns: + del data sysvals.sudoUserchown(sysvals.testdir) sysvals.outputResult(stamp, n) if 'error' in stamp: @@ -5904,13 +5984,14 @@ def runTest(n=0): return 0 def find_in_html(html, start, end, firstonly=True): - n, out = 0, [] - while n < len(html): - m = re.search(start, html[n:]) + n, cnt, out = 0, len(html), [] + while n < cnt: + e = cnt if (n + 10000 > cnt or n == 0) else n + 10000 + m = re.search(start, html[n:e]) if not m: break i = m.end() - m = re.search(end, html[n+i:]) + m = re.search(end, html[n+i:e]) if not m: break j = m.start() @@ -5945,7 +6026,7 @@ def data_from_html(file, outpath, issues, fulldetail=False): tstr = dt.strftime('%Y/%m/%d %H:%M:%S') error = find_in_html(html, '') if error: - m = re.match('[a-z]* failed in (?P

[a-z0-9_]*) phase', error) + m = re.match('[a-z0-9]* failed in (?P

\S*).*', error) if m: result = 'fail in %s' % m.group('p') else: @@ -5974,6 +6055,9 @@ def data_from_html(file, outpath, issues, fulldetail=False): elist[err[0]] += 1 for i in elist: ilist.append('%sx%d' % (i, elist[i]) if elist[i] > 1 else i) + wifi = find_in_html(html, 'Wifi Resume: ', '') + if wifi: + extra['wifi'] = wifi low = find_in_html(html, 'freeze time: ', ' ms') if low and '|' in low: issue = 'FREEZEx%d' % len(low.split('|')) @@ -6048,13 +6132,15 @@ def genHtml(subdir, force=False): for dirname, dirnames, filenames in os.walk(subdir): sysvals.dmesgfile = sysvals.ftracefile = sysvals.htmlfile = '' for filename in filenames: - if(re.match('.*_dmesg.txt', filename)): - sysvals.dmesgfile = os.path.join(dirname, filename) - elif(re.match('.*_ftrace.txt', filename)): - sysvals.ftracefile = os.path.join(dirname, filename) + file = os.path.join(dirname, filename) + if sysvals.usable(file): + if(re.match('.*_dmesg.txt', filename)): + sysvals.dmesgfile = file + elif(re.match('.*_ftrace.txt', filename)): + sysvals.ftracefile = file sysvals.setOutputFile() - if sysvals.ftracefile and sysvals.htmlfile and \ - (force or not os.path.exists(sysvals.htmlfile)): + if (sysvals.dmesgfile or sysvals.ftracefile) and sysvals.htmlfile and \ + (force or not sysvals.usable(sysvals.htmlfile)): pprint('FTRACE: %s' % sysvals.ftracefile) if sysvals.dmesgfile: pprint('DMESG : %s' % sysvals.dmesgfile) @@ -6169,9 +6255,9 @@ def configFromFile(file): sysvals.cgtest = getArgInt('cgtest', value, 0, 1, False) elif(option == 'cgphase'): d = Data(0) - if value not in d.sortedPhases(): + if value not in d.phasedef: doError('invalid phase --> (%s: %s), valid phases are %s'\ - % (option, value, d.sortedPhases()), True) + % (option, value, d.phasedef.keys()), True) sysvals.cgphase = value elif(option == 'fadd'): file = sysvals.configFile(value) @@ -6184,9 +6270,7 @@ def configFromFile(file): nums = value.split() if len(nums) != 2: doError('multi requires 2 integers (exec_count and delay)', True) - sysvals.multitest['run'] = True - sysvals.multitest['count'] = getArgInt('multi: n d (exec count)', nums[0], 2, 1000000, False) - sysvals.multitest['delay'] = getArgInt('multi: n d (delay between tests)', nums[1], 0, 3600, False) + sysvals.multiinit(nums[0], nums[1]) elif(option == 'devicefilter'): sysvals.setDeviceFilter(value) elif(option == 'expandcg'): @@ -6342,6 +6426,7 @@ def printHelp(): ' -srgap Add a visible gap in the timeline between sus/res (default: disabled)\n'\ ' -skiphtml Run the test and capture the trace logs, but skip the timeline (default: disabled)\n'\ ' -result fn Export a results table to a text file for parsing.\n'\ + ' -wifi If a wifi connection is available, check that it reconnects after resume.\n'\ ' [testprep]\n'\ ' -sync Sync the filesystems before starting the test\n'\ ' -rs on/off Enable/disable runtime suspend for all devices, restore all after test\n'\ @@ -6356,8 +6441,10 @@ def printHelp(): ' -predelay t Include t ms delay before 1st suspend (default: 0 ms)\n'\ ' -postdelay t Include t ms delay after last resume (default: 0 ms)\n'\ ' -mindev ms Discard all device blocks shorter than ms milliseconds (e.g. 0.001 for us)\n'\ - ' -multi n d Execute consecutive tests at seconds intervals. The outputs will\n'\ - ' be created in a new subdirectory with a summary page.\n'\ + ' -multi n d Execute consecutive tests at seconds intervals. If is followed\n'\ + ' by a "d", "h", or "m" execute for days, hours, or mins instead.\n'\ + ' The outputs will be created in a new subdirectory with a summary page.\n'\ + ' -maxfail n Abort a -multi run after n consecutive fails (default is 0 = never abort)\n'\ ' [debug]\n'\ ' -f Use ftrace to create device callgraphs (default: disabled)\n'\ ' -ftop Use ftrace on the top level call: "%s" (default: disabled)\n'\ @@ -6379,11 +6466,11 @@ def printHelp(): ' -modes List available suspend modes\n'\ ' -status Test to see if the system is enabled to run this tool\n'\ ' -fpdt Print out the contents of the ACPI Firmware Performance Data Table\n'\ - ' -battery Print out battery info (if available)\n'\ - ' -wifi Print out wifi connection info (if wireless-tools and device exists)\n'\ + ' -wificheck Print out wifi connection info\n'\ ' -x Test xset by toggling the given mode (on/off/standby/suspend)\n'\ ' -sysinfo Print out system info extracted from BIOS\n'\ ' -devinfo Print out the pm settings of all devices which support runtime suspend\n'\ + ' -cmdinfo Print out all the platform info collected before and after suspend/resume\n'\ ' -flist Print the list of functions currently being captured in ftrace\n'\ ' -flistall Print all functions capable of being captured in ftrace\n'\ ' -summary dir Create a summary of tests in this dir [-genhtml builds missing html]\n'\ @@ -6399,8 +6486,8 @@ def printHelp(): genhtml = False cmd = '' simplecmds = ['-sysinfo', '-modes', '-fpdt', '-flist', '-flistall', - '-devinfo', '-status', '-battery', '-xon', '-xoff', '-xstandby', - '-xsuspend', '-xinit', '-xreset', '-xstat', '-wifi'] + '-devinfo', '-status', '-xon', '-xoff', '-xstandby', '-xsuspend', + '-xinit', '-xreset', '-xstat', '-wificheck', '-cmdinfo'] if '-f' in sys.argv: sysvals.cgskip = sysvals.configFile('cgskip.txt') # loop through the command line arguments @@ -6462,8 +6549,15 @@ def printHelp(): sysvals.usedevsrc = True elif(arg == '-sync'): sysvals.sync = True + elif(arg == '-wifi'): + sysvals.wifi = True elif(arg == '-gzip'): sysvals.gzip = True + elif(arg == '-info'): + try: + val = next(args) + except: + doError('-info requires one string argument', True) elif(arg == '-rs'): try: val = next(args) @@ -6555,10 +6649,14 @@ def printHelp(): sysvals.cgexp = True elif(arg == '-srgap'): sysvals.srgap = 5 + elif(arg == '-maxfail'): + sysvals.maxfail = getArgInt('-maxfail', args, 0, 1000000) elif(arg == '-multi'): - sysvals.multitest['run'] = True - sysvals.multitest['count'] = getArgInt('-multi n d (exec count)', args, 2, 1000000) - sysvals.multitest['delay'] = getArgInt('-multi n d (delay between tests)', args, 0, 3600) + try: + c, d = next(args), next(args) + except: + doError('-multi requires two values', True) + sysvals.multiinit(c, d) elif(arg == '-o'): try: val = next(args) @@ -6655,13 +6753,6 @@ def printHelp(): elif(cmd == 'fpdt'): if not getFPDT(True): ret = 1 - elif(cmd == 'battery'): - out = getBattery() - if out: - pprint('AC Connect : %s\nBattery Charge: %d' % out) - else: - pprint('no battery found') - ret = 1 elif(cmd == 'sysinfo'): sysvals.printSystemInfo(True) elif(cmd == 'devinfo'): @@ -6679,13 +6770,15 @@ def printHelp(): ret = displayControl(cmd[1:]) elif(cmd == 'xstat'): pprint('Display Status: %s' % displayControl('stat').upper()) - elif(cmd == 'wifi'): - out = sysvals.checkWifi() - if 'device' not in out: - pprint('WIFI interface not found') + elif(cmd == 'wificheck'): + dev = sysvals.checkWifi() + if dev: + print('%s is connected' % sysvals.wifiDetails(dev)) else: - for key in sorted(out): - pprint('%6s: %s' % (key.upper(), out[key])) + print('No wifi connection found') + elif(cmd == 'cmdinfo'): + for out in sysvals.cmdinfo(False, True): + print('[%s - %s]\n%s\n' % out) sys.exit(ret) # if instructed, re-analyze existing data files @@ -6720,24 +6813,38 @@ def printHelp(): setRuntimeSuspend(True) if sysvals.display: displayControl('init') - ret = 0 + failcnt, ret = 0, 0 if sysvals.multitest['run']: # run multiple tests in a separate subdirectory if not sysvals.outdir: - s = 'suspend-x%d' % sysvals.multitest['count'] - sysvals.outdir = datetime.now().strftime(s+'-%y%m%d-%H%M%S') + if 'time' in sysvals.multitest: + s = '-%dm' % sysvals.multitest['time'] + else: + s = '-x%d' % sysvals.multitest['count'] + sysvals.outdir = datetime.now().strftime('suspend-%y%m%d-%H%M%S'+s) if not os.path.isdir(sysvals.outdir): os.makedirs(sysvals.outdir) + sysvals.sudoUserchown(sysvals.outdir) + finish = datetime.now() + if 'time' in sysvals.multitest: + finish += timedelta(minutes=sysvals.multitest['time']) for i in range(sysvals.multitest['count']): - if(i != 0): + sysvals.multistat(True, i, finish) + if i != 0 and sysvals.multitest['delay'] > 0: pprint('Waiting %d seconds...' % (sysvals.multitest['delay'])) time.sleep(sysvals.multitest['delay']) - pprint('TEST (%d/%d) START' % (i+1, sysvals.multitest['count'])) fmt = 'suspend-%y%m%d-%H%M%S' sysvals.testdir = os.path.join(sysvals.outdir, datetime.now().strftime(fmt)) - ret = runTest(i+1) - pprint('TEST (%d/%d) COMPLETE' % (i+1, sysvals.multitest['count'])) - sysvals.logmsg = '' + ret = runTest(i+1, True) + failcnt = 0 if not ret else failcnt + 1 + if sysvals.maxfail > 0 and failcnt >= sysvals.maxfail: + pprint('Maximum fail count of %d reached, aborting multitest' % (sysvals.maxfail)) + break + time.sleep(5) + sysvals.resetlog() + sysvals.multistat(False, i, finish) + if 'time' in sysvals.multitest and datetime.now() >= finish: + break if not sysvals.skiphtml: runSummary(sysvals.outdir, False, False) sysvals.sudoUserchown(sysvals.outdir) From 2b6022a32e1c6e740a5929951d9d8cbed3b162b1 Mon Sep 17 00:00:00 2001 From: Hans de Goede Date: Sun, 19 Apr 2020 17:16:13 +0200 Subject: [PATCH 05/18] ACPI: button: Drop no longer necessary Asus T200TA lid_init_state quirk Commit 17e5888e4e18 ("x86: Select HARDIRQS_SW_RESEND on x86") fixes the edge-triggered embedded-controller (WC) IRQ not being replayed after resume when woken by opening the lid, which gets signaled by the EC. This means that the lid_init_state=ACPI_BUTTON_LID_INIT_OPEN quirk for the Asus T200TA is no longer necessary, the lid now works properly without it, so drop the quirk. Fixes: 17e5888e4e18 ("x86: Select HARDIRQS_SW_RESEND on x86") Signed-off-by: Hans de Goede Signed-off-by: Rafael J. Wysocki --- drivers/acpi/button.c | 12 ------------ 1 file changed, 12 deletions(-) diff --git a/drivers/acpi/button.c b/drivers/acpi/button.c index 00112cf1532203..78cfc70cb3209c 100644 --- a/drivers/acpi/button.c +++ b/drivers/acpi/button.c @@ -87,18 +87,6 @@ static const struct dmi_system_id dmi_lid_quirks[] = { }, .driver_data = (void *)(long)ACPI_BUTTON_LID_INIT_DISABLED, }, - { - /* - * Asus T200TA, _LID keeps reporting closed after every second - * openening of the lid. Causing immediate re-suspend after - * opening every other open. Using LID_INIT_OPEN fixes this. - */ - .matches = { - DMI_MATCH(DMI_SYS_VENDOR, "ASUSTeK COMPUTER INC."), - DMI_MATCH(DMI_PRODUCT_NAME, "T200TA"), - }, - .driver_data = (void *)(long)ACPI_BUTTON_LID_INIT_OPEN, - }, { /* GP-electronic T701, _LID method points to a floating GPIO */ .matches = { From b849dd84b6ccfe32622988b79b7b073861fcf9f7 Mon Sep 17 00:00:00 2001 From: Douglas Anderson Date: Tue, 24 Mar 2020 14:48:27 -0700 Subject: [PATCH 06/18] bdev: Reduce time holding bd_mutex in sync in blkdev_close() While trying to "dd" to the block device for a USB stick, I encountered a hung task warning (blocked for > 120 seconds). I managed to come up with an easy way to reproduce this on my system (where /dev/sdb is the block device for my USB stick) with: while true; do dd if=/dev/zero of=/dev/sdb bs=4M; done With my reproduction here are the relevant bits from the hung task detector: INFO: task udevd:294 blocked for more than 122 seconds. ... udevd D 0 294 1 0x00400008 Call trace: ... mutex_lock_nested+0x40/0x50 __blkdev_get+0x7c/0x3d4 blkdev_get+0x118/0x138 blkdev_open+0x94/0xa8 do_dentry_open+0x268/0x3a0 vfs_open+0x34/0x40 path_openat+0x39c/0xdf4 do_filp_open+0x90/0x10c do_sys_open+0x150/0x3c8 ... ... Showing all locks held in the system: ... 1 lock held by dd/2798: #0: ffffff814ac1a3b8 (&bdev->bd_mutex){+.+.}, at: __blkdev_put+0x50/0x204 ... dd D 0 2798 2764 0x00400208 Call trace: ... schedule+0x8c/0xbc io_schedule+0x1c/0x40 wait_on_page_bit_common+0x238/0x338 __lock_page+0x5c/0x68 write_cache_pages+0x194/0x500 generic_writepages+0x64/0xa4 blkdev_writepages+0x24/0x30 do_writepages+0x48/0xa8 __filemap_fdatawrite_range+0xac/0xd8 filemap_write_and_wait+0x30/0x84 __blkdev_put+0x88/0x204 blkdev_put+0xc4/0xe4 blkdev_close+0x28/0x38 __fput+0xe0/0x238 ____fput+0x1c/0x28 task_work_run+0xb0/0xe4 do_notify_resume+0xfc0/0x14bc work_pending+0x8/0x14 The problem appears related to the fact that my USB disk is terribly slow and that I have a lot of RAM in my system to cache things. Specifically my writes seem to be happening at ~15 MB/s and I've got ~4 GB of RAM in my system that can be used for buffering. To write 4 GB of buffer to disk thus takes ~4000 MB / ~15 MB/s = ~267 seconds. The 267 second number is a problem because in __blkdev_put() we call sync_blockdev() while holding the bd_mutex. Any other callers who want the bd_mutex will be blocked for the whole time. The problem is made worse because I believe blkdev_put() specifically tells other tasks (namely udev) to go try to access the device at right around the same time we're going to hold the mutex for a long time. Putting some traces around this (after disabling the hung task detector), I could confirm: dd: 437.608600: __blkdev_put() right before sync_blockdev() for sdb udevd: 437.623901: blkdev_open() right before blkdev_get() for sdb dd: 661.468451: __blkdev_put() right after sync_blockdev() for sdb udevd: 663.820426: blkdev_open() right after blkdev_get() for sdb A simple fix for this is to realize that sync_blockdev() works fine if you're not holding the mutex. Also, it's not the end of the world if you sync a little early (though it can have performance impacts). Thus we can make a guess that we're going to need to do the sync and then do it without holding the mutex. We still do one last sync with the mutex but it should be much, much faster. With this, my hung task warnings for my test case are gone. Signed-off-by: Douglas Anderson Reviewed-by: Guenter Roeck Reviewed-by: Christoph Hellwig Signed-off-by: Jens Axboe --- fs/block_dev.c | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/fs/block_dev.c b/fs/block_dev.c index 52b6f646cdbd0a..59435f07911a8a 100644 --- a/fs/block_dev.c +++ b/fs/block_dev.c @@ -1893,6 +1893,16 @@ static void __blkdev_put(struct block_device *bdev, fmode_t mode, int for_part) struct gendisk *disk = bdev->bd_disk; struct block_device *victim = NULL; + /* + * Sync early if it looks like we're the last one. If someone else + * opens the block device between now and the decrement of bd_openers + * then we did a sync that we didn't need to, but that's not the end + * of the world and we want to avoid long (could be several minute) + * syncs while holding the mutex. + */ + if (bdev->bd_openers == 1) + sync_blockdev(bdev); + mutex_lock_nested(&bdev->bd_mutex, for_part); if (for_part) bdev->bd_part_count--; From d6c8e949a35d6906d6c03a50e9a9cdf4e494528a Mon Sep 17 00:00:00 2001 From: Waiman Long Date: Tue, 21 Apr 2020 09:07:55 -0400 Subject: [PATCH 07/18] blk-iocost: Fix error on iocost_ioc_vrate_adj MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Systemtap 4.2 is unable to correctly interpret the "u32 (*missed_ppm)[2]" argument of the iocost_ioc_vrate_adj trace entry defined in include/trace/events/iocost.h leading to the following error: /tmp/stapAcz0G0/stap_c89c58b83cea1724e26395efa9ed4939_6321_aux_6.c:78:8: error: expected ‘;’, ‘,’ or ‘)’ before ‘*’ token , u32[]* __tracepoint_arg_missed_ppm That argument type is indeed rather complex and hard to read. Looking at block/blk-iocost.c. It is just a 2-entry u32 array. By simplifying the argument to a simple "u32 *missed_ppm" and adjusting the trace entry accordingly, the compilation error was gone. Fixes: 7caa47151ab2 ("blkcg: implement blk-iocost") Acked-by: Steven Rostedt (VMware) Acked-by: Tejun Heo Signed-off-by: Waiman Long Signed-off-by: Jens Axboe --- block/blk-iocost.c | 4 ++-- include/trace/events/iocost.h | 6 +++--- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/block/blk-iocost.c b/block/blk-iocost.c index db35ee682294c1..3ab0c1c704b68f 100644 --- a/block/blk-iocost.c +++ b/block/blk-iocost.c @@ -1591,7 +1591,7 @@ static void ioc_timer_fn(struct timer_list *timer) vrate_min, vrate_max); } - trace_iocost_ioc_vrate_adj(ioc, vrate, &missed_ppm, rq_wait_pct, + trace_iocost_ioc_vrate_adj(ioc, vrate, missed_ppm, rq_wait_pct, nr_lagging, nr_shortages, nr_surpluses); @@ -1600,7 +1600,7 @@ static void ioc_timer_fn(struct timer_list *timer) ioc->period_us * vrate * INUSE_MARGIN_PCT, 100); } else if (ioc->busy_level != prev_busy_level || nr_lagging) { trace_iocost_ioc_vrate_adj(ioc, atomic64_read(&ioc->vtime_rate), - &missed_ppm, rq_wait_pct, nr_lagging, + missed_ppm, rq_wait_pct, nr_lagging, nr_shortages, nr_surpluses); } diff --git a/include/trace/events/iocost.h b/include/trace/events/iocost.h index 7ecaa65b7106eb..c2f580fd371b11 100644 --- a/include/trace/events/iocost.h +++ b/include/trace/events/iocost.h @@ -130,7 +130,7 @@ DEFINE_EVENT(iocg_inuse_update, iocost_inuse_reset, TRACE_EVENT(iocost_ioc_vrate_adj, - TP_PROTO(struct ioc *ioc, u64 new_vrate, u32 (*missed_ppm)[2], + TP_PROTO(struct ioc *ioc, u64 new_vrate, u32 *missed_ppm, u32 rq_wait_pct, int nr_lagging, int nr_shortages, int nr_surpluses), @@ -155,8 +155,8 @@ TRACE_EVENT(iocost_ioc_vrate_adj, __entry->old_vrate = atomic64_read(&ioc->vtime_rate);; __entry->new_vrate = new_vrate; __entry->busy_level = ioc->busy_level; - __entry->read_missed_ppm = (*missed_ppm)[READ]; - __entry->write_missed_ppm = (*missed_ppm)[WRITE]; + __entry->read_missed_ppm = missed_ppm[READ]; + __entry->write_missed_ppm = missed_ppm[WRITE]; __entry->rq_wait_pct = rq_wait_pct; __entry->nr_lagging = nr_lagging; __entry->nr_shortages = nr_shortages; From d56deb1e4e6696b2c70815beb8d1d39fc0063b1c Mon Sep 17 00:00:00 2001 From: "Ma, Jianpeng" Date: Tue, 21 Apr 2020 11:48:08 +0000 Subject: [PATCH 08/18] block: remove unused header Dax related code already removed from this file. Reviewed-by: Christoph Hellwig Signed-off-by: Jianpeng Ma Signed-off-by: Jens Axboe --- fs/block_dev.c | 1 - 1 file changed, 1 deletion(-) diff --git a/fs/block_dev.c b/fs/block_dev.c index 59435f07911a8a..93672c3f1c78c8 100644 --- a/fs/block_dev.c +++ b/fs/block_dev.c @@ -19,7 +19,6 @@ #include #include #include -#include #include #include #include From 01b2bafe57b19d9119413f138765ef57990921ce Mon Sep 17 00:00:00 2001 From: Jason Gunthorpe Date: Tue, 14 Apr 2020 12:10:50 -0300 Subject: [PATCH 09/18] pnp: Use list_for_each_entry() instead of open coding MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Aside from good practice, this avoids a warning from gcc 10: ./include/linux/kernel.h:997:3: warning: array subscript -31 is outside array bounds of ‘struct list_head[1]’ [-Warray-bounds] 997 | ((type *)(__mptr - offsetof(type, member))); }) | ~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ./include/linux/list.h:493:2: note: in expansion of macro ‘container_of’ 493 | container_of(ptr, type, member) | ^~~~~~~~~~~~ ./include/linux/pnp.h:275:30: note: in expansion of macro ‘list_entry’ 275 | #define global_to_pnp_dev(n) list_entry(n, struct pnp_dev, global_list) | ^~~~~~~~~~ ./include/linux/pnp.h:281:11: note: in expansion of macro ‘global_to_pnp_dev’ 281 | (dev) != global_to_pnp_dev(&pnp_global); \ | ^~~~~~~~~~~~~~~~~ arch/x86/kernel/rtc.c:189:2: note: in expansion of macro ‘pnp_for_each_dev’ 189 | pnp_for_each_dev(dev) { Because the common code doesn't cast the starting list_head to the containing struct. Signed-off-by: Jason Gunthorpe [ rjw: Whitespace adjustments ] Signed-off-by: Rafael J. Wysocki --- include/linux/pnp.h | 29 +++++++++-------------------- 1 file changed, 9 insertions(+), 20 deletions(-) diff --git a/include/linux/pnp.h b/include/linux/pnp.h index b18dca67253d66..c2a7cfbca7131f 100644 --- a/include/linux/pnp.h +++ b/include/linux/pnp.h @@ -220,10 +220,8 @@ struct pnp_card { #define global_to_pnp_card(n) list_entry(n, struct pnp_card, global_list) #define protocol_to_pnp_card(n) list_entry(n, struct pnp_card, protocol_list) #define to_pnp_card(n) container_of(n, struct pnp_card, dev) -#define pnp_for_each_card(card) \ - for((card) = global_to_pnp_card(pnp_cards.next); \ - (card) != global_to_pnp_card(&pnp_cards); \ - (card) = global_to_pnp_card((card)->global_list.next)) +#define pnp_for_each_card(card) \ + list_for_each_entry(card, &pnp_cards, global_list) struct pnp_card_link { struct pnp_card *card; @@ -276,14 +274,9 @@ struct pnp_dev { #define card_to_pnp_dev(n) list_entry(n, struct pnp_dev, card_list) #define protocol_to_pnp_dev(n) list_entry(n, struct pnp_dev, protocol_list) #define to_pnp_dev(n) container_of(n, struct pnp_dev, dev) -#define pnp_for_each_dev(dev) \ - for((dev) = global_to_pnp_dev(pnp_global.next); \ - (dev) != global_to_pnp_dev(&pnp_global); \ - (dev) = global_to_pnp_dev((dev)->global_list.next)) -#define card_for_each_dev(card,dev) \ - for((dev) = card_to_pnp_dev((card)->devices.next); \ - (dev) != card_to_pnp_dev(&(card)->devices); \ - (dev) = card_to_pnp_dev((dev)->card_list.next)) +#define pnp_for_each_dev(dev) list_for_each_entry(dev, &pnp_global, global_list) +#define card_for_each_dev(card, dev) \ + list_for_each_entry(dev, &(card)->devices, card_list) #define pnp_dev_name(dev) (dev)->name static inline void *pnp_get_drvdata(struct pnp_dev *pdev) @@ -437,14 +430,10 @@ struct pnp_protocol { }; #define to_pnp_protocol(n) list_entry(n, struct pnp_protocol, protocol_list) -#define protocol_for_each_card(protocol,card) \ - for((card) = protocol_to_pnp_card((protocol)->cards.next); \ - (card) != protocol_to_pnp_card(&(protocol)->cards); \ - (card) = protocol_to_pnp_card((card)->protocol_list.next)) -#define protocol_for_each_dev(protocol,dev) \ - for((dev) = protocol_to_pnp_dev((protocol)->devices.next); \ - (dev) != protocol_to_pnp_dev(&(protocol)->devices); \ - (dev) = protocol_to_pnp_dev((dev)->protocol_list.next)) +#define protocol_for_each_card(protocol, card) \ + list_for_each_entry(card, &(protocol)->cards, protocol_list) +#define protocol_for_each_dev(protocol, dev) \ + list_for_each_entry(dev, &(protocol)->devices, protocol_list) extern struct bus_type pnp_bus_type; From 09beebd8f93b3c8bf894e342f0a203a5c612478c Mon Sep 17 00:00:00 2001 From: Kai-Heng Feng Date: Tue, 21 Apr 2020 16:21:55 +0800 Subject: [PATCH 10/18] PM: sleep: core: Switch back to async_schedule_dev() Commit 8b9ec6b73277 ("PM core: Use new async_schedule_dev command") introduced a new function for better performance. However commit f2a424f6c613 ("PM / core: Introduce dpm_async_fn() helper") went back to the non-optimized version, async_schedule(). So switch back to the sync_schedule_dev() to improve performance Fixes: f2a424f6c613 ("PM / core: Introduce dpm_async_fn() helper") Signed-off-by: Kai-Heng Feng Signed-off-by: Rafael J. Wysocki --- drivers/base/power/main.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/drivers/base/power/main.c b/drivers/base/power/main.c index fdd508a78ffd66..0e07e17c2def33 100644 --- a/drivers/base/power/main.c +++ b/drivers/base/power/main.c @@ -730,7 +730,7 @@ static bool dpm_async_fn(struct device *dev, async_func_t func) if (is_async(dev)) { get_device(dev); - async_schedule(func, dev); + async_schedule_dev(func, dev); return true; } From 9da73974eb9c965dd9989befb593b8c8da9e4bdc Mon Sep 17 00:00:00 2001 From: Vamshi K Sthambamkadi Date: Wed, 22 Apr 2020 11:45:06 +0530 Subject: [PATCH 11/18] tracing: Fix memory leaks in trace_events_hist.c kmemleak report 1: [<9092c50b>] kmem_cache_alloc_trace+0x138/0x270 [<05a2c9ed>] create_field_var+0xcf/0x180 [<528a2d68>] action_create+0xe2/0xc80 [<63f50b61>] event_hist_trigger_func+0x15b5/0x1920 [<28ea5d3d>] trigger_process_regex+0x7b/0xc0 [<3138e86f>] event_trigger_write+0x4d/0xb0 [] __vfs_write+0x30/0x200 [<4f424a0d>] vfs_write+0x96/0x1b0 [] ksys_write+0x53/0xc0 [<3717101a>] __ia32_sys_write+0x15/0x20 [] do_fast_syscall_32+0x70/0x250 [<46e2629c>] entry_SYSENTER_32+0xaf/0x102 This is because save_vars[] of struct hist_trigger_data are not destroyed kmemleak report 2: [<9092c50b>] kmem_cache_alloc_trace+0x138/0x270 [<6e5e97c5>] create_var+0x3c/0x110 [] create_field_var+0xaf/0x180 [<528a2d68>] action_create+0xe2/0xc80 [<63f50b61>] event_hist_trigger_func+0x15b5/0x1920 [<28ea5d3d>] trigger_process_regex+0x7b/0xc0 [<3138e86f>] event_trigger_write+0x4d/0xb0 [] __vfs_write+0x30/0x200 [<4f424a0d>] vfs_write+0x96/0x1b0 [] ksys_write+0x53/0xc0 [<3717101a>] __ia32_sys_write+0x15/0x20 [] do_fast_syscall_32+0x70/0x250 [<46e2629c>] entry_SYSENTER_32+0xaf/0x102 struct hist_field allocated through create_var() do not initialize "ref" field to 1. The code in __destroy_hist_field() does not destroy object if "ref" is initialized to zero, the condition if (--hist_field->ref > 1) always passes since unsigned int wraps. kmemleak report 3: [] __kmalloc_track_caller+0x139/0x2b0 [] kstrdup+0x27/0x50 [<39d70006>] init_var_ref+0x58/0xd0 [<8ca76370>] create_var_ref+0x89/0xe0 [] action_create+0x38f/0xc80 [<7c146821>] event_hist_trigger_func+0x15b5/0x1920 [<07de3f61>] trigger_process_regex+0x7b/0xc0 [] event_trigger_write+0x4d/0xb0 [<19bf1512>] __vfs_write+0x30/0x200 [<64ce4d27>] vfs_write+0x96/0x1b0 [] ksys_write+0x53/0xc0 [<7d4230cd>] __ia32_sys_write+0x15/0x20 [<8eadca00>] do_fast_syscall_32+0x70/0x250 [<235cf985>] entry_SYSENTER_32+0xaf/0x102 hist_fields (system & event_name) are not freed Link: http://lkml.kernel.org/r/20200422061503.GA5151@cosmos Signed-off-by: Vamshi K Sthambamkadi Signed-off-by: Steven Rostedt (VMware) --- kernel/trace/trace_events_hist.c | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/kernel/trace/trace_events_hist.c b/kernel/trace/trace_events_hist.c index 5f6834a2bf4119..fcab11cc6833b2 100644 --- a/kernel/trace/trace_events_hist.c +++ b/kernel/trace/trace_events_hist.c @@ -3320,6 +3320,9 @@ static void __destroy_hist_field(struct hist_field *hist_field) kfree(hist_field->name); kfree(hist_field->type); + kfree(hist_field->system); + kfree(hist_field->event_name); + kfree(hist_field); } @@ -4382,6 +4385,7 @@ static struct hist_field *create_var(struct hist_trigger_data *hist_data, goto out; } + var->ref = 1; var->flags = HIST_FIELD_FL_VAR; var->var.idx = idx; var->var.hist_data = var->hist_data = hist_data; @@ -5011,6 +5015,9 @@ static void destroy_field_vars(struct hist_trigger_data *hist_data) for (i = 0; i < hist_data->n_field_vars; i++) destroy_field_var(hist_data->field_vars[i]); + + for (i = 0; i < hist_data->n_save_vars; i++) + destroy_field_var(hist_data->save_vars[i]); } static void save_field_var(struct hist_trigger_data *hist_data, From 353da87921a5ec654e7e9024e083f099f1b33c97 Mon Sep 17 00:00:00 2001 From: "Steven Rostedt (VMware)" Date: Wed, 22 Apr 2020 21:38:45 -0400 Subject: [PATCH 12/18] ftrace: Fix memory leak caused by not freeing entry in unregister_ftrace_direct() kmemleak reported the following: unreferenced object 0xffff90d47127a920 (size 32): comm "modprobe", pid 1766, jiffies 4294792031 (age 162.568s) hex dump (first 32 bytes): 00 00 00 00 00 00 00 00 22 01 00 00 00 00 ad de ........"....... 00 78 12 a7 ff ff ff ff 00 00 b6 c0 ff ff ff ff .x.............. backtrace: [<00000000bb79e72e>] register_ftrace_direct+0xcb/0x3a0 [<00000000295e4f79>] do_one_initcall+0x72/0x340 [<00000000873ead18>] do_init_module+0x5a/0x220 [<00000000974d9de5>] load_module+0x2235/0x2550 [<0000000059c3d6ce>] __do_sys_finit_module+0xc0/0x120 [<000000005a8611b4>] do_syscall_64+0x60/0x230 [<00000000a0cdc49e>] entry_SYSCALL_64_after_hwframe+0x49/0xb3 The entry used to save the direct descriptor needs to be freed when unregistering. Signed-off-by: Steven Rostedt (VMware) --- kernel/trace/ftrace.c | 1 + 1 file changed, 1 insertion(+) diff --git a/kernel/trace/ftrace.c b/kernel/trace/ftrace.c index 041694a1eb74d8..bd030b1b951485 100644 --- a/kernel/trace/ftrace.c +++ b/kernel/trace/ftrace.c @@ -5165,6 +5165,7 @@ int unregister_ftrace_direct(unsigned long ip, unsigned long addr) list_del_rcu(&direct->next); synchronize_rcu_tasks(); kfree(direct); + kfree(entry); ftrace_direct_func_count--; } } From a2806ef77ff9a965a70d6c194bb3a4801bbdb07d Mon Sep 17 00:00:00 2001 From: Nikolay Borisov Date: Mon, 13 Apr 2020 18:32:46 +0300 Subject: [PATCH 13/18] tracing: Remove DECLARE_TRACE_NOARGS This macro was intentionally broken so that the kernel code is not poluted with such noargs macro used simply as markers. This use case can be satisfied by using dummy no inline functions. Just remove it. Link: http://lkml.kernel.org/r/20200413153246.8511-1-nborisov@suse.com Signed-off-by: Nikolay Borisov Signed-off-by: Steven Rostedt (VMware) --- include/linux/tracepoint.h | 22 +--------------------- 1 file changed, 1 insertion(+), 21 deletions(-) diff --git a/include/linux/tracepoint.h b/include/linux/tracepoint.h index 1fb11daa5c5334..a1fecf31162174 100644 --- a/include/linux/tracepoint.h +++ b/include/linux/tracepoint.h @@ -156,8 +156,7 @@ static inline struct tracepoint *tracepoint_ptr_deref(tracepoint_ptr_t *p) * Note, the proto and args passed in includes "__data" as the first parameter. * The reason for this is to handle the "void" prototype. If a tracepoint * has a "void" prototype, then it is invalid to declare a function - * as "(void *, void)". The DECLARE_TRACE_NOARGS() will pass in just - * "void *data", where as the DECLARE_TRACE() will pass in "void *data, proto". + * as "(void *, void)". */ #define __DO_TRACE(tp, proto, args, cond, rcuidle) \ do { \ @@ -373,25 +372,6 @@ static inline struct tracepoint *tracepoint_ptr_deref(tracepoint_ptr_t *p) # define __tracepoint_string #endif -/* - * The need for the DECLARE_TRACE_NOARGS() is to handle the prototype - * (void). "void" is a special value in a function prototype and can - * not be combined with other arguments. Since the DECLARE_TRACE() - * macro adds a data element at the beginning of the prototype, - * we need a way to differentiate "(void *data, proto)" from - * "(void *data, void)". The second prototype is invalid. - * - * DECLARE_TRACE_NOARGS() passes "void" as the tracepoint prototype - * and "void *__data" as the callback prototype. - * - * DECLARE_TRACE() passes "proto" as the tracepoint protoype and - * "void *__data, proto" as the callback prototype. - */ -#define DECLARE_TRACE_NOARGS(name) \ - __DECLARE_TRACE(name, void, , \ - cpu_online(raw_smp_processor_id()), \ - void *__data, __data) - #define DECLARE_TRACE(name, proto, args) \ __DECLARE_TRACE(name, PARAMS(proto), PARAMS(args), \ cpu_online(raw_smp_processor_id()), \ From d013496f99c5608e0f80afd67acb1ba93c4144ea Mon Sep 17 00:00:00 2001 From: Jason Yan Date: Fri, 10 Apr 2020 15:33:12 +0800 Subject: [PATCH 14/18] tracing: Convert local functions in tracing_map.c to static Fix the following sparse warning: kernel/trace/tracing_map.c:286:6: warning: symbol 'tracing_map_array_clear' was not declared. Should it be static? kernel/trace/tracing_map.c:297:6: warning: symbol 'tracing_map_array_free' was not declared. Should it be static? kernel/trace/tracing_map.c:319:26: warning: symbol 'tracing_map_array_alloc' was not declared. Should it be static? Link: http://lkml.kernel.org/r/20200410073312.38855-1-yanaijie@huawei.com Reported-by: Hulk Robot Signed-off-by: Jason Yan Signed-off-by: Steven Rostedt (VMware) --- kernel/trace/tracing_map.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/kernel/trace/tracing_map.c b/kernel/trace/tracing_map.c index 9e31bfc818ff82..74738c9856f164 100644 --- a/kernel/trace/tracing_map.c +++ b/kernel/trace/tracing_map.c @@ -283,7 +283,7 @@ int tracing_map_add_key_field(struct tracing_map *map, return idx; } -void tracing_map_array_clear(struct tracing_map_array *a) +static void tracing_map_array_clear(struct tracing_map_array *a) { unsigned int i; @@ -294,7 +294,7 @@ void tracing_map_array_clear(struct tracing_map_array *a) memset(a->pages[i], 0, PAGE_SIZE); } -void tracing_map_array_free(struct tracing_map_array *a) +static void tracing_map_array_free(struct tracing_map_array *a) { unsigned int i; @@ -316,7 +316,7 @@ void tracing_map_array_free(struct tracing_map_array *a) kfree(a); } -struct tracing_map_array *tracing_map_array_alloc(unsigned int n_elts, +static struct tracing_map_array *tracing_map_array_alloc(unsigned int n_elts, unsigned int entry_size) { struct tracing_map_array *a; From 9dd44c7e998e66f749348d07a700034f513ea9ff Mon Sep 17 00:00:00 2001 From: Damien Le Moal Date: Thu, 23 Apr 2020 12:02:37 +0900 Subject: [PATCH 15/18] null_blk: Fix zoned command handling For write operations issued to a null_blk device with zoned mode enabled, the state and write pointer position of the zone targeted by the command should be checked before badblocks and memory backing are handled as the write may be first failed due to, for instance, a sector position not aligned with the zone write pointer. This order of checking for errors reflects more accuratly the behavior of physical zoned devices. Furthermore, the write pointer position of the target zone should be incremented only and only if no errors are reported by badblocks and memory backing handling. To fix this, introduce the small helper function null_process_cmd() which execute null_handle_badblocks() and null_handle_memory_backed() and use this function in null_zone_write() to correctly handle write requests to zoned null devices depending on the type and state of the write target zone. Also call this function in null_handle_zoned() to process read requests to zoned null devices. null_process_cmd() is called directly from null_handle_cmd() for regular null devices, resulting in no functional change for these type of devices. To have symmetric names, the function null_handle_zoned() is renamed to null_process_zoned_cmd(). Signed-off-by: Damien Le Moal Reviewed-by: Chaitanya Kulkarni Reviewed-by: Christoph Hellwig Signed-off-by: Jens Axboe --- drivers/block/null_blk.h | 15 +++++++++------ drivers/block/null_blk_main.c | 35 +++++++++++++++++++++++----------- drivers/block/null_blk_zoned.c | 24 +++++++++++++---------- 3 files changed, 47 insertions(+), 27 deletions(-) diff --git a/drivers/block/null_blk.h b/drivers/block/null_blk.h index 62b660821dbcc1..83320cbed85b61 100644 --- a/drivers/block/null_blk.h +++ b/drivers/block/null_blk.h @@ -85,14 +85,18 @@ struct nullb { char disk_name[DISK_NAME_LEN]; }; +blk_status_t null_process_cmd(struct nullb_cmd *cmd, + enum req_opf op, sector_t sector, + unsigned int nr_sectors); + #ifdef CONFIG_BLK_DEV_ZONED int null_zone_init(struct nullb_device *dev); void null_zone_exit(struct nullb_device *dev); int null_report_zones(struct gendisk *disk, sector_t sector, unsigned int nr_zones, report_zones_cb cb, void *data); -blk_status_t null_handle_zoned(struct nullb_cmd *cmd, - enum req_opf op, sector_t sector, - sector_t nr_sectors); +blk_status_t null_process_zoned_cmd(struct nullb_cmd *cmd, + enum req_opf op, sector_t sector, + sector_t nr_sectors); size_t null_zone_valid_read_len(struct nullb *nullb, sector_t sector, unsigned int len); #else @@ -102,9 +106,8 @@ static inline int null_zone_init(struct nullb_device *dev) return -EINVAL; } static inline void null_zone_exit(struct nullb_device *dev) {} -static inline blk_status_t null_handle_zoned(struct nullb_cmd *cmd, - enum req_opf op, sector_t sector, - sector_t nr_sectors) +static inline blk_status_t null_process_zoned_cmd(struct nullb_cmd *cmd, + enum req_opf op, sector_t sector, sector_t nr_sectors) { return BLK_STS_NOTSUPP; } diff --git a/drivers/block/null_blk_main.c b/drivers/block/null_blk_main.c index 4e1c0712278eaf..8335e2b04aac46 100644 --- a/drivers/block/null_blk_main.c +++ b/drivers/block/null_blk_main.c @@ -1276,6 +1276,25 @@ static inline void nullb_complete_cmd(struct nullb_cmd *cmd) } } +blk_status_t null_process_cmd(struct nullb_cmd *cmd, + enum req_opf op, sector_t sector, + unsigned int nr_sectors) +{ + struct nullb_device *dev = cmd->nq->dev; + blk_status_t ret; + + if (dev->badblocks.shift != -1) { + ret = null_handle_badblocks(cmd, sector, nr_sectors); + if (ret != BLK_STS_OK) + return ret; + } + + if (dev->memory_backed) + return null_handle_memory_backed(cmd, op); + + return BLK_STS_OK; +} + static blk_status_t null_handle_cmd(struct nullb_cmd *cmd, sector_t sector, sector_t nr_sectors, enum req_opf op) { @@ -1294,17 +1313,11 @@ static blk_status_t null_handle_cmd(struct nullb_cmd *cmd, sector_t sector, goto out; } - if (nullb->dev->badblocks.shift != -1) { - cmd->error = null_handle_badblocks(cmd, sector, nr_sectors); - if (cmd->error != BLK_STS_OK) - goto out; - } - - if (dev->memory_backed) - cmd->error = null_handle_memory_backed(cmd, op); - - if (!cmd->error && dev->zoned) - cmd->error = null_handle_zoned(cmd, op, sector, nr_sectors); + if (dev->zoned) + cmd->error = null_process_zoned_cmd(cmd, op, + sector, nr_sectors); + else + cmd->error = null_process_cmd(cmd, op, sector, nr_sectors); out: nullb_complete_cmd(cmd); diff --git a/drivers/block/null_blk_zoned.c b/drivers/block/null_blk_zoned.c index 673618d8222a6e..2e9add7d89a4be 100644 --- a/drivers/block/null_blk_zoned.c +++ b/drivers/block/null_blk_zoned.c @@ -126,11 +126,16 @@ static blk_status_t null_zone_write(struct nullb_cmd *cmd, sector_t sector, struct nullb_device *dev = cmd->nq->dev; unsigned int zno = null_zone_no(dev, sector); struct blk_zone *zone = &dev->zones[zno]; + blk_status_t ret; + + trace_nullb_zone_op(cmd, zno, zone->cond); + + if (zone->type == BLK_ZONE_TYPE_CONVENTIONAL) + return null_process_cmd(cmd, REQ_OP_WRITE, sector, nr_sectors); switch (zone->cond) { case BLK_ZONE_COND_FULL: /* Cannot write to a full zone */ - cmd->error = BLK_STS_IOERR; return BLK_STS_IOERR; case BLK_ZONE_COND_EMPTY: case BLK_ZONE_COND_IMP_OPEN: @@ -143,19 +148,18 @@ static blk_status_t null_zone_write(struct nullb_cmd *cmd, sector_t sector, if (zone->cond != BLK_ZONE_COND_EXP_OPEN) zone->cond = BLK_ZONE_COND_IMP_OPEN; + ret = null_process_cmd(cmd, REQ_OP_WRITE, sector, nr_sectors); + if (ret != BLK_STS_OK) + return ret; + zone->wp += nr_sectors; if (zone->wp == zone->start + zone->len) zone->cond = BLK_ZONE_COND_FULL; - break; - case BLK_ZONE_COND_NOT_WP: - break; + return BLK_STS_OK; default: /* Invalid zone condition */ return BLK_STS_IOERR; } - - trace_nullb_zone_op(cmd, zno, zone->cond); - return BLK_STS_OK; } static blk_status_t null_zone_mgmt(struct nullb_cmd *cmd, enum req_opf op, @@ -216,8 +220,8 @@ static blk_status_t null_zone_mgmt(struct nullb_cmd *cmd, enum req_opf op, return BLK_STS_OK; } -blk_status_t null_handle_zoned(struct nullb_cmd *cmd, enum req_opf op, - sector_t sector, sector_t nr_sectors) +blk_status_t null_process_zoned_cmd(struct nullb_cmd *cmd, enum req_opf op, + sector_t sector, sector_t nr_sectors) { switch (op) { case REQ_OP_WRITE: @@ -229,6 +233,6 @@ blk_status_t null_handle_zoned(struct nullb_cmd *cmd, enum req_opf op, case REQ_OP_ZONE_FINISH: return null_zone_mgmt(cmd, op, sector); default: - return BLK_STS_OK; + return null_process_cmd(cmd, op, sector, nr_sectors); } } From d205bde78fa53e1ce256b1f7f65ede9696d73ee5 Mon Sep 17 00:00:00 2001 From: Damien Le Moal Date: Thu, 23 Apr 2020 12:02:38 +0900 Subject: [PATCH 16/18] null_blk: Cleanup zoned device initialization Move all zoned mode related code from null_blk_main.c to null_blk_zoned.c, avoiding an ugly #ifdef in the process. Rename null_zone_init() into null_init_zoned_dev(), null_zone_exit() into null_free_zoned_dev() and add the new function null_register_zoned_dev() to finalize the zoned dev setup before add_disk(). Signed-off-by: Damien Le Moal Reviewed-by: Christoph Hellwig Reviewed-by: Johannes Thumshirn Signed-off-by: Jens Axboe --- drivers/block/null_blk.h | 14 ++++++++++---- drivers/block/null_blk_main.c | 27 +++++++-------------------- drivers/block/null_blk_zoned.c | 21 +++++++++++++++++++-- 3 files changed, 36 insertions(+), 26 deletions(-) diff --git a/drivers/block/null_blk.h b/drivers/block/null_blk.h index 83320cbed85b61..81b311c9d781ef 100644 --- a/drivers/block/null_blk.h +++ b/drivers/block/null_blk.h @@ -90,8 +90,9 @@ blk_status_t null_process_cmd(struct nullb_cmd *cmd, unsigned int nr_sectors); #ifdef CONFIG_BLK_DEV_ZONED -int null_zone_init(struct nullb_device *dev); -void null_zone_exit(struct nullb_device *dev); +int null_init_zoned_dev(struct nullb_device *dev, struct request_queue *q); +int null_register_zoned_dev(struct nullb *nullb); +void null_free_zoned_dev(struct nullb_device *dev); int null_report_zones(struct gendisk *disk, sector_t sector, unsigned int nr_zones, report_zones_cb cb, void *data); blk_status_t null_process_zoned_cmd(struct nullb_cmd *cmd, @@ -100,12 +101,17 @@ blk_status_t null_process_zoned_cmd(struct nullb_cmd *cmd, size_t null_zone_valid_read_len(struct nullb *nullb, sector_t sector, unsigned int len); #else -static inline int null_zone_init(struct nullb_device *dev) +static inline int null_init_zoned_dev(struct nullb_device *dev, + struct request_queue *q) { pr_err("CONFIG_BLK_DEV_ZONED not enabled\n"); return -EINVAL; } -static inline void null_zone_exit(struct nullb_device *dev) {} +static inline int null_register_zoned_dev(struct nullb *nullb) +{ + return -ENODEV; +} +static inline void null_free_zoned_dev(struct nullb_device *dev) {} static inline blk_status_t null_process_zoned_cmd(struct nullb_cmd *cmd, enum req_opf op, sector_t sector, sector_t nr_sectors) { diff --git a/drivers/block/null_blk_main.c b/drivers/block/null_blk_main.c index 8335e2b04aac46..8efd8778e2095e 100644 --- a/drivers/block/null_blk_main.c +++ b/drivers/block/null_blk_main.c @@ -580,7 +580,7 @@ static void null_free_dev(struct nullb_device *dev) if (!dev) return; - null_zone_exit(dev); + null_free_zoned_dev(dev); badblocks_exit(&dev->badblocks); kfree(dev); } @@ -1618,19 +1618,12 @@ static int null_gendisk_register(struct nullb *nullb) disk->queue = nullb->q; strncpy(disk->disk_name, nullb->disk_name, DISK_NAME_LEN); -#ifdef CONFIG_BLK_DEV_ZONED if (nullb->dev->zoned) { - if (queue_is_mq(nullb->q)) { - int ret = blk_revalidate_disk_zones(disk); - if (ret) - return ret; - } else { - blk_queue_chunk_sectors(nullb->q, - nullb->dev->zone_size_sects); - nullb->q->nr_zones = blkdev_nr_zones(disk); - } + int ret = null_register_zoned_dev(nullb); + + if (ret) + return ret; } -#endif add_disk(disk); return 0; @@ -1786,14 +1779,9 @@ static int null_add_dev(struct nullb_device *dev) } if (dev->zoned) { - rv = null_zone_init(dev); + rv = null_init_zoned_dev(dev, nullb->q); if (rv) goto out_cleanup_blk_queue; - - nullb->q->limits.zoned = BLK_ZONED_HM; - blk_queue_flag_set(QUEUE_FLAG_ZONE_RESETALL, nullb->q); - blk_queue_required_elevator_features(nullb->q, - ELEVATOR_F_ZBD_SEQ_WRITE); } nullb->q->queuedata = nullb; @@ -1822,8 +1810,7 @@ static int null_add_dev(struct nullb_device *dev) return 0; out_cleanup_zone: - if (dev->zoned) - null_zone_exit(dev); + null_free_zoned_dev(dev); out_cleanup_blk_queue: blk_cleanup_queue(nullb->q); out_cleanup_tags: diff --git a/drivers/block/null_blk_zoned.c b/drivers/block/null_blk_zoned.c index 2e9add7d89a4be..9e4bcdad1a80f7 100644 --- a/drivers/block/null_blk_zoned.c +++ b/drivers/block/null_blk_zoned.c @@ -13,7 +13,7 @@ static inline unsigned int null_zone_no(struct nullb_device *dev, sector_t sect) return sect >> ilog2(dev->zone_size_sects); } -int null_zone_init(struct nullb_device *dev) +int null_init_zoned_dev(struct nullb_device *dev, struct request_queue *q) { sector_t dev_size = (sector_t)dev->size * 1024 * 1024; sector_t sector = 0; @@ -61,10 +61,27 @@ int null_zone_init(struct nullb_device *dev) sector += dev->zone_size_sects; } + q->limits.zoned = BLK_ZONED_HM; + blk_queue_flag_set(QUEUE_FLAG_ZONE_RESETALL, q); + blk_queue_required_elevator_features(q, ELEVATOR_F_ZBD_SEQ_WRITE); + + return 0; +} + +int null_register_zoned_dev(struct nullb *nullb) +{ + struct request_queue *q = nullb->q; + + if (queue_is_mq(q)) + return blk_revalidate_disk_zones(nullb->disk); + + blk_queue_chunk_sectors(q, nullb->dev->zone_size_sects); + q->nr_zones = blkdev_nr_zones(nullb->disk); + return 0; } -void null_zone_exit(struct nullb_device *dev) +void null_free_zoned_dev(struct nullb_device *dev) { kvfree(dev->zones); } From 89e7c85026a0f8522e4017de70f4a87513874520 Mon Sep 17 00:00:00 2001 From: John Oldman Date: Thu, 23 Apr 2020 10:07:42 +0100 Subject: [PATCH 17/18] ata: sata_inic162x fix a spelling issue Fixed a warning message spelling issue. Signed-off-by: John Oldman Signed-off-by: Jens Axboe --- drivers/ata/sata_inic162x.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/drivers/ata/sata_inic162x.c b/drivers/ata/sata_inic162x.c index a6b76cc12a6619..e517bd8822a5fa 100644 --- a/drivers/ata/sata_inic162x.c +++ b/drivers/ata/sata_inic162x.c @@ -145,7 +145,7 @@ enum { /* PORT_IDMA_CTL bits */ IDMA_CTL_RST_ATA = (1 << 2), /* hardreset ATA bus */ - IDMA_CTL_RST_IDMA = (1 << 5), /* reset IDMA machinary */ + IDMA_CTL_RST_IDMA = (1 << 5), /* reset IDMA machinery */ IDMA_CTL_GO = (1 << 7), /* IDMA mode go */ IDMA_CTL_ATA_NIEN = (1 << 8), /* ATA IRQ disable */ From bc0c4d1e176eeb614dc8734fc3ace34292771f11 Mon Sep 17 00:00:00 2001 From: Linus Torvalds Date: Fri, 24 Apr 2020 11:10:58 -0700 Subject: [PATCH 18/18] mm: check that mm is still valid in madvise() IORING_OP_MADVISE can end up basically doing mprotect() on the VM of another process, which means that it can race with our crazy core dump handling which accesses the VM state without holding the mmap_sem (because it incorrectly thinks that it is the final user). This is clearly a core dumping problem, but we've never fixed it the right way, and instead have the notion of "check that the mm is still ok" using mmget_still_valid() after getting the mmap_sem for writing in any situation where we're not the original VM thread. See commit 04f5866e41fb ("coredump: fix race condition between mmget_not_zero()/get_task_mm() and core dumping") for more background on this whole mmget_still_valid() thing. You might want to have a barf bag handy when you do. We're discussing just fixing this properly in the only remaining core dumping routines. But even if we do that, let's make do_madvise() do the right thing, and then when we fix core dumping, we can remove all these mmget_still_valid() checks. Reported-and-tested-by: Jann Horn Fixes: c1ca757bd6f4 ("io_uring: add IORING_OP_MADVISE") Acked-by: Jens Axboe Signed-off-by: Linus Torvalds --- mm/madvise.c | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/mm/madvise.c b/mm/madvise.c index 4bb30ed6c8d216..8cbd8c1bfe159e 100644 --- a/mm/madvise.c +++ b/mm/madvise.c @@ -27,6 +27,7 @@ #include #include #include +#include #include @@ -1090,6 +1091,23 @@ int do_madvise(unsigned long start, size_t len_in, int behavior) if (write) { if (down_write_killable(¤t->mm->mmap_sem)) return -EINTR; + + /* + * We may have stolen the mm from another process + * that is undergoing core dumping. + * + * Right now that's io_ring, in the future it may + * be remote process management and not "current" + * at all. + * + * We need to fix core dumping to not do this, + * but for now we have the mmget_still_valid() + * model. + */ + if (!mmget_still_valid(current->mm)) { + up_write(¤t->mm->mmap_sem); + return -EINTR; + } } else { down_read(¤t->mm->mmap_sem); }

', '