Codestin Search App

provokateurin · 2025-05-20T11:02:16Z

Resolves: Basic auth requests always set up the filesystem #31357

Summary

Unlike previous attempts I did not try to change much of the logic, just stopped calling the GC in each request from base.php and moved that to a background job.
This brings down the base query count when calling /index.php/204 with basic auth credentials via curl from 22 to 14, because the FS doesn't need to be setup anymore before the controller is even called.

Checklist

Code is properly formatted
Sign-off message is added to all commits
Tests (unit, integration, api and/or acceptance) are included
Screenshots before/after for front-end changes
Documentation (manuals or wiki) has been updated or is not required
Backports requested where applicable (ex: critical bugfixes)

susnux

Seems to work fine

provokateurin · 2025-05-20T13:03:04Z

Then the single response should contain a property "{http://open-collaboration-services.org/ns}share-permissions" with value "31" # SharingContext::theSingleResponseShouldContainAPropertyWithValue()
 Property "{http://open-collaboration-services.org/ns}share-permissions" found with value "15", expected "31" (Exception)

The test failure looks really suspicious, but I don't see how it could be related 🤔

provokateurin · 2025-05-26T08:48:59Z

The test failure looks really suspicious, but I don't see how it could be related 🤔

I can reliably reproduce it locally and on master it is just fine.

provokateurin · 2025-05-26T09:29:27Z

As far as I can tell the FS isn't correctly setup any more, as WebDAV currently relies on the way this file cache garbage collection already sets up the FS (really bad I know). I think this is similar to how previous attempts went, but I'll give it another shot.

susnux · 2025-05-26T13:50:43Z

So the files webdav plugin needs to call the setup manager?

artonge · 2025-05-26T14:28:55Z

apps/files/lib/BackgroundJob/CacheGarbageCollection.php

+		$this->userManager->callForSeenUsers(function (IUser $user) use ($cache): void {
+			$userId = $user->getUID();
+
+			try {
+				$cache->gc($userId);
+			} catch (Exception $e) {
+				$this->logger->warning('Exception when running cache gc.', [
+					'app' => 'files',
+					'user' => $userId,
+					'exception' => $e,
+				]);
+			}
+		});


Will this scale correctly if there are millions of users? I would at least add an offset, like in trashbin : so the job is able to pick up from the latest user.

server/apps/files_trashbin/lib/BackgroundJob/ExpireTrash.php

Lines 32 to 71 in fa2ed6b

protected function run($argument) {

$backgroundJob = $this->appConfig->getValueString('files_trashbin', 'background_job_expire_trash', 'yes');

if ($backgroundJob === 'no') {

return;

}

$maxAge = $this->expiration->getMaxAgeAsTimestamp();

if (!$maxAge) {

return;

}

$stopTime = time() + 60 * 30; // Stops after 30 minutes.

$offset = $this->appConfig->getValueInt('files_trashbin', 'background_job_expire_trash_offset', 0);

$users = $this->userManager->getSeenUsers($offset);

foreach ($users as $user) {

try {

$uid = $user->getUID();

if (!$this->setupFS($uid)) {

continue;

}

$dirContent = Helper::getTrashFiles('/', $uid, 'mtime');

Trashbin::deleteExpiredFiles($dirContent, $uid);

} catch (\Throwable $e) {

$this->logger->error('Error while expiring trashbin for user ' . $user->getUID(), ['exception' => $e]);

}

$offset++;

if ($stopTime < time()) {

$this->appConfig->setValueInt('files_trashbin', 'background_job_expire_trash_offset', $offset);

\OC_Util::tearDownFS();

return;

}

}

$this->appConfig->setValueInt('files_trashbin', 'background_job_expire_trash_offset', 0);

\OC_Util::tearDownFS();

}

Note that some work is also ongoing to support parallel run of the job, as it is taking too long to go through all the users with only one instance of the job: #52825

You could use a similar approach as for trashbin and versions where a job is scheduled for a user when needed. But I also understand that the gc call was run on every request, so this might not be practical.

server/apps/files_trashbin/lib/Trashbin.php

Lines 854 to 863 in fa2ed6b

private static function scheduleExpire($user) {

// let the admin disable auto expire

/** @var Application $application */

$application = Server::get(Application::class);

$expiration = $application->getContainer()->query('Expiration');

if ($expiration->isEnabled()) {

Server::get(IBus::class)->push(new Expire($user));

}

}

You are right that this might not be very performant.
It might not be so bad in reality, as the chunking v1 should not be used by any client anymore and thus the GC operation should be quite fast.
So it's mostly setting up the FS that takes all the time.
What do you think?

I would go directly to what I did for the trashbin. But if you think parallel job won't be needed, I would at least add the offset logic, as for sure, it will be needed for large instances.

Thinking about it, can we come up with a better way to know if gc is needed for a user? Not sure how those files are denoted, maybe an SQL query would do the job.

True, maybe we can just get a list of users that have at least one file in the cache, then it should be way cheaper.

provokateurin · 2025-05-27T08:02:01Z

So the files webdav plugin needs to call the setup manager?

I think so yes, but so far I have not found the right place where this needs to happen 😅

provokateurin · 2025-05-27T11:04:40Z

I think I managed to initialize the FS in the right place, but unfortunately had to hijack the auth for it as there is no way to correctly run code in beforeMethod but after the auth is done.
It could probably be optimized further by only initializing the FS if WebDAV is called and not for and of the other *DAV services (or do they also rely on the FS?).

Signed-off-by: provokateurin <[email protected]>

provokateurin · 2025-05-27T13:13:39Z

The more I look into this the more it seems like this code is actually dead.
I could only find this place where the cache is created and I could not find any place where this is access via DI or the server container:

server/lib/private/Server.php

Line 586 in c879bab

return new Cache\File();

I also just removed this and run a bunch of phpunit tests and there were no failures caused by it.
There might be places that it is still used somewhere that I couldn't find, but I will create another PR to just rip out this code (but keep the fix for initializing the FS for *DAV correctly) and see what happens.

provokateurin · 2025-05-27T13:24:34Z

#53141

provokateurin added this to the Nextcloud 32 milestone May 20, 2025

provokateurin requested review from ArtificialOwl, artonge, come-nc and skjnldsv as code owners May 20, 2025 11:02

provokateurin added the 3. to review Waiting for reviews label May 20, 2025

provokateurin requested review from icewind1991 and szaimen as code owners May 20, 2025 11:02

provokateurin added the performance 🚀 label May 20, 2025

provokateurin requested review from a team, nfebe and susnux as code owners May 20, 2025 11:02

szaimen removed their request for review May 20, 2025 12:01

skjnldsv approved these changes May 20, 2025

View reviewed changes

susnux approved these changes May 20, 2025

View reviewed changes

provokateurin force-pushed the perf/files/cache-garbage-collection-background-job branch from b991c0e to 432611a Compare May 26, 2025 06:09

provokateurin added 2. developing Work in progress and removed 3. to review Waiting for reviews labels May 26, 2025

provokateurin marked this pull request as draft May 26, 2025 08:49

artonge reviewed May 26, 2025

View reviewed changes

provokateurin force-pushed the perf/files/cache-garbage-collection-background-job branch from 432611a to b39e743 Compare May 27, 2025 11:03

provokateurin added 2 commits May 27, 2025 14:17

fix(dav): Initialize the FS for the user right after authenticating

c88512e

Signed-off-by: provokateurin <[email protected]>

perf(files): Move cache garbage collection to a background job

daf3659

Signed-off-by: provokateurin <[email protected]>

provokateurin force-pushed the perf/files/cache-garbage-collection-background-job branch from b39e743 to daf3659 Compare May 27, 2025 12:19

provokateurin mentioned this pull request May 27, 2025

perf(base): Stop setting up the FS for every basic auth request #53141

Merged

4 tasks

provokateurin closed this Jul 11, 2025

susnux deleted the perf/files/cache-garbage-collection-background-job branch July 11, 2025 13:48

skjnldsv modified the milestones: Nextcloud 32, Nextcloud 33 Sep 28, 2025

	protected function run($argument) {
	$backgroundJob = $this->appConfig->getValueString('files_trashbin', 'background_job_expire_trash', 'yes');
	if ($backgroundJob === 'no') {
	return;
	}

	$maxAge = $this->expiration->getMaxAgeAsTimestamp();
	if (!$maxAge) {
	return;
	}

	$stopTime = time() + 60 * 30; // Stops after 30 minutes.
	$offset = $this->appConfig->getValueInt('files_trashbin', 'background_job_expire_trash_offset', 0);
	$users = $this->userManager->getSeenUsers($offset);

	foreach ($users as $user) {
	try {
	$uid = $user->getUID();
	if (!$this->setupFS($uid)) {
	continue;
	}
	$dirContent = Helper::getTrashFiles('/', $uid, 'mtime');
	Trashbin::deleteExpiredFiles($dirContent, $uid);
	} catch (\Throwable $e) {
	$this->logger->error('Error while expiring trashbin for user ' . $user->getUID(), ['exception' => $e]);
	}

	$offset++;

	if ($stopTime < time()) {
	$this->appConfig->setValueInt('files_trashbin', 'background_job_expire_trash_offset', $offset);
	\OC_Util::tearDownFS();
	return;
	}
	}

	$this->appConfig->setValueInt('files_trashbin', 'background_job_expire_trash_offset', 0);
	\OC_Util::tearDownFS();
	}

	private static function scheduleExpire($user) {
	// let the admin disable auto expire
	/** @var Application $application */
	$application = Server::get(Application::class);
	$expiration = $application->getContainer()->query('Expiration');
	if ($expiration->isEnabled()) {
	Server::get(IBus::class)->push(new Expire($user));
	}
	}

Uh oh!

Conversation

provokateurin commented May 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Checklist

Uh oh!

susnux left a comment

Choose a reason for hiding this comment

Uh oh!

provokateurin commented May 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

provokateurin commented May 26, 2025

Uh oh!

provokateurin commented May 26, 2025

Uh oh!

susnux commented May 26, 2025

Uh oh!

artonge May 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

provokateurin May 27, 2025

Choose a reason for hiding this comment

Uh oh!

artonge May 27, 2025

Choose a reason for hiding this comment

Uh oh!

provokateurin May 27, 2025

Choose a reason for hiding this comment

Uh oh!

provokateurin commented May 27, 2025

Uh oh!

provokateurin commented May 27, 2025

Uh oh!

provokateurin commented May 27, 2025

Uh oh!

provokateurin commented May 27, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

provokateurin commented May 20, 2025 •

edited

Loading

provokateurin commented May 20, 2025 •

edited

Loading

artonge May 26, 2025 •

edited

Loading