X Tutup
Skip to content

Use app config to disable background jobs and prevent them from running#40969

Merged
IljaN merged 2 commits intomasterfrom
disable_jobs
Sep 6, 2023
Merged

Use app config to disable background jobs and prevent them from running#40969
IljaN merged 2 commits intomasterfrom
disable_jobs

Conversation

@jvillafanez
Copy link
Member

@jvillafanez jvillafanez commented Sep 4, 2023

Description

Allow background jobs to be disabled via app config.
You can use occ config:app:set backgroundjob jobs_disabled --value 19,5 to disable the jobs with the id 19 and 5.
The value for the key is a comma-separated list of job ids. The job ids can be checked with occ background:queue:status
You can use occ config:app:delete backgroundjob jobs_disabled to remove the option set.

Note that this only affects to the cron execution. You can still run the disabled jobs manually with occ background:queue:execute <jobid>
Furthermore, a disabled job that tried to run will have the "reserved_at" column filled. This means that you'll have to wait 12 hours for the next attempt to run the job even if it's removed from the disabled list.

Related Issue

https://github.com/owncloud/enterprise/issues/5978

Motivation and Context

This is intended to disable jobs that could use too many resources or take a lot of time, without removing them from the list. Note that all the background jobs should run unless there is a problem with them.

How Has This Been Tested?

In a fresh installation:

  1. Setup a disabled job with occ config:app:set backgroundjob jobs_disabled --value <jobid>
  2. Run occ system:cron. A warning log should appear for the job marked as disabled. "reserved_at" column in the DB is filled for the target job id.
  3. Run occ system:cron again. Disabled job won't run until 12 hours have passed, so no new log message will be shown.

Screenshots (if appropriate):

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Database schema changes (next release will require increase of minor version instead of patch)
  • Breaking change (fix or feature that would cause existing functionality to change)
  • Technical debt
  • Tests only (no source changes)

Checklist:

  • Code changes
  • Unit tests added
  • Acceptance tests added
  • Documentation ticket raised:
  • Changelog item, see TEMPLATE

@jvillafanez jvillafanez self-assigned this Sep 4, 2023
@owncloud owncloud deleted a comment from update-docs bot Sep 4, 2023
@sonarqubecloud
Copy link

sonarqubecloud bot commented Sep 4, 2023

Kudos, SonarCloud Quality Gate passed!    Quality Gate passed

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities
Security Hotspot A 0 Security Hotspots
Code Smell A 0 Code Smells

50.0% 50.0% Coverage
0.0% 0.0% Duplication

@jnweiger
Copy link
Contributor

jnweiger commented Sep 5, 2023

Are these job IDs constant over time?
I was thinking, when e.g. job 25 gets removed, something else that wants to create a new job could get the ID 25.

@jvillafanez
Copy link
Member Author

jvillafanez commented Sep 5, 2023

As far as I know, there isn't a guarantee that the ids will remain constant, but they are incremental. New jobs will use the latest id available provided by the DB.
For periodic jobs, the job id should remain constant as long as the job isn't removed somehow.

@IljaN IljaN merged commit baa9a64 into master Sep 6, 2023
@delete-merged-branch delete-merged-branch bot deleted the disable_jobs branch September 6, 2023 12:14
@cdamken
Copy link
Contributor

cdamken commented Sep 6, 2023

I installed in my cloud and disabled some jobs:

root@snoopy5:/var/www/owncloud# occ config:app:set backgroundjob jobs_disabled --value 10,25
Config value jobs_disabled for app backgroundjob set to 10,25

Lets wait and see

@cdamken
Copy link
Contributor

cdamken commented Sep 6, 2023

Something looks weird, the job 25 does not appear anymore but the 10 is still there:

root@snoopy5:~# occ b:q:s
+--------+----------------------------------------------------------+-----------------------------+---------------------------+---------------------------+---------------------------+------------------------+
| Job ID | Job                                                      | Job Arguments               | Last Run                  | Last Checked              | Reserved At               | Execution Duration (s) |
+--------+----------------------------------------------------------+-----------------------------+---------------------------+---------------------------+---------------------------+------------------------+
| 1      | OC\Log\Rotate                                            | /home/www-data/owncloud.log | 2023-09-06T14:00:05+00:00 | 2023-09-06T14:00:05+00:00 | N/A                       | 0                      |
| 2      | OCA\Activity\BackgroundJob\EmailNotification             |                             | 2023-09-06T14:00:05+00:00 | 2023-09-06T14:00:05+00:00 | N/A                       | 0                      |
| 3      | OCA\Activity\BackgroundJob\ExpireActivities              |                             | 2023-09-06T08:40:01+00:00 | 2023-09-06T14:00:07+00:00 | N/A                       | 0                      |
| 4      | OCA\DAV\BackgroundJob\CleanProperties                    |                             | 2023-09-06T08:40:01+00:00 | 2023-09-06T14:00:05+00:00 | N/A                       | 0                      |
| 5      | OCA\DAV\CardDAV\SyncJob                                  |                             | 2023-09-06T08:40:01+00:00 | 2023-09-06T14:00:08+00:00 | N/A                       | 1                      |
| 6      | OCA\Federation\SyncJob                                   |                             | 2023-09-06T08:40:02+00:00 | 2023-09-06T14:00:01+00:00 | N/A                       | 0                      |
| 7      | OCA\Files\BackgroundJob\CleanupFileLocks                 |                             | 2023-09-06T13:55:01+00:00 | 2023-09-06T14:00:01+00:00 | N/A                       | 0                      |
| 8      | OCA\Files\BackgroundJob\CleanupPersistentFileLocks       |                             | 2023-09-06T13:45:02+00:00 | 2023-09-06T14:00:01+00:00 | N/A                       | 0                      |
| 9      | OCA\Files\BackgroundJob\DeleteOrphanedItems              |                             | 2023-09-06T14:00:01+00:00 | 2023-09-06T14:00:01+00:00 | N/A                       | 0                      |
| 10     | OCA\Files\BackgroundJob\PreviewCleanupJob                |                             | 2023-09-06T13:00:01+00:00 | 2023-09-06T13:10:05+00:00 | 2023-09-06T13:10:05+00:00 | 94                     |
| 11     | OCA\Files\BackgroundJob\ScanFiles                        |                             | 2023-09-06T14:00:01+00:00 | 2023-09-06T14:00:01+00:00 | N/A                       | 4                      |
| 12     | OCA\Files_Sharing\DeleteOrphanedSharesJob                |                             | 2023-09-06T13:45:06+00:00 | 2023-09-06T14:00:05+00:00 | N/A                       | 0                      |
| 13     | OCA\Files_Sharing\ExpireSharesJob                        |                             | 2023-09-06T08:40:07+00:00 | 2023-09-06T14:00:05+00:00 | N/A                       | 0                      |
| 14     | OCA\Files_Sharing\External\ScanExternalSharesJob         |                             | 2023-09-06T14:00:05+00:00 | 2023-09-06T14:00:05+00:00 | N/A                       | 0                      |
| 15     | OCA\Files_Trashbin\BackgroundJob\ExpireTrash             |                             | 2023-09-06T14:00:05+00:00 | 2023-09-06T14:00:05+00:00 | N/A                       | 2                      |
| 16     | OCA\Files_Versions\BackgroundJob\ExpireVersions          |                             | 2023-09-06T13:35:07+00:00 | 2023-09-06T14:00:07+00:00 | N/A                       | 5                      |
| 17     | OCA\Market\CheckUpdateBackgroundJob                      |                             | 2023-09-06T08:35:06+00:00 | 2023-09-06T14:00:07+00:00 | N/A                       | 0                      |
| 18     | OCA\Music\BackgroundJob\Cleanup                          |                             | 2023-09-06T14:00:07+00:00 | 2023-09-06T14:00:07+00:00 | N/A                       | 0                      |
| 19     | OCA\Music\BackgroundJob\PodcastUpdateCheck               |                             | 2023-09-06T14:00:07+00:00 | 2023-09-06T14:00:07+00:00 | N/A                       | 0                      |
| 20     | OCA\OAuth2\BackgroundJob\CleanUp                         |                             | 2023-09-06T08:35:07+00:00 | 2023-09-06T14:00:07+00:00 | N/A                       | 0                      |
| 21     | OCA\Richdocuments\BackgroundJob\CleanupExpiredWopiTokens |                             | 2023-09-06T14:00:07+00:00 | 2023-09-06T14:00:07+00:00 | N/A                       | 0                      |
| 22     | OCA\Sccuot\Cron\SnapshotJob                              |                             | 2023-09-06T14:00:07+00:00 | 2023-09-06T14:00:07+00:00 | N/A                       | 0                      |
| 23     | OCA\UpdateNotification\Notification\BackgroundJob        |                             | 2023-09-06T08:35:07+00:00 | 2023-09-06T14:00:07+00:00 | N/A                       | 0                      |
| 24     | OCA\UpdateNotification\ResetTokenBackgroundJob           |                             | 2023-09-06T14:00:07+00:00 | 2023-09-06T14:00:07+00:00 | N/A                       | 0                      |
| 26     | OC\Authentication\Token\DefaultTokenCleanupJob           |                             | 2023-09-06T14:00:07+00:00 | 2023-09-06T14:00:07+00:00 | N/A                       | 0                      |
+--------+----------------------------------------------------------+-----------------------------+---------------------------+---------------------------+---------------------------+------------------------+
root@snoopy5:~# occ_db "select * from oc_jobs"
+----+----------------------------------------------------------+----------------------------------+------------+--------------+-------------+--------------------+
| id | class                                                    | argument                         | last_run   | last_checked | reserved_at | execution_duration |
+----+----------------------------------------------------------+----------------------------------+------------+--------------+-------------+--------------------+
|  1 | OC\Log\Rotate                                            | "\/home\/www-data\/owncloud.log" | 1694008805 |   1694008805 |           0 |                  0 |
|  2 | OCA\Activity\BackgroundJob\EmailNotification             | null                             | 1694008805 |   1694008805 |           0 |                  0 |
|  3 | OCA\Activity\BackgroundJob\ExpireActivities              | null                             | 1693989601 |   1694008807 |           0 |                  0 |
|  4 | OCA\DAV\BackgroundJob\CleanProperties                    | null                             | 1693989601 |   1694008805 |           0 |                  0 |
|  5 | OCA\DAV\CardDAV\SyncJob                                  | null                             | 1693989601 |   1694008808 |           0 |                  1 |
|  6 | OCA\Federation\SyncJob                                   | null                             | 1693989602 |   1694008801 |           0 |                  0 |
|  7 | OCA\Files\BackgroundJob\CleanupFileLocks                 | null                             | 1694008501 |   1694008801 |           0 |                  0 |
|  8 | OCA\Files\BackgroundJob\CleanupPersistentFileLocks       | null                             | 1694007902 |   1694008801 |           0 |                  0 |
|  9 | OCA\Files\BackgroundJob\DeleteOrphanedItems              | null                             | 1694008801 |   1694008801 |           0 |                  0 |
| 10 | OCA\Files\BackgroundJob\PreviewCleanupJob                | null                             | 1694005201 |   1694005805 |  1694005805 |                 94 |
| 11 | OCA\Files\BackgroundJob\ScanFiles                        | null                             | 1694008801 |   1694008801 |           0 |                  4 |
| 12 | OCA\Files_Sharing\DeleteOrphanedSharesJob                | null                             | 1694007906 |   1694008805 |           0 |                  0 |
| 13 | OCA\Files_Sharing\ExpireSharesJob                        | null                             | 1693989607 |   1694008805 |           0 |                  0 |
| 14 | OCA\Files_Sharing\External\ScanExternalSharesJob         | null                             | 1694008805 |   1694008805 |           0 |                  0 |
| 15 | OCA\Files_Trashbin\BackgroundJob\ExpireTrash             | null                             | 1694008805 |   1694008805 |           0 |                  2 |
| 16 | OCA\Files_Versions\BackgroundJob\ExpireVersions          | null                             | 1694007307 |   1694008807 |           0 |                  5 |
| 17 | OCA\Market\CheckUpdateBackgroundJob                      | null                             | 1693989306 |   1694008807 |           0 |                  0 |
| 18 | OCA\Music\BackgroundJob\Cleanup                          | null                             | 1694008807 |   1694008807 |           0 |                  0 |
| 19 | OCA\Music\BackgroundJob\PodcastUpdateCheck               | null                             | 1694008807 |   1694008807 |           0 |                  0 |
| 20 | OCA\OAuth2\BackgroundJob\CleanUp                         | null                             | 1693989307 |   1694008807 |           0 |                  0 |
| 21 | OCA\Richdocuments\BackgroundJob\CleanupExpiredWopiTokens | null                             | 1694008807 |   1694008807 |           0 |                  0 |
| 22 | OCA\Sccuot\Cron\SnapshotJob                              | null                             | 1694008807 |   1694008807 |           0 |                  0 |
| 23 | OCA\UpdateNotification\Notification\BackgroundJob        | null                             | 1693989307 |   1694008807 |           0 |                  0 |
| 24 | OCA\UpdateNotification\ResetTokenBackgroundJob           | null                             | 1694008807 |   1694008807 |           0 |                  0 |
| 25 | OC\Core\Jobs\PreviewCleanupJob                           | null                             |          0 |   1693989306 |  1693989306 |                  0 |
| 26 | \OC\Authentication\Token\DefaultTokenCleanupJob          | null                             | 1694008807 |   1694008807 |           0 |                  0 |
+----+----------------------------------------------------------+----------------------------------+------------+--------------+-------------+--------------------+

@jvillafanez
Copy link
Member Author

jvillafanez commented Sep 6, 2023

Job 25 is likely an invalid job (I'm not finding the code for it in the master branch). By default, the occ background:queue:status command shows only valid jobs, that's why the job 25 is missing from that list.
We added an option recently to show also invalid jobs, but I'm not sure if it's out yet. Check the help of that command.

@phil-davis
Copy link
Contributor

phil-davis commented Sep 6, 2023

We added an option recently to show also invalid jobs, but I'm not sure if it's out yet.

PR #40846 released in 10.13.0

@jnweiger
Copy link
Contributor

jnweiger commented Sep 6, 2023 •
edited
Loading

Furthermore, a disabled job that tried to run will have the "reserved_at" column filled. This means that you'll have to wait 12 hours for the next attempt to run the job even if it's removed from the disabled list.

Seen while testing:

  • Running the job manually clears the reserved_at field. This can be used as a workaround instead of waiting 12h.
  • occ b:q:s could be enhanced to take the new jobs_disabled variable into account.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

1 more reviewer

@IljaN IljaN IljaN approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

@jvillafanez jvillafanez

Labels

3 - To Review

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@jvillafanez @jnweiger @cdamken @phil-davis @IljaN
X Tutup