- Regression of forgejo/forgejo!11776 (and forgejo/forgejo!11881)
- Scope of the transaction is moved to a per-package cleanup rule basis.
This is also a enhancement for scaling (already deployed on Codeberg for a while).
- Package cleanup is now run with `RetryTx`, because rebuilding
repository files runs `RetryTx` and it could indicate to retry the whole
transaction.
- Previously it would error and say running `RetryTx` in a
transaction was not possible, this is now possible. Nested `RetryTx` is
always allowed, matching of which errors to retry is still the responsible
of the inner `RetryTx`.
Co-authored-by: Mathieu Fenniak <mathieu@fenniak.net>
Reviewed-on: https://codeberg.org/forgejo/forgejo/pulls/12446
Reviewed-by: Mathieu Fenniak <mfenniak@noreply.codeberg.org>
This change fixes an issue that makes Forgejo clean up too many versions of a container package even though it should keep them according to the rules set for the package.
The issue affects multi-platform container images.
Forgejo adds a package version for each platform (for example `linux/amd64`, `linux/arm64`) in addition to the actual tag (for example `0.6.0` or `latest`).
This results in rows in the table `package_version` similar to this (unimportant columns omitted for brevity):
| **lower_version** | **created_unix** |
|---|---|
| `latest` | `1768742887`|
| `0.6.0` | `1768742886` |
| `sha256:038e...` | `1768742886` |
| `sha256:fc38...` | `1768742886` |
| `0.5.0` | `1768742864` |
| `sha256:806d...` | `1768742864` |
| `sha256:0a19...` | `1768742864` |
| `0.4.0` | `1768742848` |
| `...` | `...` |
The code assumes that the first `<keep count>` entries can be ignored and considers every entry after `<keep count>` as eligible for cleanup.
That doesn't work for multi-platform container images because, for `<keep count>=5`, it considers version `0.4.0` as eligible.
## Checklist
The [contributor guide](https://forgejo.org/docs/next/contributor/) contains information that will be helpful to first time contributors. There also are a few [conditions for merging Pull Requests in Forgejo repositories](https://codeberg.org/forgejo/governance/src/branch/main/PullRequestsAgreement.md). You are also welcome to join the [Forgejo development chatroom](https://matrix.to/#/#forgejo-development:matrix.org).
### Tests
- I added test coverage for Go changes...
- [x] in their respective `*_test.go` for unit tests.
- [ ] in the `tests/integration` directory if it involves interactions with a live Forgejo server.
- I added test coverage for JavaScript changes...
- [ ] in `web_src/js/*.test.js` if it can be unit tested.
- [ ] in `tests/e2e/*.test.e2e.js` if it requires interactions with a live Forgejo server (see also the [developer guide for JavaScript testing](https://codeberg.org/forgejo/forgejo/src/branch/forgejo/tests/e2e/README.md#end-to-end-tests)).
### Documentation
- [ ] I created a pull request [to the documentation](https://codeberg.org/forgejo/docs) to explain to Forgejo users how to use this change.
- [x] I did not document these changes and I do not expect someone else to do it.
### Release notes
- [x] This change will be noticed by a Forgejo user or admin (feature, bug fix, performance, etc.). I suggest to include a release note for this change.
- [ ] This change is not visible to a Forgejo user or admin (refactor, dependency upgrade, etc.). I think there is no need to add a release note for this change.
*The decision if the pull request will be shown in the release notes is up to the mergers / release team.*
The content of the `release-notes/<pull request number>.md` file will serve as the basis for the release notes. If the file does not exist, the title of the pull request will be used instead.
<!--start release-notes-assistant-->
## Release notes
<!--URL:https://codeberg.org/forgejo/forgejo-->
- Bug fixes
- [PR](https://codeberg.org/forgejo/forgejo/pulls/11246): <!--number 11246 --><!--line 0 --><!--description Y2xlYW51cCBvZiBtdWx0aS1wbGF0Zm9ybSBjb250YWluZXIgaW1hZ2Vz-->cleanup of multi-platform container images<!--description-->
<!--end release-notes-assistant-->
Reviewed-on: https://codeberg.org/forgejo/forgejo/pulls/11246
Reviewed-by: Michael Kriese <michael.kriese@gmx.de>
Reviewed-by: Mathieu Fenniak <mfenniak@noreply.codeberg.org>
Co-authored-by: wandhydrant <wandhydrant@noreply.codeberg.org>
Co-committed-by: wandhydrant <wandhydrant@noreply.codeberg.org>
The package cleanup routine checks every container version for whether it is referenced by a multi-platform manifest, which appears to be a performance problem indicated by CPU profiling collected in #9358 on SQLite systems. This PR removes that check completely, which isn't necessary since #4698 added a much more performant mass-cleanup of these dangling platform versions.
May fix#9358 completely, but it leaves fundamental scalability concerns with SQLite due to long-running transactions. The transactions will be shorter with this change. Requires end-user testing to confirm if sufficiently fixed.
## Checklist
The [contributor guide](https://forgejo.org/docs/next/contributor/) contains information that will be helpful to first time contributors. There also are a few [conditions for merging Pull Requests in Forgejo repositories](https://codeberg.org/forgejo/governance/src/branch/main/PullRequestsAgreement.md). You are also welcome to join the [Forgejo development chatroom](https://matrix.to/#/#forgejo-development:matrix.org).
### Tests
- I added test coverage for Go changes...
- [x] in their respective `*_test.go` for unit tests.
- [x] in the `tests/integration` directory if it involves interactions with a live Forgejo server.
- I added test coverage for JavaScript changes...
- [ ] in `web_src/js/*.test.js` if it can be unit tested.
- [ ] in `tests/e2e/*.test.e2e.js` if it requires interactions with a live Forgejo server (see also the [developer guide for JavaScript testing](https://codeberg.org/forgejo/forgejo/src/branch/forgejo/tests/e2e/README.md#end-to-end-tests)).
### Documentation
- [ ] I created a pull request [to the documentation](https://codeberg.org/forgejo/docs) to explain to Forgejo users how to use this change.
- [x] I did not document these changes and I do not expect someone else to do it.
### Release notes
- [ ] I do not want this change to show in the release notes.
- [x] I want the title to show in the release notes with a link to this pull request.
- [ ] I want the content of the `release-notes/<pull request number>.md` to be be used for the release notes instead of the title.
<!--start release-notes-assistant-->
## Release notes
<!--URL:https://codeberg.org/forgejo/forgejo-->
- Bug fixes
- [PR](https://codeberg.org/forgejo/forgejo/pulls/10297): <!--number 10297 --><!--line 0 --><!--description cmVkdWNlIHJ1bnRpbWUgb2YgY29udGFpbmVyIGNsZWFudXAgYnkgcmVseWluZyBvbiBtYXNzIGRpZ2VzdCBjbGVhbnVw-->reduce runtime of container cleanup by relying on mass digest cleanup<!--description-->
<!--end release-notes-assistant-->
Reviewed-on: https://codeberg.org/forgejo/forgejo/pulls/10297
Reviewed-by: Michael Kriese <michael.kriese@gmx.de>
Reviewed-by: Earl Warren <earl-warren@noreply.codeberg.org>
Co-authored-by: Mathieu Fenniak <mathieu@fenniak.net>
Co-committed-by: Mathieu Fenniak <mathieu@fenniak.net>
If the keep count of a cleanup rule is greater than the number of available packages, it fails with:
```
panic(boundsError{x: int64(x), signed: true, y: y, code: boundsSliceB})
.../packages/packages.go:175
.../routers/web/org/setting_packages.go:108
```
Regression of https://codeberg.org/forgejo/forgejo/pulls/9219/files
Refs https://codeberg.org/forgejo/forgejo/issues/9461
<!--start release-notes-assistant-->
## Release notes
<!--URL:https://codeberg.org/forgejo/forgejo-->
- Bug fixes
- [PR](https://codeberg.org/forgejo/forgejo/pulls/9468): <!--number 9468 --><!--line 0 --><!--description cGFja2FnZSBjbGVhbmVkIHJ1bGUgZmFpbHMgaWYgdGhlIGtlZXAgY291bnQgaXMgdG9vIGhpZ2g=-->package cleaned rule fails if the keep count is too high<!--description-->
<!--end release-notes-assistant-->
Reviewed-on: https://codeberg.org/forgejo/forgejo/pulls/9468
Reviewed-by: Gusted <gusted@noreply.codeberg.org>
Reviewed-by: Mathieu Fenniak <mfenniak@noreply.codeberg.org>
Co-authored-by: Earl Warren <contact@earl-warren.org>
Co-committed-by: Earl Warren <contact@earl-warren.org>
Before it has only processed the newest 200 (or 50 for default `MAX_RESPONSE_ITEMS: 50`) versions.
After it processes all versions.
<!--start release-notes-assistant-->
## Release notes
<!--URL:https://codeberg.org/forgejo/forgejo-->
- Bug fixes
- [PR](https://codeberg.org/forgejo/forgejo/pulls/9219): <!--number 9219 --><!--line 0 --><!--description cGFja2FnZSBjbGVhbnVwIHJ1bGVzIGFyZSBub3QgYXBwbGllZCB3aGVuIHRoZXJlIGFyZSBtb3JlIHRoYW4gMjAwIHBhY2thZ2VzIChkZXBlbmRzIG9uIGBNQVhfUkVTUE9OU0VfSVRFTVNgKQ==-->package cleanup rules are not applied when there are more than 200 packages (depends on `MAX_RESPONSE_ITEMS`)<!--description-->
<!--end release-notes-assistant-->
Reviewed-on: https://codeberg.org/forgejo/forgejo/pulls/9219
Reviewed-by: Earl Warren <earl-warren@noreply.codeberg.org>
Co-authored-by: Michael Kriese <michael.kriese@visualon.de>
Co-committed-by: Michael Kriese <michael.kriese@visualon.de>
Co-authored-by: Aleksandr Gamzin alexgamz1119@gmail.com
Adds support for the Apt-Rpm registry of the Alt Lunux distribution.
Alt Linux uses RPM packages to store and distribute software to its users. But the logic of the Alt Linux package registry is different from the Red Hat package registry.
I have added support for the Alt Linux package registry.
## Checklist
The [contributor guide](https://forgejo.org/docs/next/contributor/) contains information that will be helpful to first time contributors. There also are a few [conditions for merging Pull Requests in Forgejo repositories](https://codeberg.org/forgejo/governance/src/branch/main/PullRequestsAgreement.md). You are also welcome to join the [Forgejo development chatroom](https://matrix.to/#/#forgejo-development:matrix.org).
### Tests
- I added test coverage for Go changes...
- [ ] in their respective `*_test.go` for unit tests.
- [x] in the `tests/integration` directory if it involves interactions with a live Forgejo server.
- I added test coverage for JavaScript changes...
- [ ] in `web_src/js/*.test.js` if it can be unit tested.
- [ ] in `tests/e2e/*.test.e2e.js` if it requires interactions with a live Forgejo server (see also the [developer guide for JavaScript testing](https://codeberg.org/forgejo/forgejo/src/branch/forgejo/tests/e2e/README.md#end-to-end-tests)).
### Documentation
- [x] I created a pull request [to the documentation](https://codeberg.org/forgejo/docs) to explain to Forgejo users how to use this change.
- [ ] I did not document these changes and I do not expect someone else to do it.
### Release notes
- [ ] I do not want this change to show in the release notes.
- [x] I want the title to show in the release notes with a link to this pull request.
- [ ] I want the content of the `release-notes/<pull request number>.md` to be be used for the release notes instead of the title.
Co-authored-by: Aleksandr Gamzin <gamzin@altlinux.org>
Reviewed-on: https://codeberg.org/forgejo/forgejo/pulls/6351
Reviewed-by: Earl Warren <earl-warren@noreply.codeberg.org>
Co-authored-by: Alex619829 <alex619829@noreply.codeberg.org>
Co-committed-by: Alex619829 <alex619829@noreply.codeberg.org>
The inventory of the sha256:* images and the manifest index that
reference them is incomplete because it does not take into account any
image older than the expiration limit. As a result some sha256:* will
be considered orphaned although they are referenced from a manifest
index that was created more recently than the expiration limit.
There must not be any filtering based on the creation time when
building the inventory. The expiration limit must only be taken into
account when deleting orphaned images: those that are more recent than
the expiration limit must not be deleted.
This limit is specially important because it protects against a race
between a cleanup task and an ongoing mirroring task. A mirroring
task (such as skopeo sync) will first upload sha256:* images and then
create the corresponding manifest index. If a cleanup races against
it, the sha256:* images that are not yet referenced will be deleted
without skopeo noticing and the published index manifest that happens
at a later time will contain references to non-existent images.
Replace a double select with a simple select.
The complication originates from the initial implementation which
deleted packages instead of selecting them. It was justified to
workaround a problem in MySQL. But it is just a waste of resources
when collecting a list of IDs.
I noticed the `BuildAllRepositoryFiles` function under the Alpine folder
is unused and I thought it was a bug.
But I'm not sure about this. Was it on purpose?
Hello there,
Cargo Index over HTTP is now prefered over git for package updates: we
should not force users who do not need the GIT repo to have the repo
created/updated on each publish (it can still be created in the packages
settings).
The current behavior when publishing is to check if the repo exist and
create it on the fly if not, then update it's content.
Cargo HTTP Index does not rely on the repo itself so this will be
useless for everyone not using the git protocol for cargo registry.
This PR only disable the creation on the fly of the repo when publishing
a crate.
This is linked to #26844 (error 500 when trying to publish a crate if
user is missing write access to the repo) because it's now optional.
---------
Co-authored-by: KN4CK3R <admin@oldschoolhack.me>
Until now expired package data gets deleted daily by a cronjob. The
admin page shows the size of all packages and the size of unreferenced
data. The users (#25035, #20631) expect the deletion of this data if
they run the cronjob from the admin page but the job only deletes data
older than 24h.
This PR adds a new button which deletes all expired data.
---------
Co-authored-by: silverwind <me@silverwind.io>
Co-authored-by: @awkwardbunny
This PR adds a Debian package registry. You can follow [this
tutorial](https://www.baeldung.com/linux/create-debian-package) to build
a *.deb package for testing. Source packages are not supported at the
moment and I did not find documentation of the architecture "all" and
how these packages should be treated.
---------
Co-authored-by: Brian Hong <brian@hongs.me>
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
