- Regression of forgejo/forgejo!11776 (and forgejo/forgejo!11881)
- Scope of the transaction is moved to a per-package cleanup rule basis.
This is also a enhancement for scaling (already deployed on Codeberg for a while).
- Package cleanup is now run with `RetryTx`, because rebuilding
repository files runs `RetryTx` and it could indicate to retry the whole
transaction.
- Previously it would error and say running `RetryTx` in a
transaction was not possible, this is now possible. Nested `RetryTx` is
always allowed, matching of which errors to retry is still the responsible
of the inner `RetryTx`.
Co-authored-by: Mathieu Fenniak <mathieu@fenniak.net>
Reviewed-on: https://codeberg.org/forgejo/forgejo/pulls/12446
Reviewed-by: Mathieu Fenniak <mfenniak@noreply.codeberg.org>
`DefaultMaxInSize` is an internal parameter for limiting the size of `field IN (...)` clauses in DB queries, which is a reasonable thing to do -- in addition to the errors noted when [originally introduced](https://github.com/go-gitea/gitea/pull/4594), there are technical limits that apply to each of PostgreSQL, MySQL, and SQLite which would prevent an unbounded size for a query like this. However: the size is incredibly small at 50, and, the implementation of `DefaultMaxInSize` is really wasteful with copy-and-paste coding.
This PR:
- introduces `GetByIDs` which fetches a `map[int64]*Model` from the database for an array of ID values, while respecting `IN` clause size limits
- introduces `GetByFieldIn` which fetches a `map[int64][]*Model` from the database for an array of field values, while respecting `IN` clause size limits
- uses `slices.Chunk` for other locations where queries are too complex for these implementations
- bumps the `DefaultMaxInSize` parameter from 50 to 500, a conservative increase well under known limits, but 10x the current value:
- PostgreSQL supports up to 1GB query text size with 65,535 parameters, but I've experienced performance degradation at high value counts
- MySQL supports 64MB query text size without known limits of parameter count
- SQLite supports 32,766 parameters in a query
## Checklist
The [contributor guide](https://forgejo.org/docs/next/contributor/) contains information that will be helpful to first time contributors. All work and communication must conform to Forgejo's [AI Agreement](https://codeberg.org/forgejo/governance/src/branch/main/AIAgreement.md). There also are a few [conditions for merging Pull Requests in Forgejo repositories](https://codeberg.org/forgejo/governance/src/branch/main/PullRequestsAgreement.md). You are also welcome to join the [Forgejo development chatroom](https://matrix.to/#/#forgejo-development:matrix.org).
### Tests for Go changes
- I added test coverage for Go changes...
- [x] in their respective `*_test.go` for unit tests.
- Refactored functions are assumed to be covered by existing tests to some extent; that assumption is probably wrong but the changes here are relatively easily reviewed for correctness as well.
- [ ] in the `tests/integration` directory if it involves interactions with a live Forgejo server.
- I ran...
- [x] `make pr-go` before pushing
### Documentation
- [ ] I created a pull request [to the documentation](https://codeberg.org/forgejo/docs) to explain to Forgejo users how to use this change.
- [x] I did not document these changes and I do not expect someone else to do it.
### Release notes
- [ ] This change will be noticed by a Forgejo user or admin (feature, bug fix, performance, etc.). I suggest to include a release note for this change.
- [x] This change is not visible to a Forgejo user or admin (refactor, dependency upgrade, etc.). I think there is no need to add a release note for this change.
Reviewed-on: https://codeberg.org/forgejo/forgejo/pulls/11999
Reviewed-by: Andreas Ahlenstorf <aahlenst@noreply.codeberg.org>
Co-authored-by: Mathieu Fenniak <mathieu@fenniak.net>
Co-committed-by: Mathieu Fenniak <mathieu@fenniak.net>
Fixes#11438.
Whenever a "unique constraint violation" error is encountered by package mutation, detect if a `xorm.ErrUniqueConstraintViolation` error occurs. If it does, retry the entire transaction.
## Checklist
The [contributor guide](https://forgejo.org/docs/next/contributor/) contains information that will be helpful to first time contributors. There also are a few [conditions for merging Pull Requests in Forgejo repositories](https://codeberg.org/forgejo/governance/src/branch/main/PullRequestsAgreement.md). You are also welcome to join the [Forgejo development chatroom](https://matrix.to/#/#forgejo-development:matrix.org).
### Tests for Go changes
(can be removed for JavaScript changes)
- I added test coverage for Go changes...
- [ ] in their respective `*_test.go` for unit tests.
- [ ] in the `tests/integration` directory if it involves interactions with a live Forgejo server.
- I ran...
- [ ] `make pr-go` before pushing
### Documentation
- [ ] I created a pull request [to the documentation](https://codeberg.org/forgejo/docs) to explain to Forgejo users how to use this change.
- [x] I did not document these changes and I do not expect someone else to do it.
### Release notes
- [x] This change will be noticed by a Forgejo user or admin (feature, bug fix, performance, etc.). I suggest to include a release note for this change.
- [ ] This change is not visible to a Forgejo user or admin (refactor, dependency upgrade, etc.). I think there is no need to add a release note for this change.
Reviewed-on: https://codeberg.org/forgejo/forgejo/pulls/11776
Reviewed-by: Andreas Ahlenstorf <aahlenst@noreply.codeberg.org>
Co-authored-by: Mathieu Fenniak <mathieu@fenniak.net>
Co-committed-by: Mathieu Fenniak <mathieu@fenniak.net>
Adds support for `optional.Option[T]` to be used on an xorm schema struct to represent nullable fields. The `optional.None[T]()` value will be stored in the database as `NULL`.
```go
type OptionString struct {
ID int64 `xorm:"pk autoincr"`
StringField optional.Option[string]
}
```
Before this change, it is possible to represent a nullable field in two reasonable ways: , or as a `sql.Null[T]` (eg. `StringField sql.Null[string]`). The problems with these are:
- as a pointer (eg. `StringField *string`) -- but this introduces the risk of panics when `nil` values are dereferenced, and makes it difficult to use literals in structure creation (although `new()` in Go 1.26 would reduce this issue when Forgejo is upgraded to it)
- as a `sql.Null[T]` -- but this "leaks" references to the `database/sql` package for anything that interacts with Forgejo models, and it's API is awkward as nothing gates you into checking the `Valid` field before you access and use the `V` field
`optional.Option[T]` addresses these points and provides a single way to use an optional primitive type, with a safe check-before-access interface, which can be used consistently throughout model code and other application code. Figuring out the best way to handle this became a blocker to me for [adding foreign keys to nullable fields](https://codeberg.org/forgejo/discussions/issues/385#issuecomment-10218316) in database models, which is what drove me to implement this solution.
## Notes: Filtering on `Option[T]` Fields
It is supported and functional to perform queries with xorm beans with non-None `Option` values. For example:
```go
cond := &OptionString{
StringField: optional.Some("hello"),
}
err := db.GetEngine(t.Context()).Find(&arr, cond)
```
will generate a database query `WHERE string_field = 'hello'`, and correctly filter the records.
It is **not** supported to perform queries with `None` values, for two reasons:
- xorm cannot distinguish between an explicit `&OptionString{ StringField: optional.None[string]() }`, and `&OptionString{}`. Both of them have the `StringField` field set to the zero-value of `Option[String]`.
- For this SQL query to be formatted correctly, it would require `WHERE string_field IS NOT NULL`, not `WHERE string_field = NULL`. This is not how xorm generated bean-based queries.
This is similar to the risk that exists with any other field querying on its zero-value with xorm. It's an unfortunate structural limitation of xorm, and can lead to developers believing database queries are performing filtering that they are not.
(perhaps we can mitigate this risk with semgrep or other automated tooling in the future)
## Checklist
The [contributor guide](https://forgejo.org/docs/next/contributor/) contains information that will be helpful to first time contributors. There also are a few [conditions for merging Pull Requests in Forgejo repositories](https://codeberg.org/forgejo/governance/src/branch/main/PullRequestsAgreement.md). You are also welcome to join the [Forgejo development chatroom](https://matrix.to/#/#forgejo-development:matrix.org).
### Tests for Go changes
(can be removed for JavaScript changes)
- I added test coverage for Go changes...
- [x] in their respective `*_test.go` for unit tests.
- [ ] in the `tests/integration` directory if it involves interactions with a live Forgejo server.
- I ran...
- [x] `make pr-go` before pushing
### Documentation
- [ ] I created a pull request [to the documentation](https://codeberg.org/forgejo/docs) to explain to Forgejo users how to use this change.
- [x] I did not document these changes and I do not expect someone else to do it.
### Release notes
- [ ] This change will be noticed by a Forgejo user or admin (feature, bug fix, performance, etc.). I suggest to include a release note for this change.
- [x] This change is not visible to a Forgejo user or admin (refactor, dependency upgrade, etc.). I think there is no need to add a release note for this change.
Reviewed-on: https://codeberg.org/forgejo/forgejo/pulls/11553
Reviewed-by: Gusted <gusted@noreply.codeberg.org>
Co-authored-by: Mathieu Fenniak <mathieu@fenniak.net>
Co-committed-by: Mathieu Fenniak <mathieu@fenniak.net>
This modifies usernames of ActivityPub accounts to use the @example@example.tld
format with an additional optional port component (e.g. @user@example.tld:42).
This allows accounts from ActivityPub servers with more relaxed username
requirements than those of Forgejo's to interact with Forgejo. Forgejo would
also follow a "de facto" standard of ActivityPub implementations.
By separating different information using @'s, we also gain future
opportunities to store more information about ActivityPub accounts internally,
so that we won't have to rely on e.g. the amount of dashes in a username as
my migration currently does.
Continuation of Aravinth's work: https://codeberg.org/forgejo/forgejo/pulls/4778
Reviewed-on: https://codeberg.org/forgejo/forgejo/pulls/9254
Reviewed-by: jerger <jerger@noreply.codeberg.org>
Reviewed-by: Ellen Εμιλία Άννα Zscheile <fogti@noreply.codeberg.org>
Reviewed-by: Gusted <gusted@noreply.codeberg.org>
Co-authored-by: Panagiotis "Ivory" Vasilopoulos <git@n0toose.net>
Co-committed-by: Panagiotis "Ivory" Vasilopoulos <git@n0toose.net>
This PR fixes a number of typos throughout the entire repository. Running https://github.com/crate-ci/typos and then changing all occurrences that I naively deemed "safe enough".
Reviewed-on: https://codeberg.org/forgejo/forgejo/pulls/10753
Reviewed-by: Gusted <gusted@noreply.codeberg.org>
Co-authored-by: Christoph Mewes <christoph@kubermatic.com>
Co-committed-by: Christoph Mewes <christoph@kubermatic.com>
In support of adding foreign keys to the `action_runner_token` table, this PR also had to:
- Add detection and error if a table with "soft delete" is used with a foreign key, because it causes a tricky to track down foreign key violation
- Remove unused xorm "soft delete" capability on the `action_runner_token` table
- Change the `RepoID` and `OwnerID` fields to use the value `NULL` to indicate that this scope wasn't valid for the token, rather than the value `0`
## Checklist
The [contributor guide](https://forgejo.org/docs/next/contributor/) contains information that will be helpful to first time contributors. There also are a few [conditions for merging Pull Requests in Forgejo repositories](https://codeberg.org/forgejo/governance/src/branch/main/PullRequestsAgreement.md). You are also welcome to join the [Forgejo development chatroom](https://matrix.to/#/#forgejo-development:matrix.org).
### Tests
- I added test coverage for Go changes...
- [x] in their respective `*_test.go` for unit tests.
- [ ] in the `tests/integration` directory if it involves interactions with a live Forgejo server.
- I added test coverage for JavaScript changes...
- [ ] in `web_src/js/*.test.js` if it can be unit tested.
- [ ] in `tests/e2e/*.test.e2e.js` if it requires interactions with a live Forgejo server (see also the [developer guide for JavaScript testing](https://codeberg.org/forgejo/forgejo/src/branch/forgejo/tests/e2e/README.md#end-to-end-tests)).
### Documentation
- [ ] I created a pull request [to the documentation](https://codeberg.org/forgejo/docs) to explain to Forgejo users how to use this change.
- [x] I did not document these changes and I do not expect someone else to do it.
### Release notes
- [ ] I do not want this change to show in the release notes.
- [x] I want the title to show in the release notes with a link to this pull request.
- [ ] I want the content of the `release-notes/<pull request number>.md` to be be used for the release notes instead of the title.
Reviewed-on: https://codeberg.org/forgejo/forgejo/pulls/10756
Reviewed-by: Andreas Ahlenstorf <aahlenst@noreply.codeberg.org>
Co-authored-by: Mathieu Fenniak <mathieu@fenniak.net>
Co-committed-by: Mathieu Fenniak <mathieu@fenniak.net>
```
NAME:
forgejo doctor cleanup-commit-status - Cleanup extra records in commit_status table
USAGE:
forgejo doctor cleanup-commit-status
DESCRIPTION:
Forgejo suffered from a bug which caused the creation of more entries in the
"commit_status" table than necessary. This operation removes the redundant
data caused by the bug. Removing this data is almost always safe.
These reundant records can be accessed by users through the API, making it
possible, but unlikely, that removing it could have an impact to
integrating services (API: /repos/{owner}/{repo}/commits/{ref}/statuses).
It is safe to run while Forgejo is online.
On very large Forgejo instances, the performance of operation will improve
if the buffer-size option is used with large values. Approximately 130 MB of
memory is required for every 100,000 records in the buffer.
Bug reference: https://codeberg.org/forgejo/forgejo/issues/10671
OPTIONS:
--help, -h show help
--custom-path string, -C string Set custom path (defaults to '{WorkPath}/custom')
--config string, -c string Set custom config file (defaults to '{WorkPath}/custom/conf/app.ini')
--work-path string, -w string Set Forgejo's working path (defaults to the directory of the Forgejo binary)
--verbose, -V Show process details
--dry-run Report statistics from the operation but do not modify the database
--buffer-size int Record count per query while iterating records; larger values are typically faster but use more memory (default: 100000)
--delete-chunk-size int Number of records to delete per DELETE query (default: 1000)
```
The cleanup effectively performs `SELECT * FROM commit_status ORDER BY repo_id, sha, context, index, id`, and iterates through the records. Whenever `index, id` changes without the other fields changing, then it's a useless record that can be deleted. The major complication is doing that at scale without bringing the entire database table into memory, which is performed through a new iteration method `IterateByKeyset`.
Manually tested against a 455,303 record table in PostgreSQL, MySQL, and SQLite, which was reduced to 10,781 records, dropping 97.5% of the records.
Reviewed-on: https://codeberg.org/forgejo/forgejo/pulls/10686
Reviewed-by: Gusted <gusted@noreply.codeberg.org>
Co-authored-by: Mathieu Fenniak <mathieu@fenniak.net>
Co-committed-by: Mathieu Fenniak <mathieu@fenniak.net>
- Create `modules/testimport/import.go` to centralize blank import needed for tests (in order to run the `init` function) to simplify maintenance.
- Remove the imports that are not needed.
Reviewed-on: https://codeberg.org/forgejo/forgejo/pulls/10662
Reviewed-by: Michael Kriese <michael.kriese@gmx.de>
Reviewed-by: Gusted <gusted@noreply.codeberg.org>
Co-authored-by: limiting-factor <limiting-factor@posteo.com>
Co-committed-by: limiting-factor <limiting-factor@posteo.com>
This PR migrates the unmaintaiend `lib/pq` library to `jackc/pgx`, which is the de-facto standard lib in go for postgres connections these days.
Some implementation notes:
We register both `pgx` and `postgresschema` driver names (for backward comp). We can't register `postgres` as this one is still used by `lib/pq` imported by `go-chi/session`, which is in use when users go for the "postgres" session type in the "Session config.
It is questionable if anyone is really using the "postgres" driver option in the session config - but for consistency, it would be good to also migrate to `pgx` there, especially as the code lives within Forgejo under [go-chi/session](https://code.forgejo.org/go-chi/session).
`pgx` supports multi-host notation in the connection string. New tests have been added therefore.
`pgx` also allows for connection string parameters such as `?default_query_exec_mode=simple_protocol`. This should possibly allow running with `pgbouncer` "transaction" mode instead of "session", which could substantially enhance Postgres query handling.
## Checklist
### Tests
- I added test coverage for Go changes...
- [x] in their respective `*_test.go` for unit tests.
- [ ] in the `tests/integration` directory if it involves interactions with a live Forgejo server.
- I added test coverage for JavaScript changes...
- [ ] in `web_src/js/*.test.js` if it can be unit tested.
- [ ] in `tests/e2e/*.test.e2e.js` if it requires interactions with a live Forgejo server (see also the [developer guide for JavaScript testing](https://codeberg.org/forgejo/forgejo/src/branch/forgejo/tests/e2e/README.md#end-to-end-tests)).
### Documentation
- [ ] I created a pull request [to the documentation](https://codeberg.org/forgejo/docs) to explain to Forgejo users how to use this change.
- [x] I did not document these changes and I do not expect someone else to do it.
### Release notes
- [ ] I do not want this change to show in the release notes.
- [x] I want the title to show in the release notes with a link to this pull request.
- [ ] I want the content of the `release-notes/<pull request number>.md` to be be used for the release notes instead of the title.
Reviewed-on: https://codeberg.org/forgejo/forgejo/pulls/10219
Reviewed-by: Mathieu Fenniak <mfenniak@noreply.codeberg.org>
Co-authored-by: pat-s <patrick.schratz@gmail.com>
Co-committed-by: pat-s <patrick.schratz@gmail.com>
Within Codeberg we are looking into distributing the database queries, we tried forgejo/forgejo!7212 on several occasions but never got it to work.
After a long debugging session in a staging environment I was able to find two bugs that made it impossible for this feature to work: forgejo/docs!1587 which resulted in replica engines never being configured and used if you followed the documentation. The other bug is what this patch intends to fix. In order to do some database operation, you need the database engine - it will first look if one is set for the context (only useful for transactions) and otherwise create a new session of the engine from the master engine `x`. The problem is that `x` is explicitly set to be the master engine and not the engine group (that includes the replica engines) - Unless the code uses `DefaultContext`, which is almost nowhere used after some great refactoring in Gitea to use the passed context, it did not use the replica engines.
Get engine from the `DefaultContext` (which is set to the enginegroup) and create a new session from that.
20f8572b92/models/db/engine.go (L220-L231)
And `SetDefaultEngine` is called from 20f8572b92/models/db/engine.go (L212)
Where `eng` is the engine group.
## Test
1. Configure database replicas.
2. Start Forgejo.
3. Verify Forgejo loads.
4. Stop the database replicas.
5. Verify Forgejo shows 500 errors.
Reviewed-on: https://codeberg.org/forgejo/forgejo/pulls/10140
Reviewed-by: Mathieu Fenniak <mfenniak@noreply.codeberg.org>
Co-authored-by: Gusted <postmaster@gusted.xyz>
Co-committed-by: Gusted <postmaster@gusted.xyz>
Although #9922 was deployed to Codeberg, it was reported on Matrix that a user observed a `-1` pull request count.
@Gusted checked and verified that the stats stored in redis appeared incorrect, and that no errors occurred on Codeberg that included the repo ID (eg. deadlocks, SQL queries).
```
127.0.0.1:6379> GET Repo:CountPulls:924266
"1"
127.0.0.1:6379> GET Repo:CountPullsClosed:924266
"2"
```
One possible cause is that when `UpdateRepoIssueNumbers` is invoked and invalidates the cache key for the repository, it is currently in a transaction; the next request for that cached count could be computed before the transaction is committed and the update is visible. It's been verified that `UpdateRepoIssueNumbers` is called within a transaction in most interactions (I put a panic in it if `db.InTransaction(ctx)`, and most related tests failed).
This PR fixes that hole by performing the cache invalidation in an `AfterTx()` hook which is invoked after the transaction is committed to the database.
(Another possible cause is documented in #10127)
Reviewed-on: https://codeberg.org/forgejo/forgejo/pulls/10130
Reviewed-by: Gusted <gusted@noreply.codeberg.org>
Co-authored-by: Mathieu Fenniak <mathieu@fenniak.net>
Co-committed-by: Mathieu Fenniak <mathieu@fenniak.net>
Adds foreign keys to the table pull_request which are covered by the doctor's db consistency check:
- issue_id -> issue
- base_repo_id -> repository
Note that other fields that look like references -- `head_repo_id` and `merger` -- are not covered by the db consistency check and therefore out-of-scope for the first phase of foreign keys. They're on my list for future more detailed evaluation.
In addition to running automated tests (and making a few tweaks to get them to pass), I performed manual testing of:
- Deleting the base repo of a pull request -- the repo is deleted without error and the pull request is deleted as well.
- Deleting the issue behind a PR via an issue delete API call -- this code-path handles deleting a PR correctly.
## Checklist
The [contributor guide](https://forgejo.org/docs/next/contributor/) contains information that will be helpful to first time contributors. There also are a few [conditions for merging Pull Requests in Forgejo repositories](https://codeberg.org/forgejo/governance/src/branch/main/PullRequestsAgreement.md). You are also welcome to join the [Forgejo development chatroom](https://matrix.to/#/#forgejo-development:matrix.org).
### Tests
- I added test coverage for Go changes...
- [ ] in their respective `*_test.go` for unit tests.
- [ ] in the `tests/integration` directory if it involves interactions with a live Forgejo server.
- I added test coverage for JavaScript changes...
- [ ] in `web_src/js/*.test.js` if it can be unit tested.
- [ ] in `tests/e2e/*.test.e2e.js` if it requires interactions with a live Forgejo server (see also the [developer guide for JavaScript testing](https://codeberg.org/forgejo/forgejo/src/branch/forgejo/tests/e2e/README.md#end-to-end-tests)).
### Documentation
- [ ] I created a pull request [to the documentation](https://codeberg.org/forgejo/docs) to explain to Forgejo users how to use this change.
- [x] I did not document these changes and I do not expect someone else to do it.
### Release notes
- [ ] I do not want this change to show in the release notes.
- [x] I want the title to show in the release notes with a link to this pull request.
- [ ] I want the content of the `release-notes/<pull request number>.md` to be be used for the release notes instead of the title.
Reviewed-on: https://codeberg.org/forgejo/forgejo/pulls/9832
Reviewed-by: Earl Warren <earl-warren@noreply.codeberg.org>
Co-authored-by: Mathieu Fenniak <mathieu@fenniak.net>
Co-committed-by: Mathieu Fenniak <mathieu@fenniak.net>
In response to the issue #9755, @Gusted had manually created an index on `action_task` in preparation for the v13 migration of Codeberg. The restart of v13 dropped the index because `Sync`'s default behaviour is to do so. I'm inclined to think this doesn't make sense...
Manually tested:
- Confirmed that index with `IDX_...` prefix will be dropped if present and unexpected
- Confirmed that index is not dropped w/ this patch
## Checklist
The [contributor guide](https://forgejo.org/docs/next/contributor/) contains information that will be helpful to first time contributors. There also are a few [conditions for merging Pull Requests in Forgejo repositories](https://codeberg.org/forgejo/governance/src/branch/main/PullRequestsAgreement.md). You are also welcome to join the [Forgejo development chatroom](https://matrix.to/#/#forgejo-development:matrix.org).
### Tests
- I added test coverage for Go changes...
- [ ] in their respective `*_test.go` for unit tests.
- [ ] in the `tests/integration` directory if it involves interactions with a live Forgejo server.
- I added test coverage for JavaScript changes...
- [ ] in `web_src/js/*.test.js` if it can be unit tested.
- [ ] in `tests/e2e/*.test.e2e.js` if it requires interactions with a live Forgejo server (see also the [developer guide for JavaScript testing](https://codeberg.org/forgejo/forgejo/src/branch/forgejo/tests/e2e/README.md#end-to-end-tests)).
### Documentation
- [ ] I created a pull request [to the documentation](https://codeberg.org/forgejo/docs) to explain to Forgejo users how to use this change.
- [x] I did not document these changes and I do not expect someone else to do it.
### Release notes
- [ ] I do not want this change to show in the release notes.
- [x] I want the title to show in the release notes with a link to this pull request.
- [ ] I want the content of the `release-notes/<pull request number>.md` to be be used for the release notes instead of the title.
Reviewed-on: https://codeberg.org/forgejo/forgejo/pulls/9796
Reviewed-by: Gusted <gusted@noreply.codeberg.org>
Co-authored-by: Mathieu Fenniak <mathieu@fenniak.net>
Co-committed-by: Mathieu Fenniak <mathieu@fenniak.net>
Discovered while testing #9708:
Fresh install databases, which is also a process used by the integration tests, create tables by using `SyncAllTables`. The order that tables are created is ungoverned -- it occurs based upon the order that Go calls each module's `init` to register their schema models. With the current foreign keys in the database, this does not yet cause an error. But it will shortly.
I've manually tested this fix in this way:
- The correct order to create tables is that indicated by `foreignKeySortInsert`. This creates tables that are *referenced* by foreign keys before creating tables that do the referencing (eg. `user` before `tracked_time`).
- If I modify this PR and sort the keys the opposite way `foreignKeySortDelete`, then integration tests fail:
```
??? [TestLogger] 2025/10/15 22:37:58 models/db/engine.go:270:InitEngineWithMigration() [E] [Error SQL Query] CREATE TABLE IF NOT EXISTS "gtestschema"."tracked_time" ("id" BIGSERIAL PRIMARY KEY NOT NULL, "issue_id" BIGINT NULL, "user_id" BIGINT NULL, "created_unix" BIGINT NULL, "time" BIGINT NOT NULL, "deleted" BOOL DEFAULT false NOT NULL, CONSTRAINT "tracked_time_issue_id_fkey" FOREIGN KEY ("issue_id") REFERENCES "gtestschema"."issue" ("id"), CONSTRAINT "tracked_time_user_id_fkey" FOREIGN KEY ("user_id") REFERENCES "gtestschema"."user" ("id")); [] - pq: relation "gtestschema.issue" does not exist
??? [TestLogger] 2025/10/15 22:37:58 routers/common/db.go:36:InitDBEngine() [E] ORM engine initialization attempt #1/10 failed.
```
Therefore this PR which doesn't appear to fix anything today fixes a latent bug that will occur shortly (possibly in #9397, possibly when another foreign key is added, possibly if Go changes the order in which `init` functions are invoked).
## Checklist
The [contributor guide](https://forgejo.org/docs/next/contributor/) contains information that will be helpful to first time contributors. There also are a few [conditions for merging Pull Requests in Forgejo repositories](https://codeberg.org/forgejo/governance/src/branch/main/PullRequestsAgreement.md). You are also welcome to join the [Forgejo development chatroom](https://matrix.to/#/#forgejo-development:matrix.org).
### Tests
- I added test coverage for Go changes...
- [ ] in their respective `*_test.go` for unit tests.
- [ ] in the `tests/integration` directory if it involves interactions with a live Forgejo server.
- I added test coverage for JavaScript changes...
- [ ] in `web_src/js/*.test.js` if it can be unit tested.
- [ ] in `tests/e2e/*.test.e2e.js` if it requires interactions with a live Forgejo server (see also the [developer guide for JavaScript testing](https://codeberg.org/forgejo/forgejo/src/branch/forgejo/tests/e2e/README.md#end-to-end-tests)).
### Documentation
- [ ] I created a pull request [to the documentation](https://codeberg.org/forgejo/docs) to explain to Forgejo users how to use this change.
- [x] I did not document these changes and I do not expect someone else to do it.
### Release notes
- [x] I do not want this change to show in the release notes.
- [ ] I want the title to show in the release notes with a link to this pull request.
- [ ] I want the content of the `release-notes/<pull request number>.md` to be be used for the release notes instead of the title.
Reviewed-on: https://codeberg.org/forgejo/forgejo/pulls/9709
Reviewed-by: Michael Kriese <michael.kriese@gmx.de>
Co-authored-by: Mathieu Fenniak <mathieu@fenniak.net>
Co-committed-by: Mathieu Fenniak <mathieu@fenniak.net>
Noted in #9557 -- as more foreign keys are added, a couple test locations (`reverseproxy_test.go` and `cmd_admin_test.go`) that truncate fixture data have a growing list of impacted tables. This PR introduces a new `TruncateBeansCascade` which can be used for tests, as well as enhancing some helper functions that I expect will be used later to support automatic operation ordering.
## Checklist
The [contributor guide](https://forgejo.org/docs/next/contributor/) contains information that will be helpful to first time contributors. There also are a few [conditions for merging Pull Requests in Forgejo repositories](https://codeberg.org/forgejo/governance/src/branch/main/PullRequestsAgreement.md). You are also welcome to join the [Forgejo development chatroom](https://matrix.to/#/#forgejo-development:matrix.org).
### Tests
- I added test coverage for Go changes...
- [x] in their respective `*_test.go` for unit tests. (**Implicitly tested** by virtue of usage in a test.)
- [ ] in the `tests/integration` directory if it involves interactions with a live Forgejo server.
- I added test coverage for JavaScript changes...
- [ ] in `web_src/js/*.test.js` if it can be unit tested.
- [ ] in `tests/e2e/*.test.e2e.js` if it requires interactions with a live Forgejo server (see also the [developer guide for JavaScript testing](https://codeberg.org/forgejo/forgejo/src/branch/forgejo/tests/e2e/README.md#end-to-end-tests)).
### Documentation
- [ ] I created a pull request [to the documentation](https://codeberg.org/forgejo/docs) to explain to Forgejo users how to use this change.
- [x] I did not document these changes and I do not expect someone else to do it.
### Release notes
- [x] I do not want this change to show in the release notes.
- [ ] I want the title to show in the release notes with a link to this pull request.
- [ ] I want the content of the `release-notes/<pull request number>.md` to be be used for the release notes instead of the title.
Reviewed-on: https://codeberg.org/forgejo/forgejo/pulls/9684
Reviewed-by: Gusted <gusted@noreply.codeberg.org>
Co-authored-by: Mathieu Fenniak <mathieu@fenniak.net>
Co-committed-by: Mathieu Fenniak <mathieu@fenniak.net>
Fixes#9644.
Rewrites `db.Iterate` so that it performs DB queries in this format:
- First: `SELECT ...columns... FROM table ORDER BY id LIMIT ...buffer-size...`
- Subsequent buffer fills: adding a `WHERE id > ...last-id-from-previous...`
This approach:
- Prevents records from being missed or returned twice
- Returns records in a predictable order
- Should be faster, by virtue of using database indexes on the primary key to perform the query
- Doesn't rely on any unpredictable database behaviour when using `LIMIT` and `OFFSET` without an `ORDER BY`
- (Downside: does require reflection to read field values off Go structures for the primary key value)
Expands the automated tests to include the predicted failure case identified in #9644, which verified the previous broken behaviour, as well as verifying that the `cond` parameter is applied which was previously not covered by test automation.
Reviewed-on: https://codeberg.org/forgejo/forgejo/pulls/9657
Reviewed-by: Earl Warren <earl-warren@noreply.codeberg.org>
Co-authored-by: Mathieu Fenniak <mathieu@fenniak.net>
Co-committed-by: Mathieu Fenniak <mathieu@fenniak.net>
Adds four foreign keys:
- stopwatch -- issue_id -> issue, user_id -> user
- tracked_time -- issue_id -> issue, user_id -> user
The majority of work encompassed in this PR is updating testing and support infrastructure to support foreign keys:
- `models/db/foreign_keys.go` adds new capabilities to sort registered tables into the right insertion order to avoid violating foreign keys
- `RecreateTables`, used by migration testing and the `doctor recreate-table` CLI, has been updated to support tables with foreign keys; new restrictions require that FK-related tables be rebuilt at the same time
- test fixture data is inserted in foreign-key order, and deleted in the reverse
An upgrade to xorm v1.3.9-forgejo.2 is incorporated in this PR, as two unexpected behaviors in the foreign key schema management were discovered during development of the updated `RecreateTables` routine.
Work in this PR is laid out to be reviewed easier commit-by-commit.
## Checklist
The [contributor guide](https://forgejo.org/docs/next/contributor/) contains information that will be helpful to first time contributors. There also are a few [conditions for merging Pull Requests in Forgejo repositories](https://codeberg.org/forgejo/governance/src/branch/main/PullRequestsAgreement.md). You are also welcome to join the [Forgejo development chatroom](https://matrix.to/#/#forgejo-development:matrix.org).
### Tests
- I added test coverage for Go changes...
- [x] in their respective `*_test.go` for unit tests.
- [x] in the `tests/integration` directory if it involves interactions with a live Forgejo server.
- I added test coverage for JavaScript changes...
- [ ] in `web_src/js/*.test.js` if it can be unit tested.
- [ ] in `tests/e2e/*.test.e2e.js` if it requires interactions with a live Forgejo server (see also the [developer guide for JavaScript testing](https://codeberg.org/forgejo/forgejo/src/branch/forgejo/tests/e2e/README.md#end-to-end-tests)).
### Documentation
- [ ] I created a pull request [to the documentation](https://codeberg.org/forgejo/docs) to explain to Forgejo users how to use this change.
- [x] I did not document these changes and I do not expect someone else to do it.
### Release notes
- [ ] I do not want this change to show in the release notes.
- [ ] I want the title to show in the release notes with a link to this pull request.
- [x] I want the content of the `release-notes/<pull request number>.md` to be be used for the release notes instead of the title.
Reviewed-on: https://codeberg.org/forgejo/forgejo/pulls/9373
Reviewed-by: Earl Warren <earl-warren@noreply.codeberg.org>
Reviewed-by: Gusted <gusted@noreply.codeberg.org>
Co-authored-by: Mathieu Fenniak <mathieu@fenniak.net>
Co-committed-by: Mathieu Fenniak <mathieu@fenniak.net>
MariaDB supports "INSERT ... RETURNING ..." since 10.5.0, so this patch makes `mysqlGetNextResourceIndex` use that, provided the query is run on MariaDB of a sufficient version. If it's not supported it proceeds as it always did.
Reviewed-on: https://codeberg.org/forgejo/forgejo/pulls/8691
Reviewed-by: Gusted <gusted@noreply.codeberg.org>
Co-authored-by: BtbN <btbn@btbn.de>
Co-committed-by: BtbN <btbn@btbn.de>
- Add the written HTTP status after completing the HTTP response. This makes it easier to find that one request that returns a different status code (ref. https://codeberg.org/Codeberg/Community/issues/2049#issue-1972600)
- Add the affected amount of rows and last insert ID after the SQL query is done, I have not yet a concrete use-case but this might help with debugging which ID corresponds to some SQL query that someone might want to take a closer look at and if some SQL query affects more than necessary amount of rows.
Reviewed-on: https://codeberg.org/forgejo/forgejo/pulls/8680
Reviewed-by: Earl Warren <earl-warren@noreply.codeberg.org>
Co-authored-by: Gusted <postmaster@gusted.xyz>
Co-committed-by: Gusted <postmaster@gusted.xyz>
- Older code (154 places to be exact) that want to do transactions uses `TxContext`. If the context is not already in a transaction then it uses `DefaultContext` for the new transaction.
- Not reusing the context it was given leads to two problems: for tracing (forgejo/forgejo#6470) any SQL code that is executed inside this transaction is not shown in the trace. If the context is cancelled then the transaction is not cancelled and will always complete (given there's no SQL error).
- When this function was introduced it didn't take a parent context, hence the usage of `DefaultContext`. When `parentCtx` was introduced it was only used to avoid nested transactions and use the parent's transaction. I do not see any reasons why the parent context cannot be reused, it is reused in the newer transaction function `WithTx` without any known problems.
Reviewed-on: https://codeberg.org/forgejo/forgejo/pulls/8464
Reviewed-by: Otto <otto@codeberg.org>
Reviewed-by: Earl Warren <earl-warren@noreply.codeberg.org>
Co-authored-by: Gusted <postmaster@gusted.xyz>
Co-committed-by: Gusted <postmaster@gusted.xyz>
- A mistake that is often made is to not put a empty fixture for tables that gets populated in a test and should be cleaned up between runs. The CI does not detect such issues as these never arise on the first run of the test suite and are only noticed when developers run a test for a second time, unless you are aware of this behavior (which very few do as this is not documented and pure folk knowledge) can end up in chasing a bug that does not exist for hours. forgejo/forgejo#7041, forgejo/forgejo#7419, meissa/forgejo#21, forgejo/forgejo#5771
- Because we now own the fixture loader (forgejo/forgejo#7715), its rather simple to ensure that all tables that did not receive fixtures should be cleaned between runs and removes the need to place empty fixture files.
Reviewed-on: https://codeberg.org/forgejo/forgejo/pulls/8353
Reviewed-by: Earl Warren <earl-warren@noreply.codeberg.org>
Reviewed-by: Otto <otto@codeberg.org>
Co-authored-by: Gusted <postmaster@gusted.xyz>
Co-committed-by: Gusted <postmaster@gusted.xyz>
The following will trigger a XORM warning:
```
type Repository struct {
ID int64 `xorm:"pk autoincr"`
Topics []string `xorm:"TEXT JSON NOT NULL"`
}
```
that looks like:
```
[W] Table repository Column topics db default is '', struct default is
```
it cannot be resolved because:
- SQLite requires a default when there is a NOT NULL
- MySQL forbids a default for TEXT
Reviewed-on: https://codeberg.org/forgejo/forgejo/pulls/8021
Reviewed-by: Gusted <gusted@noreply.codeberg.org>
Co-authored-by: Earl Warren <contact@earl-warren.org>
Co-committed-by: Earl Warren <contact@earl-warren.org>
- The default engine is no longer guaranteed to be of the type `*xorm.Engine`, so instead return the interface `db.Engine`.
- Regression of forgejo/forgejo#7212
# Testing
1. Install a Forgejo instance via the setup screen.
Reviewed-on: https://codeberg.org/forgejo/forgejo/pulls/7452
Reviewed-by: Earl Warren <earl-warren@noreply.codeberg.org>
Co-authored-by: Gusted <postmaster@gusted.xyz>
Co-committed-by: Gusted <postmaster@gusted.xyz>
- For every process that is spawned (every new non-trivial goroutine
such as http requests, queues or tasks) start a [execution
tracer](https://pkg.go.dev/runtime/trace). This allows very precise
diagnosis of how each individual process over a time period.
- It's safe and [fast](https://go.dev/blog/execution-traces-2024#low-overhead-tracing) to
be run in production, hence no setting to disable this. There's only
noticable overhead when tracing is actually performed and not continuous.
- Proper tracing support would mean the codebase would be full of
`trace.WithRegion` and `trace.Log`, which feels premature for this patch
as there's no real-world usage yet to indicate which places would need
this the most. So far only Git commands and SQL queries receive somewhat
proper tracing support given that these are used throughout the codebase.
- Make git commands a new process type.
- Add tracing to diagnosis zip file.
use errors.New to replace fmt.Errorf with no parameters
Signed-off-by: RiceChuan <lc582041246@gmail.com>
(cherry picked from commit dfd75944992fc6508ec891b4c29715c23e59e4ed)
Now that my colleague just posted a wonderful blog post https://blog.datalad.org/posts/forgejo-runner-podman-deployment/ on forgejo runner, some time I will try to add that damn codespell action to work on CI here ;) meanwhile some typos managed to sneak in and this PR should address them (one change might be functional in a test -- not sure if would cause a fail or not)
### Release notes
- [ ] I do not want this change to show in the release notes.
- [ ] I want the title to show in the release notes with a link to this pull request.
- [ ] I want the content of the `release-notes/<pull request number>.md` to be be used for the release notes instead of the title.
Reviewed-on: https://codeberg.org/forgejo/forgejo/pulls/4857
Reviewed-by: Earl Warren <earl-warren@noreply.codeberg.org>
Co-authored-by: Yaroslav Halchenko <debian@onerussian.com>
Co-committed-by: Yaroslav Halchenko <debian@onerussian.com>
have repo OrderBy definitions defined in one place and use a single type
for OrderBy database options
(cherry picked from commit bb04311b0b5b7a28f94c4bc409db1c4a04bcef17)
The previous implementation will start multiple POST requests from the
frontend when moving a column and another bug is moving the default
column will never be remembered in fact.
- [x] This PR will allow the default column to move to a non-first
position
- [x] And it also uses one request instead of multiple requests when
moving the columns
- [x] Use a star instead of a pin as the icon for setting the default
column action
- [x] Inserted new column will be append to the end
- [x] Fix#30701 the newly added issue will be append to the end of the
default column
- [x] Fix when deleting a column, all issues in it will be displayed
from UI but database records exist.
- [x] Add a limitation for columns in a project to 20. So the sorting
will not be overflow because it's int8.
---------
Co-authored-by: silverwind <me@silverwind.io>
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
(cherry picked from commit a303c973e0264dab45a787c4afa200e183e0d953)
Conflicts:
routers/web/web.go
e91733468ef726fc9365aa4820cdd5f2ddfdaa23 Add missing database transaction for new issue (#29490) was not cherry-picked
services/issue/issue.go
fe6792dff3 Enable/disable owner and repo projects independently (#28805) was not cherry-picked
Noteable additions:
- `redefines-builtin-id` forbid variable names that shadow go builtins
- `empty-lines` remove unnecessary empty lines that `gofumpt` does not
remove for some reason
- `superfluous-else` eliminate more superfluous `else` branches
Rules are also sorted alphabetically and I cleaned up various parts of
`.golangci.yml`.
(cherry picked from commit 74f0c84fa4245a20ce6fb87dac1faf2aeeded2a2)
Conflicts:
.golangci.yml
apply the linter recommendations to Forgejo code as well
Unify the behaviors of "user create" and "user change-password".
Co-authored-by: KN4CK3R <admin@oldschoolhack.me>
(cherry picked from commit 4c6e2da088cf092a9790df5c84b7b338508fede7)
Conflicts:
- cmd/admin_user_create.go
Resolved by favoring Gitea's version of the conflicting areas.
- docs/content/administration/command-line.en-us.md
Removed, Gitea specific.
- If the database returns a error in integration tests, it should be
marked as a failure of the test.
- Ref: https://codeberg.org/forgejo/forgejo/issues/2962 (this should
help with logging the SQL that is resulting in the error).
Help #29999, or its tests cannot pass.
Also, add some comments to clarify the usage of `TxContext`.
I don't check all usages of `TxContext` because there are too many
(almost 140+). It's a better idea to replace them with `WithTx` instead
of checking them one by one. However, that may be another refactoring
PR.
(cherry picked from commit c6c4d66004c70b24abc8048b39b660b8361a0395)
- I found this while doing some unrelated testing in Forgejo. It wasn't
my intention to log failed SQL queries if they were cancelled (which can
happen quite frequently for larger instances) as in those cases it's not
interesting to know which SQL query was run. My intentation was only to
log an SQL query if there was an error reported by the database.
- Ref #2140
So the caller can check log events at the desired level instead of
being limited to the default level log.INFO
(cherry picked from commit 2fbf5f9555)
(cherry picked from commit e2137a3147)
