jojo/models/unittest/unit_tests.go

170 lines
5.1 KiB
Go
Raw Normal View History

// Copyright 2016 The Gitea Authors. All rights reserved.
// SPDX-License-Identifier: MIT
package unittest
import (
"math"
"testing"
"forgejo.org/models/db"
fix: reduce deadlocks merging PRs w/ async label stat recalcs (#9868) The intent of this change is to reduce the scope of deadlock issues identified in #9785. I've identified other deadlock issues from synthetic testing, so this is not a complete fix, but it's a partial fix. This design was discussed in #9785 and this is the most basic implementation, with a very small scope of work converted to use it. Introduces a new `forgejo.org/services/stats` module which allows for the queuing and routing of recalc requests for object stats; in this case, the "number of issues" that are assigned to a label, and the number of closed issues that are assigned to a label. The reasons that these calculations are performed asynchronously through a queue are: - User operations that are common and performance-sensitive don't have to wait for recalculations that don't need to be exactly up-to-date at all times. For example, merging a pull request will be a faster operation; as it closes an issue, it needs to recalculate `label.num_closed_issues` for every label attached to the PR. - Database deadlocks that can occur between concurrent operations -- for example, if you were holding a lock on an issue while recalculating a label's count of open issues -- can be broken by making the recalculation occur outside of the transaction. ## Checklist The [contributor guide](https://forgejo.org/docs/next/contributor/) contains information that will be helpful to first time contributors. There also are a few [conditions for merging Pull Requests in Forgejo repositories](https://codeberg.org/forgejo/governance/src/branch/main/PullRequestsAgreement.md). You are also welcome to join the [Forgejo development chatroom](https://matrix.to/#/#forgejo-development:matrix.org). ### Tests - I added test coverage for Go changes... - [x] in their respective `*_test.go` for unit tests. - [ ] in the `tests/integration` directory if it involves interactions with a live Forgejo server. - I added test coverage for JavaScript changes... - [ ] in `web_src/js/*.test.js` if it can be unit tested. - [ ] in `tests/e2e/*.test.e2e.js` if it requires interactions with a live Forgejo server (see also the [developer guide for JavaScript testing](https://codeberg.org/forgejo/forgejo/src/branch/forgejo/tests/e2e/README.md#end-to-end-tests)). ### Documentation - [ ] I created a pull request [to the documentation](https://codeberg.org/forgejo/docs) to explain to Forgejo users how to use this change. - [x] I did not document these changes and I do not expect someone else to do it. - Internal developer documentation is present. ### Release notes - [ ] I do not want this change to show in the release notes. - [ ] I want the title to show in the release notes with a link to this pull request. - [x] I want the content of the `release-notes/<pull request number>.md` to be be used for the release notes instead of the title. Reviewed-on: https://codeberg.org/forgejo/forgejo/pulls/9868 Reviewed-by: Gusted <gusted@noreply.codeberg.org> Co-authored-by: Mathieu Fenniak <mathieu@fenniak.net> Co-committed-by: Mathieu Fenniak <mathieu@fenniak.net>
2025-10-31 02:12:36 +01:00
"forgejo.org/services/stats"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"xorm.io/builder"
)
// Code in this file is mainly used by unittest.CheckConsistencyFor, which is not in the unit test for various reasons.
// In the future if we can decouple CheckConsistencyFor into separate unit test code, then this file can be moved into unittest package too.
// NonexistentID an ID that will never exist
const NonexistentID = int64(math.MaxInt64)
type testCond struct {
query any
args []any
}
type testOrderBy string
// Cond create a condition with arguments for a test
func Cond(query any, args ...any) any {
return &testCond{query: query, args: args}
}
// OrderBy creates "ORDER BY" a test query
func OrderBy(orderBy string) any {
return testOrderBy(orderBy)
}
func whereOrderConditions(e db.Engine, conditions []any) db.Engine {
orderBy := "id" // query must have the "ORDER BY", otherwise the result is not deterministic
for _, condition := range conditions {
switch cond := condition.(type) {
case *testCond:
e = e.Where(cond.query, cond.args...)
case testOrderBy:
orderBy = string(cond)
default:
e = e.Where(cond)
}
}
return e.OrderBy(orderBy)
}
// LoadBeanIfExists loads beans from fixture database if exist
func LoadBeanIfExists(bean any, conditions ...any) (bool, error) {
e := db.GetEngine(db.DefaultContext)
return whereOrderConditions(e, conditions).Get(bean)
}
// BeanExists for testing, check if a bean exists
func BeanExists(t testing.TB, bean any, conditions ...any) bool {
exists, err := LoadBeanIfExists(bean, conditions...)
require.NoError(t, err)
return exists
}
// AssertExistsAndLoadBean assert that a bean exists and load it from the test database
fix: mark code comments as Outdated based upon line-of-code existence in current PR commit (#12054) Currently when a commit is pushed to a branch, code comments are marked as Outdated if a `git blame` on the current commit's code returns the same commit as the `git blame` did when the comment was originally created. This implementation doesn't make sense: - It doesn't handle the case correctly where the same line of code exists unaltered in the new commit, but it has been relocated (eg. new lines entered or removed above the location). - It falsely keeps the commit valid if the line of code that the comment was made upon has been removed, if, coincidentally, the line of code that now exists at the commit came from the same source commit. For example, if the line of code that the comment was on was deleted, but the next line of code came from the same commit, the comment will be kept as valid. This PR uses the logic introduced in #12015, using a `git blame --reverse` -- the commit & line that was identified as having the comment on it is reversed, and if it still exists in the new head, then the comment is considered valid. Otherwise it is marked as outdated. Automated tests are added primarily by revising the automated tests in #12015 -- a comment in an existing test case was marked as outdated, even though it shouldn't have been. ## Checklist The [contributor guide](https://forgejo.org/docs/next/contributor/) contains information that will be helpful to first time contributors. All work and communication must conform to Forgejo's [AI Agreement](https://codeberg.org/forgejo/governance/src/branch/main/AIAgreement.md). There also are a few [conditions for merging Pull Requests in Forgejo repositories](https://codeberg.org/forgejo/governance/src/branch/main/PullRequestsAgreement.md). You are also welcome to join the [Forgejo development chatroom](https://matrix.to/#/#forgejo-development:matrix.org). ### Tests for Go changes - I added test coverage for Go changes... - [ ] in their respective `*_test.go` for unit tests. - [x] in the `tests/integration` directory if it involves interactions with a live Forgejo server. - I ran... - [x] `make pr-go` before pushing ### Documentation - [ ] I created a pull request [to the documentation](https://codeberg.org/forgejo/docs) to explain to Forgejo users how to use this change. - [x] I did not document these changes and I do not expect someone else to do it. ### Release notes - [x] This change will be noticed by a Forgejo user or admin (feature, bug fix, performance, etc.). I suggest to include a release note for this change. - [ ] This change is not visible to a Forgejo user or admin (refactor, dependency upgrade, etc.). I think there is no need to add a release note for this change. <!--start release-notes-assistant--> ## Release notes <!--URL:https://codeberg.org/forgejo/forgejo--> - Bug fixes - [PR](https://codeberg.org/forgejo/forgejo/pulls/12054): <!--number 12054 --><!--line 0 --><!--description bWFyayBjb2RlIGNvbW1lbnRzIGFzIE91dGRhdGVkIGJhc2VkIHVwb24gbGluZS1vZi1jb2RlIGV4aXN0ZW5jZSBpbiBjdXJyZW50IFBSIGNvbW1pdA==-->mark code comments as Outdated based upon line-of-code existence in current PR commit<!--description--> <!--end release-notes-assistant--> Reviewed-on: https://codeberg.org/forgejo/forgejo/pulls/12054 Reviewed-by: Andreas Ahlenstorf <aahlenst@noreply.codeberg.org> Co-authored-by: Mathieu Fenniak <mathieu@fenniak.net> Co-committed-by: Mathieu Fenniak <mathieu@fenniak.net>
2026-04-11 23:10:34 +02:00
func AssertExistsAndLoadBean[T any](t require.TestingT, bean T, conditions ...any) T {
exists, err := LoadBeanIfExists(bean, conditions...)
require.NoError(t, err)
assert.True(t, exists,
"Expected to find %+v (of type %T, with conditions %+v), but did not",
bean, bean, conditions)
return bean
}
// AssertExistsAndLoadMap assert that a row exists and load it from the test database
func AssertExistsAndLoadMap(t testing.TB, table string, conditions ...any) map[string]string {
e := db.GetEngine(db.DefaultContext).Table(table)
res, err := whereOrderConditions(e, conditions).Query()
require.NoError(t, err)
assert.Len(t, res, 1,
"Expected to find one row in %s (with conditions %+v), but found %d",
table, conditions, len(res),
)
if len(res) == 1 {
rec := map[string]string{}
for k, v := range res[0] {
rec[k] = string(v)
}
return rec
}
return nil
}
// GetCount get the count of a bean
func GetCount(t testing.TB, bean any, conditions ...any) int {
e := db.GetEngine(db.DefaultContext)
for _, condition := range conditions {
switch cond := condition.(type) {
case *testCond:
e = e.Where(cond.query, cond.args...)
default:
e = e.Where(cond)
}
}
count, err := e.Count(bean)
require.NoError(t, err)
return int(count)
}
// AssertNotExistsBean assert that a bean does not exist in the test database
func AssertNotExistsBean(t testing.TB, bean any, conditions ...any) {
exists, err := LoadBeanIfExists(bean, conditions...)
require.NoError(t, err)
assert.False(t, exists)
}
// AssertExistsIf asserts that a bean exists or does not exist, depending on
// what is expected.
func AssertExistsIf(t testing.TB, expected bool, bean any, conditions ...any) {
exists, err := LoadBeanIfExists(bean, conditions...)
require.NoError(t, err)
assert.Equal(t, expected, exists)
}
// AssertSuccessfulInsert assert that beans is successfully inserted
func AssertSuccessfulInsert(t testing.TB, beans ...any) {
err := db.Insert(db.DefaultContext, beans...)
require.NoError(t, err)
}
// AssertSuccessfulDelete assert that beans is successfully deleted
func AssertSuccessfulDelete(t require.TestingT, beans ...any) {
err := db.DeleteBeans(db.DefaultContext, beans...)
require.NoError(t, err)
}
// AssertCount assert the count of a bean
func AssertCount(t testing.TB, bean, expected any) bool {
return assert.EqualValues(t, expected, GetCount(t, bean))
}
// AssertInt64InRange assert value is in range [low, high]
func AssertInt64InRange(t testing.TB, low, high, value int64) {
assert.True(t, value >= low && value <= high,
"Expected value in range [%d, %d], found %d", low, high, value)
}
// GetCountByCond get the count of database entries matching bean
func GetCountByCond(t testing.TB, tableName string, cond builder.Cond) int64 {
e := db.GetEngine(db.DefaultContext)
count, err := e.Table(tableName).Where(cond).Count()
require.NoError(t, err)
return count
}
// AssertCountByCond test the count of database entries matching bean
func AssertCountByCond(t testing.TB, tableName string, cond builder.Cond, expected int) bool {
return assert.EqualValues(t, expected, GetCountByCond(t, tableName, cond),
"Failed consistency test, the counted bean (of table %s) was %+v", tableName, cond)
}
fix: reduce deadlocks merging PRs w/ async label stat recalcs (#9868) The intent of this change is to reduce the scope of deadlock issues identified in #9785. I've identified other deadlock issues from synthetic testing, so this is not a complete fix, but it's a partial fix. This design was discussed in #9785 and this is the most basic implementation, with a very small scope of work converted to use it. Introduces a new `forgejo.org/services/stats` module which allows for the queuing and routing of recalc requests for object stats; in this case, the "number of issues" that are assigned to a label, and the number of closed issues that are assigned to a label. The reasons that these calculations are performed asynchronously through a queue are: - User operations that are common and performance-sensitive don't have to wait for recalculations that don't need to be exactly up-to-date at all times. For example, merging a pull request will be a faster operation; as it closes an issue, it needs to recalculate `label.num_closed_issues` for every label attached to the PR. - Database deadlocks that can occur between concurrent operations -- for example, if you were holding a lock on an issue while recalculating a label's count of open issues -- can be broken by making the recalculation occur outside of the transaction. ## Checklist The [contributor guide](https://forgejo.org/docs/next/contributor/) contains information that will be helpful to first time contributors. There also are a few [conditions for merging Pull Requests in Forgejo repositories](https://codeberg.org/forgejo/governance/src/branch/main/PullRequestsAgreement.md). You are also welcome to join the [Forgejo development chatroom](https://matrix.to/#/#forgejo-development:matrix.org). ### Tests - I added test coverage for Go changes... - [x] in their respective `*_test.go` for unit tests. - [ ] in the `tests/integration` directory if it involves interactions with a live Forgejo server. - I added test coverage for JavaScript changes... - [ ] in `web_src/js/*.test.js` if it can be unit tested. - [ ] in `tests/e2e/*.test.e2e.js` if it requires interactions with a live Forgejo server (see also the [developer guide for JavaScript testing](https://codeberg.org/forgejo/forgejo/src/branch/forgejo/tests/e2e/README.md#end-to-end-tests)). ### Documentation - [ ] I created a pull request [to the documentation](https://codeberg.org/forgejo/docs) to explain to Forgejo users how to use this change. - [x] I did not document these changes and I do not expect someone else to do it. - Internal developer documentation is present. ### Release notes - [ ] I do not want this change to show in the release notes. - [ ] I want the title to show in the release notes with a link to this pull request. - [x] I want the content of the `release-notes/<pull request number>.md` to be be used for the release notes instead of the title. Reviewed-on: https://codeberg.org/forgejo/forgejo/pulls/9868 Reviewed-by: Gusted <gusted@noreply.codeberg.org> Co-authored-by: Mathieu Fenniak <mathieu@fenniak.net> Co-committed-by: Mathieu Fenniak <mathieu@fenniak.net>
2025-10-31 02:12:36 +01:00
func FlushAsyncCalcs(t testing.TB) {
require.NoError(t, stats.Flush(t.Context()))
}