## Context
the three commits in this series are the first step towards the goal of removing the special casing around `JWT_SECRET`, which is used for various modules via `GetGeneralTokenSigningSecret()`. Ultimately, I want to work towards enabling seamless migration away from general use of the common secret. To enable this, we need proper secret/key rotation support, that is, we need to allow for configuration of additional secrets/keys which are accepted for token validation, but not used to issue tokens.
I have this _Verifier_ support basically implemented, but this PR is not it.
This PR contains cleanup refactoring which I worked on before writing the _Verifier_ support, because I noticed that the existing secret/key handling across modules was inconsistent and required duplicated code.
I am submitting this part now to allow for incremental review of not too large a diff, and because these commits remained unchanged during two weeks since I moved on the the next task.
## The problem being addressed
Configuration of JWT signing secrets/keys was inconsistent:
Under `[oauth2]` the full configuration set was supported:
- `JWT_SIGNING_ALGORITHM` configured the algorithm
- `JWT_SECRET` configured a literal secret for symmetric algorithms
- `JWT_SECRET_URI` configured a `file:` uri of a secret for symmetric algorithms
- `JWT_SIGNING_PRIVATE_KEY_FILE` configured a file for asymmetric algorithms
For `[server]`, the LFS module only supported `LFS_JWT_SECRET`, and the signing method was hardcoded to `HS256`
For `[actions]`, only asymmetric signing methods were supported via `ID_TOKEN_SIGNING_ALGORITHM` and `ID_TOKEN_SIGNING_PRIVATE_KEY_FILE`.
## ini unification
The proposed code centralizes ini parsing to always support the following ini keys:
- `[pfx]SIGNING_ALGORITHM` determines the algorithm
- `[pfx]SECRET` is a literal secret for symmetric algorithms
- `[pfx]SECRET_URI` is the uri of a secret for symmetric algorithms
- `[pfx]SIGNING_PRIVATE_KEY_FILE` is a file with a private key for asymmetric algorithms
`[pfx]` is specific to the module and chosen to support the existing ini keys
Centralizing this code and unifying the ini keys will come handy for at least the following reasons:
- consistent behavior across modules is easier to understand
- less duplicated code
- easier to expand later, which is my main motivation
## implementation notes
as might be apparent by the _take3_ branch name, this is the third iteration of this patch series. The main reason why I abandoned the other two is that I first tried to move all the key initialization into the code called from settings.go when the ini file is parsed. But that lead to a lot of friction with test cases, because private key files which are configured, but do not exist will get created and hence require a writable `AppDataPath` and additional clean up.
To avoid a lot of noise and complications in test cases, I kept the existing two stage process, where
- the settings component creates missing symmetric signing keys and writes them to the .ini
- the settings component creates a simple configuration struct
- which is then used from the module init to create the actual key, which also includes creating a private key file if asymmetric crypto is configured and the key file does not exist.
I would have wished this patch was a net negative in terms of LOCs, but I hope it contributes to clarity and many added lines are in test cases.
## Commits
Because sometimes PRs are merged as squashes with the PR text remaining, I am repeating here the individual messages of the individual commits for future reference:
### Refactor signing key initalization and oauth2 use of it
This commit is the first in a series towards the goal of addressing the
FIXME comment in modules/setting/oauth2.go to remove
GeneralTokenSigningSecret
To do it properly, the task also requires addition of signing secret/key
rotation: We ultimately want to be able to change a signing key, but
continue to accept the previous one. This is particularly relevant to
offer a path from GeneralTokenSigningSecret aka JWT_SECRET to new,
specific component key configuration, where it should be possible to add
the former JWT_SECRET as a key accepted for verification to enable a
seamless transition.
This perspective, in turn, calls for refactoring of the existing secret
initialization code to centralize the common functions of parsing
signing key related configuration directives: The oauth2 module
currently is the only component accepting symmetric and asymmetric keys,
with the limitation of the symmetric key being also the
GeneralTokenSigningSecret. Other components either enforce HS256 or
public key algorithms.
We should really give the choice of algorithm selection and avoid code
duplication in other places, so this commit
- generalizes setting parsing into a configuration struct: A prefix can
be provided, with which the common configuration directives are
processed:
- [pfx]SIGNING_ALGORITHM determines the algorithm
- [pfx]SECRET is a literal secret for symmetric algorithms
- [pfx]SECRET_URI is the uri of a secret for symmetric algorithms
- [pfx]SIGNING_PRIVATE_KEY_FILE is a file with a private key for asymmetric algorithms
- which is then accepted by jwtx.InitSigningKey() to create an actual
signing key
The reasons for the two stage process are explained in a long-ish
comment in modules/setting/security.go. In short, other options would
either violate sensible module boundaries or cause too much friction.
These other options have actually been tried, this is take 3 of the
proposed changes.
### Refactor services/lfs: Change token code to use SigningKey
This now also enables use of token algorithms other than HS256.
In this case, signing key initialization also happens during settings
initialization, because LFS is also used in CLI commands.
### Refactor api/actions to use new signingkey API
This now also enables use of symmetric token algorithms.
Reviewed-on: https://codeberg.org/forgejo/forgejo/pulls/11194
Reviewed-by: Gusted <gusted@noreply.codeberg.org>
Co-authored-by: Nils Goroll <nils.goroll@uplex.de>
Co-committed-by: Nils Goroll <nils.goroll@uplex.de>
Related to #11035, this test case used `require.Errorf()` where it could use `require.Error()` to make it clear that this is not a case of confusion with `require.ErrorContains()`
The other cases in the same function already use `require.Error()`, so this clearly looks like a consistency glitch.
Reviewed-on: https://codeberg.org/forgejo/forgejo/pulls/11037
Reviewed-by: Gusted <gusted@noreply.codeberg.org>
Co-authored-by: Nils Goroll <nils.goroll@uplex.de>
Co-committed-by: Nils Goroll <nils.goroll@uplex.de>
Fixes#8893 by using a 30 second initialization timeout on the minio storage.
Reviewed-on: https://codeberg.org/forgejo/forgejo/pulls/8897
Reviewed-by: Earl Warren <earl-warren@noreply.codeberg.org>
Reviewed-by: Gusted <gusted@noreply.codeberg.org>
Co-authored-by: Mathieu Fenniak <mathieu@fenniak.net>
Co-committed-by: Mathieu Fenniak <mathieu@fenniak.net>
Fixes#8529, supersedes #8597
Reverts the workaround of #8541 (while keeping the test) and ignore Close error (like #8527 did).
https://github.com/minio/minio-go/pull/2137 makes the test pass without the `io.NopCloser`:
```
docker run --rm -e MINIO_DOMAIN=minio -e MINIO_ROOT_USER=123456 -e MINIO_ROOT_PASSWORD=12345678 -p 9000:9000 data.forgejo.org/oci/bitnami/minio:2024.8.17
```
```
TEST_MINIO_ENDPOINT=localhost:9000 go test -v -run ^TestMinioStorageIterator$ ./modules/storage
```
Wrapping the `io.Reader` in an `io.NopCloser` also hides all other methods (like `WriteTo` or `Seek`), so is not ideal.
As a further "line of defense", the Close error is ignored (if saveAsPackageBlob was succesful, we don't really care if Close fails).
Co-authored-by: Renovate Bot <forgejo-renovate-action@forgejo.org>
Reviewed-on: https://codeberg.org/forgejo/forgejo/pulls/8816
Reviewed-by: Gusted <gusted@noreply.codeberg.org>
Co-authored-by: oliverpool <git@olivier.pfad.fr>
Co-committed-by: oliverpool <git@olivier.pfad.fr>
Fixes#8529, reverts #8527.
I was able to reproduce the problem in a test:
- it triggered only when the reader was an io.Reader
- and the size was provided (-1 takes another code path in minio)
287b1f21e1 should fail when running:
```
docker run --rm -e MINIO_DOMAIN=minio -e MINIO_ROOT_USER=123456 -e MINIO_ROOT_PASSWORD=12345678 -p 9000:9000 data.forgejo.org/oci/bitnami/minio:2024.8.17
```
and
```
TEST_MINIO_ENDPOINT=localhost:9000 go test -v -run ^TestMinioStorageIterator$ ./modules/storage
```
### Tests
- I added test coverage for Go changes...
- [x] in their respective `*_test.go` for unit tests.
- [ ] in the `tests/integration` directory if it involves interactions with a live Forgejo server.
### Documentation
- [ ] I created a pull request [to the documentation](https://codeberg.org/forgejo/docs) to explain to Forgejo users how to use this change.
- [x] I did not document these changes and I do not expect someone else to do it.
### Release notes
- [x] I do not want this change to show in the release notes.
- [ ] I want the title to show in the release notes with a link to this pull request.
- [ ] I want the content of the `release-notes/<pull request number>.md` to be be used for the release notes instead of the title.
Reviewed-on: https://codeberg.org/forgejo/forgejo/pulls/8541
Reviewed-by: Michael Kriese <michael.kriese@gmx.de>
Reviewed-by: Earl Warren <earl-warren@noreply.codeberg.org>
Co-authored-by: oliverpool <git@olivier.pfad.fr>
Co-committed-by: oliverpool <git@olivier.pfad.fr>
Closes https://github.com/go-gitea/gitea/issues/30296
- Adds a DB fixture for actions artifacts
- Adds artifacts test files
- Clears artifacts test files between each run
- Note: I initially initialized the artifacts only for artifacts tests,
but because the files are small it only takes ~8ms, so I changed it to
always run in test setup for simplicity
- Fix some otherwise flaky tests by making them not depend on previous
tests
(cherry picked from commit 66971e591e5dddd5b6dc1572ac48f4e4ab29b8e0)
Conflicts:
- tests/integration/api_actions_artifact_test.go
Conflict resolved by manually changing the tested artifact
name from "artifact" to "artifact-download"
- tests/integration/api_actions_artifact_v4_test.go
Conflict resolved by manually updating the tested artifact
names, and adjusting the test case only present in our tree.
- tests/test_utils.go
Resolved by manually copying the added function.
Fix#28121
I did some tests and found that the `missing signature key` error is
caused by an incorrect `Content-Type` header. Gitea correctly sets the
`Content-Type` header when serving files.
348d1d0f32/routers/api/packages/container/container.go (L712-L717)
However, when `SERVE_DIRECT` is enabled, the `Content-Type` header may
be set to an incorrect value by the storage service. To fix this issue,
we can use query parameters to override response header values.
https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObject.html
<img width="600px"
src="https://github.com/user-attachments/assets/f2ff90f0-f1df-46f9-9680-b8120222c555"
/>
In this PR, I introduced a new parameter to the `URL` method to support
additional parameters.
```
URL(path, name string, reqParams url.Values) (*url.URL, error)
```
---
Most S3-like services support specifying the content type when storing
objects. However, Gitea always use `application/octet-stream`.
Therefore, I believe we also need to improve the `Save` method to
support storing objects with the correct content type.
b7fb20e73e/modules/storage/minio.go (L214-L221)
(cherry picked from commit 0690cb076bf63f71988a709f62a9c04660b51a4f)
Conflicts:
- modules/storage/azureblob.go
Dropped the change, as we do not support Azure blob storage.
- modules/storage/helper.go
Resolved by adjusting their `discardStorage` to our
`DiscardStorage`
- routers/api/actions/artifacts.go
routers/api/actions/artifactsv4.go
routers/web/repo/actions/view.go
routers/web/repo/download.go
Resolved the conflicts by manually adding the new `nil`
parameter to the `storage.Attachments.URL()` calls.
Originally conflicted due to differences in the if expression
above these calls.
We wanted to be able to use the IAM role provided by the EC2 instance
metadata in order to access S3 via the Minio configuration. To do this,
a new credentials chain is added that will check the following locations
for credentials when an access key is not provided. In priority order,
they are:
1. MINIO_ prefixed environment variables
2. AWS_ prefixed environment variables
3. a minio credentials file
4. an aws credentials file
5. EC2 instance metadata
(cherry picked from commit c0880e7695346997c6a93f05cd01634cb3ad03ee)
Conflicts:
docs/content/administration/config-cheat-sheet.en-us.md
does not exist in Forgejo
(cherry picked from commit f9d75d4705)
(cherry picked from commit 64f76f4ab2)
(cherry picked from commit 5d02454155)
[CI] Forgejo Actions workflows
(cherry picked from commit 3ff59b5379)
(cherry picked from commit 8af826a6f7)
(cherry picked from commit d7c09d9cc8)
[CI] use the docker label instead of ubuntu-latest
(cherry picked from commit b6a6470db6)
[CI] all tests need compliance before proceeding
(cherry picked from commit b35c496f2c)
(cherry picked from commit 36a4148a8e)
(cherry picked from commit 7ffcffa653)
(cherry picked from commit 8a246d296e)
(cherry picked from commit dd0b6e1826)
[CI] Forgejo Actions based release process (squash) MySQL optimization
Refs: https://codeberg.org/forgejo/forgejo/issues/976
(cherry picked from commit b4b8c489e6)
(cherry picked from commit 1e861db4af)
(cherry picked from commit a6c0e00330)
(cherry picked from commit f97b336465)
(cherry picked from commit 6d65d5f0d6f798556a8f9e547896be03a5ee2f87)
(cherry picked from commit 79bfbadbed)
(cherry picked from commit e86c40a34a)
(cherry picked from commit da0c454adb)
(cherry picked from commit b49d892cda)
[CI] enable minio tests
(cherry picked from commit 4d8f438031)
(cherry picked from commit c4eeb0a61e)
[CI] Forgejo Actions based CI for PR & branches (squash) cleanup
(cherry picked from commit 80eb20e842)
(cherry picked from commit d2ff589858)
(cherry picked from commit f6eedecb67)
(cherry picked from commit cf458091e2)
(cherry picked from commit ddd322cb2d)
(cherry picked from commit f0f5729b64)
Conflicts:
.github/workflows/pull-db-tests.yml
https://codeberg.org/forgejo/forgejo/pulls/1573
(cherry picked from commit bb347aedd4)
(cherry picked from commit 0f5ebe8c3e)
(cherry picked from commit 9101cb4715)
(cherry picked from commit ab118fe4ef)
[CI] Forgejo Actions based CI for PR & branches (squash) use node:20-bookworm
No longer use the custom test-env image, it is unecessary technical
debt.
Also upgrade to bitnami/minio:2023.8.31 to align with what Gitea tests
(cherry picked from commit d9b77fd273)
(cherry picked from commit ef8f099f9e)
(cherry picked from commit 6d3c675d20)
(cherry picked from commit fc00ff45f2)
(cherry picked from commit b7f02b9846)
(cherry picked from commit d23d86e567)
(cherry picked from commit e68e65460c)
(cherry picked from commit 96f93a1fdb)
(cherry picked from commit 2ff3080018)
(cherry picked from commit 8ea00bcff4)
(cherry picked from commit 85be961970)
(cherry picked from commit 387b279fc3)
(cherry picked from commit 219be7aa85)
(cherry picked from commit 929b75ee39)
(cherry picked from commit dd42a30d6a)
Should BucketExists (HeadBucket) fail because of an error related to
the connection rather than the existence of the bucket, no information
is available and the admin is left guessing.
https://docs.aws.amazon.com/AmazonS3/latest/API/API_HeadBucket.html
> This action is useful to determine if a bucket exists and you have
> permission to access it. The action returns a 200 OK if the bucket
> exists and you have permission to access it.
>
> If the bucket does not exist or you do not have permission to access
> it, the HEAD request returns a generic 400 Bad Request, 403
> Forbidden or 404 Not Found code. A message body is not included, so
> you cannot determine the exception beyond these error codes.
GetBucketVersioning is used instead and exclusively dedicated to
asserting if using the connection does not return a BadRequest.
If it does the NewMinioStorage logs an error and returns. Otherwise
it keeps going knowing that BucketExists is not going to fail for
reasons unrelated to the existence of the bucket and the permissions
to access it.
(cherry picked from commit d1df4b3bc62e5e61893a923f1c4b58f084eb03af)
Refs: https://codeberg.org/forgejo/forgejo/issues/1338
Previously, `err` was defined above, checked for `err == nil` and used
nowhere else.
Hence, the result of `convertMinioErr` would always be `nil`.
This leads to a NPE further down the line.
That is not intentional, it should convert the error of the most recent
operation, not one of its predecessors.
Found through
https://discord.com/channels/322538954119184384/322538954119184384/1143185780206993550.
For some reason, the permission of the client_id and secret may cannot
create bucket, so now we will check whether bucket does exist first and
then try to create a bucket if it doesn't exist.
Try to fix#25984
Co-authored-by: silverwind <me@silverwind.io>
The MinIO client isn't redirecting to the correct AWS endpoint if a
non-default data center is used.
In my use case I created an AWS bucket at `eu-central-1` region. Because
of the missing region initialization of the client the default
`us-east-1` API endpoint is used returning a `301 Moved Permanently`
response that's not handled properly by MinIO client. This in return
aborts using S3 storage on AWS as the `BucketExists()` call will fail
with the http moved error.
MinIO client trace shows the issue:
```text
---------START-HTTP---------
HEAD / HTTP/1.1
Host: xxxxxxxxxxx-prod-gitea-data.s3.dualstack.us-east-1.amazonaws.com
User-Agent: MinIO (windows; amd64) minio-go/v7.0.61
Authorization: AWS4-HMAC-SHA256 Credential=**REDACTED**/20230809/accesspoint.eu-central-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=**REDACTED**
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20230809T141143Z
HTTP/1.1 301 Moved Permanently
Connection: close
Content-Type: application/xml
Date: Wed, 09 Aug 2023 14:11:43 GMT
Server: AmazonS3
X-Amz-Bucket-Region: eu-central-1
X-Amz-Id-2: UK7wfeYi0HcTcytNvQ3wTAZ5ZP1mOSMnvRZ9Fz4xXzeNsS47NB/KfFx2unFxo3L7XckHpMNPPVo=
X-Amz-Request-Id: S1V2MJV8SZ11GEVN
---------END-HTTP---------
```
Co-authored-by: Heiko Besemann <heiko.besemann@qbeyond.de>
Follow up #22405Fix#20703
This PR rewrites storage configuration read sequences with some breaks
and tests. It becomes more strict than before and also fixed some
inherit problems.
- Move storage's MinioConfig struct into setting, so after the
configuration loading, the values will be stored into the struct but not
still on some section.
- All storages configurations should be stored on one section,
configuration items cannot be overrided by multiple sections. The
prioioty of configuration is `[attachment]` > `[storage.attachments]` |
`[storage.customized]` > `[storage]` > `default`
- For extra override configuration items, currently are `SERVE_DIRECT`,
`MINIO_BASE_PATH`, `MINIO_BUCKET`, which could be configured in another
section. The prioioty of the override configuration is `[attachment]` >
`[storage.attachments]` > `default`.
- Add more tests for storages configurations.
- Update the storage documentations.
---------
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
minio storage iterator shows different behavior with local fs iterator.
in local fs storage:
``` go
s.IterateObjects("prefix", func(path,obj)
println(path) // show "prefix/xxx.file"
})
```
in minio storage:
```go
s.IterateObjects("prefix", func(path,obj)
println(path) // show "xxx.file"
})
```
I think local fs is correct, minio use wrong `basePath` to trim storage
path prefix.
---------
Co-authored-by: Giteabot <teabot@gitea.io>
Since #23493 has conflicts with latest commits, this PR is my proposal
for fixing #23371
Details are in the comments
And refactor the `modules/options` module, to make it always use
"filepath" to access local files.
Benefits:
* No need to do `util.CleanPath(strings.ReplaceAll(p, "\\", "/"))),
"/")` any more (not only one before)
* The function behaviors are clearly defined
Support to iterator subdirectory in ObjectStorage for
ObjectStorage.Iterator method.
It's required for https://github.com/go-gitea/gitea/pull/22738 to make
artifact files cleanable.
---------
Co-authored-by: Jason Song <i@wolfogre.com>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
As suggest by Go developers, use `filepath.WalkDir` instead of
`filepath.Walk` because [*Walk is less efficient than WalkDir,
introduced in Go 1.16, which avoids calling `os.Lstat` on every file or
directory visited](https://pkg.go.dev/path/filepath#Walk).
This proposition address that, in a similar way as
https://github.com/go-gitea/gitea/pull/22392 did.
Co-authored-by: zeripath <art27@cantab.net>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
The PR #21198 introduced a probable security vulnerability which
resulted in making all storage files be marked as executable.
This PR ensures that these are forcibly marked as non-executable.
Fix#22161
Signed-off-by: Andrew Thornton <art27@cantab.net>
A complement to #21985.
I overlooked it because the name of the switch is `StartServer`, not
`Enabled`. I believe the weird name is a legacy, but renaming is out of
scope.
Change all license headers to comply with REUSE specification.
Fix#16132
Co-authored-by: flynnnnnnnnnn <flynnnnnnnnnn@github>
Co-authored-by: John Olheiser <john.olheiser@gmail.com>
Go creates temporary files as 600, but sometimes we want the group to be able to read them (for example,
for another user to back up the storage)
This PR applies the umask to the renamed tmp files in local storage.
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
* Clean paths when looking in Storage
Ensure paths are clean for minio aswell as local storage.
Use url.Path not RequestURI/EscapedPath in storageHandler.
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Apply suggestions from code review
Co-authored-by: Lauris BH <lauris@nix.lv>
The io/ioutil package has been deprecated as of Go 1.16, see
https://golang.org/doc/go1.16#ioutil. This commit replaces the existing
io/ioutil functions with their new definitions in io and os packages.
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
Storage.Iterate provides the path and an open object. On windows using
local storage means that the objects will be locked thus preventing clean
from deleting them.
This PR simply closes the objects early.
Fix#16932
Signed-off-by: Andrew Thornton <art27@cantab.net>
