Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cleanup references to Bearer tokens and OIDC authentication for API usage #2176

Merged
merged 1 commit into from
Nov 19, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 3 additions & 4 deletions docs/site/content/en/docs/Concepts/users/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,10 +14,6 @@ We define 3 levels of access to each item (test, run, dataset or schema):
- public: available even to non-authenticated users (for reading)
- protected: available to all authenticated users that have the `viewer` role (see below)
- private: available only to users who 'own' this data - those who have the team role.

In addition to these 3 levels, runs and schemas can have a 'token' (randomly generated string): everyone who knows this token can read the record. This token is reset any time the restriction level changes.

Tests can have tokens, too: you can have an arbitrary number of tokens, each with a subset of read, modify and upload privileges.


## Users and roles
Expand All @@ -32,6 +28,9 @@ There are few generic roles automatically created during initial realm import.

The `admin` role is a system-wide role and is not restricted to a particular teams.

## API Keys

Users can generate an API key, that will provide programatic access to the Horreum API with the same authorization permissions as the user who created the API Key.

## User authentication

Expand Down
6 changes: 0 additions & 6 deletions docs/site/content/en/docs/Deployment/bare-metal.md
Original file line number Diff line number Diff line change
Expand Up @@ -135,12 +135,6 @@ QUARKUS_MAILER_LOGIN=disabled
# By default webhook notifications that fail to verify TLS integrity fail; set this to ignore verification result.
# HORREUM_HOOK_TLS_INSECURE=true

# This is an offline token for the __user_reader user. You can obtain that with
# curl -s -X POST https://keycloak.example.com/auth/realms/horreum/protocol/openid-connect/token \
# -H 'content-type: application/x-www-form-urlencoded' \
# -d 'username=__user_reader&password='$PASSWORD'&grant_type=password&client_id=horreum-ui&scope=offline_access' \
# | jq -r .refresh_token
HORREUM_KEYCLOAK_USER_READER_TOKEN=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
```

With all this in you can finally start Horreum as any other Java application (note: dependencies in `repo/target/lib/` must be present):
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ API operations that support Filter Query provide an HTTP request parameter named
To use the Filter Query with an operation that returns a [Run](/docs/concepts/core-concepts/#run) data can be done as follows.

```bash
curl -s 'http://localhost:8080/api/sql/1/queryrun?query=$.*&array=true' -H 'content-type: application/json' -H 'Authorization: Bearer '$TOKEN
curl -s 'http://localhost:8080/api/sql/1/queryrun?query=$.*&array=true' -H 'content-type: application/json' -H 'X-Horreum-API-Key: '$API_KEY
{
"valid": true,
"jsonpath": "$.*",
Expand All @@ -29,7 +29,7 @@ The response contains an enclosing JSON object that is common for Filter Query o
To filter the above API operation to retrieve only the *results* property of the uploaded [Run](/docs/concepts/core-concepts/#run) object the *query* parameter is defined as `$.results`

```bash
curl -s 'http://localhost:8080/api/sql/1/queryrun?query=$.results&array=true' -H 'content-type: application/json' -H 'Authorization: Bearer '$TOKEN
curl -s 'http://localhost:8080/api/sql/1/queryrun?query=$.results&array=true' -H 'content-type: application/json' -H 'X-Horreum-API-Key: '$API_KEY
{
"valid": true,
"jsonpath": "$.results",
Expand Down
10 changes: 5 additions & 5 deletions docs/site/content/en/docs/Tasks/import-export/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,18 +44,18 @@ Click on Schema/Test and there is a button where you can select either _Import S
```bash
curl 'http://localhost:8080/api/schema/import/' \
-s -X POST -H 'content-type: application/json' \
-H 'Authorization: Bearer '$TOKEN \
-H 'X-Horreum-API-Key: '$API_KEY \
-d @/path/to/schema.json
```

If you are unfamiliar with creating the auth token please see [Upload Run](/docs/tasks/upload-new-run/).
If you are unfamiliar with generating an API Key, please see [Upload Run](/docs/tasks/api-keys/).

## Import Tests

```bash
curl 'http://localhost:8080/api/test/import/' \
-s -X POST -H 'content-type: application/json' \
-H 'Authorization: Bearer '$TOKEN \
-H 'X-Horreum-API-Key: '$API_KEY \
-d @/path/to/test.json
```

Expand All @@ -64,7 +64,7 @@ curl 'http://localhost:8080/api/test/import/' \
```bash
SCHEMAID='123'
curl 'http://localhost:8080/api/schema/export/?id='$SCHEMAID \
-H 'Authorization: Bearer '$TOKEN \
-H 'X-Horreum-API-Key: '$API_KEY \
-O --output-dir /path/to/folder
```

Expand All @@ -74,6 +74,6 @@ curl 'http://localhost:8080/api/schema/export/?id='$SCHEMAID \
TESTID='123'
curl 'http://localhost:8080/api/test/export/?id=$TESTID' \
-s -X POST -H 'content-type: application/json' \
-H 'Authorization: Bearer '$TOKEN \
-H 'X-Horreum-API-Key: '$API_KEY \
-O --output-dir /path/to/folder
```
6 changes: 1 addition & 5 deletions docs/site/content/en/docs/Tasks/manage-reports/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,10 +25,6 @@ Available Report configurations
The same task can be repeated using the web API to delete a `Report`. Copy and paste this into your shell. Note, modify the REPORT_ID parameter. The response to expect is a 204 HTTP response code for a successful deletion.

```bash
TOKEN=$(curl -s http://localhost:8180/realms/horreum/protocol/openid-connect/token \
-d 'username=user' -d 'password=secret' \
-d 'grant_type=password' -d 'client_id=horreum-ui' \
| jq -r .access_token)
REPORT_ID=<your_report_id_here>
curl 'http://localhost:8080/api/report/table/'$REPORT_ID -H 'content-type: application/json' -H 'Authorization: Bearer '$TOKEN --request DELETE -v
curl 'http://localhost:8080/api/report/table/'$REPORT_ID -H 'content-type: application/json' -H 'X-Horreum-API-Key: '$API_KEY --request DELETE -v
```
40 changes: 4 additions & 36 deletions docs/site/content/en/docs/Tasks/upload-run/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,44 +7,11 @@ weight: 3

Horreum accepts any valid **JSON** as the input. To get maximum out of Horreum, though, it is recommended to categorize the input using [JSON schema](https://json-schema.org/).

There are two principal ways to authorize operations:

- Authentication against OIDC provider (Keycloak): This is the standard way that you use when accessing Horreum UI - you use your credentials to get a JSON Web Token (JWT) and this is stored in the browser session. When accessing Horreum over the REST API you need to use this for [Bearer Authentication](https://datatracker.ietf.org/doc/html/rfc6750#section-2.1). The authorization is based on the teams and roles within those teams that you have.
- Horreum API Keys: See more in [API keys](/docs/tasks/api-keys). These replace the so called "Horreum Tokens" that were used in the past in operations that accepted a `token` parameter.
Operations are authorized via API Keys, for details on how to generate an API Key, please refer to [API keys](/docs/tasks/api-keys).

If you're running your tests in Jenkins you can skip a lot of the complexity below using [Horreum Plugin](https://plugins.jenkins.io/horreum/). This plugin supports both Jenkins Pipeline and Freeform jobs.

## Getting JWT token

New data can be uploaded into Horreum only by authorized users. This user must have the permission to upload for given team, e.g. if you'll use `dev-team` as the owner this role is called `dev-uploader` and it is a composition of the team role (`dev-team`) and `uploader` role. You can read more about user management [here](/docs/concepts/users).

```bash
TOKEN=$(curl -s http://localhost:8180/realms/horreum/protocol/openid-connect/token \
-d 'username=user' -d 'password=secret' \
-d 'grant_type=password' -d 'client_id=horreum-ui' \
| jq -r .access_token)
```

A note on JWT token issuer: OIDC-enabled applications usually validate the URL that issued the request vs. URL of the authentication server the application is configured to use - if those don't match you receive 403 Forbidden response without further information. Had you used `http://localhost:8180` as `KEYCLOAK_URL` in the example above you would get rejected in developer mode with the default infrastructure, even though `localhost` resolves to `127.0.0.1` - the URL has to match to what you have in `horreum-backend/.env` as `QUARKUS_OIDC_AUTH_SERVER_URL`. You can disable this check with `-Dquarkus.oidc.token.issuer=any` but that is definitely not recommended in production.

## Using offline JWT token

Access token has very limited lifespan; when you want to perform the upload from CI script and don't want to store the password inside you can keep an offline token. This token cannot be used directly as an access token; instead you can store it and use it to obtain a regular short-lived access token:

```bash
OFFLINE_TOKEN=$(curl -s http://localhost:8180/realms/horreum/protocol/openid-connect/token \
-d 'username=user' -d 'password=secret' \
-d 'grant_type=password' -d 'client_id=horreum-ui' -d 'scope=offline_access' \
| jq -r .refresh_token)
TOKEN=$(curl -s http://localhost:8180/realms/horreum/protocol/openid-connect/token \
-d 'refresh_token='$OFFLINE_TOKEN \
-d 'grant_type=refresh_token' -d 'client_id=horreum-ui' \
| jq -r .access_token)
```

Note that the offline token also expires eventually, by default after 30 days.

## Uploading the data
## Uploading Data

There are several mandatory parameters for the upload:

Expand All @@ -59,14 +26,15 @@ Optionally you can also set `schema` with URI of the JSON schema, overriding (or
The upload itself can look like:

```bash
API_KEY='HUSR_00000000_0000_0000_0000_000000000000'
TEST='$.info.benchmark'
START='2021-08-01T10:35:22.00Z'
STOP='2021-08-01T10:40:28.00Z'
OWNER='dev-team'
ACCESS='PUBLIC'
curl 'http://localhost:8080/api/run/data?test='$TEST'&start='$START'&stop='$STOP'&owner='$OWNER'&access='$ACCESS \
-s -X POST -H 'content-type: application/json' \
-H 'Authorization: Bearer '$TOKEN \
-H 'X-Horreum-API-Key: '$API_KEY \
-d @/path/to/data.json
```

Expand Down
18 changes: 13 additions & 5 deletions docs/site/content/en/docs/Tutorials/create-test-run/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ We assume that you already went through the [previous tutorial](/docs/tutorials/
User logged in
{{% /imgproc %}}

### 1. Create Test

Press the **New Test** button and fill the test name. Test names must be unique within Horreum.

Expand All @@ -25,6 +26,8 @@ Create new test

Click on **Save** button on left side at the bottom - a blue banner will confirm that the test was created.

### 2. Prepare test data

Now we can prepare the JSON document representing our benchmark results - open `/tmp/run.json` with your favorite text editor and paste in this:

```json
Expand All @@ -34,23 +37,28 @@ Now we can prepare the JSON document representing our benchmark results - open `
}
```

### 3. Generate API Key

To interact with the REST API, you will need an API Key, please follow the instructions to [generate an API Key](/docs/tasks/api-keys/)

### 4. Upload Data

Now copy and paste this into shell; the reply will be just `1` - the ID of the newly uploaded run.

```bash
TOKEN=$(curl -s http://localhost:8180/realms/horreum/protocol/openid-connect/token \
-d 'username=user' -d 'password=secret' \
-d 'grant_type=password' -d 'client_id=horreum-ui' \
| jq -r .access_token)
API_KEY='HUSR_00000000_0000_0000_0000_000000000000'
TEST='Foobar'
START='2021-08-01T10:35:22.00Z'
STOP='2021-08-01T10:40:28.00Z'
OWNER='dev-team'
ACCESS='PUBLIC'
curl 'http://localhost:8080/api/run/data?test='$TEST'&start='$START'&stop='$STOP'&owner='$OWNER'&access='$ACCESS \
-s -H 'content-type: application/json' -H 'Authorization: Bearer '$TOKEN \
-s -H 'content-type: application/json' -H 'X-Horreum-API-Key: '$API_KEY \
-d @/tmp/run.json
```

### 5. View Data in Horreum

Let's navigate into the tests overview (main page) by clicking the **Tests** link in the main menu on the top left of the page:

{{% imgproc tests Fit "1200x300" %}}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,7 @@ The query can be executed by making a call to the Horreum API;

```bash
$ curl 'http://localhost:8080/api/run/data?test='$TEST'&start='$START'&stop='$STOP'&owner='$OWNER'&access='$ACCESS \
-s -H 'content-type: application/json' -H 'Authorization: Bearer '$TOKEN \
-s -H 'content-type: application/json' -H 'X-Horreum-API-Key: '$API_KEY \
-d @/tmp/collector_query.json
```

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,7 @@ The document query can then be sumitted to the Horreum API;

```bash
$ curl 'http://localhost:8080/api/run/data?test='$TEST'&start='$START'&stop='$STOP'&owner='$OWNER'&access='$ACCESS \
-s -H 'content-type: application/json' -H 'Authorization: Bearer '$TOKEN \
-s -H 'content-type: application/json' -H 'X-Horreum-API-Key: '$API_KEY \
-d @/tmp/elastic_payload.json
```

Expand Down Expand Up @@ -101,7 +101,7 @@ The query can be executed by making a call to the Horreum API;

```bash
$ curl 'http://localhost:8080/api/run/data?test='$TEST'&start='$START'&stop='$STOP'&owner='$OWNER'&access='$ACCESS \
-s -H 'content-type: application/json' -H 'Authorization: Bearer '$TOKEN \
-s -H 'content-type: application/json' -H 'X-Horreum-API-Key: '$API_KEY \
-d @/tmp/elastic_query.json
```

Expand Down