Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[W.I.P] Portal upload or download image #224

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

[W.I.P] Portal upload or download image #224

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
167 changes: 167 additions & 0 deletions proposals/images/portal-upordown-image/data-flow.drawio
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,167 @@
<mxfile host="65bd71144e">
<diagram id="gFRJlOlQtpl9VA4PU4Et" name="Page-1">
<mxGraphModel dx="1179" dy="909" grid="1" gridSize="10" guides="1" tooltips="1" connect="1" arrows="1" fold="1" page="1" pageScale="1" pageWidth="1200" pageHeight="1600" math="0" shadow="0">
<root>
<mxCell id="0"/>
<mxCell id="1" parent="0"/>
<mxCell id="9" style="edgeStyle=none;html=1;startArrow=none;" edge="1" parent="1" source="14">
<mxGeometry relative="1" as="geometry">
<mxPoint x="180" y="530" as="targetPoint"/>
</mxGeometry>
</mxCell>
<mxCell id="3" value="Harbor Portal" style="rounded=1;whiteSpace=wrap;html=1;" vertex="1" parent="1">
<mxGeometry x="130" y="160" width="100" height="40" as="geometry"/>
</mxCell>
<mxCell id="10" style="edgeStyle=none;html=1;startArrow=none;" edge="1" parent="1" source="19">
<mxGeometry relative="1" as="geometry">
<mxPoint x="390" y="530" as="targetPoint"/>
</mxGeometry>
</mxCell>
<mxCell id="4" value="Middleware Auth" style="rounded=1;whiteSpace=wrap;html=1;" vertex="1" parent="1">
<mxGeometry x="340" y="160" width="100" height="40" as="geometry"/>
</mxCell>
<mxCell id="11" style="edgeStyle=none;html=1;startArrow=none;" edge="1" parent="1" source="23">
<mxGeometry relative="1" as="geometry">
<mxPoint x="590" y="530" as="targetPoint"/>
</mxGeometry>
</mxCell>
<mxCell id="5" value="Harbor Core&amp;nbsp;" style="rounded=1;whiteSpace=wrap;html=1;" vertex="1" parent="1">
<mxGeometry x="540" y="160" width="100" height="40" as="geometry"/>
</mxCell>
<mxCell id="12" style="edgeStyle=none;html=1;startArrow=none;" edge="1" parent="1" source="27">
<mxGeometry relative="1" as="geometry">
<mxPoint x="800" y="530" as="targetPoint"/>
</mxGeometry>
</mxCell>
<mxCell id="7" value="Registry" style="rounded=1;whiteSpace=wrap;html=1;" vertex="1" parent="1">
<mxGeometry x="750" y="160" width="100" height="40" as="geometry"/>
</mxCell>
<mxCell id="13" style="edgeStyle=none;html=1;" edge="1" parent="1" source="8">
<mxGeometry relative="1" as="geometry">
<mxPoint x="1010" y="530" as="targetPoint"/>
</mxGeometry>
</mxCell>
<mxCell id="8" value="DB" style="rounded=1;whiteSpace=wrap;html=1;" vertex="1" parent="1">
<mxGeometry x="960" y="160" width="100" height="40" as="geometry"/>
</mxCell>
<mxCell id="15" value="" style="edgeStyle=none;html=1;endArrow=none;" edge="1" parent="1" source="3" target="14">
<mxGeometry relative="1" as="geometry">
<mxPoint x="180" y="901.1764705882351" as="targetPoint"/>
<mxPoint x="180" y="200" as="sourcePoint"/>
</mxGeometry>
</mxCell>
<mxCell id="14" value="" style="rounded=1;whiteSpace=wrap;html=1;" vertex="1" parent="1">
<mxGeometry x="170" y="260" width="20" height="210" as="geometry"/>
</mxCell>
<mxCell id="17" style="edgeStyle=none;html=1;entryX=0.022;entryY=0.072;entryDx=0;entryDy=0;entryPerimeter=0;" edge="1" parent="1" source="16" target="14">
<mxGeometry relative="1" as="geometry"/>
</mxCell>
<mxCell id="16" value="User" style="shape=umlActor;verticalLabelPosition=bottom;verticalAlign=top;html=1;outlineConnect=0;" vertex="1" parent="1">
<mxGeometry x="10" y="245" width="30" height="60" as="geometry"/>
</mxCell>
<mxCell id="18" value="upload/download" style="text;html=1;strokeColor=none;fillColor=none;align=center;verticalAlign=middle;whiteSpace=wrap;rounded=0;" vertex="1" parent="1">
<mxGeometry x="60" y="240" width="80" height="30" as="geometry"/>
</mxCell>
<mxCell id="20" value="" style="edgeStyle=none;html=1;endArrow=none;" edge="1" parent="1" source="4" target="19">
<mxGeometry relative="1" as="geometry">
<mxPoint x="390" y="910" as="targetPoint"/>
<mxPoint x="390" y="200" as="sourcePoint"/>
</mxGeometry>
</mxCell>
<mxCell id="19" value="" style="rounded=1;whiteSpace=wrap;html=1;" vertex="1" parent="1">
<mxGeometry x="380" y="290" width="20" height="170" as="geometry"/>
</mxCell>
<mxCell id="21" value="" style="endArrow=classic;html=1;exitX=1.01;exitY=0.18;exitDx=0;exitDy=0;entryX=-0.027;entryY=0.048;entryDx=0;entryDy=0;entryPerimeter=0;exitPerimeter=0;" edge="1" parent="1" source="14" target="19">
<mxGeometry width="50" height="50" relative="1" as="geometry">
<mxPoint x="280" y="410" as="sourcePoint"/>
<mxPoint x="330" y="360" as="targetPoint"/>
</mxGeometry>
</mxCell>
<mxCell id="22" value="access verify" style="text;html=1;strokeColor=none;fillColor=none;align=center;verticalAlign=middle;whiteSpace=wrap;rounded=0;" vertex="1" parent="1">
<mxGeometry x="230" y="260" width="90" height="30" as="geometry"/>
</mxCell>
<mxCell id="24" value="" style="edgeStyle=none;html=1;endArrow=none;" edge="1" parent="1" source="5" target="23">
<mxGeometry relative="1" as="geometry">
<mxPoint x="590" y="910" as="targetPoint"/>
<mxPoint x="590" y="200" as="sourcePoint"/>
</mxGeometry>
</mxCell>
<mxCell id="23" value="" style="rounded=1;whiteSpace=wrap;html=1;" vertex="1" parent="1">
<mxGeometry x="580" y="310" width="20" height="130" as="geometry"/>
</mxCell>
<mxCell id="25" value="" style="endArrow=classic;html=1;exitX=1.071;exitY=0.122;exitDx=0;exitDy=0;exitPerimeter=0;entryX=0;entryY=0;entryDx=0;entryDy=0;" edge="1" parent="1" source="19" target="23">
<mxGeometry width="50" height="50" relative="1" as="geometry">
<mxPoint x="520" y="330" as="sourcePoint"/>
<mxPoint x="570" y="280" as="targetPoint"/>
</mxGeometry>
</mxCell>
<mxCell id="26" value="upload/download" style="text;html=1;strokeColor=none;fillColor=none;align=center;verticalAlign=middle;whiteSpace=wrap;rounded=0;" vertex="1" parent="1">
<mxGeometry x="440" y="280" width="120" height="30" as="geometry"/>
</mxCell>
<mxCell id="28" value="" style="rounded=1;whiteSpace=wrap;html=1;" vertex="1" parent="1">
<mxGeometry x="1000" y="330" width="20" height="130" as="geometry"/>
</mxCell>
<mxCell id="29" value="" style="endArrow=classic;html=1;exitX=1.027;exitY=0.161;exitDx=0;exitDy=0;exitPerimeter=0;entryX=0;entryY=0;entryDx=0;entryDy=0;" edge="1" parent="1" source="23" target="28">
<mxGeometry width="50" height="50" relative="1" as="geometry">
<mxPoint x="710" y="320" as="sourcePoint"/>
<mxPoint x="760" y="270" as="targetPoint"/>
</mxGeometry>
</mxCell>
<mxCell id="31" value="" style="edgeStyle=none;html=1;endArrow=none;" edge="1" parent="1" source="7" target="27">
<mxGeometry relative="1" as="geometry">
<mxPoint x="800" y="910" as="targetPoint"/>
<mxPoint x="800" y="200" as="sourcePoint"/>
</mxGeometry>
</mxCell>
<mxCell id="27" value="" style="rounded=1;whiteSpace=wrap;html=1;" vertex="1" parent="1">
<mxGeometry x="790" y="370" width="20" height="60" as="geometry"/>
</mxCell>
<mxCell id="32" value="" style="endArrow=classic;html=1;exitX=1.079;exitY=0.531;exitDx=0;exitDy=0;entryX=-0.072;entryY=0.152;entryDx=0;entryDy=0;entryPerimeter=0;exitPerimeter=0;" edge="1" parent="1" source="23" target="27">
<mxGeometry width="50" height="50" relative="1" as="geometry">
<mxPoint x="620" y="340" as="sourcePoint"/>
<mxPoint x="670" y="290" as="targetPoint"/>
</mxGeometry>
</mxCell>
<mxCell id="33" value="push tar file from&amp;nbsp;multipart.File" style="text;html=1;strokeColor=none;fillColor=none;align=center;verticalAlign=middle;whiteSpace=wrap;rounded=0;" vertex="1" parent="1">
<mxGeometry x="600" y="350" width="180" height="30" as="geometry"/>
</mxCell>
<mxCell id="34" value="" style="endArrow=classic;html=1;exitX=0.97;exitY=1;exitDx=0;exitDy=0;exitPerimeter=0;entryX=1.006;entryY=0.791;entryDx=0;entryDy=0;entryPerimeter=0;" edge="1" parent="1" source="35" target="23">
<mxGeometry width="50" height="50" relative="1" as="geometry">
<mxPoint x="680" y="380" as="sourcePoint"/>
<mxPoint x="730" y="330" as="targetPoint"/>
</mxGeometry>
</mxCell>
<mxCell id="35" value="pull image write to http.Response" style="text;html=1;align=center;verticalAlign=middle;resizable=0;points=[];autosize=1;strokeColor=none;fillColor=none;" vertex="1" parent="1">
<mxGeometry x="595" y="380" width="200" height="30" as="geometry"/>
</mxCell>
<mxCell id="36" value="" style="endArrow=classic;html=1;exitX=-0.038;exitY=0.87;exitDx=0;exitDy=0;exitPerimeter=0;entryX=1.024;entryY=0.785;entryDx=0;entryDy=0;entryPerimeter=0;" edge="1" parent="1" source="23" target="19">
<mxGeometry width="50" height="50" relative="1" as="geometry">
<mxPoint x="410" y="400" as="sourcePoint"/>
<mxPoint x="460" y="350" as="targetPoint"/>
</mxGeometry>
</mxCell>
<mxCell id="37" value="push result or pull tar file" style="text;html=1;strokeColor=none;fillColor=none;align=center;verticalAlign=middle;whiteSpace=wrap;rounded=0;" vertex="1" parent="1">
<mxGeometry x="420" y="390" width="140" height="30" as="geometry"/>
</mxCell>
<mxCell id="38" value="" style="endArrow=classic;html=1;exitX=-0.038;exitY=0.87;exitDx=0;exitDy=0;exitPerimeter=0;entryX=1.066;entryY=0.905;entryDx=0;entryDy=0;entryPerimeter=0;" edge="1" parent="1" target="14">
<mxGeometry width="50" height="50" relative="1" as="geometry">
<mxPoint x="380" y="449.9999999999999" as="sourcePoint"/>
<mxPoint x="201.24" y="450.35" as="targetPoint"/>
</mxGeometry>
</mxCell>
<mxCell id="39" value="busybox.tar" style="rounded=1;whiteSpace=wrap;html=1;" vertex="1" parent="1">
<mxGeometry x="50" y="436" width="60" height="50" as="geometry"/>
</mxCell>
<mxCell id="40" value="" style="endArrow=classic;html=1;exitX=0.005;exitY=0.961;exitDx=0;exitDy=0;exitPerimeter=0;entryX=1;entryY=0.5;entryDx=0;entryDy=0;" edge="1" parent="1" source="14" target="39">
<mxGeometry width="50" height="50" relative="1" as="geometry">
<mxPoint x="210" y="330" as="sourcePoint"/>
<mxPoint x="260" y="280" as="targetPoint"/>
</mxGeometry>
</mxCell>
<mxCell id="41" value="get concurrent num" style="text;html=1;align=center;verticalAlign=middle;resizable=0;points=[];autosize=1;strokeColor=none;fillColor=none;" vertex="1" parent="1">
<mxGeometry x="615" y="300" width="130" height="30" as="geometry"/>
</mxCell>
</root>
</mxGraphModel>
</diagram>
</mxfile>
Binary file added proposals/images/portal-upordown-image/data-flow.png
View file
Open in desktop
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
149 changes: 149 additions & 0 deletions proposals/new/portal-upordown-image.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,149 @@
# Upload or Download Artifacte from portal

Author: Leng Rongfu

## Abstract
Current users, upload artifacts to the current registry from the Harbor portal, or download artifacts.


## Background
Currently, push/pull images must be able to access the registry service. However, in some enterprises, users are denied access to the /v2 interface of the core service on their local machines, and can only access the Harbor portal page. Therefore, it is more convenient to upload or download images on the page.


## User Stories

### Story 1
As a system administrator, you can upload or download images from any project.

### Story 2
As a project administrator, you can upload or download images from the current project.

### Story 3
As a user, when you have the push image permission for a certain project, you can upload images to that project.

### Story 4
As a user, when you have the pull image permission for a certain project, you can download images to that project.

### Story 6
As a user, when you not have the pull or push image permission for a certain project, you not can download or upload images to that project.

## Premise
1, The portal can be accessed normally.

## Proposal

We propose the following solutions:

1. Add "Upload Image" and "Download Image" buttons to the Action section on the Repositories detail page.
2. When a user has the push image permission, the "Upload Image" button is clickable.
3. When a user has the pull image permission, the "Download Image" button is clickable.
4. Because it consumes resources, system administrators can set the number of concurrent uploads or downloads allowed.

### APIs for upload/download image

1. Upload image

```
POST /api/v2.0/projects/{project}/repositories/{repositorie}/artifacts/upload?tag=v1
Content-type: form-data
Body: file

```

2. Download image

```
GET /api/v2.0/projects/{project}/repositories/{repositorie}/artifacts/download?tag=v1

```


### DB scheme

> ConfigEntry is System Setting use table.
```
type ConfigEntry struct {
ID int64 `orm:"pk;auto;column(id)" json:"-"`
Key string `orm:"column(k)" json:"k"`
Value string `orm:"column(v)" json:"v"`
}
```
Key: `upload_concurrent`
Key: `download_concurrent`

### Configuration Manager

Add the function to configure the limit of concurrent uploads or downloads on the System Setting page, and store it in the ConfigEntry database table through the Configuration Controller.


Use `src/controller/config/controller.go` `Controller` to manager concurrent.


### ArtifactAPI

```go
type ArtifactAPI interface {
UploadArtifact(ctx context.Context, params artifact.UploadArtifactParams) middleware.Responder
DownloadArtifact(ctx context.Context, params artifact.DownloadArtifactParams) middleware.Responder
}

type UploadArtifactParams {
HTTPRequest *http.Request `json:"-"`
XRequestID *string
tag string
ProjectName string
RepositoryName string
}

type DownloadArtifactParams{
HTTPRequest *http.Request `json:"-"`
XRequestID *string
tag string
ProjectName string
RepositoryName string
}
```

### Solution

![data-flow](../images/portal-upordown-image/data-flow.png)

Image uploading and downloading are performed in the Harbor-core service. During uploading, the input stream is read from the http.Request and then directly pushed to the registry. During downloading, the image is pulled from the registry and then directly written into the http.ResponseWriter.

#### Use skopeo tools

use `skopeo` command tools to push or pull image.

```go
func uploadHandler(w http.ResponseWriter, r *http.Request) {
file, _, err := r.FormFile("file")
cmd := exec.Command("skopeo", "copy","docker-archive:/dev/stdin", "docker://registry.com/library/busybox:v1")
cmd.Stdin = file
cmd.Stdout = os.Stdout
cmd.Stderr = os.Stderr
}
func downloadHandler(w http.ResponseWriter, r *http.Request) {
cmd := exec.Command("skopeo", "copy", "--format=v2s2", "--quiet", "docker://registry.com/library/busybox:v1", "docker-archive:/dev/stdout")
cmd.Stdin = os.Stdin
cmd.Stdout = w
cmd.Stderr = os.Stderr
}
```

#### Use Regclient

use `github.com/regclient/regclient` project to push or pull image.

```go
func uploadHandler(w http.ResponseWriter, r *http.Request) {
file, _, err := r.FormFile("file")
imageRef, err := ref.New("registry.com/library/busybox:v1")
err = regClient.ImageImport(context.Background(), imageRef, &file)
}

func downloadHandler(w http.ResponseWriter, r *http.Request) {
imageRef, err := ref.New("registry.com/library/busybox:v1")
regClient.ImageExport(context.Background(), imageRef, w)
}
```