amorfo77/restic
1
0 Fork 0
mirror of https://github.com/restic/restic.git synced 2026-10-13 05:27:13 +00:00
Code Issues Projects Releases Wiki Activity

Update dependencies, enable pruning for vendor/

Browse Source 
So, `dep` got an nice new feature to remove tests and non-go files from
`vendor/`, and this brings the size of the vendor directory from ~300MiB
down to ~20MiB. We don that now.
This commit is contained in:
Alexander Neumann committed 2018-08-01 21:32:15 +02:00
1 parent 3422c1ca83
commit bff635bc5f
6741 files changed
+26940 -4902031
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

vendor/github.com/kurin/blazer/.gitignore
Generated
-26
View File
@@ -1,26 +0,0 @@
bin/bonfire/bonfire
# Compiled Object files, Static and Dynamic libs (Shared Objects)
*.o
*.a
*.so
# Folders
_obj
_test
# Architecture specific extensions/prefixes
*.[568vq]
[568vq].out
*.cgo1.go
*.cgo2.c
_cgo_defun.c
_cgo_gotypes.go
_cgo_export.*
_testmain.go
*.exe
*.test
*.prof
vendor/github.com/kurin/blazer/.travis.yml
Generated Vendored
-13
View File
@@ -1,13 +0,0 @@
language: go
go:
- tip
branches:
only:
- master
before_script: go run internal/bin/cleanup/cleanup.go
script:
- go test -v ./base ./b2 ./x/...
- go vet -v ./base ./b2 ./x/...
vendor/github.com/kurin/blazer/CONTRIBUTING.md
Generated Vendored
-27
View File
@@ -1,27 +0,0 @@
Want to contribute? Great! First, read this page (including the small print at the end).
### Before you contribute
Before we can use your code, you must sign the
[Google Individual Contributor License Agreement]
(https://cla.developers.google.com/about/google-individual)
(CLA), which you can do online. The CLA is necessary mainly because you own the
copyright to your changes, even after your contribution becomes part of our
codebase, so we need your permission to use and distribute your code. We also
need to be sure of various other things—for instance that you'll tell us if you
know that your code infringes on other people's patents. You don't have to sign
the CLA until after you've submitted your code for review and a member has
approved it, but you must do it before we can put your code into our codebase.
Before you start working on a larger contribution, you should get in touch with
us first through the issue tracker with your idea so that we can help out and
possibly guide you. Coordinating up front makes it much easier to avoid
frustration later on.
### Code reviews
All submissions, including submissions by project members, require review. We
use Github pull requests for this purpose.
### The small print
Contributions made by corporations are covered by a different agreement than
the one above, the
[Software Grant and Corporate Contributor License Agreement]
(https://cla.developers.google.com/about/google-corporate).
vendor/github.com/kurin/blazer/README.md
Generated Vendored
-130
View File
@@ -1,130 +0,0 @@
Blazer
====
[![GoDoc](https://godoc.org/github.com/kurin/blazer/b2?status.svg)](https://godoc.org/github.com/kurin/blazer/b2)
[![Build Status](https://travis-ci.org/kurin/blazer.svg)](https://travis-ci.org/kurin/blazer)
Blazer is a Golang client library for Backblaze's B2 object storage service.
It is designed for simple integration with existing applications that may
already be using S3 and Google Cloud Storage, by exporting only a few standard
Go types.
It implements and satisfies the [B2 integration
checklist](https://www.backblaze.com/b2/docs/integration_checklist.html),
automatically handling error recovery, reauthentication, and other low-level
aspects, making it suitable to upload very large files, or over multi-day time
scales.
```go
import "github.com/kurin/blazer/b2"
```
## Examples
### Copy a file into B2
```go
func copyFile(ctx context.Context, bucket *b2.Bucket, src, dst string) error {
f, err := file.Open(src)
if err != nil {
return err
}
defer f.Close()
obj := bucket.Object(dst)
w := obj.NewWriter(ctx)
if _, err := io.Copy(w, f); err != nil {
w.Close()
return err
}
return w.Close()
}
```
If the file is less than 100MB, Blazer will simply buffer the file and use the
`b2_upload_file` API to send the file to Backblaze. If the file is greater
than 100MB, Blazer will use B2's large file support to upload the file in 100MB
chunks.
### Copy a file into B2, with multiple concurrent uploads
Uploading a large file with multiple HTTP connections is simple:
```go
func copyFile(ctx context.Context, bucket *b2.Bucket, writers int, src, dst string) error {
f, err := file.Open(src)
if err != nil {
return err
}
defer f.Close()
w := bucket.Object(dst).NewWriter(ctx)
w.ConcurrentUploads = writers
if _, err := io.Copy(w, f); err != nil {
w.Close()
return err
}
return w.Close()
}
```
This will automatically split the file into `writers` chunks of 100MB uploads.
Note that 100MB is the smallest chunk size that B2 supports.
### Download a file from B2
Downloading is as simple as uploading:
```go
func downloadFile(ctx context.Context, bucket *b2.Bucket, downloads int, src, dst string) error {
r := bucket.Object(src).NewReader(ctx)
defer r.Close()
f, err := file.Create(dst)
if err != nil {
return err
}
r.ConcurrentDownloads = downloads
if _, err := io.Copy(f, r); err != nil {
f.Close()
return err
}
return f.Close()
}
```
### List all objects in a bucket
```go
func printObjects(ctx context.Context, bucket *b2.Bucket) error {
iterator := bucket.List(ctx)
for iterator.Next() {
fmt.Println(itrator.Object())
}
return iterator.Err()
}
```
### Grant temporary auth to a file
Say you have a number of files in a private bucket, and you want to allow other
people to download some files. This is possible to do by issuing a temporary
authorization token for the prefix of the files you want to share.
```go
token, err := bucket.AuthToken(ctx, "photos", time.Hour)
```
If successful, `token` is then an authorization token valid for one hour, which
can be set in HTTP GET requests.
The hostname to use when downloading files via HTTP is account-specific and can
be found via the BaseURL method:
```go
base := bucket.BaseURL()
```
---
This is not an official Google product.
vendor/github.com/kurin/blazer/b2/b2_test.go
Generated Vendored
-869
View File
@@ -1,869 +0,0 @@
// Copyright 2016, Google
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package b2
import (
"bytes"
"context"
"crypto/sha1"
"fmt"
"io"
"io/ioutil"
"net/http"
"sort"
"strings"
"sync"
"testing"
"time"
)
const (
bucketName = "b2-tests"
smallFileName = "Teeny Tiny"
largeFileName = "BigBytes"
)
var gmux = &sync.Mutex{}
type testError struct {
retry bool
backoff time.Duration
reauth bool
reupload bool
}
func (t testError) Error() string {
return fmt.Sprintf("retry %v; backoff %v; reauth %v; reupload %v", t.retry, t.backoff, t.reauth, t.reupload)
}
type errCont struct {
errMap map[string]map[int]error
opMap map[string]int
}
func (e *errCont) getError(name string) error {
if e.errMap == nil {
return nil
}
if e.opMap == nil {
e.opMap = make(map[string]int)
}
i := e.opMap[name]
e.opMap[name]++
return e.errMap[name][i]
}
type testRoot struct {
errs *errCont
auths int
bucketMap map[string]map[string]string
}
func (t *testRoot) authorizeAccount(context.Context, string, string, clientOptions) error {
t.auths++
return nil
}
func (t *testRoot) backoff(err error) time.Duration {
e, ok := err.(testError)
if !ok {
return 0
}
return e.backoff
}
func (t *testRoot) reauth(err error) bool {
e, ok := err.(testError)
if !ok {
return false
}
return e.reauth
}
func (t *testRoot) reupload(err error) bool {
e, ok := err.(testError)
if !ok {
return false
}
return e.reupload
}
func (t *testRoot) transient(err error) bool {
e, ok := err.(testError)
if !ok {
return false
}
return e.retry || e.reupload || e.backoff > 0
}
func (t *testRoot) createKey(context.Context, string, []string, time.Duration, string, string) (b2KeyInterface, error) {
return nil, nil
}
func (t *testRoot) listKeys(context.Context, int, string) ([]b2KeyInterface, string, error) {
return nil, "", nil
}
func (t *testRoot) createBucket(_ context.Context, name, _ string, _ map[string]string, _ []LifecycleRule) (b2BucketInterface, error) {
if err := t.errs.getError("createBucket"); err != nil {
return nil, err
}
if _, ok := t.bucketMap[name]; ok {
return nil, fmt.Errorf("%s: bucket exists", name)
}
m := make(map[string]string)
t.bucketMap[name] = m
return &testBucket{
n: name,
errs: t.errs,
files: m,
}, nil
}
func (t *testRoot) listBuckets(context.Context) ([]b2BucketInterface, error) {
var b []b2BucketInterface
for k, v := range t.bucketMap {
b = append(b, &testBucket{
n: k,
errs: t.errs,
files: v,
})
}
return b, nil
}
type testBucket struct {
n string
errs *errCont
files map[string]string
}
func (t *testBucket) name() string { return t.n }
func (t *testBucket) btype() string { return "allPrivate" }
func (t *testBucket) attrs() *BucketAttrs { return nil }
func (t *testBucket) deleteBucket(context.Context) error { return nil }
func (t *testBucket) updateBucket(context.Context, *BucketAttrs) error { return nil }
func (t *testBucket) id() string { return "" }
func (t *testBucket) getUploadURL(context.Context) (b2URLInterface, error) {
if err := t.errs.getError("getUploadURL"); err != nil {
return nil, err
}
return &testURL{
files: t.files,
}, nil
}
func (t *testBucket) startLargeFile(_ context.Context, name, _ string, _ map[string]string) (b2LargeFileInterface, error) {
return &testLargeFile{
name: name,
parts: make(map[int][]byte),
files: t.files,
errs: t.errs,
}, nil
}
func (t *testBucket) listFileNames(ctx context.Context, count int, cont, pfx, del string) ([]b2FileInterface, string, error) {
var f []string
gmux.Lock()
defer gmux.Unlock()
for name := range t.files {
f = append(f, name)
}
sort.Strings(f)
idx := sort.SearchStrings(f, cont)
var b []b2FileInterface
var next string
for i := idx; i < len(f) && i-idx < count; i++ {
b = append(b, &testFile{
n: f[i],
s: int64(len(t.files[f[i]])),
files: t.files,
})
if i+1 < len(f) {
next = f[i+1]
}
if i+1 == len(f) {
next = ""
}
}
return b, next, nil
}
func (t *testBucket) listFileVersions(ctx context.Context, count int, a, b, c, d string) ([]b2FileInterface, string, string, error) {
x, y, z := t.listFileNames(ctx, count, a, c, d)
return x, y, "", z
}
func (t *testBucket) listUnfinishedLargeFiles(ctx context.Context, count int, cont string) ([]b2FileInterface, string, error) {
return nil, "", fmt.Errorf("testBucket.listUnfinishedLargeFiles(ctx, %d, %q): not implemented", count, cont)
}
func (t *testBucket) downloadFileByName(_ context.Context, name string, offset, size int64) (b2FileReaderInterface, error) {
gmux.Lock()
defer gmux.Unlock()
f := t.files[name]
end := int(offset + size)
if end >= len(f) {
end = len(f)
}
if int(offset) >= len(f) {
return nil, errNoMoreContent
}
return &testFileReader{
b: ioutil.NopCloser(bytes.NewBufferString(f[offset:end])),
s: end - int(offset),
n: name,
}, nil
}
func (t *testBucket) hideFile(context.Context, string) (b2FileInterface, error) { return nil, nil }
func (t *testBucket) getDownloadAuthorization(context.Context, string, time.Duration, string) (string, error) {
return "", nil
}
func (t *testBucket) baseURL() string { return "" }
func (t *testBucket) file(id, name string) b2FileInterface { return nil }
type testURL struct {
files map[string]string
}
func (t *testURL) reload(context.Context) error { return nil }
func (t *testURL) uploadFile(_ context.Context, r io.Reader, _ int, name, _, _ string, _ map[string]string) (b2FileInterface, error) {
buf := &bytes.Buffer{}
if _, err := io.Copy(buf, r); err != nil {
return nil, err
}
gmux.Lock()
defer gmux.Unlock()
t.files[name] = buf.String()
return &testFile{
n: name,
s: int64(len(t.files[name])),
files: t.files,
}, nil
}
type testLargeFile struct {
name string
parts map[int][]byte
files map[string]string
errs *errCont
}
func (t *testLargeFile) finishLargeFile(context.Context) (b2FileInterface, error) {
var total []byte
gmux.Lock()
defer gmux.Unlock()
for i := 1; i <= len(t.parts); i++ {
total = append(total, t.parts[i]...)
}
t.files[t.name] = string(total)
return &testFile{
n: t.name,
s: int64(len(total)),
files: t.files,
}, nil
}
func (t *testLargeFile) getUploadPartURL(context.Context) (b2FileChunkInterface, error) {
gmux.Lock()
defer gmux.Unlock()
return &testFileChunk{
parts: t.parts,
errs: t.errs,
}, nil
}
type testFileChunk struct {
parts map[int][]byte
errs *errCont
}
func (t *testFileChunk) reload(context.Context) error { return nil }
func (t *testFileChunk) uploadPart(_ context.Context, r io.Reader, _ string, _, index int) (int, error) {
if err := t.errs.getError("uploadPart"); err != nil {
return 0, err
}
buf := &bytes.Buffer{}
i, err := io.Copy(buf, r)
if err != nil {
return int(i), err
}
gmux.Lock()
defer gmux.Unlock()
t.parts[index] = buf.Bytes()
return int(i), nil
}
type testFile struct {
n string
s int64
t time.Time
a string
files map[string]string
}
func (t *testFile) name() string { return t.n }
func (t *testFile) size() int64 { return t.s }
func (t *testFile) timestamp() time.Time { return t.t }
func (t *testFile) status() string { return t.a }
func (t *testFile) compileParts(int64, map[int]string) b2LargeFileInterface {
panic("not implemented")
}
func (t *testFile) getFileInfo(context.Context) (b2FileInfoInterface, error) {
return nil, nil
}
func (t *testFile) listParts(context.Context, int, int) ([]b2FilePartInterface, int, error) {
return nil, 0, nil
}
func (t *testFile) deleteFileVersion(context.Context) error {
gmux.Lock()
defer gmux.Unlock()
delete(t.files, t.n)
return nil
}
type testFileReader struct {
b io.ReadCloser
s int
n string
}
func (t *testFileReader) Read(p []byte) (int, error) { return t.b.Read(p) }
func (t *testFileReader) Close() error { return nil }
func (t *testFileReader) stats() (int, string, string, map[string]string) { return t.s, "", "", nil }
func (t *testFileReader) id() string { return t.n }
type zReader struct{}
var pattern = []byte{0x02, 0x80, 0xff, 0x1a, 0xcc, 0x63, 0x22}
func (zReader) Read(p []byte) (int, error) {
for i := 0; i+len(pattern) < len(p); i += len(pattern) {
copy(p[i:], pattern)
}
return len(p), nil
}
type zReadSeeker struct {
size int64
pos int64
}
func (rs *zReadSeeker) Read(p []byte) (int, error) {
for i := rs.pos; ; i++ {
j := int(i - rs.pos)
if j >= len(p) || i >= rs.size {
var rtn error
if i >= rs.size {
rtn = io.EOF
}
rs.pos = i
return j, rtn
}
f := int(i) % len(pattern)
p[j] = pattern[f]
}
}
func (rs *zReadSeeker) Seek(offset int64, whence int) (int64, error) {
switch whence {
case io.SeekStart:
rs.pos = offset
case io.SeekEnd:
rs.pos = rs.size + offset
}
return rs.pos, nil
}
func TestReaderFrom(t *testing.T) {
ctx := context.Background()
ctx, cancel := context.WithTimeout(ctx, 10*time.Second)
defer cancel()
table := []struct {
size, pos int64
}{
{
size: 10,
},
}
for _, e := range table {
client := &Client{
backend: &beRoot{
b2i: &testRoot{
bucketMap: make(map[string]map[string]string),
errs: &errCont{},
},
},
}
bucket, err := client.NewBucket(ctx, bucketName, &BucketAttrs{Type: Private})
if err != nil {
t.Fatal(err)
}
defer func() {
if err := bucket.Delete(ctx); err != nil {
t.Error(err)
}
}()
r := &zReadSeeker{pos: e.pos, size: e.size}
w := bucket.Object("writer").NewWriter(ctx)
n, err := w.ReadFrom(r)
if err != nil {
t.Errorf("ReadFrom(): %v", err)
}
if n != e.size {
t.Errorf("ReadFrom(): got %d bytes, wanted %d bytes", n, e.size)
}
}
}
func TestReauth(t *testing.T) {
ctx := context.Background()
ctx, cancel := context.WithTimeout(ctx, 10*time.Second)
defer cancel()
root := &testRoot{
bucketMap: make(map[string]map[string]string),
errs: &errCont{
errMap: map[string]map[int]error{
"createBucket": {0: testError{reauth: true}},
},
},
}
client := &Client{
backend: &beRoot{
b2i: root,
},
}
auths := root.auths
if _, err := client.NewBucket(ctx, "fun", &BucketAttrs{Type: Private}); err != nil {
t.Errorf("bucket should not err, got %v", err)
}
if root.auths != auths+1 {
t.Errorf("client should have re-authenticated; did not")
}
}
func TestBackoff(t *testing.T) {
ctx := context.Background()
ctx, cancel := context.WithTimeout(ctx, 10*time.Second)
defer cancel()
var calls []time.Duration
ch := make(chan time.Time)
close(ch)
after = func(d time.Duration) <-chan time.Time {
calls = append(calls, d)
return ch
}
table := []struct {
root *testRoot
want int
}{
{
root: &testRoot{
bucketMap: make(map[string]map[string]string),
errs: &errCont{
errMap: map[string]map[int]error{
"createBucket": {
0: testError{backoff: time.Second},
1: testError{backoff: 2 * time.Second},
},
},
},
},
want: 2,
},
{
root: &testRoot{
bucketMap: make(map[string]map[string]string),
errs: &errCont{
errMap: map[string]map[int]error{
"getUploadURL": {
0: testError{retry: true},
},
},
},
},
want: 1,
},
}
var total int
for _, ent := range table {
client := &Client{
backend: &beRoot{
b2i: ent.root,
},
}
b, err := client.NewBucket(ctx, "fun", &BucketAttrs{Type: Private})
if err != nil {
t.Fatal(err)
}
o := b.Object("foo")
w := o.NewWriter(ctx)
if _, err := io.Copy(w, bytes.NewBufferString("foo")); err != nil {
t.Fatal(err)
}
if err := w.Close(); err != nil {
t.Fatal(err)
}
total += ent.want
}
if len(calls) != total {
t.Errorf("got %d calls, wanted %d", len(calls), total)
}
}
func TestBackoffWithoutRetryAfter(t *testing.T) {
ctx := context.Background()
ctx, cancel := context.WithTimeout(ctx, 10*time.Second)
defer cancel()
var calls []time.Duration
ch := make(chan time.Time)
close(ch)
after = func(d time.Duration) <-chan time.Time {
calls = append(calls, d)
return ch
}
root := &testRoot{
bucketMap: make(map[string]map[string]string),
errs: &errCont{
errMap: map[string]map[int]error{
"createBucket": {
0: testError{retry: true},
1: testError{retry: true},
},
},
},
}
client := &Client{
backend: &beRoot{
b2i: root,
},
}
if _, err := client.NewBucket(ctx, "fun", &BucketAttrs{Type: Private}); err != nil {
t.Errorf("bucket should not err, got %v", err)
}
if len(calls) != 2 {
t.Errorf("wrong number of backoff calls; got %d, want 2", len(calls))
}
}
type badTransport struct{}
func (badTransport) RoundTrip(r *http.Request) (*http.Response, error) {
return &http.Response{
Status: "700 What",
StatusCode: 700,
Body: ioutil.NopCloser(bytes.NewBufferString("{}")),
Request: r,
}, nil
}
func TestCustomTransport(t *testing.T) {
ctx := context.Background()
// Sorta fragile but...
_, err := NewClient(ctx, "abcd", "efgh", Transport(badTransport{}))
if err == nil {
t.Error("NewClient returned successfully, expected an error")
}
if !strings.Contains(err.Error(), "700") {
t.Errorf("Expected nonsense error code 700, got %v", err)
}
}
func TestReaderDoubleClose(t *testing.T) {
ctx := context.Background()
client := &Client{
backend: &beRoot{
b2i: &testRoot{
bucketMap: make(map[string]map[string]string),
errs: &errCont{},
},
},
}
bucket, err := client.NewBucket(ctx, "bucket", &BucketAttrs{Type: Private})
if err != nil {
t.Fatal(err)
}
o, _, err := writeFile(ctx, bucket, "file", 10, 10)
if err != nil {
t.Fatal(err)
}
r := o.NewReader(ctx)
// Read to EOF, and then read some more.
if _, err := io.Copy(ioutil.Discard, r); err != nil {
t.Fatal(err)
}
if _, err := io.Copy(ioutil.Discard, r); err != nil {
t.Fatal(err)
}
}
func TestReadWrite(t *testing.T) {
ctx := context.Background()
ctx, cancel := context.WithTimeout(ctx, 10*time.Second)
defer cancel()
client := &Client{
backend: &beRoot{
b2i: &testRoot{
bucketMap: make(map[string]map[string]string),
errs: &errCont{},
},
},
}
bucket, err := client.NewBucket(ctx, bucketName, &BucketAttrs{Type: Private})
if err != nil {
t.Fatal(err)
}
defer func() {
if err := bucket.Delete(ctx); err != nil {
t.Error(err)
}
}()
sobj, wsha, err := writeFile(ctx, bucket, smallFileName, 1e6+42, 1e8)
if err != nil {
t.Fatal(err)
}
defer func() {
if err := sobj.Delete(ctx); err != nil {
t.Error(err)
}
}()
if err := readFile(ctx, sobj, wsha, 1e5, 10); err != nil {
t.Error(err)
}
lobj, wshaL, err := writeFile(ctx, bucket, largeFileName, 1e6-1e5, 1e4)
if err != nil {
t.Fatal(err)
}
defer func() {
if err := lobj.Delete(ctx); err != nil {
t.Error(err)
}
}()
if err := readFile(ctx, lobj, wshaL, 1e7, 10); err != nil {
t.Error(err)
}
}
func TestReadRangeReturnsRight(t *testing.T) {
ctx := context.Background()
ctx, cancel := context.WithTimeout(ctx, 10*time.Second)
defer cancel()
client := &Client{
backend: &beRoot{
b2i: &testRoot{
bucketMap: make(map[string]map[string]string),
errs: &errCont{},
},
},
}
bucket, err := client.NewBucket(ctx, bucketName, &BucketAttrs{Type: Private})
if err != nil {
t.Fatal(err)
}
defer func() {
if err := bucket.Delete(ctx); err != nil {
t.Error(err)
}
}()
obj, _, err := writeFile(ctx, bucket, "file", 1e6+42, 1e8)
if err != nil {
t.Fatal(err)
}
r := obj.NewRangeReader(ctx, 200, 1400)
r.ChunkSize = 1000
i, err := io.Copy(ioutil.Discard, r)
if err != nil {
t.Error(err)
}
if i != 1400 {
t.Errorf("NewRangeReader(_, 200, 1400): want 1400, got %d", i)
}
}
func TestWriterReturnsError(t *testing.T) {
ctx := context.Background()
ctx, cancel := context.WithTimeout(ctx, 10*time.Second)
defer cancel()
client := &Client{
backend: &beRoot{
b2i: &testRoot{
bucketMap: make(map[string]map[string]string),
errs: &errCont{
errMap: map[string]map[int]error{
"uploadPart": {
0: testError{},
1: testError{},
2: testError{},
3: testError{},
4: testError{},
5: testError{},
6: testError{},
},
},
},
},
},
}
bucket, err := client.NewBucket(ctx, bucketName, &BucketAttrs{Type: Private})
if err != nil {
t.Fatal(err)
}
w := bucket.Object("test").NewWriter(ctx)
r := io.LimitReader(zReader{}, 1e7)
w.ChunkSize = 1e4
w.ConcurrentUploads = 4
if _, err := io.Copy(w, r); err == nil {
t.Fatalf("io.Copy: should have returned an error")
}
}
func TestFileBuffer(t *testing.T) {
r := io.LimitReader(zReader{}, 1e8)
w, err := newFileBuffer("")
if err != nil {
t.Fatal(err)
}
defer w.Close()
if _, err := io.Copy(w, r); err != nil {
t.Fatal(err)
}
bReader, err := w.Reader()
if err != nil {
t.Fatal(err)
}
hsh := sha1.New()
if _, err := io.Copy(hsh, bReader); err != nil {
t.Fatal(err)
}
hshText := fmt.Sprintf("%x", hsh.Sum(nil))
if hshText != w.Hash() {
t.Errorf("hashes are not equal: bufferWriter is %q, read buffer is %q", w.Hash(), hshText)
}
}
func TestNonBuffer(t *testing.T) {
table := []struct {
str string
off int64
len int64
want string
}{
{
str: "a string",
off: 0,
len: 3,
want: "a s",
},
{
str: "a string",
off: 3,
len: 1,
want: "t",
},
{
str: "a string",
off: 3,
len: 5,
want: "tring",
},
}
for _, e := range table {
nb := newNonBuffer(strings.NewReader(e.str), e.off, e.len)
want := fmt.Sprintf("%s%x", e.want, sha1.Sum([]byte(e.str[int(e.off):int(e.off+e.len)])))
r, err := nb.Reader()
if err != nil {
t.Error(err)
continue
}
got, err := ioutil.ReadAll(r)
if err != nil {
t.Errorf("ioutil.ReadAll(%#v): %v", e, err)
continue
}
if want != string(got) {
t.Errorf("ioutil.ReadAll(%#v): got %q, want %q", e, string(got), want)
}
}
}
func writeFile(ctx context.Context, bucket *Bucket, name string, size int64, csize int) (*Object, string, error) {
r := io.LimitReader(zReader{}, size)
o := bucket.Object(name)
f := o.NewWriter(ctx)
h := sha1.New()
w := io.MultiWriter(f, h)
f.ConcurrentUploads = 5
f.ChunkSize = csize
n, err := io.Copy(w, r)
if err != nil {
return nil, "", err
}
if n != size {
return nil, "", fmt.Errorf("io.Copy(): wrote %d bytes; wanted %d bytes", n, size)
}
if err := f.Close(); err != nil {
return nil, "", err
}
return o, fmt.Sprintf("%x", h.Sum(nil)), nil
}
func readFile(ctx context.Context, obj *Object, sha string, chunk, concur int) error {
r := obj.NewReader(ctx)
r.ChunkSize = chunk
r.ConcurrentDownloads = concur
h := sha1.New()
if _, err := io.Copy(h, r); err != nil {
return err
}
if err := r.Close(); err != nil {
return err
}
rsha := fmt.Sprintf("%x", h.Sum(nil))
if sha != rsha {
return fmt.Errorf("bad hash: got %s, want %s", rsha, sha)
}
return nil
}
vendor/github.com/kurin/blazer/b2/integration_test.go
Generated Vendored
-1273
View File
File diff suppressed because it is too large. Load diff
vendor/github.com/kurin/blazer/base/integration_test.go
Generated Vendored
-649
View File
@@ -1,649 +0,0 @@
// Copyright 2016, Google
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package base
import (
"bytes"
"crypto/sha1"
"encoding/json"
"fmt"
"io"
"os"
"reflect"
"strings"
"testing"
"time"
"github.com/kurin/blazer/x/transport"
"context"
)
const (
apiID = "B2_ACCOUNT_ID"
apiKey = "B2_SECRET_KEY"
)
const (
bucketName = "base-tests"
smallFileName = "TeenyTiny"
largeFileName = "BigBytes"
)
type zReader struct{}
func (zReader) Read(p []byte) (int, error) {
return len(p), nil
}
func TestStorage(t *testing.T) {
id := os.Getenv(apiID)
key := os.Getenv(apiKey)
if id == "" || key == "" {
t.Skipf("B2_ACCOUNT_ID or B2_SECRET_KEY unset; skipping integration tests")
}
ctx := context.Background()
// b2_authorize_account
b2, err := AuthorizeAccount(ctx, id, key, UserAgent("blazer-base-test"))
if err != nil {
t.Fatal(err)
}
// b2_create_bucket
infoKey := "key"
infoVal := "val"
m := map[string]string{infoKey: infoVal}
rules := []LifecycleRule{
{
Prefix: "what/",
DaysNewUntilHidden: 5,
},
}
bname := id + "-" + bucketName
bucket, err := b2.CreateBucket(ctx, bname, "", m, rules)
if err != nil {
t.Fatal(err)
}
if bucket.Info[infoKey] != infoVal {
t.Errorf("%s: bucketInfo[%q] got %q, want %q", bucket.Name, infoKey, bucket.Info[infoKey], infoVal)
}
if len(bucket.LifecycleRules) != 1 {
t.Errorf("%s: lifecycle rules: got %d rules, wanted 1", bucket.Name, len(bucket.LifecycleRules))
}
defer func() {
// b2_delete_bucket
if err := bucket.DeleteBucket(ctx); err != nil {
t.Error(err)
}
}()
// b2_update_bucket
bucket.Info["new"] = "yay"
bucket.LifecycleRules = nil // Unset options should be a noop.
newBucket, err := bucket.Update(ctx)
if err != nil {
t.Errorf("%s: update bucket: %v", bucket.Name, err)
return
}
bucket = newBucket
if bucket.Info["new"] != "yay" {
t.Errorf("%s: info key \"new\": got %s, want \"yay\"", bucket.Name, bucket.Info["new"])
}
if len(bucket.LifecycleRules) != 1 {
t.Errorf("%s: lifecycle rules: got %d rules, wanted 1", bucket.Name, len(bucket.LifecycleRules))
}
// b2_list_buckets
buckets, err := b2.ListBuckets(ctx)
if err != nil {
t.Fatal(err)
}
var found bool
for _, bucket := range buckets {
if bucket.Name == bname {
found = true
break
}
}
if !found {
t.Errorf("%s: new bucket not found", bname)
}
// b2_get_upload_url
ue, err := bucket.GetUploadURL(ctx)
if err != nil {
t.Fatal(err)
}
// b2_upload_file
smallFile := io.LimitReader(zReader{}, 1024*50) // 50k
hash := sha1.New()
buf := &bytes.Buffer{}
w := io.MultiWriter(hash, buf)
if _, err := io.Copy(w, smallFile); err != nil {
t.Error(err)
}
smallSHA1 := fmt.Sprintf("%x", hash.Sum(nil))
smallInfoMap := map[string]string{
"one": "1",
"two": "2",
}
file, err := ue.UploadFile(ctx, buf, buf.Len(), smallFileName, "application/octet-stream", smallSHA1, smallInfoMap)
if err != nil {
t.Fatal(err)
}
defer func() {
// b2_delete_file_version
if err := file.DeleteFileVersion(ctx); err != nil {
t.Error(err)
}
}()
// b2_start_large_file
largeInfoMap := map[string]string{
"one_billion": "1e9",
"two_trillion": "2eSomething, I guess 2e12",
}
lf, err := bucket.StartLargeFile(ctx, largeFileName, "application/octet-stream", largeInfoMap)
if err != nil {
t.Fatal(err)
}
// b2_get_upload_part_url
fc, err := lf.GetUploadPartURL(ctx)
if err != nil {
t.Fatal(err)
}
// b2_upload_part
largeFile := io.LimitReader(zReader{}, 10e6) // 10M
for i := 0; i < 2; i++ {
r := io.LimitReader(largeFile, 5e6) // 5M
hash := sha1.New()
buf := &bytes.Buffer{}
w := io.MultiWriter(hash, buf)
if _, err := io.Copy(w, r); err != nil {
t.Error(err)
}
if _, err := fc.UploadPart(ctx, buf, fmt.Sprintf("%x", hash.Sum(nil)), buf.Len(), i+1); err != nil {
t.Error(err)
}
}
// b2_finish_large_file
lfile, err := lf.FinishLargeFile(ctx)
if err != nil {
t.Fatal(err)
}
// b2_get_file_info
smallInfo, err := file.GetFileInfo(ctx)
if err != nil {
t.Fatal(err)
}
compareFileAndInfo(t, smallInfo, smallFileName, smallSHA1, smallInfoMap)
largeInfo, err := lfile.GetFileInfo(ctx)
if err != nil {
t.Fatal(err)
}
compareFileAndInfo(t, largeInfo, largeFileName, "none", largeInfoMap)
defer func() {
if err := lfile.DeleteFileVersion(ctx); err != nil {
t.Error(err)
}
}()
clf, err := bucket.StartLargeFile(ctx, largeFileName, "application/octet-stream", nil)
if err != nil {
t.Fatal(err)
}
// b2_cancel_large_file
if err := clf.CancelLargeFile(ctx); err != nil {
t.Fatal(err)
}
// b2_list_file_names
files, _, err := bucket.ListFileNames(ctx, 100, "", "", "")
if err != nil {
t.Fatal(err)
}
if len(files) != 2 {
t.Errorf("expected 2 files, got %d: %v", len(files), files)
}
// b2_download_file_by_name
fr, err := bucket.DownloadFileByName(ctx, smallFileName, 0, 0)
if err != nil {
t.Fatal(err)
}
if fr.SHA1 != smallSHA1 {
t.Errorf("small file SHAs don't match: got %q, want %q", fr.SHA1, smallSHA1)
}
lbuf := &bytes.Buffer{}
if _, err := io.Copy(lbuf, fr); err != nil {
t.Fatal(err)
}
if lbuf.Len() != fr.ContentLength {
t.Errorf("small file retreived lengths don't match: got %d, want %d", lbuf.Len(), fr.ContentLength)
}
// b2_hide_file
hf, err := bucket.HideFile(ctx, smallFileName)
if err != nil {
t.Fatal(err)
}
defer func() {
if err := hf.DeleteFileVersion(ctx); err != nil {
t.Error(err)
}
}()
// b2_list_file_versions
files, _, _, err = bucket.ListFileVersions(ctx, 100, "", "", "", "")
if err != nil {
t.Fatal(err)
}
if len(files) != 3 {
t.Errorf("expected 3 files, got %d: %v", len(files), files)
}
// b2_get_download_authorization
if _, err := bucket.GetDownloadAuthorization(ctx, "foo/", 24*time.Hour, "attachment"); err != nil {
t.Errorf("failed to get download auth token: %v", err)
}
}
func TestUploadAuthAfterConnectionHang(t *testing.T) {
id := os.Getenv(apiID)
key := os.Getenv(apiKey)
if id == "" || key == "" {
t.Skipf("B2_ACCOUNT_ID or B2_SECRET_KEY unset; skipping integration tests")
}
ctx := context.Background()
hung := make(chan struct{})
// An http.RoundTripper that dies and hangs after sending ~10k bytes.
hang := func() {
close(hung)
select {}
}
tport := transport.WithFailures(nil, transport.AfterNBytes(10000, hang))
b2, err := AuthorizeAccount(ctx, id, key, Transport(tport))
if err != nil {
t.Fatal(err)
}
bname := id + "-" + bucketName
bucket, err := b2.CreateBucket(ctx, bname, "", nil, nil)
if err != nil {
t.Fatal(err)
}
defer func() {
if err := bucket.DeleteBucket(ctx); err != nil {
t.Error(err)
}
}()
ue, err := bucket.GetUploadURL(ctx)
if err != nil {
t.Fatal(err)
}
smallFile := io.LimitReader(zReader{}, 1024*50) // 50k
hash := sha1.New()
buf := &bytes.Buffer{}
w := io.MultiWriter(hash, buf)
if _, err := io.Copy(w, smallFile); err != nil {
t.Error(err)
}
smallSHA1 := fmt.Sprintf("%x", hash.Sum(nil))
go func() {
ue.UploadFile(ctx, buf, buf.Len(), smallFileName, "application/octet-stream", smallSHA1, nil)
}()
<-hung
// Do the whole thing again with the same upload auth, before the remote end
// notices we're gone.
smallFile = io.LimitReader(zReader{}, 1024*50) // 50k again
buf.Reset()
if _, err := io.Copy(buf, smallFile); err != nil {
t.Error(err)
}
file, err := ue.UploadFile(ctx, buf, buf.Len(), smallFileName, "application/octet-stream", smallSHA1, nil)
if err == nil {
t.Error("expected an error, got none")
if err := file.DeleteFileVersion(ctx); err != nil {
t.Error(err)
}
}
if Action(err) != AttemptNewUpload {
t.Errorf("Action(%v): got %v, want AttemptNewUpload", err, Action(err))
}
}
func TestCancelledContextCancelsHTTPRequest(t *testing.T) {
id := os.Getenv(apiID)
key := os.Getenv(apiKey)
if id == "" || key == "" {
t.Skipf("B2_ACCOUNT_ID or B2_SECRET_KEY unset; skipping integration tests")
}
ctx := context.Background()
tport := transport.WithFailures(nil, transport.MatchPathSubstring("b2_upload_file"), transport.FailureRate(1), transport.Stall(2*time.Second))
b2, err := AuthorizeAccount(ctx, id, key, Transport(tport))
if err != nil {
t.Fatal(err)
}
bname := id + "-" + bucketName
bucket, err := b2.CreateBucket(ctx, bname, "", nil, nil)
if err != nil {
t.Fatal(err)
}
defer func() {
if err := bucket.DeleteBucket(ctx); err != nil {
t.Error(err)
}
}()
ue, err := bucket.GetUploadURL(ctx)
if err != nil {
t.Fatal(err)
}
smallFile := io.LimitReader(zReader{}, 1024*50) // 50k
hash := sha1.New()
buf := &bytes.Buffer{}
w := io.MultiWriter(hash, buf)
if _, err := io.Copy(w, smallFile); err != nil {
t.Error(err)
}
smallSHA1 := fmt.Sprintf("%x", hash.Sum(nil))
cctx, cancel := context.WithCancel(ctx)
go func() {
time.Sleep(1)
cancel()
}()
if _, err := ue.UploadFile(cctx, buf, buf.Len(), smallFileName, "application/octet-stream", smallSHA1, nil); err != context.Canceled {
t.Errorf("expected canceled context, but got %v", err)
}
}
func TestDeadlineExceededContextCancelsHTTPRequest(t *testing.T) {
id := os.Getenv(apiID)
key := os.Getenv(apiKey)
if id == "" || key == "" {
t.Skipf("B2_ACCOUNT_ID or B2_SECRET_KEY unset; skipping integration tests")
}
ctx := context.Background()
tport := transport.WithFailures(nil, transport.MatchPathSubstring("b2_upload_file"), transport.FailureRate(1), transport.Stall(2*time.Second))
b2, err := AuthorizeAccount(ctx, id, key, Transport(tport))
if err != nil {
t.Fatal(err)
}
bname := id + "-" + bucketName
bucket, err := b2.CreateBucket(ctx, bname, "", nil, nil)
if err != nil {
t.Fatal(err)
}
defer func() {
if err := bucket.DeleteBucket(ctx); err != nil {
t.Error(err)
}
}()
ue, err := bucket.GetUploadURL(ctx)
if err != nil {
t.Fatal(err)
}
smallFile := io.LimitReader(zReader{}, 1024*50) // 50k
hash := sha1.New()
buf := &bytes.Buffer{}
w := io.MultiWriter(hash, buf)
if _, err := io.Copy(w, smallFile); err != nil {
t.Error(err)
}
smallSHA1 := fmt.Sprintf("%x", hash.Sum(nil))
cctx, cancel := context.WithTimeout(ctx, time.Second)
defer cancel()
if _, err := ue.UploadFile(cctx, buf, buf.Len(), smallFileName, "application/octet-stream", smallSHA1, nil); err != context.DeadlineExceeded {
t.Errorf("expected deadline exceeded error, but got %v", err)
}
}
func compareFileAndInfo(t *testing.T, info *FileInfo, name, sha1 string, imap map[string]string) {
if info.Name != name {
t.Errorf("got %q, want %q", info.Name, name)
}
if info.SHA1 != sha1 {
t.Errorf("got %q, want %q", info.SHA1, sha1)
}
if !reflect.DeepEqual(info.Info, imap) {
t.Errorf("got %v, want %v", info.Info, imap)
}
}
// from https://www.backblaze.com/b2/docs/string_encoding.html
var testCases = `[
{"fullyEncoded": "%20", "minimallyEncoded": "+", "string": " "},
{"fullyEncoded": "%21", "minimallyEncoded": "!", "string": "!"},
{"fullyEncoded": "%22", "minimallyEncoded": "%22", "string": "\""},
{"fullyEncoded": "%23", "minimallyEncoded": "%23", "string": "#"},
{"fullyEncoded": "%24", "minimallyEncoded": "$", "string": "$"},
{"fullyEncoded": "%25", "minimallyEncoded": "%25", "string": "%"},
{"fullyEncoded": "%26", "minimallyEncoded": "%26", "string": "&"},
{"fullyEncoded": "%27", "minimallyEncoded": "'", "string": "'"},
{"fullyEncoded": "%28", "minimallyEncoded": "(", "string": "("},
{"fullyEncoded": "%29", "minimallyEncoded": ")", "string": ")"},
{"fullyEncoded": "%2A", "minimallyEncoded": "*", "string": "*"},
{"fullyEncoded": "%2B", "minimallyEncoded": "%2B", "string": "+"},
{"fullyEncoded": "%2C", "minimallyEncoded": "%2C", "string": ","},
{"fullyEncoded": "%2D", "minimallyEncoded": "-", "string": "-"},
{"fullyEncoded": "%2E", "minimallyEncoded": ".", "string": "."},
{"fullyEncoded": "/", "minimallyEncoded": "/", "string": "/"},
{"fullyEncoded": "%30", "minimallyEncoded": "0", "string": "0"},
{"fullyEncoded": "%31", "minimallyEncoded": "1", "string": "1"},
{"fullyEncoded": "%32", "minimallyEncoded": "2", "string": "2"},
{"fullyEncoded": "%33", "minimallyEncoded": "3", "string": "3"},
{"fullyEncoded": "%34", "minimallyEncoded": "4", "string": "4"},
{"fullyEncoded": "%35", "minimallyEncoded": "5", "string": "5"},
{"fullyEncoded": "%36", "minimallyEncoded": "6", "string": "6"},
{"fullyEncoded": "%37", "minimallyEncoded": "7", "string": "7"},
{"fullyEncoded": "%38", "minimallyEncoded": "8", "string": "8"},
{"fullyEncoded": "%39", "minimallyEncoded": "9", "string": "9"},
{"fullyEncoded": "%3A", "minimallyEncoded": ":", "string": ":"},
{"fullyEncoded": "%3B", "minimallyEncoded": ";", "string": ";"},
{"fullyEncoded": "%3C", "minimallyEncoded": "%3C", "string": "<"},
{"fullyEncoded": "%3D", "minimallyEncoded": "=", "string": "="},
{"fullyEncoded": "%3E", "minimallyEncoded": "%3E", "string": ">"},
{"fullyEncoded": "%3F", "minimallyEncoded": "%3F", "string": "?"},
{"fullyEncoded": "%40", "minimallyEncoded": "@", "string": "@"},
{"fullyEncoded": "%41", "minimallyEncoded": "A", "string": "A"},
{"fullyEncoded": "%42", "minimallyEncoded": "B", "string": "B"},
{"fullyEncoded": "%43", "minimallyEncoded": "C", "string": "C"},
{"fullyEncoded": "%44", "minimallyEncoded": "D", "string": "D"},
{"fullyEncoded": "%45", "minimallyEncoded": "E", "string": "E"},
{"fullyEncoded": "%46", "minimallyEncoded": "F", "string": "F"},
{"fullyEncoded": "%47", "minimallyEncoded": "G", "string": "G"},
{"fullyEncoded": "%48", "minimallyEncoded": "H", "string": "H"},
{"fullyEncoded": "%49", "minimallyEncoded": "I", "string": "I"},
{"fullyEncoded": "%4A", "minimallyEncoded": "J", "string": "J"},
{"fullyEncoded": "%4B", "minimallyEncoded": "K", "string": "K"},
{"fullyEncoded": "%4C", "minimallyEncoded": "L", "string": "L"},
{"fullyEncoded": "%4D", "minimallyEncoded": "M", "string": "M"},
{"fullyEncoded": "%4E", "minimallyEncoded": "N", "string": "N"},
{"fullyEncoded": "%4F", "minimallyEncoded": "O", "string": "O"},
{"fullyEncoded": "%50", "minimallyEncoded": "P", "string": "P"},
{"fullyEncoded": "%51", "minimallyEncoded": "Q", "string": "Q"},
{"fullyEncoded": "%52", "minimallyEncoded": "R", "string": "R"},
{"fullyEncoded": "%53", "minimallyEncoded": "S", "string": "S"},
{"fullyEncoded": "%54", "minimallyEncoded": "T", "string": "T"},
{"fullyEncoded": "%55", "minimallyEncoded": "U", "string": "U"},
{"fullyEncoded": "%56", "minimallyEncoded": "V", "string": "V"},
{"fullyEncoded": "%57", "minimallyEncoded": "W", "string": "W"},
{"fullyEncoded": "%58", "minimallyEncoded": "X", "string": "X"},
{"fullyEncoded": "%59", "minimallyEncoded": "Y", "string": "Y"},
{"fullyEncoded": "%5A", "minimallyEncoded": "Z", "string": "Z"},
{"fullyEncoded": "%5B", "minimallyEncoded": "%5B", "string": "["},
{"fullyEncoded": "%5C", "minimallyEncoded": "%5C", "string": "\\"},
{"fullyEncoded": "%5D", "minimallyEncoded": "%5D", "string": "]"},
{"fullyEncoded": "%5E", "minimallyEncoded": "%5E", "string": "^"},
{"fullyEncoded": "%5F", "minimallyEncoded": "_", "string": "_"},
{"fullyEncoded": "%60", "minimallyEncoded": "%60", "string": "` + "`" + `"},
{"fullyEncoded": "%61", "minimallyEncoded": "a", "string": "a"},
{"fullyEncoded": "%62", "minimallyEncoded": "b", "string": "b"},
{"fullyEncoded": "%63", "minimallyEncoded": "c", "string": "c"},
{"fullyEncoded": "%64", "minimallyEncoded": "d", "string": "d"},
{"fullyEncoded": "%65", "minimallyEncoded": "e", "string": "e"},
{"fullyEncoded": "%66", "minimallyEncoded": "f", "string": "f"},
{"fullyEncoded": "%67", "minimallyEncoded": "g", "string": "g"},
{"fullyEncoded": "%68", "minimallyEncoded": "h", "string": "h"},
{"fullyEncoded": "%69", "minimallyEncoded": "i", "string": "i"},
{"fullyEncoded": "%6A", "minimallyEncoded": "j", "string": "j"},
{"fullyEncoded": "%6B", "minimallyEncoded": "k", "string": "k"},
{"fullyEncoded": "%6C", "minimallyEncoded": "l", "string": "l"},
{"fullyEncoded": "%6D", "minimallyEncoded": "m", "string": "m"},
{"fullyEncoded": "%6E", "minimallyEncoded": "n", "string": "n"},
{"fullyEncoded": "%6F", "minimallyEncoded": "o", "string": "o"},
{"fullyEncoded": "%70", "minimallyEncoded": "p", "string": "p"},
{"fullyEncoded": "%71", "minimallyEncoded": "q", "string": "q"},
{"fullyEncoded": "%72", "minimallyEncoded": "r", "string": "r"},
{"fullyEncoded": "%73", "minimallyEncoded": "s", "string": "s"},
{"fullyEncoded": "%74", "minimallyEncoded": "t", "string": "t"},
{"fullyEncoded": "%75", "minimallyEncoded": "u", "string": "u"},
{"fullyEncoded": "%76", "minimallyEncoded": "v", "string": "v"},
{"fullyEncoded": "%77", "minimallyEncoded": "w", "string": "w"},
{"fullyEncoded": "%78", "minimallyEncoded": "x", "string": "x"},
{"fullyEncoded": "%79", "minimallyEncoded": "y", "string": "y"},
{"fullyEncoded": "%7A", "minimallyEncoded": "z", "string": "z"},
{"fullyEncoded": "%7B", "minimallyEncoded": "%7B", "string": "{"},
{"fullyEncoded": "%7C", "minimallyEncoded": "%7C", "string": "|"},
{"fullyEncoded": "%7D", "minimallyEncoded": "%7D", "string": "}"},
{"fullyEncoded": "%7E", "minimallyEncoded": "~", "string": "~"},
{"fullyEncoded": "%7F", "minimallyEncoded": "%7F", "string": "\u007f"},
{"fullyEncoded": "%E8%87%AA%E7%94%B1", "minimallyEncoded": "%E8%87%AA%E7%94%B1", "string": "\u81ea\u7531"},
{"fullyEncoded": "%F0%90%90%80", "minimallyEncoded": "%F0%90%90%80", "string": "\ud801\udc00"}
]`
type testCase struct {
Full string `json:"fullyEncoded"`
Min string `json:"minimallyEncoded"`
Raw string `json:"string"`
}
func TestEscapes(t *testing.T) {
dec := json.NewDecoder(strings.NewReader(testCases))
var tcs []testCase
if err := dec.Decode(&tcs); err != nil {
t.Fatal(err)
}
for _, tc := range tcs {
en := escape(tc.Raw)
if !(en == tc.Full || en == tc.Min) {
t.Errorf("encode %q: got %q, want %q or %q", tc.Raw, en, tc.Min, tc.Full)
}
m, err := unescape(tc.Min)
if err != nil {
t.Errorf("decode %q: %v", tc.Min, err)
}
if m != tc.Raw {
t.Errorf("decode %q: got %q, want %q", tc.Min, m, tc.Raw)
}
f, err := unescape(tc.Full)
if err != nil {
t.Errorf("decode %q: %v", tc.Full, err)
}
if f != tc.Raw {
t.Errorf("decode %q: got %q, want %q", tc.Full, f, tc.Raw)
}
}
}
func TestUploadDownloadFilenameEscaping(t *testing.T) {
filename := "file%foo.txt"
id := os.Getenv(apiID)
key := os.Getenv(apiKey)
if id == "" || key == "" {
t.Skipf("B2_ACCOUNT_ID or B2_SECRET_KEY unset; skipping integration tests")
}
ctx := context.Background()
// b2_authorize_account
b2, err := AuthorizeAccount(ctx, id, key, UserAgent("blazer-base-test"))
if err != nil {
t.Fatal(err)
}
// b2_create_bucket
bname := id + "-" + bucketName
bucket, err := b2.CreateBucket(ctx, bname, "", nil, nil)
if err != nil {
t.Fatal(err)
}
defer func() {
// b2_delete_bucket
if err := bucket.DeleteBucket(ctx); err != nil {
t.Error(err)
}
}()
// b2_get_upload_url
ue, err := bucket.GetUploadURL(ctx)
if err != nil {
t.Fatal(err)
}
// b2_upload_file
smallFile := io.LimitReader(zReader{}, 128)
hash := sha1.New()
buf := &bytes.Buffer{}
w := io.MultiWriter(hash, buf)
if _, err := io.Copy(w, smallFile); err != nil {
t.Error(err)
}
smallSHA1 := fmt.Sprintf("%x", hash.Sum(nil))
file, err := ue.UploadFile(ctx, buf, buf.Len(), filename, "application/octet-stream", smallSHA1, nil)
if err != nil {
t.Fatal(err)
}
defer func() {
// b2_delete_file_version
if err := file.DeleteFileVersion(ctx); err != nil {
t.Error(err)
}
}()
// b2_download_file_by_name
fr, err := bucket.DownloadFileByName(ctx, filename, 0, 0)
if err != nil {
t.Fatal(err)
}
lbuf := &bytes.Buffer{}
if _, err := io.Copy(lbuf, fr); err != nil {
t.Fatal(err)
}
}
vendor/github.com/kurin/blazer/base/strings_test.go
Generated Vendored
-52
View File
@@ -1,52 +0,0 @@
package base
import (
"fmt"
"testing"
)
func TestEncodeDecode(t *testing.T) {
// crashes identified by go-fuzz
origs := []string{
"&\x020000",
"&\x020000\x9c",
"&\x020\x9c0",
"&\x0230j",
"&\x02\x98000",
"&\x02\x983\xc8j00",
"00\x000",
"00\x0000",
"00\x0000000000000",
"\x11\x030",
}
for _, orig := range origs {
escaped := escape(orig)
unescaped, err := unescape(escaped)
if err != nil {
t.Errorf("%s: orig: %#v, escaped: %#v, unescaped: %#v\n", err.Error(), orig, escaped, unescaped)
continue
}
if unescaped != orig {
t.Errorf("expected: %#v, got: %#v", orig, unescaped)
}
}
}
// hook for go-fuzz: https://github.com/dvyukov/go-fuzz
func Fuzz(data []byte) int {
orig := string(data)
escaped := escape(orig)
unescaped, err := unescape(escaped)
if err != nil {
return 0
}
if unescaped != orig {
panic(fmt.Sprintf("unescaped: \"%#v\", != orig: \"%#v\"", unescaped, orig))
}
return 1
}
vendor/github.com/kurin/blazer/bin/b2keys/b2keys.go
Generated Vendored
-98
View File
@@ -1,98 +0,0 @@
// b2keys is a small utility for managing Backblaze B2 keys.
package main
import (
"context"
"flag"
"fmt"
"os"
"time"
"github.com/google/subcommands"
"github.com/kurin/blazer/b2"
)
const (
apiID = "B2_ACCOUNT_ID"
apiKey = "B2_SECRET_KEY"
)
func main() {
subcommands.Register(&create{}, "")
flag.Parse()
ctx := context.Background()
os.Exit(int(subcommands.Execute(ctx)))
}
type create struct {
d *time.Duration
bucket *string
pfx *string
}
func (c *create) Name() string { return "create" }
func (c *create) Synopsis() string { return "create a new application key" }
func (c *create) Usage() string {
return "b2keys create [-bucket bucket] [-duration duration] [-prefix pfx] name capability [capability ...]"
}
func (c *create) SetFlags(fs *flag.FlagSet) {
c.d = fs.Duration("duration", 0, "the lifetime of the new key")
c.bucket = fs.String("bucket", "", "limit the key to the given bucket")
c.pfx = fs.String("prefix", "", "limit the key to the objects starting with prefix")
}
func (c *create) Execute(ctx context.Context, f *flag.FlagSet, _ ...interface{}) subcommands.ExitStatus {
id := os.Getenv(apiID)
key := os.Getenv(apiKey)
if id == "" || key == "" {
fmt.Fprintf(os.Stderr, "both %s and %s must be set in the environment", apiID, apiKey)
return subcommands.ExitUsageError
}
args := f.Args()
if len(args) < 2 {
fmt.Fprintf(os.Stderr, "%s\n", c.Usage())
return subcommands.ExitUsageError
}
name := args[0]
caps := args[1:]
var opts []b2.KeyOption
if *c.d > 0 {
opts = append(opts, b2.Lifetime(*c.d))
}
if *c.pfx != "" {
opts = append(opts, b2.Prefix(*c.pfx))
}
opts = append(opts, b2.Capabilities(caps...))
client, err := b2.NewClient(ctx, id, key, b2.UserAgent("b2keys"))
if err != nil {
fmt.Fprintf(os.Stderr, "%v\n", err)
return subcommands.ExitFailure
}
var cr creater = client
if *c.bucket != "" {
bucket, err := client.Bucket(ctx, *c.bucket)
if err != nil {
fmt.Fprintf(os.Stderr, "%v\n", err)
return subcommands.ExitFailure
}
cr = bucket
}
b2key, err := cr.CreateKey(ctx, name, opts...)
if err != nil {
fmt.Fprintf(os.Stderr, "%v\n", err)
return subcommands.ExitFailure
}
fmt.Printf("key=%s, secret=%s\n", b2key.ID(), b2key.Secret())
return subcommands.ExitSuccess
}
type creater interface {
CreateKey(context.Context, string, ...b2.KeyOption) (*b2.Key, error)
}
vendor/github.com/kurin/blazer/bin/bonfire/bonfire.go
Generated Vendored
-43
View File
@@ -1,43 +0,0 @@
package main
import (
"context"
"fmt"
"net/http"
"github.com/kurin/blazer/bonfire"
"github.com/kurin/blazer/internal/pyre"
)
type superManager struct {
*bonfire.LocalBucket
bonfire.FS
}
func main() {
ctx := context.Background()
mux := http.NewServeMux()
fs := bonfire.FS("/tmp/b2")
bm := &bonfire.LocalBucket{Port: 8822}
if err := pyre.RegisterServerOnMux(ctx, &pyre.Server{
Account: bonfire.Localhost(8822),
LargeFile: fs,
Bucket: bm,
}, mux); err != nil {
fmt.Println(err)
return
}
sm := superManager{
LocalBucket: bm,
FS: fs,
}
pyre.RegisterLargeFileManagerOnMux(fs, mux)
pyre.RegisterSimpleFileManagerOnMux(fs, mux)
pyre.RegisterDownloadManagerOnMux(sm, mux)
fmt.Println("ok")
fmt.Println(http.ListenAndServe("localhost:8822", mux))
}
vendor/github.com/kurin/blazer/bonfire/bonfire.go
Generated Vendored
-257
View File
@@ -1,257 +0,0 @@
// Copyright 2018, Google
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
// Package bonfire implements the B2 service.
package bonfire
import (
"crypto/sha1"
"encoding/json"
"errors"
"fmt"
"io"
"os"
"path/filepath"
"sort"
"strconv"
"sync"
"github.com/kurin/blazer/internal/pyre"
)
type FS string
func (f FS) open(fp string) (io.WriteCloser, error) {
if err := os.MkdirAll(filepath.Dir(fp), 0755); err != nil {
return nil, err
}
return os.Create(fp)
}
func (f FS) PartWriter(id string, part int) (io.WriteCloser, error) {
fp := filepath.Join(string(f), id, fmt.Sprintf("%d", part))
return f.open(fp)
}
func (f FS) Writer(bucket, name, id string) (io.WriteCloser, error) {
fp := filepath.Join(string(f), bucket, name, id)
return f.open(fp)
}
func (f FS) Parts(id string) ([]string, error) {
dir := filepath.Join(string(f), id)
file, err := os.Open(dir)
if err != nil {
return nil, err
}
defer file.Close()
fs, err := file.Readdir(0)
if err != nil {
return nil, err
}
shas := make([]string, len(fs)-1)
for _, fi := range fs {
if fi.Name() == "info" {
continue
}
i, err := strconv.ParseInt(fi.Name(), 10, 32)
if err != nil {
return nil, err
}
p, err := os.Open(filepath.Join(dir, fi.Name()))
if err != nil {
return nil, err
}
sha := sha1.New()
if _, err := io.Copy(sha, p); err != nil {
p.Close()
return nil, err
}
p.Close()
shas[int(i)-1] = fmt.Sprintf("%x", sha.Sum(nil))
}
return shas, nil
}
type fi struct {
Name string
Bucket string
}
func (f FS) Start(bucketId, fileName, fileId string, bs []byte) error {
w, err := f.open(filepath.Join(string(f), fileId, "info"))
if err != nil {
return err
}
if err := json.NewEncoder(w).Encode(fi{Name: fileName, Bucket: bucketId}); err != nil {
w.Close()
return err
}
return w.Close()
}
func (f FS) Finish(fileId string) error {
r, err := os.Open(filepath.Join(string(f), fileId, "info"))
if err != nil {
return err
}
defer r.Close()
var info fi
if err := json.NewDecoder(r).Decode(&info); err != nil {
return err
}
shas, err := f.Parts(fileId) // oh my god this is terrible
if err != nil {
return err
}
w, err := f.open(filepath.Join(string(f), info.Bucket, info.Name, fileId))
if err != nil {
return err
}
for i := 1; i <= len(shas); i++ {
r, err := os.Open(filepath.Join(string(f), fileId, fmt.Sprintf("%d", i)))
if err != nil {
w.Close()
return err
}
if _, err := io.Copy(w, r); err != nil {
w.Close()
r.Close()
return err
}
r.Close()
}
if err := w.Close(); err != nil {
return err
}
return os.RemoveAll(filepath.Join(string(f), fileId))
}
func (f FS) ObjectByName(bucket, name string) (pyre.DownloadableObject, error) {
dir := filepath.Join(string(f), bucket, name)
d, err := os.Open(dir)
if err != nil {
return nil, err
}
defer d.Close()
fis, err := d.Readdir(0)
if err != nil {
return nil, err
}
sort.Slice(fis, func(i, j int) bool { return fis[i].ModTime().Before(fis[j].ModTime()) })
o, err := os.Open(filepath.Join(dir, fis[0].Name()))
if err != nil {
return nil, err
}
return do{
o: o,
size: fis[0].Size(),
}, nil
}
type do struct {
size int64
o *os.File
}
func (d do) Size() int64 { return d.size }
func (d do) Reader() io.ReaderAt { return d.o }
func (d do) Close() error { return d.o.Close() }
func (f FS) Get(fileId string) ([]byte, error) { return nil, nil }
type Localhost int
func (l Localhost) String() string { return fmt.Sprintf("http://localhost:%d", l) }
func (l Localhost) UploadHost(id string) (string, error) { return l.String(), nil }
func (Localhost) Authorize(string, string) (string, error) { return "ok", nil }
func (Localhost) CheckCreds(string, string) error { return nil }
func (l Localhost) APIRoot(string) string { return l.String() }
func (l Localhost) DownloadRoot(string) string { return l.String() }
func (Localhost) Sizes(string) (int32, int32) { return 1e5, 1 }
func (l Localhost) UploadPartHost(fileId string) (string, error) { return l.String(), nil }
type LocalBucket struct {
Port int
mux sync.Mutex
b map[string][]byte
nti map[string]string
}
func (lb *LocalBucket) AddBucket(id, name string, bs []byte) error {
lb.mux.Lock()
defer lb.mux.Unlock()
if lb.b == nil {
lb.b = make(map[string][]byte)
}
if lb.nti == nil {
lb.nti = make(map[string]string)
}
lb.b[id] = bs
lb.nti[name] = id
return nil
}
func (lb *LocalBucket) RemoveBucket(id string) error {
lb.mux.Lock()
defer lb.mux.Unlock()
if lb.b == nil {
lb.b = make(map[string][]byte)
}
delete(lb.b, id)
return nil
}
func (lb *LocalBucket) UpdateBucket(id string, rev int, bs []byte) error {
return errors.New("no")
}
func (lb *LocalBucket) ListBuckets(acct string) ([][]byte, error) {
lb.mux.Lock()
defer lb.mux.Unlock()
var bss [][]byte
for _, bs := range lb.b {
bss = append(bss, bs)
}
return bss, nil
}
func (lb *LocalBucket) GetBucket(id string) ([]byte, error) {
lb.mux.Lock()
defer lb.mux.Unlock()
bs, ok := lb.b[id]
if !ok {
return nil, errors.New("not found")
}
return bs, nil
}
func (lb *LocalBucket) GetBucketID(name string) (string, error) {
lb.mux.Lock()
defer lb.mux.Unlock()
id, ok := lb.nti[name]
if !ok {
return "", errors.New("not found")
}
return id, nil
}
vendor/github.com/kurin/blazer/examples/simple/simple.go
Generated Vendored
-134
View File
@@ -1,134 +0,0 @@
// Copyright 2017, Google
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
// This is a simple program that will copy named files into or out of B2.
//
// To copy a file into B2:
//
// B2_ACCOUNT_ID=foo B2_ACCOUNT_KEY=bar simple /path/to/file b2://bucket/path/to/dst
//
// To copy a file out:
//
// B2_ACCOUNT_ID=foo B2_ACCOUNT_KEY=bar simple b2://bucket/path/to/file /path/to/dst
package main
import (
"context"
"flag"
"fmt"
"io"
"net/url"
"os"
"strings"
"github.com/kurin/blazer/b2"
)
func main() {
flag.Parse()
b2id := os.Getenv("B2_ACCOUNT_ID")
b2key := os.Getenv("B2_ACCOUNT_KEY")
args := flag.Args()
if len(args) != 2 {
fmt.Printf("Usage:\n\nsimple [src] [dst]\n")
return
}
src, dst := args[0], args[1]
ctx := context.Background()
c, err := b2.NewClient(ctx, b2id, b2key)
if err != nil {
fmt.Println(err)
return
}
var r io.ReadCloser
var w io.WriteCloser
if strings.HasPrefix(src, "b2://") {
reader, err := b2Reader(ctx, c, src)
if err != nil {
fmt.Println(err)
return
}
r = reader
} else {
f, err := os.Open(src)
if err != nil {
fmt.Println(err)
return
}
r = f
}
// Readers do not need their errors checked on close. (Also it's a little
// silly to defer this in main(), but.)
defer r.Close()
if strings.HasPrefix(dst, "b2://") {
writer, err := b2Writer(ctx, c, dst)
if err != nil {
fmt.Println(err)
return
}
w = writer
} else {
f, err := os.Create(dst)
if err != nil {
fmt.Println(err)
return
}
w = f
}
// Copy and check error.
if _, err := io.Copy(w, r); err != nil {
fmt.Println(err)
return
}
// It is very important to check the error of the writer.
if err := w.Close(); err != nil {
fmt.Println(err)
}
}
func b2Reader(ctx context.Context, c *b2.Client, path string) (io.ReadCloser, error) {
o, err := b2Obj(ctx, c, path)
if err != nil {
return nil, err
}
return o.NewReader(ctx), nil
}
func b2Writer(ctx context.Context, c *b2.Client, path string) (io.WriteCloser, error) {
o, err := b2Obj(ctx, c, path)
if err != nil {
return nil, err
}
return o.NewWriter(ctx), nil
}
func b2Obj(ctx context.Context, c *b2.Client, path string) (*b2.Object, error) {
uri, err := url.Parse(path)
if err != nil {
return nil, err
}
bucket, err := c.Bucket(ctx, uri.Host)
if err != nil {
return nil, err
}
// B2 paths must not begin with /, so trim it here.
return bucket.Object(strings.TrimPrefix(uri.Path, "/")), nil
}
vendor/github.com/kurin/blazer/internal/b2assets/data/status.html
Generated Vendored
-36
View File
@@ -1,36 +0,0 @@
<!DOCTYPE html>
<html>
<head>
<title>b2 client status</title>
</head>
<body>
{{$methods := methods .}}
{{$durations := durations .}}
{{$table := table .}}
<h1>count by code</h1>
<table>
{{range $method := $methods}}
<tr>
<td>{{$method}}</td>
{{range $duration := $durations}}
<td>{{index $table $method $duration}}</td>
{{end}}
</tr>
{{end}}
</table>
<h1>uploads</h1>
{{range $name, $val := .Writers}}
<h2>{{ $name }}</h2>
{{range $id, $prog := $val.Progress}}
{{inc $id}} <progress value="{{$prog}}" max="1"></progress><br />
{{end}}
{{end}}
<h1>downloads</h1>
{{range $name, $val := .Readers}}
<h2>{{ $name }}</h2>
{{range $id, $prog := $val.Progress}}
{{inc $id}} <progress value="{{$prog}}" max="1"></progress><br />
{{end}}
{{end}}
</body>
</html>
vendor/github.com/kurin/blazer/internal/bin/cleanup/cleanup.go
Generated Vendored
-71
View File
@@ -1,71 +0,0 @@
package main
import (
"context"
"fmt"
"os"
"strings"
"sync"
"github.com/kurin/blazer/b2"
)
const (
apiID = "B2_ACCOUNT_ID"
apiKey = "B2_SECRET_KEY"
)
func main() {
id := os.Getenv(apiID)
key := os.Getenv(apiKey)
ctx := context.Background()
client, err := b2.NewClient(ctx, id, key)
if err != nil {
fmt.Println(err)
return
}
buckets, err := client.ListBuckets(ctx)
if err != nil {
fmt.Println(err)
return
}
var kill []string
for _, bucket := range buckets {
if strings.HasPrefix(bucket.Name(), fmt.Sprintf("%s-b2-tests-", id)) {
kill = append(kill, bucket.Name())
}
if bucket.Name() == fmt.Sprintf("%s-consistobucket", id) || bucket.Name() == fmt.Sprintf("%s-base-tests", id) {
kill = append(kill, bucket.Name())
}
}
var wg sync.WaitGroup
for _, name := range kill {
wg.Add(1)
go func(name string) {
defer wg.Done()
fmt.Println("removing", name)
if err := killBucket(ctx, client, name); err != nil {
fmt.Println(err)
}
}(name)
}
wg.Wait()
}
func killBucket(ctx context.Context, client *b2.Client, name string) error {
bucket, err := client.NewBucket(ctx, name, nil)
if b2.IsNotExist(err) {
return nil
}
if err != nil {
return err
}
defer bucket.Delete(ctx)
iter := bucket.List(ctx, b2.ListHidden())
for iter.Next() {
if err := iter.Object().Delete(ctx); err != nil {
fmt.Println(err)
}
}
return iter.Err()
}
vendor/github.com/kurin/blazer/internal/pyre/api.go
Generated Vendored
-354
View File
@@ -1,354 +0,0 @@
// Copyright 2018, Google
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package pyre
import (
"context"
"encoding/base64"
"errors"
"fmt"
"net"
"net/http"
"os"
"reflect"
"strings"
"github.com/golang/protobuf/proto"
"github.com/google/uuid"
"github.com/grpc-ecosystem/grpc-gateway/runtime"
"google.golang.org/grpc"
"google.golang.org/grpc/metadata"
pb "github.com/kurin/blazer/internal/pyre/proto"
)
type apiErr struct {
Status int `json:"status"`
Code string `json:"code"`
Message string `json:"message"`
}
func serveMuxOptions() []runtime.ServeMuxOption {
return []runtime.ServeMuxOption{
runtime.WithMarshalerOption("*", &runtime.JSONPb{}),
runtime.WithProtoErrorHandler(func(ctx context.Context, mux *runtime.ServeMux, m runtime.Marshaler, rw http.ResponseWriter, req *http.Request, err error) {
aErr := apiErr{
Status: 400,
Code: "uh oh",
Message: err.Error(),
}
rw.WriteHeader(aErr.Status)
if err := m.NewEncoder(rw).Encode(aErr); err != nil {
fmt.Fprintln(os.Stdout, err)
}
}),
}
}
func getAuth(ctx context.Context) (string, error) {
md, ok := metadata.FromIncomingContext(ctx)
if !ok {
return "", errors.New("no metadata")
}
data := md.Get("authorization")
if len(data) == 0 {
return "", nil
}
return data[0], nil
}
func RegisterServerOnMux(ctx context.Context, srv *Server, mux *http.ServeMux) error {
rmux := runtime.NewServeMux(serveMuxOptions()...)
l, err := net.Listen("tcp", "localhost:0")
if err != nil {
return err
}
gsrv := grpc.NewServer()
if err := pb.RegisterPyreServiceHandlerFromEndpoint(ctx, rmux, l.Addr().String(), []grpc.DialOption{grpc.WithInsecure()}); err != nil {
return err
}
pb.RegisterPyreServiceServer(gsrv, srv)
mux.Handle("/b2api/v1/", rmux)
go gsrv.Serve(l)
go func() {
<-ctx.Done()
gsrv.GracefulStop()
}()
return nil
}
type AccountManager interface {
Authorize(acct, key string) (string, error)
CheckCreds(token, api string) error
APIRoot(acct string) string
DownloadRoot(acct string) string
UploadPartHost(fileID string) (string, error)
UploadHost(id string) (string, error)
Sizes(acct string) (recommended, minimum int32)
}
type BucketManager interface {
AddBucket(id, name string, bs []byte) error
RemoveBucket(id string) error
UpdateBucket(id string, rev int, bs []byte) error
ListBuckets(acct string) ([][]byte, error)
GetBucket(id string) ([]byte, error)
}
type LargeFileOrganizer interface {
Start(bucketID, fileName, fileID string, bs []byte) error
Get(fileID string) ([]byte, error)
Parts(fileID string) ([]string, error)
Finish(fileID string) error
}
type Server struct {
Account AccountManager
Bucket BucketManager
LargeFile LargeFileOrganizer
List ListManager
}
func (s *Server) AuthorizeAccount(ctx context.Context, req *pb.AuthorizeAccountRequest) (*pb.AuthorizeAccountResponse, error) {
auth, err := getAuth(ctx)
if err != nil {
return nil, err
}
if !strings.HasPrefix(auth, "Basic ") {
return nil, errors.New("basic auth required")
}
auth = strings.TrimPrefix(auth, "Basic ")
bs, err := base64.StdEncoding.DecodeString(auth)
if err != nil {
return nil, err
}
split := strings.Split(string(bs), ":")
if len(split) != 2 {
return nil, errors.New("bad auth")
}
acct, key := split[0], split[1]
token, err := s.Account.Authorize(acct, key)
if err != nil {
return nil, err
}
rec, min := s.Account.Sizes(acct)
return &pb.AuthorizeAccountResponse{
AuthorizationToken: token,
ApiUrl: s.Account.APIRoot(acct),
DownloadUrl: s.Account.DownloadRoot(acct),
RecommendedPartSize: rec,
MinimumPartSize: rec,
AbsoluteMinimumPartSize: min,
}, nil
}
func (s *Server) ListBuckets(ctx context.Context, req *pb.ListBucketsRequest) (*pb.ListBucketsResponse, error) {
resp := &pb.ListBucketsResponse{}
buckets, err := s.Bucket.ListBuckets(req.AccountId)
if err != nil {
return nil, err
}
for _, bs := range buckets {
var bucket pb.Bucket
if err := proto.Unmarshal(bs, &bucket); err != nil {
return nil, err
}
resp.Buckets = append(resp.Buckets, &bucket)
}
return resp, nil
}
func (s *Server) CreateBucket(ctx context.Context, req *pb.Bucket) (*pb.Bucket, error) {
req.BucketId = uuid.New().String()
bs, err := proto.Marshal(req)
if err != nil {
return nil, err
}
if err := s.Bucket.AddBucket(req.BucketId, req.BucketName, bs); err != nil {
return nil, err
}
return req, nil
}
func (s *Server) DeleteBucket(ctx context.Context, req *pb.Bucket) (*pb.Bucket, error) {
bs, err := s.Bucket.GetBucket(req.BucketId)
if err != nil {
return nil, err
}
var bucket pb.Bucket
if err := proto.Unmarshal(bs, &bucket); err != nil {
return nil, err
}
if err := s.Bucket.RemoveBucket(req.BucketId); err != nil {
return nil, err
}
return &bucket, nil
}
func (s *Server) GetUploadUrl(ctx context.Context, req *pb.GetUploadUrlRequest) (*pb.GetUploadUrlResponse, error) {
host, err := s.Account.UploadHost(req.BucketId)
if err != nil {
return nil, err
}
return &pb.GetUploadUrlResponse{
UploadUrl: fmt.Sprintf("%s/b2api/v1/b2_upload_file/%s", host, req.BucketId),
BucketId: req.BucketId,
}, nil
}
func (s *Server) StartLargeFile(ctx context.Context, req *pb.StartLargeFileRequest) (*pb.StartLargeFileResponse, error) {
fileID := uuid.New().String()
resp := &pb.StartLargeFileResponse{
FileId: fileID,
FileName: req.FileName,
BucketId: req.BucketId,
ContentType: req.ContentType,
FileInfo: req.FileInfo,
}
bs, err := proto.Marshal(resp)
if err != nil {
return nil, err
}
if err := s.LargeFile.Start(req.BucketId, req.FileName, fileID, bs); err != nil {
return nil, err
}
return resp, nil
}
func (s *Server) GetUploadPartUrl(ctx context.Context, req *pb.GetUploadPartUrlRequest) (*pb.GetUploadPartUrlResponse, error) {
host, err := s.Account.UploadPartHost(req.FileId)
if err != nil {
return nil, err
}
return &pb.GetUploadPartUrlResponse{
UploadUrl: fmt.Sprintf("%s/b2api/v1/b2_upload_part/%s", host, req.FileId),
}, nil
}
func (s *Server) FinishLargeFile(ctx context.Context, req *pb.FinishLargeFileRequest) (*pb.FinishLargeFileResponse, error) {
parts, err := s.LargeFile.Parts(req.FileId)
if err != nil {
return nil, err
}
if !reflect.DeepEqual(parts, req.PartSha1Array) {
return nil, errors.New("sha1 array mismatch")
}
if err := s.LargeFile.Finish(req.FileId); err != nil {
return nil, err
}
return &pb.FinishLargeFileResponse{}, nil
}
func (s *Server) ListFileVersions(ctx context.Context, req *pb.ListFileVersionsRequest) (*pb.ListFileVersionsResponse, error) {
return nil, nil
}
type objTuple struct {
name, version string
}
type ListManager interface {
// NextN returns the next n objects, sorted by lexicographical order by name,
// beginning at and including, if it exists, fileName. If withPrefix is not
// empty, it only returns names that begin with that prefix. If skipPrefix
// is not empty, then the no files with that prefix are returned. If the two
// conflict, skipPrefix wins (i.e., do not return the entry).
//
// If fewer than n entries are returned, this signifies that no more names
// exist that meet these criteria.
NextN(bucketID, fileName, withPrefix, skipPrefix string, n int) ([]VersionedObject, error)
}
type VersionedObject interface {
Name() string
NextNVersions(begin string, n int) ([]string, error)
}
func getDirNames(lm ListManager, bucket, name, prefix, delim string, n int) ([]string, error) {
var sfx string
var out []string
for n > 0 {
vo, err := lm.NextN(bucket, name, prefix, sfx, 1)
if err != nil {
return nil, err
}
if len(vo) == 0 {
return out, nil
}
v := vo[0]
name = v.Name()
suffix := name[len(prefix):]
i := strings.Index(suffix, delim)
if i < 0 {
sfx = ""
out = append(out, name)
name += "\000"
n--
continue
}
sfx = v.Name()[:len(prefix)+i+1]
out = append(out, sfx)
n--
}
return out, nil
}
//func getNextObjects(lm ListManager, bucket, name, prefix, delimiter string, n int) ([]VersionedObject, error) {
// if delimiter == "" {
// return lm.NextN(bucket, name, prefix, "", n)
// }
// afterPfx := strings.TrimPrefix(name, prefix)
// i := strings.Index(afterPfx, delimiter)
// if i == 0 {
//
// }
// if i < 0 {
// return lm.NextN(bucket, name, prefix, "", n)
// }
// skipPfx := name[:len(prefix)+i]
// // TO
//}
//
//func listFileVersions(lm ListManager, bucket, name, version, prefix, delimiter string, n int) ([]objTuple, error) {
// var tups []objTuple
// var got int
// for {
// objs, err := getNextObjects(bucket, name, prefix, delimiter, n-got)
// if err != nil {
// return nil, err
// }
// if len(objs) == 0 {
// break
// }
// for _, o := range objs {
// var begin string
// if len(tups) == 0 {
// begin = version
// }
// vers, err := lm.NextNVersions(begin, n-got)
// if err != nil {
// return nil, err
// }
// got += len(vers)
// for _, ver := range vers {
// tups = append(tups, objTuple{name: o.Name(), version: ver})
// }
// if got >= n {
// return tups[:n], nil
// }
// }
// }
// return tups, nil
//}
vendor/github.com/kurin/blazer/internal/pyre/api_test.go
Generated Vendored
-137
View File
@@ -1,137 +0,0 @@
// Copyright 2018, Google
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package pyre
import (
"reflect"
"sort"
"strings"
"sync"
"testing"
)
type testVersionedObject struct {
name string
versions []string
}
func (t testVersionedObject) Name() string { return t.name }
func (t testVersionedObject) NextNVersions(b string, n int) ([]string, error) {
var out []string
var seen bool
if b == "" {
seen = true
}
for _, v := range t.versions {
if b == v {
seen = true
}
if !seen {
continue
}
if len(out) >= n {
return out, nil
}
out = append(out, v)
}
return out, nil
}
type testListManager struct {
objs map[string][]string
m sync.Mutex
}
func (t *testListManager) NextN(b, fn, pfx, spfx string, n int) ([]VersionedObject, error) {
t.m.Lock()
defer t.m.Unlock()
var out []VersionedObject
var keys []string
for k := range t.objs {
keys = append(keys, k)
}
sort.Strings(keys)
for _, k := range keys {
if k < fn {
continue
}
if !strings.HasPrefix(k, pfx) {
continue
}
if spfx != "" && strings.HasPrefix(k, spfx) {
continue
}
out = append(out, testVersionedObject{name: k, versions: t.objs[k]})
n--
if n <= 0 {
return out, nil
}
}
return out, nil
}
func TestGetDirNames(t *testing.T) {
table := []struct {
lm ListManager
name string
pfx string
delim string
num int
want []string
}{
{
lm: &testListManager{
objs: map[string][]string{
"/usr/local/etc/foo/bar": {"a"},
"/usr/local/etc/foo/baz": {"a"},
"/usr/local/etc/foo": {"a"},
"/usr/local/etc/fool": {"a"},
},
},
num: 2,
pfx: "/usr/local/etc/",
delim: "/",
want: []string{"/usr/local/etc/foo", "/usr/local/etc/foo/"},
},
{
lm: &testListManager{
objs: map[string][]string{
"/usr/local/etc/foo/bar": {"a"},
"/usr/local/etc/foo/baz": {"a"},
"/usr/local/etc/foo": {"a"},
"/usr/local/etc/fool": {"a"},
"/usr/local/etc/bar": {"a"},
},
},
num: 4,
pfx: "/usr/local/etc/",
delim: "/",
want: []string{"/usr/local/etc/bar", "/usr/local/etc/foo", "/usr/local/etc/foo/", "/usr/local/etc/fool"},
},
}
for _, e := range table {
got, err := getDirNames(e.lm, "", e.name, e.pfx, e.delim, e.num)
if err != nil {
t.Error(err)
continue
}
if !reflect.DeepEqual(got, e.want) {
t.Errorf("getDirNames(%v, %q, %q, %q, %d): got %v, want %v", e.lm, e.name, e.pfx, e.delim, e.num, got, e.want)
}
}
}
vendor/github.com/kurin/blazer/internal/pyre/download.go
Generated Vendored
-136
View File
@@ -1,136 +0,0 @@
// Copyright 2018, Google
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package pyre
import (
"fmt"
"io"
"net/http"
"strconv"
"strings"
)
type DownloadableObject interface {
Size() int64
Reader() io.ReaderAt
io.Closer
}
type DownloadManager interface {
ObjectByName(bucketID, name string) (DownloadableObject, error)
GetBucketID(bucket string) (string, error)
GetBucket(id string) ([]byte, error)
}
type downloadServer struct {
dm DownloadManager
}
type downloadRequest struct {
off, n int64
}
func parseDownloadHeaders(r *http.Request) (*downloadRequest, error) {
rang := r.Header.Get("Range")
if rang == "" {
return &downloadRequest{}, nil
}
if !strings.HasPrefix(rang, "bytes=") {
return nil, fmt.Errorf("unknown range format: %q", rang)
}
rang = strings.TrimPrefix(rang, "bytes=")
if !strings.Contains(rang, "-") {
return nil, fmt.Errorf("unknown range format: %q", rang)
}
parts := strings.Split(rang, "-")
off, err := strconv.ParseInt(parts[0], 10, 64)
if err != nil {
return nil, err
}
end, err := strconv.ParseInt(parts[1], 10, 64)
if err != nil {
return nil, err
}
return &downloadRequest{
off: off,
n: (end + 1) - off,
}, nil
}
func (fs *downloadServer) serveWholeObject(rw http.ResponseWriter, obj DownloadableObject) {
rw.Header().Set("Content-Length", fmt.Sprintf("%d", obj.Size()))
sr := io.NewSectionReader(obj.Reader(), 0, obj.Size())
if _, err := io.Copy(rw, sr); err != nil {
http.Error(rw, err.Error(), 503)
fmt.Println("no reader", err)
}
}
func (fs *downloadServer) servePartialObject(rw http.ResponseWriter, obj DownloadableObject, off, len int64) {
if off >= obj.Size() {
http.Error(rw, "hell naw", 416)
fmt.Printf("range not good (%d-%d for %d)\n", off, len, obj.Size())
return
}
if off+len > obj.Size() {
len = obj.Size() - off
}
sr := io.NewSectionReader(obj.Reader(), off, len)
rw.Header().Set("Content-Length", fmt.Sprintf("%d", len))
rw.WriteHeader(206) // this goes after headers are set
if _, err := io.Copy(rw, sr); err != nil {
fmt.Println("bad read:", err)
}
}
func (fs *downloadServer) ServeHTTP(rw http.ResponseWriter, r *http.Request) {
req, err := parseDownloadHeaders(r)
if err != nil {
http.Error(rw, err.Error(), 503)
fmt.Println("weird header")
return
}
path := strings.TrimPrefix(r.URL.Path, "/")
parts := strings.Split(path, "/")
if len(parts) < 3 {
http.Error(rw, err.Error(), 404)
fmt.Println("weird file")
return
}
bucket := parts[1]
bid, err := fs.dm.GetBucketID(bucket)
if err != nil {
http.Error(rw, err.Error(), 503)
fmt.Println("no bucket:", err)
return
}
file := strings.Join(parts[2:], "/")
obj, err := fs.dm.ObjectByName(bid, file)
if err != nil {
http.Error(rw, err.Error(), 503)
fmt.Println("no reader", err)
return
}
defer obj.Close()
if req.off == 0 && req.n == 0 {
fs.serveWholeObject(rw, obj)
return
}
fs.servePartialObject(rw, obj, req.off, req.n)
}
func RegisterDownloadManagerOnMux(d DownloadManager, mux *http.ServeMux) {
mux.Handle("/file/", &downloadServer{dm: d})
}
vendor/github.com/kurin/blazer/internal/pyre/large.go
Generated Vendored
-91
View File
@@ -1,91 +0,0 @@
// Copyright 2018, Google
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package pyre
import (
"encoding/json"
"fmt"
"io"
"net/http"
"strconv"
"strings"
)
const uploadFilePartPrefix = "/b2api/v1/b2_upload_part/"
type LargeFileManager interface {
PartWriter(id string, part int) (io.WriteCloser, error)
}
type largeFileServer struct {
fm LargeFileManager
}
type uploadPartRequest struct {
ID string `json:"fileId"`
Part int `json:"partNumber"`
Size int64 `json:"contentLength"`
Hash string `json:"contentSha1"`
}
func parseUploadPartHeaders(r *http.Request) (uploadPartRequest, error) {
var ur uploadPartRequest
ur.Hash = r.Header.Get("X-Bz-Content-Sha1")
part, err := strconv.ParseInt(r.Header.Get("X-Bz-Part-Number"), 10, 64)
if err != nil {
return ur, err
}
ur.Part = int(part)
size, err := strconv.ParseInt(r.Header.Get("Content-Length"), 10, 64)
if err != nil {
return ur, err
}
ur.Size = size
ur.ID = strings.TrimPrefix(r.URL.Path, uploadFilePartPrefix)
return ur, nil
}
func (fs *largeFileServer) ServeHTTP(rw http.ResponseWriter, r *http.Request) {
req, err := parseUploadPartHeaders(r)
if err != nil {
http.Error(rw, err.Error(), 500)
fmt.Println("oh no")
return
}
w, err := fs.fm.PartWriter(req.ID, req.Part)
if err != nil {
http.Error(rw, err.Error(), 500)
fmt.Println("oh no")
return
}
if _, err := io.Copy(w, io.LimitReader(r.Body, req.Size)); err != nil {
w.Close()
http.Error(rw, err.Error(), 500)
fmt.Println("oh no")
return
}
if err := w.Close(); err != nil {
http.Error(rw, err.Error(), 500)
fmt.Println("oh no")
return
}
if err := json.NewEncoder(rw).Encode(req); err != nil {
fmt.Println("oh no")
}
}
func RegisterLargeFileManagerOnMux(f LargeFileManager, mux *http.ServeMux) {
mux.Handle(uploadFilePartPrefix, &largeFileServer{fm: f})
}
vendor/github.com/kurin/blazer/internal/pyre/proto/pyre.pb.go
Generated Vendored
-1896
View File
File diff suppressed because it is too large. Load diff
vendor/github.com/kurin/blazer/internal/pyre/proto/pyre.pb.gw.go
Generated Vendored
-484
View File
@@ -1,484 +0,0 @@
// Code generated by protoc-gen-grpc-gateway. DO NOT EDIT.
// source: proto/pyre.proto
/*
Package pyre_proto is a reverse proxy.
It translates gRPC into RESTful JSON APIs.
*/
package pyre_proto
import (
"io"
"net/http"
"github.com/golang/protobuf/proto"
"github.com/grpc-ecosystem/grpc-gateway/runtime"
"github.com/grpc-ecosystem/grpc-gateway/utilities"
"golang.org/x/net/context"
"google.golang.org/grpc"
"google.golang.org/grpc/codes"
"google.golang.org/grpc/grpclog"
"google.golang.org/grpc/status"
)
var _ codes.Code
var _ io.Reader
var _ status.Status
var _ = runtime.String
var _ = utilities.NewDoubleArray
func request_PyreService_AuthorizeAccount_0(ctx context.Context, marshaler runtime.Marshaler, client PyreServiceClient, req *http.Request, pathParams map[string]string) (proto.Message, runtime.ServerMetadata, error) {
var protoReq AuthorizeAccountRequest
var metadata runtime.ServerMetadata
msg, err := client.AuthorizeAccount(ctx, &protoReq, grpc.Header(&metadata.HeaderMD), grpc.Trailer(&metadata.TrailerMD))
return msg, metadata, err
}
func request_PyreService_ListBuckets_0(ctx context.Context, marshaler runtime.Marshaler, client PyreServiceClient, req *http.Request, pathParams map[string]string) (proto.Message, runtime.ServerMetadata, error) {
var protoReq ListBucketsRequest
var metadata runtime.ServerMetadata
if err := marshaler.NewDecoder(req.Body).Decode(&protoReq); err != nil && err != io.EOF {
return nil, metadata, status.Errorf(codes.InvalidArgument, "%v", err)
}
msg, err := client.ListBuckets(ctx, &protoReq, grpc.Header(&metadata.HeaderMD), grpc.Trailer(&metadata.TrailerMD))
return msg, metadata, err
}
func request_PyreService_CreateBucket_0(ctx context.Context, marshaler runtime.Marshaler, client PyreServiceClient, req *http.Request, pathParams map[string]string) (proto.Message, runtime.ServerMetadata, error) {
var protoReq Bucket
var metadata runtime.ServerMetadata
if err := marshaler.NewDecoder(req.Body).Decode(&protoReq); err != nil && err != io.EOF {
return nil, metadata, status.Errorf(codes.InvalidArgument, "%v", err)
}
msg, err := client.CreateBucket(ctx, &protoReq, grpc.Header(&metadata.HeaderMD), grpc.Trailer(&metadata.TrailerMD))
return msg, metadata, err
}
func request_PyreService_DeleteBucket_0(ctx context.Context, marshaler runtime.Marshaler, client PyreServiceClient, req *http.Request, pathParams map[string]string) (proto.Message, runtime.ServerMetadata, error) {
var protoReq Bucket
var metadata runtime.ServerMetadata
if err := marshaler.NewDecoder(req.Body).Decode(&protoReq); err != nil && err != io.EOF {
return nil, metadata, status.Errorf(codes.InvalidArgument, "%v", err)
}
msg, err := client.DeleteBucket(ctx, &protoReq, grpc.Header(&metadata.HeaderMD), grpc.Trailer(&metadata.TrailerMD))
return msg, metadata, err
}
func request_PyreService_GetUploadUrl_0(ctx context.Context, marshaler runtime.Marshaler, client PyreServiceClient, req *http.Request, pathParams map[string]string) (proto.Message, runtime.ServerMetadata, error) {
var protoReq GetUploadUrlRequest
var metadata runtime.ServerMetadata
if err := marshaler.NewDecoder(req.Body).Decode(&protoReq); err != nil && err != io.EOF {
return nil, metadata, status.Errorf(codes.InvalidArgument, "%v", err)
}
msg, err := client.GetUploadUrl(ctx, &protoReq, grpc.Header(&metadata.HeaderMD), grpc.Trailer(&metadata.TrailerMD))
return msg, metadata, err
}
func request_PyreService_StartLargeFile_0(ctx context.Context, marshaler runtime.Marshaler, client PyreServiceClient, req *http.Request, pathParams map[string]string) (proto.Message, runtime.ServerMetadata, error) {
var protoReq StartLargeFileRequest
var metadata runtime.ServerMetadata
if err := marshaler.NewDecoder(req.Body).Decode(&protoReq); err != nil && err != io.EOF {
return nil, metadata, status.Errorf(codes.InvalidArgument, "%v", err)
}
msg, err := client.StartLargeFile(ctx, &protoReq, grpc.Header(&metadata.HeaderMD), grpc.Trailer(&metadata.TrailerMD))
return msg, metadata, err
}
func request_PyreService_GetUploadPartUrl_0(ctx context.Context, marshaler runtime.Marshaler, client PyreServiceClient, req *http.Request, pathParams map[string]string) (proto.Message, runtime.ServerMetadata, error) {
var protoReq GetUploadPartUrlRequest
var metadata runtime.ServerMetadata
if err := marshaler.NewDecoder(req.Body).Decode(&protoReq); err != nil && err != io.EOF {
return nil, metadata, status.Errorf(codes.InvalidArgument, "%v", err)
}
msg, err := client.GetUploadPartUrl(ctx, &protoReq, grpc.Header(&metadata.HeaderMD), grpc.Trailer(&metadata.TrailerMD))
return msg, metadata, err
}
func request_PyreService_FinishLargeFile_0(ctx context.Context, marshaler runtime.Marshaler, client PyreServiceClient, req *http.Request, pathParams map[string]string) (proto.Message, runtime.ServerMetadata, error) {
var protoReq FinishLargeFileRequest
var metadata runtime.ServerMetadata
if err := marshaler.NewDecoder(req.Body).Decode(&protoReq); err != nil && err != io.EOF {
return nil, metadata, status.Errorf(codes.InvalidArgument, "%v", err)
}
msg, err := client.FinishLargeFile(ctx, &protoReq, grpc.Header(&metadata.HeaderMD), grpc.Trailer(&metadata.TrailerMD))
return msg, metadata, err
}
func request_PyreService_ListFileVersions_0(ctx context.Context, marshaler runtime.Marshaler, client PyreServiceClient, req *http.Request, pathParams map[string]string) (proto.Message, runtime.ServerMetadata, error) {
var protoReq ListFileVersionsRequest
var metadata runtime.ServerMetadata
if err := marshaler.NewDecoder(req.Body).Decode(&protoReq); err != nil && err != io.EOF {
return nil, metadata, status.Errorf(codes.InvalidArgument, "%v", err)
}
msg, err := client.ListFileVersions(ctx, &protoReq, grpc.Header(&metadata.HeaderMD), grpc.Trailer(&metadata.TrailerMD))
return msg, metadata, err
}
// RegisterPyreServiceHandlerFromEndpoint is same as RegisterPyreServiceHandler but
// automatically dials to "endpoint" and closes the connection when "ctx" gets done.
func RegisterPyreServiceHandlerFromEndpoint(ctx context.Context, mux *runtime.ServeMux, endpoint string, opts []grpc.DialOption) (err error) {
conn, err := grpc.Dial(endpoint, opts...)
if err != nil {
return err
}
defer func() {
if err != nil {
if cerr := conn.Close(); cerr != nil {
grpclog.Infof("Failed to close conn to %s: %v", endpoint, cerr)
}
return
}
go func() {
<-ctx.Done()
if cerr := conn.Close(); cerr != nil {
grpclog.Infof("Failed to close conn to %s: %v", endpoint, cerr)
}
}()
}()
return RegisterPyreServiceHandler(ctx, mux, conn)
}
// RegisterPyreServiceHandler registers the http handlers for service PyreService to "mux".
// The handlers forward requests to the grpc endpoint over "conn".
func RegisterPyreServiceHandler(ctx context.Context, mux *runtime.ServeMux, conn *grpc.ClientConn) error {
return RegisterPyreServiceHandlerClient(ctx, mux, NewPyreServiceClient(conn))
}
// RegisterPyreServiceHandlerClient registers the http handlers for service PyreService
// to "mux". The handlers forward requests to the grpc endpoint over the given implementation of "PyreServiceClient".
// Note: the gRPC framework executes interceptors within the gRPC handler. If the passed in "PyreServiceClient"
// doesn't go through the normal gRPC flow (creating a gRPC client etc.) then it will be up to the passed in
// "PyreServiceClient" to call the correct interceptors.
func RegisterPyreServiceHandlerClient(ctx context.Context, mux *runtime.ServeMux, client PyreServiceClient) error {
mux.Handle("GET", pattern_PyreService_AuthorizeAccount_0, func(w http.ResponseWriter, req *http.Request, pathParams map[string]string) {
ctx, cancel := context.WithCancel(req.Context())
defer cancel()
if cn, ok := w.(http.CloseNotifier); ok {
go func(done <-chan struct{}, closed <-chan bool) {
select {
case <-done:
case <-closed:
cancel()
}
}(ctx.Done(), cn.CloseNotify())
}
inboundMarshaler, outboundMarshaler := runtime.MarshalerForRequest(mux, req)
rctx, err := runtime.AnnotateContext(ctx, mux, req)
if err != nil {
runtime.HTTPError(ctx, mux, outboundMarshaler, w, req, err)
return
}
resp, md, err := request_PyreService_AuthorizeAccount_0(rctx, inboundMarshaler, client, req, pathParams)
ctx = runtime.NewServerMetadataContext(ctx, md)
if err != nil {
runtime.HTTPError(ctx, mux, outboundMarshaler, w, req, err)
return
}
forward_PyreService_AuthorizeAccount_0(ctx, mux, outboundMarshaler, w, req, resp, mux.GetForwardResponseOptions()...)
})
mux.Handle("POST", pattern_PyreService_ListBuckets_0, func(w http.ResponseWriter, req *http.Request, pathParams map[string]string) {
ctx, cancel := context.WithCancel(req.Context())
defer cancel()
if cn, ok := w.(http.CloseNotifier); ok {
go func(done <-chan struct{}, closed <-chan bool) {
select {
case <-done:
case <-closed:
cancel()
}
}(ctx.Done(), cn.CloseNotify())
}
inboundMarshaler, outboundMarshaler := runtime.MarshalerForRequest(mux, req)
rctx, err := runtime.AnnotateContext(ctx, mux, req)
if err != nil {
runtime.HTTPError(ctx, mux, outboundMarshaler, w, req, err)
return
}
resp, md, err := request_PyreService_ListBuckets_0(rctx, inboundMarshaler, client, req, pathParams)
ctx = runtime.NewServerMetadataContext(ctx, md)
if err != nil {
runtime.HTTPError(ctx, mux, outboundMarshaler, w, req, err)
return
}
forward_PyreService_ListBuckets_0(ctx, mux, outboundMarshaler, w, req, resp, mux.GetForwardResponseOptions()...)
})
mux.Handle("POST", pattern_PyreService_CreateBucket_0, func(w http.ResponseWriter, req *http.Request, pathParams map[string]string) {
ctx, cancel := context.WithCancel(req.Context())
defer cancel()
if cn, ok := w.(http.CloseNotifier); ok {
go func(done <-chan struct{}, closed <-chan bool) {
select {
case <-done:
case <-closed:
cancel()
}
}(ctx.Done(), cn.CloseNotify())
}
inboundMarshaler, outboundMarshaler := runtime.MarshalerForRequest(mux, req)
rctx, err := runtime.AnnotateContext(ctx, mux, req)
if err != nil {
runtime.HTTPError(ctx, mux, outboundMarshaler, w, req, err)
return
}
resp, md, err := request_PyreService_CreateBucket_0(rctx, inboundMarshaler, client, req, pathParams)
ctx = runtime.NewServerMetadataContext(ctx, md)
if err != nil {
runtime.HTTPError(ctx, mux, outboundMarshaler, w, req, err)
return
}
forward_PyreService_CreateBucket_0(ctx, mux, outboundMarshaler, w, req, resp, mux.GetForwardResponseOptions()...)
})
mux.Handle("POST", pattern_PyreService_DeleteBucket_0, func(w http.ResponseWriter, req *http.Request, pathParams map[string]string) {
ctx, cancel := context.WithCancel(req.Context())
defer cancel()
if cn, ok := w.(http.CloseNotifier); ok {
go func(done <-chan struct{}, closed <-chan bool) {
select {
case <-done:
case <-closed:
cancel()
}
}(ctx.Done(), cn.CloseNotify())
}
inboundMarshaler, outboundMarshaler := runtime.MarshalerForRequest(mux, req)
rctx, err := runtime.AnnotateContext(ctx, mux, req)
if err != nil {
runtime.HTTPError(ctx, mux, outboundMarshaler, w, req, err)
return
}
resp, md, err := request_PyreService_DeleteBucket_0(rctx, inboundMarshaler, client, req, pathParams)
ctx = runtime.NewServerMetadataContext(ctx, md)
if err != nil {
runtime.HTTPError(ctx, mux, outboundMarshaler, w, req, err)
return
}
forward_PyreService_DeleteBucket_0(ctx, mux, outboundMarshaler, w, req, resp, mux.GetForwardResponseOptions()...)
})
mux.Handle("POST", pattern_PyreService_GetUploadUrl_0, func(w http.ResponseWriter, req *http.Request, pathParams map[string]string) {
ctx, cancel := context.WithCancel(req.Context())
defer cancel()
if cn, ok := w.(http.CloseNotifier); ok {
go func(done <-chan struct{}, closed <-chan bool) {
select {
case <-done:
case <-closed:
cancel()
}
}(ctx.Done(), cn.CloseNotify())
}
inboundMarshaler, outboundMarshaler := runtime.MarshalerForRequest(mux, req)
rctx, err := runtime.AnnotateContext(ctx, mux, req)
if err != nil {
runtime.HTTPError(ctx, mux, outboundMarshaler, w, req, err)
return
}
resp, md, err := request_PyreService_GetUploadUrl_0(rctx, inboundMarshaler, client, req, pathParams)
ctx = runtime.NewServerMetadataContext(ctx, md)
if err != nil {
runtime.HTTPError(ctx, mux, outboundMarshaler, w, req, err)
return
}
forward_PyreService_GetUploadUrl_0(ctx, mux, outboundMarshaler, w, req, resp, mux.GetForwardResponseOptions()...)
})
mux.Handle("POST", pattern_PyreService_StartLargeFile_0, func(w http.ResponseWriter, req *http.Request, pathParams map[string]string) {
ctx, cancel := context.WithCancel(req.Context())
defer cancel()
if cn, ok := w.(http.CloseNotifier); ok {
go func(done <-chan struct{}, closed <-chan bool) {
select {
case <-done:
case <-closed:
cancel()
}
}(ctx.Done(), cn.CloseNotify())
}
inboundMarshaler, outboundMarshaler := runtime.MarshalerForRequest(mux, req)
rctx, err := runtime.AnnotateContext(ctx, mux, req)
if err != nil {
runtime.HTTPError(ctx, mux, outboundMarshaler, w, req, err)
return
}
resp, md, err := request_PyreService_StartLargeFile_0(rctx, inboundMarshaler, client, req, pathParams)
ctx = runtime.NewServerMetadataContext(ctx, md)
if err != nil {
runtime.HTTPError(ctx, mux, outboundMarshaler, w, req, err)
return
}
forward_PyreService_StartLargeFile_0(ctx, mux, outboundMarshaler, w, req, resp, mux.GetForwardResponseOptions()...)
})
mux.Handle("POST", pattern_PyreService_GetUploadPartUrl_0, func(w http.ResponseWriter, req *http.Request, pathParams map[string]string) {
ctx, cancel := context.WithCancel(req.Context())
defer cancel()
if cn, ok := w.(http.CloseNotifier); ok {
go func(done <-chan struct{}, closed <-chan bool) {
select {
case <-done:
case <-closed:
cancel()
}
}(ctx.Done(), cn.CloseNotify())
}
inboundMarshaler, outboundMarshaler := runtime.MarshalerForRequest(mux, req)
rctx, err := runtime.AnnotateContext(ctx, mux, req)
if err != nil {
runtime.HTTPError(ctx, mux, outboundMarshaler, w, req, err)
return
}
resp, md, err := request_PyreService_GetUploadPartUrl_0(rctx, inboundMarshaler, client, req, pathParams)
ctx = runtime.NewServerMetadataContext(ctx, md)
if err != nil {
runtime.HTTPError(ctx, mux, outboundMarshaler, w, req, err)
return
}
forward_PyreService_GetUploadPartUrl_0(ctx, mux, outboundMarshaler, w, req, resp, mux.GetForwardResponseOptions()...)
})
mux.Handle("POST", pattern_PyreService_FinishLargeFile_0, func(w http.ResponseWriter, req *http.Request, pathParams map[string]string) {
ctx, cancel := context.WithCancel(req.Context())
defer cancel()
if cn, ok := w.(http.CloseNotifier); ok {
go func(done <-chan struct{}, closed <-chan bool) {
select {
case <-done:
case <-closed:
cancel()
}
}(ctx.Done(), cn.CloseNotify())
}
inboundMarshaler, outboundMarshaler := runtime.MarshalerForRequest(mux, req)
rctx, err := runtime.AnnotateContext(ctx, mux, req)
if err != nil {
runtime.HTTPError(ctx, mux, outboundMarshaler, w, req, err)
return
}
resp, md, err := request_PyreService_FinishLargeFile_0(rctx, inboundMarshaler, client, req, pathParams)
ctx = runtime.NewServerMetadataContext(ctx, md)
if err != nil {
runtime.HTTPError(ctx, mux, outboundMarshaler, w, req, err)
return
}
forward_PyreService_FinishLargeFile_0(ctx, mux, outboundMarshaler, w, req, resp, mux.GetForwardResponseOptions()...)
})
mux.Handle("POST", pattern_PyreService_ListFileVersions_0, func(w http.ResponseWriter, req *http.Request, pathParams map[string]string) {
ctx, cancel := context.WithCancel(req.Context())
defer cancel()
if cn, ok := w.(http.CloseNotifier); ok {
go func(done <-chan struct{}, closed <-chan bool) {
select {
case <-done:
case <-closed:
cancel()
}
}(ctx.Done(), cn.CloseNotify())
}
inboundMarshaler, outboundMarshaler := runtime.MarshalerForRequest(mux, req)
rctx, err := runtime.AnnotateContext(ctx, mux, req)
if err != nil {
runtime.HTTPError(ctx, mux, outboundMarshaler, w, req, err)
return
}
resp, md, err := request_PyreService_ListFileVersions_0(rctx, inboundMarshaler, client, req, pathParams)
ctx = runtime.NewServerMetadataContext(ctx, md)
if err != nil {
runtime.HTTPError(ctx, mux, outboundMarshaler, w, req, err)
return
}
forward_PyreService_ListFileVersions_0(ctx, mux, outboundMarshaler, w, req, resp, mux.GetForwardResponseOptions()...)
})
return nil
}
var (
pattern_PyreService_AuthorizeAccount_0 = runtime.MustPattern(runtime.NewPattern(1, []int{2, 0, 2, 1, 2, 2}, []string{"b2api", "v1", "b2_authorize_account"}, ""))
pattern_PyreService_ListBuckets_0 = runtime.MustPattern(runtime.NewPattern(1, []int{2, 0, 2, 1, 2, 2}, []string{"b2api", "v1", "b2_list_buckets"}, ""))
pattern_PyreService_CreateBucket_0 = runtime.MustPattern(runtime.NewPattern(1, []int{2, 0, 2, 1, 2, 2}, []string{"b2api", "v1", "b2_create_bucket"}, ""))
pattern_PyreService_DeleteBucket_0 = runtime.MustPattern(runtime.NewPattern(1, []int{2, 0, 2, 1, 2, 2}, []string{"b2api", "v1", "b2_delete_bucket"}, ""))
pattern_PyreService_GetUploadUrl_0 = runtime.MustPattern(runtime.NewPattern(1, []int{2, 0, 2, 1, 2, 2}, []string{"b2api", "v1", "b2_get_upload_url"}, ""))
pattern_PyreService_StartLargeFile_0 = runtime.MustPattern(runtime.NewPattern(1, []int{2, 0, 2, 1, 2, 2}, []string{"b2api", "v1", "b2_start_large_file"}, ""))
pattern_PyreService_GetUploadPartUrl_0 = runtime.MustPattern(runtime.NewPattern(1, []int{2, 0, 2, 1, 2, 2}, []string{"b2api", "v1", "b2_get_upload_part_url"}, ""))
pattern_PyreService_FinishLargeFile_0 = runtime.MustPattern(runtime.NewPattern(1, []int{2, 0, 2, 1, 2, 2}, []string{"b2api", "v1", "b2_finish_large_file"}, ""))
pattern_PyreService_ListFileVersions_0 = runtime.MustPattern(runtime.NewPattern(1, []int{2, 0, 2, 1, 2, 2}, []string{"b2api", "v1", "b2_list_file_versions"}, ""))
)
var (
forward_PyreService_AuthorizeAccount_0 = runtime.ForwardResponseMessage
forward_PyreService_ListBuckets_0 = runtime.ForwardResponseMessage
forward_PyreService_CreateBucket_0 = runtime.ForwardResponseMessage
forward_PyreService_DeleteBucket_0 = runtime.ForwardResponseMessage
forward_PyreService_GetUploadUrl_0 = runtime.ForwardResponseMessage
forward_PyreService_StartLargeFile_0 = runtime.ForwardResponseMessage
forward_PyreService_GetUploadPartUrl_0 = runtime.ForwardResponseMessage
forward_PyreService_FinishLargeFile_0 = runtime.ForwardResponseMessage
forward_PyreService_ListFileVersions_0 = runtime.ForwardResponseMessage
)
vendor/github.com/kurin/blazer/internal/pyre/proto/pyre.proto
Generated Vendored
-335
View File
@@ -1,335 +0,0 @@
// Copyright 2018, Google
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
syntax = "proto3";
import "google/api/annotations.proto";
package pyre.proto;
message AuthorizeAccountRequest {}
message AuthorizeAccountResponse {
// The identifier for the account.
string account_id = 1;
// An authorization token to use with all calls, other than
// b2_authorize_account, that need an Authorization header. This
// authorization token is valid for at most 24 hours.
string authorization_token = 2;
// The base URL to use for all API calls except for uploading and downloading
// files.
string api_url = 3;
// The base URL to use for downloading files.
string download_url = 4;
// The recommended size for each part of a large file. We recommend using
// this part size for optimal upload performance.
int32 recommended_part_size = 5;
// The smallest possible size of a part of a large file (except the last
// one). This is smaller than the recommended part size. If you use it, you
// may find that it takes longer overall to upload a large file.
int32 absolute_minimum_part_size = 6;
int32 minimum_part_size = 7; // alias for recommended_part_size
}
message ListBucketsRequest {
// The ID of your account.
string account_id = 1;
// When specified, the result will be a list containing just this bucket, if
// it's present in the account, or no buckets if the account does not have a
// bucket with this ID.
string bucket_id = 2;
// When specified, the result will be a list containing just this bucket, if
// it's present in the account, or no buckets if the account does not have a
// bucket with this ID.
string bucket_name = 3;
// If present, B2 will use it as a filter for bucket types returned in the
// list buckets response. If not present, only buckets with bucket types
// "allPublic", "allPrivate" and "snapshot" will be returned. A special
// filter value of ["all"] will return all bucket types.
//
// If present, it must be in the form of a json array of strings containing
// valid bucket types in quotes and separated by a comma. Valid bucket types
// include "allPrivate", "allPublic", "snapshot", and other values added in
// the future.
//
// A bad request error will be returned if "all" is used with other bucket
// types, this field is empty, or invalid bucket types are requested.
repeated string bucket_types = 4;
}
message LifecycleRule {
// After a file is uploaded, the number of days before it can be hidden.
int32 days_from_uploading_to_hiding = 1;
// After a file is hidden, the number of days before it can be deleted.
int32 days_from_hiding_to_deleting = 2;
// The rule applies to files whose names start with this prefix.
string file_name_prefix = 3;
}
message CorsRule {
// A name for humans to recognize the rule in a user interface. Names must be
// unique within a bucket. Names can consist of upper-case and lower-case
// English letters, numbers, and "-". No other characters are allowed. A name
// must be at least 6 characters long, and can be at most 50 characters long.
// These are all allowed names: myPhotosSite, allowAnyHttps,
// backblaze-images. Names that start with "b2-" are reserved for Backblaze
// use.
string cors_rule_name = 1;
// A non-empty list specifying which origins the rule covers. Each value may
// have one of many formats:
//
// * The origin can be fully specified, such as http://www.example.com:8180
// or https://www.example.com:4433.
//
// * The origin can omit a default port, such as https://www.example.com.
//
// * The origin may have a single '*' as part of the domain name, such as
// https://*.example.com, https://*:8443 or https://*.
//
// * The origin may be 'https' to match any origin that uses HTTPS. (This is
// broader than 'https://*' because it matches any port.)
//
// * Finally, the origin can be a single '*' to match any origin.
//
// If any entry is "*", it must be the only entry. There can be at most one
// "https" entry and no entry after it may start with "https:".
repeated string allowed_origins = 2;
// A list specifying which operations the rule allows. At least one value
// must be specified. All values must be from the following list. More values
// may be added to this list at any time.
//
// b2_download_file_by_name
// b2_download_file_by_id
// b2_upload_file
// b2_upload_part
repeated string allowed_operations = 3;
// If present, this is a list of headers that are allowed in a pre-flight
// OPTIONS's request's Access-Control-Request-Headers header value. Each
// value may have one of many formats:
//
// * It may be a complete header name, such as x-bz-content-sha1.
//
// * It may end with an asterisk, such as x-bz-info-*.
//
// * Finally, it may be a single '*' to match any header.
//
// If any entry is "*", it must be the only entry in the list. If this list
// is missing, it is be treated as if it is a list with no entries.
repeated string allowed_headers = 4;
// If present, this is a list of headers that may be exposed to an
// application inside the client (eg. exposed to Javascript in a browser).
// Each entry in the list must be a complete header name (eg.
// "x-bz-content-sha1"). If this list is missing or empty, no headers will be
// exposed.
repeated string expose_headers = 5;
// This specifies the maximum number of seconds that a browser may cache the
// response to a preflight request. The value must not be negative and it
// must not be more than 86,400 seconds (one day).
int32 max_age_seconds = 6;
}
message Bucket {
string account_id = 1;
string bucket_id = 2;
string bucket_name = 3;
string bucket_type = 4;
map<string, string> bucket_info = 5;
repeated CorsRule cores_rules = 6;
repeated LifecycleRule lifecycle_rules = 7;
int32 revision = 8;
}
message ListBucketsResponse {
repeated Bucket buckets = 1;
}
message GetUploadUrlRequest {
string bucket_id = 1;
}
message GetUploadUrlResponse {
string bucket_id = 1;
string upload_url = 2;
string authorization_token = 3;
}
message UploadFileResponse {
string file_id = 1;
string file_name = 2;
string account_id = 3;
string bucket_id = 4;
int32 content_length = 5;
string content_sha1 = 6;
string content_type = 7;
map<string, string> file_info = 8;
string action = 9;
int64 upload_timestamp = 10;
}
message StartLargeFileRequest {
string bucket_id = 1;
string file_name = 2;
string content_type = 3;
map<string, string> file_info = 4;
}
message StartLargeFileResponse {
string file_id = 1;
string file_name = 2;
string account_id = 3;
string bucket_id = 4;
string content_type = 5;
map<string, string> file_info = 6;
int64 upload_timestamp = 7;
}
message GetUploadPartUrlRequest {
string file_id = 1;
}
message GetUploadPartUrlResponse {
string file_id = 1;
string upload_url = 2;
string authorization_token = 3;
}
message FinishLargeFileRequest {
string file_id = 1;
repeated string part_sha1_array = 2;
// string sha1 = 3;
}
message FinishLargeFileResponse {
string file_id = 1;
string file_name = 2;
string account_id = 3;
string bucket_id = 4;
int64 content_length = 5;
string content_sha1 = 6; // always "none"
string content_type = 7;
map<string, string> file_info = 8;
string action = 9;
int64 upload_timestamp = 10;
}
message ListFileVersionsRequest {
string bucket_id = 1;
string start_file_name = 2;
string start_file_id = 3;
int32 max_file_count = 4;
string prefix = 5;
string delimiter = 6;
}
message ListFileVersionsResponse {
repeated File files = 1;
string next_file_name = 2;
string next_file_id = 3;
}
message File {
string file_id = 1;
string file_name = 2;
int64 content_length = 3;
string content_type = 4;
string content_sha1 = 5;
map<string, string> file_info = 6;
string action = 7;
int64 size = 8;
int64 upload_timestamp = 9;
}
service PyreService {
// Used to log in to the B2 API. Returns an authorization token that can be
// used for account-level operations, and a URL that should be used as the
// base URL for subsequent API calls.
rpc AuthorizeAccount(AuthorizeAccountRequest) returns (AuthorizeAccountResponse) {
option (google.api.http) = {
get: "/b2api/v1/b2_authorize_account"
};
}
// Lists buckets associated with an account, in alphabetical order by bucket
// name.
rpc ListBuckets(ListBucketsRequest) returns (ListBucketsResponse) {
option (google.api.http) = {
post: "/b2api/v1/b2_list_buckets"
body: "*"
};
}
// Creates a new bucket. A bucket belongs to the account used to create it.
//
// Buckets can be named. The name must be globally unique. No account can use
// a bucket with the same name. Buckets are assigned a unique bucketId which
// is used when uploading, downloading, or deleting files.
//
// There is a limit of 100 buckets per account.
rpc CreateBucket(Bucket) returns (Bucket) {
option (google.api.http) = {
post: "/b2api/v1/b2_create_bucket"
body: "*"
};
}
// Deletes the bucket specified. Only buckets that contain no version of any
// files can be deleted.
rpc DeleteBucket(Bucket) returns (Bucket) {
option (google.api.http) = {
post: "/b2api/v1/b2_delete_bucket"
body: "*"
};
}
rpc GetUploadUrl(GetUploadUrlRequest) returns (GetUploadUrlResponse) {
option (google.api.http) = {
post: "/b2api/v1/b2_get_upload_url"
body: "*"
};
}
// Prepares for uploading the parts of a large file.
rpc StartLargeFile(StartLargeFileRequest) returns (StartLargeFileResponse) {
option (google.api.http) = {
post: "/b2api/v1/b2_start_large_file"
body: "*"
};
}
// Gets an URL to use for uploading parts of a large file.
rpc GetUploadPartUrl(GetUploadPartUrlRequest) returns (GetUploadPartUrlResponse) {
option (google.api.http) = {
post: "/b2api/v1/b2_get_upload_part_url"
body: "*"
};
}
// Converts the parts that have been uploaded into a single B2 file.
rpc FinishLargeFile(FinishLargeFileRequest) returns (FinishLargeFileResponse) {
option (google.api.http) = {
post: "/b2api/v1/b2_finish_large_file"
body: "*"
};
}
// Lists all of the versions of all of the files contained in one bucket, in
// alphabetical order by file name, and by reverse of date/time uploaded for
// versions of files with the same name.
rpc ListFileVersions(ListFileVersionsRequest) returns (ListFileVersionsResponse) {
option (google.api.http) = {
post: "/b2api/v1/b2_list_file_versions"
body: "*"
};
}
}
vendor/github.com/kurin/blazer/internal/pyre/pyre.go
Generated Vendored
-20
View File
@@ -1,20 +0,0 @@
// Copyright 2018, Google
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
// Package pyre provides a gRPC-based implementation of B2, as well as a
// RESTful gateway on top of it.
package pyre
//go:generate protoc -I/usr/local/include -I. -I$GOPATH/src -I$GOPATH/src/github.com/grpc-ecosystem/grpc-gateway/third_party/googleapis --grpc-gateway_out=logtostderr=true:. proto/pyre.proto
//go:generate protoc -I/usr/local/include -I. -I$GOPATH/src -I$GOPATH/src/github.com/grpc-ecosystem/grpc-gateway/third_party/googleapis --go_out=plugins=grpc:. proto/pyre.proto
vendor/github.com/kurin/blazer/internal/pyre/simple.go
Generated Vendored
-109
View File
@@ -1,109 +0,0 @@
// Copyright 2018, Google
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package pyre
import (
"encoding/json"
"fmt"
"io"
"net/http"
"strconv"
"strings"
"github.com/google/uuid"
"github.com/kurin/blazer/internal/b2types"
)
const uploadFilePrefix = "/b2api/v1/b2_upload_file/"
type SimpleFileManager interface {
Writer(bucket, name, id string) (io.WriteCloser, error)
}
type simpleFileServer struct {
fm SimpleFileManager
}
type uploadRequest struct {
name string
contentType string
size int64
sha1 string
bucket string
info map[string]string
}
func parseUploadHeaders(r *http.Request) (*uploadRequest, error) {
ur := &uploadRequest{info: make(map[string]string)}
ur.name = r.Header.Get("X-Bz-File-Name")
ur.contentType = r.Header.Get("Content-Type")
ur.sha1 = r.Header.Get("X-Bz-Content-Sha1")
size, err := strconv.ParseInt(r.Header.Get("Content-Length"), 10, 64)
if err != nil {
return nil, err
}
ur.size = size
for k := range r.Header {
if !strings.HasPrefix("X-Bz-Info-", k) {
continue
}
name := strings.TrimPrefix("X-Bz-Info-", k)
ur.info[name] = r.Header.Get(k)
}
ur.bucket = strings.TrimPrefix(r.URL.Path, uploadFilePrefix)
return ur, nil
}
func (fs *simpleFileServer) ServeHTTP(rw http.ResponseWriter, r *http.Request) {
req, err := parseUploadHeaders(r)
if err != nil {
http.Error(rw, err.Error(), 500)
fmt.Println("oh no")
return
}
id := uuid.New().String()
w, err := fs.fm.Writer(req.bucket, req.name, id)
if err != nil {
http.Error(rw, err.Error(), 500)
fmt.Println("oh no")
return
}
if _, err := io.Copy(w, io.LimitReader(r.Body, req.size)); err != nil {
w.Close()
http.Error(rw, err.Error(), 500)
fmt.Println("oh no")
return
}
if err := w.Close(); err != nil {
http.Error(rw, err.Error(), 500)
fmt.Println("oh no")
return
}
resp := &b2types.UploadFileResponse{
FileID: id,
Name: req.name,
SHA1: req.sha1,
BucketID: req.bucket,
}
if err := json.NewEncoder(rw).Encode(resp); err != nil {
http.Error(rw, err.Error(), 500)
fmt.Println("oh no")
return
}
}
func RegisterSimpleFileManagerOnMux(f SimpleFileManager, mux *http.ServeMux) {
mux.Handle(uploadFilePrefix, &simpleFileServer{fm: f})
}
vendor/github.com/kurin/blazer/x/consistent/consistent.go
Generated Vendored
-363
View File
@@ -1,363 +0,0 @@
// Copyright 2016, Google
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
// Package consistent implements an experimental interface for using B2 as a
// coordination primitive.
package consistent
import (
"bytes"
"context"
"crypto/rand"
"encoding/base64"
"encoding/json"
"errors"
"fmt"
"io"
"io/ioutil"
"reflect"
"github.com/kurin/blazer/b2"
)
const metaKey = "blazer-meta-key-no-touchie"
var (
errUpdateConflict = errors.New("update conflict")
errNotInGroup = errors.New("not in group")
)
// NewGroup creates a new consistent Group for the given bucket.
func NewGroup(bucket *b2.Bucket, name string) *Group {
return &Group{
name: name,
b: bucket,
}
}
// Group represents a collection of B2 objects that can be modified in a
// consistent way. Objects in the same group contend with each other for
// updates, but there can only be so many (maximum of 10; fewer if there are
// other bucket attributes set) groups in a given bucket.
type Group struct {
name string
b *b2.Bucket
ba *b2.BucketAttrs
}
// Operate calls f with the contents of the group object given by name, and
// updates that object with the output of f if f returns no error. Operate
// guarantees that no other callers have modified the contents of name in the
// meantime (as long as all other callers are using this package). It may call
// f any number of times and, as a result, the potential data transfer is
// unbounded. Callers should have f fail after a given number of attempts if
// this is unacceptable.
//
// The io.Reader that f returns is guaranteed to be read until at least the
// first error. Callers must ensure that this is sufficient for the reader to
// clean up after itself.
func (g *Group) OperateStream(ctx context.Context, name string, f func(io.Reader) (io.Reader, error)) error {
for {
r, err := g.NewReader(ctx, name)
if err != nil && err != errNotInGroup {
return err
}
out, err := f(r)
r.Close()
if err != nil {
return err
}
defer io.Copy(ioutil.Discard, out) // ensure the reader is read
w, err := g.NewWriter(ctx, r.Key, name)
if err != nil {
return err
}
if _, err := io.Copy(w, out); err != nil {
return err
}
if err := w.Close(); err != nil {
if err == errUpdateConflict {
continue
}
return err
}
return nil
}
}
// Operate uses OperateStream to act on byte slices.
func (g *Group) Operate(ctx context.Context, name string, f func([]byte) ([]byte, error)) error {
return g.OperateStream(ctx, name, func(r io.Reader) (io.Reader, error) {
b, err := ioutil.ReadAll(r)
if b2.IsNotExist(err) {
b = nil
err = nil
}
if err != nil {
return nil, err
}
bs, err := f(b)
if err != nil {
return nil, err
}
return bytes.NewReader(bs), nil
})
}
// OperateJSON is a convenience function for transforming JSON data in B2 in a
// consistent way. Callers should pass a function f which accepts a pointer to
// a struct of a given type and transforms it into another struct (ideally but
// not necessarily of the same type). Callers should also pass an example
// struct, t, or a pointer to it, that is the same type. t will not be
// altered. If there is no existing file, f will be called with an pointer to
// an empty struct of type t. Otherwise, it will be called with a pointer to a
// struct filled out with the given JSON.
func (g *Group) OperateJSON(ctx context.Context, name string, t interface{}, f func(interface{}) (interface{}, error)) error {
jsonType := reflect.TypeOf(t)
for jsonType.Kind() == reflect.Ptr {
jsonType = jsonType.Elem()
}
return g.OperateStream(ctx, name, func(r io.Reader) (io.Reader, error) {
in := reflect.New(jsonType).Interface()
if err := json.NewDecoder(r).Decode(in); err != nil && err != io.EOF && !b2.IsNotExist(err) {
return nil, err
}
out, err := f(in)
if err != nil {
return nil, err
}
pr, pw := io.Pipe()
go func() { pw.CloseWithError(json.NewEncoder(pw).Encode(out)) }()
return closeAfterReading{rc: pr}, nil
})
}
// closeAfterReading closes the underlying reader on the first non-nil error
type closeAfterReading struct {
rc io.ReadCloser
}
func (car closeAfterReading) Read(p []byte) (int, error) {
n, err := car.rc.Read(p)
if err != nil {
car.rc.Close()
}
return n, err
}
// Writer is an io.ReadCloser.
type Writer struct {
ctx context.Context
wc io.WriteCloser
name string
suffix string
key string
g *Group
}
// Write implements io.Write.
func (w Writer) Write(p []byte) (int, error) { return w.wc.Write(p) }
// Close writes any remaining data into B2 and updates the group to reflect the
// contents of the new object. If the group object has been modified, Close()
// will fail.
func (w Writer) Close() error {
if err := w.wc.Close(); err != nil {
return err
}
// TODO: maybe see if you can cut down on calls to info()
for {
ci, err := w.g.info(w.ctx)
if err != nil {
// Replacement failed; delete the new version.
w.g.b.Object(w.name + "/" + w.suffix).Delete(w.ctx)
return err
}
old, ok := ci.Locations[w.name]
if ok && old != w.key {
w.g.b.Object(w.name + "/" + w.suffix).Delete(w.ctx)
return errUpdateConflict
}
ci.Locations[w.name] = w.suffix
if err := w.g.save(w.ctx, ci); err != nil {
if err == errUpdateConflict {
continue
}
w.g.b.Object(w.name + "/" + w.suffix).Delete(w.ctx)
return err
}
// Replacement successful; delete the old version.
w.g.b.Object(w.name + "/" + w.key).Delete(w.ctx)
return nil
}
}
// Reader is an io.ReadCloser. Key must be passed to NewWriter.
type Reader struct {
r io.ReadCloser
Key string
}
func (r Reader) Read(p []byte) (int, error) {
if r.r == nil {
return 0, io.EOF
}
return r.r.Read(p)
}
func (r Reader) Close() error {
if r.r == nil {
return nil
}
return r.r.Close()
}
// NewWriter creates a Writer and prepares it to be updated. The key argument
// should come from the Key field of a Reader; if Writer.Close() returns with
// no error, then the underlying group object was successfully updated from the
// data available from the Reader with no intervening writes. New objects can
// be created with an empty key.
func (g *Group) NewWriter(ctx context.Context, key, name string) (Writer, error) {
suffix, err := random()
if err != nil {
return Writer{}, err
}
return Writer{
ctx: ctx,
wc: g.b.Object(name + "/" + suffix).NewWriter(ctx),
name: name,
suffix: suffix,
key: key,
g: g,
}, nil
}
// NewReader creates a Reader with the current version of the object, as well
// as that object's update key.
func (g *Group) NewReader(ctx context.Context, name string) (Reader, error) {
ci, err := g.info(ctx)
if err != nil {
return Reader{}, err
}
suffix, ok := ci.Locations[name]
if !ok {
return Reader{}, errNotInGroup
}
return Reader{
r: g.b.Object(name + "/" + suffix).NewReader(ctx),
Key: suffix,
}, nil
}
func (g *Group) info(ctx context.Context) (*consistentInfo, error) {
attrs, err := g.b.Attrs(ctx)
if err != nil {
return nil, err
}
g.ba = attrs
imap := attrs.Info
if imap == nil {
return nil, nil
}
enc, ok := imap[metaKey+"-"+g.name]
if !ok {
return &consistentInfo{
Version: 1,
Locations: make(map[string]string),
}, nil
}
b, err := base64.StdEncoding.DecodeString(enc)
if err != nil {
return nil, err
}
ci := &consistentInfo{}
if err := json.Unmarshal(b, ci); err != nil {
return nil, err
}
if ci.Locations == nil {
ci.Locations = make(map[string]string)
}
return ci, nil
}
func (g *Group) save(ctx context.Context, ci *consistentInfo) error {
ci.Serial++
b, err := json.Marshal(ci)
if err != nil {
return err
}
s := base64.StdEncoding.EncodeToString(b)
for {
oldAI, err := g.info(ctx)
if err != nil {
return err
}
if oldAI.Serial != ci.Serial-1 {
return errUpdateConflict
}
if g.ba.Info == nil {
g.ba.Info = make(map[string]string)
}
g.ba.Info[metaKey+"-"+g.name] = s
err = g.b.Update(ctx, g.ba)
if err == nil {
return nil
}
if !b2.IsUpdateConflict(err) {
return err
}
// Bucket update conflict; try again.
}
}
// List returns a list of all the group objects.
func (g *Group) List(ctx context.Context) ([]string, error) {
ci, err := g.info(ctx)
if err != nil {
return nil, err
}
var l []string
for name := range ci.Locations {
l = append(l, name)
}
return l, nil
}
type consistentInfo struct {
Version int
// Serial is incremented for every version saved. If we ensure that
// current.Serial = 1 + previous.Serial, and that the bucket metadata is
// updated cleanly, then we know that the version we saved is the direct
// successor to the version we had. If the bucket metadata doesn't update
// cleanly, but the serial relation holds true for the new AI struct, then we
// can retry without bothering the user. However, if the serial relation no
// longer holds true, it means someone else has updated AI and we have to ask
// the user to redo everything they've done.
//
// However, it is still necessary for higher level constructs to confirm that
// the serial number they expect is good. The writer does this, for example,
// but comparing the "key" of the file it is replacing.
Serial int
Locations map[string]string
}
func random() (string, error) {
b := make([]byte, 20)
if _, err := rand.Read(b); err != nil {
return "", err
}
return fmt.Sprintf("%x", b), nil
}
vendor/github.com/kurin/blazer/x/consistent/consistent_test.go
Generated Vendored
-163
View File
@@ -1,163 +0,0 @@
package consistent
import (
"context"
"io/ioutil"
"os"
"strconv"
"sync"
"testing"
"github.com/kurin/blazer/b2"
)
const (
apiID = "B2_ACCOUNT_ID"
apiKey = "B2_SECRET_KEY"
bucketName = "consistobucket"
)
func TestOperationLive(t *testing.T) {
ctx := context.Background()
bucket, done := startLiveTest(ctx, t)
defer done()
g := NewGroup(bucket, "tester")
name := "some_kinda_name/thing.txt"
var wg sync.WaitGroup
for i := 0; i < 10; i++ {
wg.Add(1)
i := i
go func() {
defer wg.Done()
for j := 0; j < 10; j++ {
var n int
if err := g.Operate(ctx, name, func(b []byte) ([]byte, error) {
if len(b) > 0 {
i, err := strconv.Atoi(string(b))
if err != nil {
return nil, err
}
n = i
}
return []byte(strconv.Itoa(n + 1)), nil
}); err != nil {
t.Error(err)
}
t.Logf("thread %d: successful %d++", i, n)
}
}()
}
wg.Wait()
r, err := g.NewReader(ctx, name)
if err != nil {
t.Fatal(err)
}
defer r.Close()
b, err := ioutil.ReadAll(r)
if err != nil {
t.Fatal(err)
}
n, err := strconv.Atoi(string(b))
if err != nil {
t.Fatal(err)
}
if n != 100 {
t.Errorf("result: got %d, want 100", n)
}
}
type jsonThing struct {
Boop int `json:"boop_field"`
Thread int `json:"thread_id"`
}
func TestOperationJSONLive(t *testing.T) {
ctx := context.Background()
bucket, done := startLiveTest(ctx, t)
defer done()
g := NewGroup(bucket, "tester")
name := "some_kinda_json/thing.json"
var wg sync.WaitGroup
for i := 0; i < 4; i++ {
wg.Add(1)
i := i
go func() {
var n int
defer wg.Done()
for j := 0; j < 4; j++ {
// Pass both a struct and a pointer to a struct.
var face interface{}
face = jsonThing{}
if j%2 == 0 {
face = &jsonThing{}
}
if err := g.OperateJSON(ctx, name, face, func(j interface{}) (interface{}, error) {
jt := j.(*jsonThing)
n = jt.Boop
return &jsonThing{
Boop: jt.Boop + 1,
Thread: i,
}, nil
}); err != nil {
t.Error(err)
}
t.Logf("thread %d: successful %d++", i, n)
}
}()
}
wg.Wait()
if err := g.OperateJSON(ctx, name, &jsonThing{}, func(i interface{}) (interface{}, error) {
jt := i.(*jsonThing)
if jt.Boop != 16 {
t.Errorf("got %d boops; want 16", jt.Boop)
}
return nil, nil
}); err != nil {
t.Error(err)
}
}
func startLiveTest(ctx context.Context, t *testing.T) (*b2.Bucket, func()) {
id := os.Getenv(apiID)
key := os.Getenv(apiKey)
if id == "" || key == "" {
t.Skipf("B2_ACCOUNT_ID or B2_SECRET_KEY unset; skipping integration tests")
return nil, nil
}
client, err := b2.NewClient(ctx, id, key)
if err != nil {
t.Fatal(err)
return nil, nil
}
bucket, err := client.NewBucket(ctx, id+"-"+bucketName, nil)
if err != nil {
t.Fatal(err)
return nil, nil
}
f := func() {
iter := bucket.List(ctx, b2.ListHidden())
for iter.Next() {
if err := iter.Object().Delete(ctx); err != nil {
t.Error(err)
}
}
if err := iter.Err(); err != nil && !b2.IsNotExist(err) {
t.Error(err)
}
if err := bucket.Delete(ctx); err != nil && !b2.IsNotExist(err) {
t.Error(err)
}
}
return bucket, f
}
type object struct {
o *b2.Object
err error
}
vendor/github.com/kurin/blazer/x/transport/transport.go
Generated Vendored
-207
View File
@@ -1,207 +0,0 @@
// Copyright 2017, Google
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
// Package transport provides http.RoundTrippers that may be useful to clients
// of Blazer.
package transport
import (
"context"
"fmt"
"io"
"io/ioutil"
"math/rand"
"net/http"
"strings"
"sync/atomic"
"time"
)
// WithFailures returns an http.RoundTripper that wraps an existing
// RoundTripper, causing failures according to the options given. If rt is
// nil, the http.DefaultTransport is wrapped.
func WithFailures(rt http.RoundTripper, opts ...FailureOption) http.RoundTripper {
if rt == nil {
rt = http.DefaultTransport
}
o := &options{
rt: rt,
}
for _, opt := range opts {
opt(o)
}
return o
}
type options struct {
pathSubstrings []string
failureRate float64
status int
stall time.Duration
rt http.RoundTripper
msg string
trg *triggerReaderGroup
}
func (o *options) doRequest(req *http.Request) (*http.Response, error) {
if o.trg != nil && req.Body != nil {
req.Body = o.trg.new(req.Body)
}
resp, err := o.rt.RoundTrip(req)
if resp != nil && o.trg != nil {
resp.Body = o.trg.new(resp.Body)
}
return resp, err
}
func (o *options) RoundTrip(req *http.Request) (*http.Response, error) {
// TODO: fix triggering conditions
if rand.Float64() > o.failureRate {
return o.doRequest(req)
}
var match bool
if len(o.pathSubstrings) == 0 {
match = true
}
for _, ss := range o.pathSubstrings {
if strings.Contains(req.URL.Path, ss) {
match = true
break
}
}
if !match {
return o.doRequest(req)
}
if o.status > 0 {
resp := &http.Response{
Status: fmt.Sprintf("%d %s", o.status, http.StatusText(o.status)),
StatusCode: o.status,
Body: ioutil.NopCloser(strings.NewReader(o.msg)),
Request: req,
}
return resp, nil
}
if o.stall > 0 {
ctx := req.Context()
select {
case <-time.After(o.stall):
case <-ctx.Done():
}
}
return o.doRequest(req)
}
// A FailureOption specifies the kind of failure that the RoundTripper should
// display.
type FailureOption func(*options)
// MatchPathSubstring restricts the RoundTripper to URLs whose paths contain
// the given string. The default behavior is to match all paths.
func MatchPathSubstring(s string) FailureOption {
return func(o *options) {
o.pathSubstrings = append(o.pathSubstrings, s)
}
}
// FailureRate causes the RoundTripper to fail a certain percentage of the
// time. rate should be a number between 0 and 1, where 0 will never fail and
// 1 will always fail. The default is never to fail.
func FailureRate(rate float64) FailureOption {
return func(o *options) {
o.failureRate = rate
}
}
// Response simulates a given status code. The returned http.Response will
// have its Status, StatusCode, and Body (with any predefined message) set.
func Response(status int) FailureOption {
return func(o *options) {
o.status = status
}
}
// Stall simulates a network connection failure by stalling for the given
// duration.
func Stall(dur time.Duration) FailureOption {
return func(o *options) {
o.stall = dur
}
}
// If a specific Response is requested, the body will have the given message
// set.
func Body(msg string) FailureOption {
return func(o *options) {
o.msg = msg
}
}
// Trigger will raise the RoundTripper's failure rate to 100% when the given
// context is closed.
func Trigger(ctx context.Context) FailureOption {
return func(o *options) {
go func() {
<-ctx.Done()
o.failureRate = 1
}()
}
}
// AfterNBytes will call effect once (roughly) n bytes have gone over the wire.
// Both sent and received bytes are counted against the total. Only bytes in
// the body of an HTTP request are currently counted; this may change in the
// future. effect will only be called once, and it will block (allowing
// callers to simulate connection hangs).
func AfterNBytes(n int, effect func()) FailureOption {
return func(o *options) {
o.trg = &triggerReaderGroup{
bytes: int64(n),
trigger: effect,
}
}
}
type triggerReaderGroup struct {
bytes int64
trigger func()
triggered int64
}
func (rg *triggerReaderGroup) new(rc io.ReadCloser) io.ReadCloser {
return &triggerReader{
ReadCloser: rc,
bytes: &rg.bytes,
trigger: rg.trigger,
triggered: &rg.triggered,
}
}
type triggerReader struct {
io.ReadCloser
bytes *int64
trigger func()
triggered *int64
}
func (r *triggerReader) Read(p []byte) (int, error) {
n, err := r.ReadCloser.Read(p)
if atomic.AddInt64(r.bytes, -int64(n)) < 0 && atomic.CompareAndSwapInt64(r.triggered, 0, 1) {
// Can't use sync.Once because it blocks for *all* callers until Do returns.
r.trigger()
}
return n, err
}
vendor/github.com/kurin/blazer/x/window/accum_test.go
Generated Vendored
-63
View File
@@ -1,63 +0,0 @@
// Copyright 2018, Google
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package window_test
import (
"fmt"
"time"
"github.com/kurin/blazer/x/window"
)
type Accumulator struct {
w *window.Window
}
func (a Accumulator) Add(s string) {
a.w.Insert([]string{s})
}
func (a Accumulator) All() []string {
v := a.w.Reduce()
return v.([]string)
}
func NewAccum(size time.Duration) Accumulator {
r := func(i, j interface{}) interface{} {
a, ok := i.([]string)
if !ok {
a = nil
}
b, ok := j.([]string)
if !ok {
b = nil
}
for _, s := range b {
a = append(a, s)
}
return a
}
return Accumulator{w: window.New(size, time.Second, r)}
}
func Example_accumulator() {
a := NewAccum(time.Minute)
a.Add("this")
a.Add("is")
a.Add("that")
fmt.Printf("total: %v\n", a.All())
// Output:
// total: [this is that]
}
vendor/github.com/kurin/blazer/x/window/counter_test.go
Generated Vendored
-60
View File
@@ -1,60 +0,0 @@
// Copyright 2018, Google
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package window_test
import (
"fmt"
"time"
"github.com/kurin/blazer/x/window"
)
type Counter struct {
w *window.Window
}
func (c Counter) Add() {
c.w.Insert(1)
}
func (c Counter) Count() int {
v := c.w.Reduce()
return v.(int)
}
func New(size time.Duration) Counter {
r := func(i, j interface{}) interface{} {
a, ok := i.(int)
if !ok {
a = 0
}
b, ok := j.(int)
if !ok {
b = 0
}
return a + b
}
return Counter{w: window.New(size, time.Second, r)}
}
func Example_counter() {
c := New(time.Minute)
c.Add()
c.Add()
c.Add()
fmt.Printf("total: %d\n", c.Count())
// Output:
// total: 3
}
vendor/github.com/kurin/blazer/x/window/window_test.go
Generated Vendored
-103
View File
@@ -1,103 +0,0 @@
// Copyright 2018, Google
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package window
import (
"testing"
"time"
)
type epair struct {
e interface{}
t time.Time
}
func adder(i, j interface{}) interface{} {
a, ok := i.(int)
if !ok {
a = 0
}
b, ok := j.(int)
if !ok {
b = 0
}
return a + b
}
func TestWindows(t *testing.T) {
table := []struct {
size, dur time.Duration
incs []epair
look time.Time
reduce Reducer
want interface{}
}{
{
size: time.Minute,
dur: time.Second,
incs: []epair{
// year, month, day, hour, min, sec, nano
{t: time.Date(2000, 1, 1, 0, 0, 0, 0, time.UTC), e: 1},
{t: time.Date(2000, 1, 1, 0, 0, 1, 0, time.UTC), e: 1},
{t: time.Date(2000, 1, 1, 0, 0, 2, 0, time.UTC), e: 1},
{t: time.Date(2000, 1, 1, 0, 0, 3, 0, time.UTC), e: 1},
{t: time.Date(2000, 1, 1, 0, 0, 4, 0, time.UTC), e: 1},
{t: time.Date(2000, 1, 1, 0, 0, 5, 0, time.UTC), e: 1},
},
look: time.Date(2000, 1, 1, 0, 1, 0, 0, time.UTC),
want: 5,
reduce: adder,
},
{
incs: []epair{
// year, month, day, hour, min, sec, nano
{t: time.Date(2000, 1, 1, 0, 0, 0, 0, time.UTC), e: 1},
{t: time.Date(2000, 1, 1, 0, 0, 1, 0, time.UTC), e: 1},
{t: time.Date(2000, 1, 1, 0, 0, 2, 0, time.UTC), e: 1},
{t: time.Date(2000, 1, 1, 0, 0, 3, 0, time.UTC), e: 1},
{t: time.Date(2000, 1, 1, 0, 0, 4, 0, time.UTC), e: 1},
{t: time.Date(2000, 1, 1, 0, 0, 5, 0, time.UTC), e: 1},
},
want: 6,
reduce: adder,
},
{ // what happens if time goes backwards?
size: time.Minute,
dur: time.Second,
incs: []epair{
{t: time.Date(2000, 1, 1, 0, 0, 0, 0, time.UTC), e: 1},
{t: time.Date(2000, 1, 1, 0, 0, 1, 0, time.UTC), e: 1},
{t: time.Date(2000, 1, 1, 0, 0, 2, 0, time.UTC), e: 1},
{t: time.Date(2000, 1, 1, 0, 0, 3, 0, time.UTC), e: 1},
{t: time.Date(2000, 1, 1, 0, 0, 4, 0, time.UTC), e: 1},
{t: time.Date(2000, 1, 1, 0, 0, 0, 0, time.UTC), e: 1},
},
look: time.Date(2000, 1, 1, 0, 0, 30, 0, time.UTC),
want: 1,
reduce: adder,
},
}
for _, e := range table {
w := New(e.size, e.dur, e.reduce)
for _, inc := range e.incs {
w.insertAt(inc.t, inc.e)
}
ct := w.reducedAt(e.look)
if ct != e.want {
t.Errorf("reducedAt(%v) got %v, want %v", e.look, ct, e.want)
}
}
}